That Ibn 'Umar heard a man saying: "No by the Ka'bah" so Ibn 'Umar said: "Nothing is sworn by other than Allah, for I heard the Messenger of Allah (ﷺ) say: 'Whoever swears by other than Allah, he has committed disbelief or shirk.'"
Jami` at-Tirmidhi Book 20, Hadith 13