Ibn Umar heard a man swearing: No, I swear by the Ka'bah. Ibn Umar said to him: I heard the Messenger of Allah (ﷺ) say: He who swears by anyone but Allah is polytheist.
Sunan Abi Dawud Book 22, Hadith 10