I liked to enter the House (the Ka'bah) and pray therein. The Messenger of Allah (ﷺ) caught me by hand and admitted me to al-Hijr. He then said: Pray in al-Hijr when you intend to enter the House (the Ka'bah), for it is a part of the House (the Ka'bah). Your people shortened it when they built the Ka'bah, and they took it out of the House.
Sunan Abi Dawud Book 11, Hadith 308
← Previous Hadith ‘Abbas said “When the Prophet (ﷺ) came to Makkah he refused to enter the House (the Ka’bah) for there were idols in it. He ordered to take them out and they were taken out. The statues of Abraham and Isma’il were taken out and they had arrows in their hands. Apostle of Allaah(ﷺ) said “May Allaah destroy them! By Allaah, they knew that they never cast lots by arrow. He then entered the House(the Ka’bah) and uttered the takbir (Allaah is most great) in all its sides and corners. He then came out and did not pray.

Sunan Abi Dawud Book 11, Hadith 307 Next Hadith → The Prophet (ﷺ) went out from me, while he was happy, but he returned to me while he was sad. He said: I entered the Ka'bah, I know beforehand about my affair what I have come to know later I would not have entered it. I am afraid I have put my community to hardship.

Sunan Abi Dawud Book 11, Hadith 309