(the wife of the Prophet) that Allah's Messenger (ﷺ) said to her, "Do you know that when your people (Quraish) rebuilt the Ka`ba, they decreased it from its original foundation laid by Abraham?" I said, "O Allah's Messenger (ﷺ)! Why don't you rebuild it on its original foundation laid by Abraham?" He replied, "Were it not for the fact that your people are close to the Pre-Islamic Period of ignorance (i.e. they have recently become Muslims) I would have done so." The sub-narrator, `Abdullah (bin `Umar ) stated: `Aisha 'must have heard this from Allah's Messenger (ﷺ) for in my opinion Allah's Messenger (ﷺ) had not placed his hand over the two corners of the Ka`ba opposite Al-Hijr only because the Ka`ba was not rebuilt on its original foundations laid by Abraham.
Sahih al-Bukhari Book 25, Hadith 69
← Previous Hadith When the Ka`ba was built, the Prophet (ﷺ) and `Abbas went to bring stones (for its construction). Al `Abbas said to the Prophet, "Take off your waist sheet and put it on your neck." (When the Prophet (ﷺ) took it off) he fell on the ground with his eyes open towards the sky and said, "Give me my waist sheet." And he covered himself with it.

Sahih al-Bukhari Book 25, Hadith 68 Next Hadith → I asked the Prophet (ﷺ) whether the round wall (near Ka`ba) was part of the Ka`ba. The Prophet (ﷺ) replied in the affirmative. I further said, "What is wrong with them, why have they not included it in the building of the Ka`ba?" He said, "Don't you see that your people (Quraish) ran short of money (so they could not include it inside the building of Ka`ba)?" I asked, "What about its gate? Why is it so high?" He replied, "Your people did this so as to admit into it whomever they liked and prevent whomever they liked. Were your people not close to the Pre-Islamic Period of ignorance (i.e. they have recently embraced Islam) and were I not afraid that they would dislike it, surely I would have included the (area of the) wall inside the building of the Ka`ba and I would have lowered its gate to the level of the ground."

Sahih al-Bukhari Book 25, Hadith 70