“The Quraish said: ‘We are the neighbors of the House and we do not leave the sanctuary.’ Allah said: ‘Then depart from the place whence all the people depart.’”
Sunan Ibn Majah Book 25, Hadith 137