The Prophet (nay peace be upon him) commanded him to build a mosque at Ta'if where the idols were placed.
Sunan Abi Dawud Book 2, Hadith 60