I asked Allah's Messenger (ﷺ) about looking hither and thither in prayer. He replied, "It is a way of stealing by which Satan takes away (a portion) from the prayer of a person."
Sahih al-Bukhari Book 10, Hadith 145
← Previous Hadith The Prophet (ﷺ) said, "What is wrong with those people who look towards the sky during the prayer?" His talk grew stern while delivering this speech and he said, "They should stop (looking towards the sky during the prayer); otherwise their eyesight would be taken away."

Sahih al-Bukhari Book 10, Hadith 144 Next Hadith → Once the Prophet (ﷺ) prayed on a Khamisa with marks on it and said, "The marks on it diverted my attention, take this Khamisa to Abu Jahm and bring an Inbijaniya (from him.)"

Sahih al-Bukhari Book 10, Hadith 146