"Why did you go back?" He said: "I asked permission to enter three times, as the Messenger of Allah(ﷺ) enjoined upon us, then if we are given permission we should enter, otherwise we should go back." He said: "You should bring me proof of that, or else!" Then he came to a gathering of his people and asked them to swear by Allah concerning that, and they did so, so he let him go."
Sunan Ibn Majah Book 33, Hadith 50