The Messenger of Allah said: "Whoever stays with a people, then he is not to fast without their permission."
Jami` at-Tirmidhi Book 8, Hadith 108