When you tell (your brother on Friday) to be silent while the imam is giving the sermon you are guilty of idle talk.
Sunan Abi Dawud Book 2, Hadith 723