When you tell (your brother on Friday) to be silent while the imam is giving the sermon you are guilty of idle talk.

Sunan Abi Dawud Book 2, Hadith 723