If you (even) ask your companion to be quiet on Friday while the Imam is delivering the sermon, you have in fact talked irrelevance.
Sahih Muslim Book 7, Hadith 15
← Previous Hadith Abu Huraira reported that the Messenger of, Allah (ﷺ) said. He who takes a bath on Friday, the bath which is obligatory after the sexual discharge and then goes (to the mosque), he is like one who offers a she-camel as a sacrifice, and he who comes at the second hour would be like one who offers a cow, and he who comes at the third hour is live one who offers a ram with horns, and he who comes at the fourth hour is like one who offers a hen, and he who comes at the fifth hour is like one who offers an egg. And when the Imam comes out, the angels are also present and listen to the mention of God (the sermon).

Sahih Muslim Book 7, Hadith 14 Next Hadith → A hadith like this has been narrated by Abu Huraira by another chain of transmitters.

Sahih Muslim Book 7, Hadith 16