"The Prophet forbade men to perfume themselves with saffron. (Shahih)
Sunan an-Nasa'i Book 24, Hadith 88