“There is nothing standing between a person and Shirk (polytheism) except leaving the prayer, so if he leaves it he has committed Shirk.”
Sunan Ibn Majah Book 5, Hadith 278