Yoga. Is it considered shirk?

It isn't a brand new topic, for it has been covered years before this post was created. Although in a rather confusing way. Yoga has spread among the Muslims massively, as a matter of fact it has spread throughout the world massively. One major problem arose out of this, as it became a rather complicated topic. There are many scholars who say that yoga is completely prohibited in Islam, not just that but it is literal shirk if you do yoga and to my understanding that is not just the meditation part in which you recite nonsense things and worship the Sun. Question is often thrown what if you don't believe in the rubbish, not recite it at all (I mean after all reciting does absolutely nothing in the yoga process) and not worship the Sun, take it as an exercise. Still most scholars say that it is completely prohibited. Now a realization is that there is a hadith of Prophet Muhammed (peace be upon him) in which he clearly says that if you follow other religions, you become one of them. Now this has confused me furthermore, it could well be interpreted that it means if you even do yoga at all that is shirk. Shirk is in my mind and the mind of all Muslims I believe in a league of its own, it is the largest sin in Islam for a reason, I mean this is the only sin that is never forgiven and punished eternally. That is very dangerous then because if it is literal shirk, then I don't think anyone would be willing to do it. Don't get me wrong I am not promoting sins again, but this is a reality and fact that the mercy of God is given to everyone with the first pillar of Islam, only shirk is something that the merciful God never forgives if you do not repent in this life. So the question arises is yoga equal to shirk? Or is it sinful, just not shirk?