In that case, Zoroastrians also don't intend to pray to fire. It's just their Qibla. Yet, they are not called ahl ul-kitab. They believe in one God. They pray through the archangels, the amesha spentas and the ahuras. However, that's just similar to Christians praying via Jesus (pbuh). They use icons to represent them just like how Christians use icons like a cross or crucifix.
Praying towards fire is animism. Let's put it this way, if you're praying towards a portrait of a person, but you don't intend to, all Muslims unanimously consider it as idolatry. So why not fire? Why not water (Zoroastrians pray towards water too at times, other times towards the Sun)? Fire veneration plays an important role in all polytheistic and animistic religions.
To put it straight, if you're locked in prison and there are idols/images around you and you have to pray salat, you can't help it. However, if you can avoid it, it's a different matter entirely.