Actually if we look at the hadeeth saying that it's haram for women to cover the face you will see that Rasululllah used the word "Niqaab".
Niqaab in Arabic means a cloth pulled close around the face hence touching it, which is why it can be said when a man wears his Imaamah around his face that he is putting "Niqaab" across his face.
So the Ulama who say that you should cover your face whilst making Hajj with a loose piece of cloth say this because they said that Rasululllah only stated "niqaab" and didn't say "do not cover"

Allahu A'lam, this is not my position but I am saying this hoping to clear matters for insha'allah amani