why nobody speaks against tombs/grave-worshipping?

Interesting to note what Ibn Qayyim al-Jawziyyah (rahimullah) mantioned in Zad al-Ma'ad on this topic:

When the Prophet travelled from Tabuk until he was an hour away from Al Madinah and the people of Ad-Dhirar Mosque had come to him when he was preparing to leave for tabuk and said to him: "We have built a mosque for the sick and needy and for the rainy nights and we would like you to pray in it." He said "I am on the point of traveling, but when we return, if Allah wills, I will come to you." and Allah revealed that it was a mosque built by the hypocrites to cause harm and disbelief in surah at-tawbah "And there are those who put up a mosque by way of mischief and infidelity - to disunite the Believers ..." [Surah at tawbah 9:107].

The Prophet ordered this mosque to be burnt.

Ibn Qayyim comments on that and says:

We also derive from this story the permissibility of buring places of sin and disobedience, as the Prophet burnt the Masjid Ad-Dirar (The Mosque of Harm); and it is incumbent upon the Imam to destroy it, either by demolishing it or by burning it, or by altering its shape and changing its function. And if that was the case regarding Masjid Ad-Dirar, then the shrines where shirk is practised should with all the more reason be destroyed and likewise the houses of the wine merchants and those who do evil deeds; 'Umar (r) burnt down a whole village in which wine was sold. And he burn down the Palace of Sa'd when he secluded himself in it from the people. And the Prophet intended to burn the houses of those who did not attend the Friday prayer or the congregational prayer; and the only thing which prevented him from doing so was the presence therein of persons whom it was not obligatory (i.e. woman and small children).

We also understand from it that an endowment is not valid unless it is done as an act of righteousness, so a mosque which was built over a grave should be demolished, just as a dead body should be exhaumed from a mosque if it was buried therein, for the mosque and the grave do not go together in Islam. This is the religion of Islam with which Allah sent His Messenger and it has become something strange among the people, as you see. [1]

--------
Footnotes:

[1] This is a reference to the Hadeeth of the Prophet in which he said "Islam started as something strange and it will again return to being something strange, just as it started." (Narrated by Muslim, Tirmidhi, Ibn Majah, Ahmed and Ad-Darimi)