That just means you are putting your trust in a stone - instead of relying on Allah the almighty, who is in charge of everything in the universe.
I know you say that its not about believeing/disbeliving, but why wear a birthstone? We dont even celebrate birthdays - so why even wear one?
Peace