What is the hardest language to learn? That will depend on what our native tongue is. Languages are broken down into numerous groups. Each group is made up of languages from the same origin. We find the languages within our group to be the easiest to learn. The further removed we get from our language group, the more difficult it becomes.
Difficulty of a language is going to be primarily in the eyes of the student.
To myself the most difficult languages I know of would be the old whistle language of the original inhabitants of the Canary language. It was not composed of what we call words, but was a series of whistling sounds. The most complex to me would be the language of Tierra-del-Fuego. Which is actually 3 distinct languages. One spoken only man to man, another only spoken woman to woman and a third when speaking man to woman. Very distinct different languages for all three.
I personaly find all the Latin based languages to be easiest for me. The most difficult language I ever tried to learn is Cherokee. Although my wife spoke it, I never got to know more than a few words and never did master the alphabet.