Prophet [PBUH] used to pray at night.
That does not mean at 11PM!

Lets take the time zone for Madina:

Isha Prayer is at 8:03PM

Therefore, One can assume that he prayed until 10 MAX, and went to bed. (plz keep in mind that during those days there was no electricity - and supplies for Muslim Ummah was limited)

Fajr Prayer is at 5:05AM

The Prophet then got up at night (one may assume 3 o'clock or so) and prayed.

The rest can be read in the following Hadith:

Aswad Ibn Yazeed - reported from Aa'isha (Radhiallaahu Anhu) that the Prophet (Sallallaahu Alayhi Wasallam) spent his nights in prayer. She said, 'We slept during the first part of the night, then the Prophet (Sallallaahu Alayhi Wasallam) woke up and prayed. After the prayers, we enjoyed sexual relations if we so desired. When he heard the adhaan (call for morning prayer) he took the ceremonial bath by pouring water over himself. Otherwise, he performed only the ablution (wudhoo). Then he left home to pray in the Mosque.' (Bukhaari)


note: the timing is just a guess