The Harem at Topkapi palace. The harem was the private quarters of the Sultans and their families from Mehmet II in the mid-15th centry until the mid-1800s. Up to 400 concubines were brought from every corner of the empire for the Sultan and his sons. The most prized women were Serbs and Circassians. No man except the Sultan and his heirs were allowed in the harem. The women of the harem would do all chores, cooking, cleaning and entertaining. They were administered by the mother of the Sultan called the Sultan Valide. It was a heirarchical system with the women bearing sons given more preferential treatment.