Built by Francois de Caumont around 1489, the Château des Milandes was the main house of the lords of Caumont until 1535, who preferred to live in this manor house instead of the large, uncomfortable medieval castle of Château de Castelnaud-la-Chapelle.

In 1940, the entertainer Josephine Baker rented the château and then bought it in 1947.