In medieval times the castle formed part of the defences of the town of Luzech due to its dominance of the Lot River. The château de Cayx was bought by the Lefranc de Pompignan family during the eighteenth century. The noted French poet Jean-Jacques Lefranc, marquis de Pompignan spent the latter half of his life there.
The château was first fortified during the fourteenth century. Since then it has been rebuilt and renovated several times. The Pompignans built the extensive wine cellars under the château. A vine infestation devastated the vineyards during the late nineteenth century. The Pompignan family emigrated to Indo-China. They did not return to Europe until 1955. In 1965 the family's heir, Henri Marie Jean André de Laborde de Monpezat, married Princess Margrethe, then heiress presumptive to the Danish throne, becoming Prince Consort of Denmark after she succeeded to Denmark's throne.
The royal couple purchased the château and the estate in 1974. Since then they have renovated it extensively. According to the official website of the Danish monarchy, the residence has become a "relaxed setting for reunions of the entire Danish Royal Family and their French relatives".
The residence is not open to the public. A guided tour of the gardens is available during the summer. The château still produces wine.