The first castle was built on a basalt outcrop in the 12th century to provide surveillance over several roads. It was reinforced in the 14th century by Guillaume de Sam, who built the keep, a second chapel and the eastern buildings.
The castle became the property of the d'Estaing family when the sole heir, Jehanne de Murol, married Gaspard d'Estaing in the 15th century. The castle was richly decorated and, under François I d'Estaing, in the 16th century it was surrounded by a huge curtain wall with towers. Emerging unscathed from the sieges during the Catholic League, the castle was transformed when Jean d'Estaing built a more comfortable Renaissance pavilion at the foot of the inner castle. The castle was later abandoned.