The castle was built circa 1060 by William the Conqueror, who successfully conquered England in 1066. His son Henry I then built the Saint Georges church, a keep (1123) and a large hall for the Duke Court
At Christmas 1182 a royal court celebration for Christmas in the aula of Caen Castle brought together Henry II and his sons, Richard the Lionheart and John Lackland, receiving more than a thousand knights.
Caen Castle, along with all of Normandy, was handed over to the French Crown in 1204. Philip II reinforced the fortifications.
The castle saw several engagements during the Hundred Years' War (1346, 1417, 1450). The keep was pulled down in 1793 during the French Revolution, by order of the National Convention.
The castle, which was used as a barracks during World War II, was bombed in 1944 and seriously damaged.
In 1946, Michel de Boüard, an archeologist from Caen decided to start excavations in the area of the castle to bring to the light medieval traces. The musée des Beaux-Arts, which was installed in 1967, opened in 1971.