It is believed that the castle was built by Robert de Lacy around the year 1186. Clitheroe Castle is believed to be the smallest keep in the whole of England. It stands atop a 35-metre outcrop of limestone and is one of the oldest buildings in Lancashire. It is also the only remaining castle in the county which had a royalist garrison during the English Civil War.
The castle's most prominent feature is the hole in its side which was made in 1649 and was ordered by the government. It was to be put in "such condition that in might neither be a charge to the Commonwealth to keep it, nor a danger to have it kept against them".