Construction of the Llandovery Castle began in 1116 by the Normans and ceased shortly thereafter. The castle was attacked and partially destroyed by Welsh forces under Gruffydd ap Rhys however the castle remained in Norman hands. In 1158 the castle was seized by Welsh forces under Rhys ap Gruffydd, youngest son of Gruffydd ap Rhys. The castle changed hands over the next several years falling to the English under Edward I in 1277. It was again taken by the Welsh briefly in 1282 by forces under Llywelyn the Last. It was again attacked by Welsh forces during the Owain Glyndwr rebellion in 1403 and left in partial rune. The castle continued its decline from that point forward and not repaired.
It is easily accessible with a car park located below it and is a short distance from Llandovery town center, and the site of the statue of Llywelyn ap Gruffydd Fychan placed in 2001