From 1460 the Catholic Kings granted an extensive dominion to Don Andrés de Cabrera in the south zone of the current province of Madrid, who fixed as head of jurisdiction to Chinchón's Villa. It is very probable that the new gentlemen were constructed a castle, as residence and ostentation of his power. The conflict that freed itself with the commoners affected Chinchón, since not about a lot of time that had separated of the province of Segovia, it was supporting still good relations. On January 21, 1521 after length besiege the castle it falls down in hands commoners. The siege and later plunder stayed to the castle in lamentable conditions. Once recovered the castle and suffocated the movement commoner is repaired though two later generations were raising a new building on the remains of the previous one.
At the beginning of the century XVIII he had to be left, maybe coinciding with the destructions done by the Marquess of the Mine during the War of Succession (1706). One century later (XIX) one new military occupation the Polish Brigade of the marshall Víctor I transport new damages. According to documents, the Counts of Chinchón stopped living the castle from the century XVIII. It seems to be that the current castle is the second building that occupies this emplacement, since the primitive one was destroyed for with posteriority to be reconstructed by the same owners more as a palace, that as a fortification.