In the area where the castle stands, dwelling and storage structures (a hydraulic feature, silos and hut floors) belonging to a Copper Age settlement have been documented. Of the Roman settlement, a double walled enclosure can be identified, protected by a moat on the northern side that reinforced its defences. It was probably organised around a tower located to the south-west, to which the large-sized masonry blocks forming the foundations of the medieval tower would correspond. Inside, buildings would have stood, of which few traces remain, with storage structures for agricultural produce and water. The medieval fortress, built between the 13th and 14th centuries making use of Roman-period remains, consisted of a walled enclosure in the shape of an irregular trapezoid, with four towers, of which three survive today: a large rectangular tower occupying the south-western corner; a small, circular one to the north-west, of which only the base survives; and another rectangular, solid one to the north-east, which defended the castle gate. A fourth tower once stood at the south-eastern corner, lost in the 19th century. The complex is built of masonry, except for the southern wall, which is built in rammed earth. The south-western tower, or keep, is divided into three floors plus a ground floor, occupied by a cistern. In some parts of the keep, traces can still be seen of the original decoration that would have covered the walls of the fortification — a pattern of leaves or drops using sgraffito techniques, decoration that would also have covered the walls and the solid tower. The tower's original door was heavily defended, double-leaved, and would have had a machicolation-like opening for pouring liquids or stones onto attackers, as well as a portcullis. This access was sealed off when a cistern was built inside the tower, occupying almost the entire ground floor. The vaulted roof of the cistern has disappeared, but two related elements survive: a settling structure and a well for drawing water. The archaeological excavations that uncovered this water tank also recovered the original staircase embedded in the tower's north wall. Both can be seen during a visit, and it is possible to walk over the cistern. The climb up the tower ends at the terrace, which offers wide visual control of the surrounding territory, with views of sites such as El Berrueco, Andújar, Arjona and Martos — explaining why this location was chosen for the fortification. Its position on a steep slope acted as a natural defence, but the northern and western sides, being more accessible, were protected by a wide moat around 7 m across and up to 2.10 m deep, hidden until it was uncovered during recent archaeological work. The moat ran between two towers reinforced with scarps that are still visible. From the 16th century onwards, a new door was opened in the western wall, with a Renaissance doorway carved in ashlar stone that survives today. It consists of two sections: the lower one with a semicircular arch on pilasters, and the second with a large rectangular panel at its centre, bearing the coat of arms of Don Fernando de Torres and his first wife, Doña Francisca de Carvajal y Osorio. The greatest transformation was the construction of wings attached to the walls, forming a large residential space. The Keep and the southern wall retain openings that would have connected them. Today, the ground floors of these structures can still be identified, although they would originally have reached two or three storeys in height. These wings housed the palace's "privileged space" — that is, the rooms of the Count and Countess, the chapel, the crypt, the entrance hall and a gallery-corridor or atrium, in the western wing — while in the southern building, the ground floor was used for storage, with the first floor reserved for the palace's main representative space, a large hall with a fireplace carved into the tower's façade. The structures to the north and east would have been dedicated mainly to service functions, including stables, kitchens and a cellar. Archaeological work confirmed the existence of a prehistoric, Copper Age settlement on the hill where the castle now stands. The presence of sickle teeth, mills and storage silos points to an agricultural economy based on cereal farming. A hydraulic structure, underground dwelling features and post-hole structures have also been found. There does not appear to have been continuous occupation of the site, suggesting the settlement must have been temporarily abandoned. From the early 1st century BC to the 2nd century AD, there was an increase in agricultural settlements devoted to cereal farming across the countryside. During this period, the site was reoccupied as a fortified settlement, reaching its peak between the first quarter of the 1st century and the 2nd century AD, after which it was abandoned once again. In the late Middle Ages, the term "villar" identified a place with the remains of an abandoned ancient settlement. Following the conquest of the territory, the Castilians settled here while Roman structures were still visible, choosing to reuse some of them, along with building material, to construct the castle. The hamlet of El Villar de Don Pardo is already mentioned in sources in 1251, but the castle was not built until the late 13th or early 14th century, with the fortification first referred to in 1416, as a place still belonging to the Council of Jaén. Fernando Ruiz de Torres later obtained the cession of the hamlet. In the second half of the 15th century, with Teresa de Torres — married to the Constable of Castile, Miguel Lucas de Iranzo — as lady of the estate, Villardompardo became one of the defensive lines of the supporters of Henry IV against the rebellious nobility, led in the area by the Order of Calatrava. The death of her son without heirs passed the lordship to a side branch of the Torres family, which married into the family of Peter I of Portugal. Philip II made it the seat of a county when he named Don Fernando de Torres y Portugal Count of Villardompardo in 1576. Between the late 16th and early 17th centuries, the castle was converted into a palace, serving as the usual residence of the counts. Their successors, however, settled at Court in Madrid, which began a process of decline through lack of maintenance. The abolition of entailed estates under the reign of Ferdinand VII forced the inheritance to be divided, separating Villardompardo Castle from the main noble line. In the late 19th or early 20th century, the 16th-century buildings were demolished, leaving only the perimeter walls, levelling the surface and carving steps into the northern wall to temporarily turn the enclosure into a bullring. The castle has undergone various emergency consolidation works, archaeological excavations and restoration between 2012 and 2023.

It was probably later used as a storehouse and tax-collection centre by the Order of Calatrava, once the town was incorporated into the Order's mastership in the first third of the 15th century. Today only a truncated tower survives, of regular floor plan and built in masonry, known as the Casa de la Tercia (13th century).

The castle played an important role at one of the key strategic points controlling the Guadalquivir and Guadalimar rivers. Its origins probably date back to the Caliphate period, although it was destroyed by Ferdinand III in the first third of the 13th century. Once under Christian rule, the castle was rebuilt between the end of that century and the beginning of the next. Its most authentic feature is its magnificent keep, square in plan, which houses two superimposed rooms inside, covered by barrel vaults. It was declared a Site of Cultural Interest in 1985.