Most of the outer defences of the fort survive including parts of the outer works which extend into the adjacent recreation grounds to the east and west. Some of the internal buildings date to the period of use as a hospital and possibly earlier, maybe even preceding the defensive works. From the hill it is possible to see Fort Clarence tower to the west, and the remains of Fort Amherst to the north east. A ditch, wall and (reputedly) tunnels were to link the three into a single fortified Napoleonic defensive line, defending the naval docks against a (land-based) attack from the south.
Fort Pitt is also the location of the fictional duel between Mr. Tracy Tupman and Dr. Slammer in the Charles Dickens novel The Posthumous Papers of the Pickwick Club.