The property was built on a large tract of land patented in 1703 by an immigrant Scot, Ninian Beall. This land was originally named "Rock of Dumbarton�? in tribute to Beall’s homeland. In 1798 developer Samuel Jackson bought 4.5 acres of "Rock of Dumbarton" and began to construct the property that eventually became known as "Cedar Hill." In order to build this house and others in the area, Jackson employed craftsmen who had flocked to the new Federal city after its creation in 1791. Although construction began in approximately 1799, work was halted when Jackson went bankrupt. Not having an intended owner for the house, it went up for public auction, and eventually was acquired by Joseph Nourse, first Register of the U.S. Treasury. Nourse completed construction of the home and he and his family lived at "Cedar Hill" from 1804-1813. In 1813, the home was bought and renamed "Belle Vue" by Charles Carroll. During Carroll’s residence, the house would host Dolley Madison on August 24, 1814, during her flight from the White House and British invaders
In 1915, when the Dumbarton ("Que Street") Bridge was built over Rock Creek, the house was moved 100 feet to its present site, to allow for the extension of Que Street into Georgetown. The National Society of The Colonial Dames of America (NSCDA) purchased the house in 1928 and restored it to its early 19th century character, changing the house’s name to "Dumbarton House." The NSCDA declared Dumbarton House its national headquarters and opened it as a museum to the public in 1932.