The house was built in 1818 for naval hero Stephen Decatur and his wife Susan, it was the first residential property to be built on what was then known as President Park (now Lafayette Square), as the White House was just across the street.

The Decatur' only owned the home for a short time as Stephen died at a young age following a duel, Susan auctioned off the majority of the belongings and moved to a smaller home. After them the house belonged to different important political figures, including Henry Clay and Edward Livingstone. Also it was home to numerous slaves and servants. The National Trust For Historic Preservation took over the house in 1956 after it was bequeathed to them by Marie Beale, whose family has occupied the house for 84 years.

The house is one of the oldest surviving in Washington, and one of only three surviving which were built by the designer Benjamin Henry Latrobe who was America's first professional architect and engineer.

Stephen is said to haunt his former home, seen as a shadowy figure mainly in a upstairs wibdow or coming out of the backdoor, also the sound of mournful weeping as been heard this is thought to be his wife.