George Washington Parke Custis, grandson of Martha Washington and adopted son of U.S. president George Washington, built Arlington House. Naming it for a Custis property in Maryland, he saw Arlington not only as home but also as a monument to the first president, as well as a place proudly to display his large collection of items previously owned by George Washington. The house was based on a design by the Italian-born English architect George Hadfield, who had worked on the United States Capitol. Construction began in 1802 and continued over several decades before its completion in 1818. Here the young West Point graduate Robert E. Lee courted his distant cousin Mary Anna Randolph Custis, daughter of G. W. P. Custis and Mary Lee Fitzhugh Custis. Here, too, the couple married on June 30, 1831, maintaining their home at Arlington until forced out at the beginning of the Civil War. Lee described it as a place where his "affections and attachments are more strongly placed than at any other place in the World."

Arlington encompassed 1,100 acres, where Custis-owned slaves cultivated crops including corn and wheat. After the 1857 death of his father-in-law, Lee took over management of the estate, which his wife had inherited. It was a time of frustration for some slaves, who had anticipated freedom at the death of G. W. P. Custis. Lee, dealing with a complex will and large residual debt, called it "an unpleasant legacy," and did not formally free the Custis slaves until January 1863.

On April 22, 1861, at Arlington House Lee made his famous decision to resign his U.S. Army commission and side with Virginia, whose convention in Richmond had just voted to secede. Arlington's location, close to the Potomac River, meant that continued residence by Lee, or his family, would be untenable. Lee's family departed the following month, though Mary Custis Lee had said as late as May 5, 1861, that she "would not stir from this house, even if the whole northern army were to surround it." Early in the war Arlington became the headquarters of Union general Irvin McDowell while he commanded forces defending Washington. It was a hub of military activity: the house provided office and residential space, while the grounds witnessed large-scale troop movements and encampments.

Arlington House also served another purpose: in May of 1863, Union officials selected the grounds of Arlington House as the site for a Freedman's Village where newly emancipated slaves could establish themselves. Formally dedicated in December 1863, the Freedman's Village would grow into an elaborate community, complete with a hospital, schools, and other public buildings. Despite persistent efforts to remove the African American community, residents successfully resisted overtures to eliminate the village until 1900. In June 1864, the Quartermaster General of the United States Army, the Georgia-born general Montgomery C. Meigs, ordered that a section of those grounds become a military cemetery, though the first burial had occurred the previous month. Revenge motivated these two developments, at least in part; Union authorities wanted to drive home the point that Lee, considered to be a traitor, had forfeited any claim to Arlington House.

The Lee family lost ownership of the Arlington property during the war, but a U.S. Supreme Court decision in U.S. v. Lee (1882) ruled that the house had been seized without due process and resulted in a congressional act in 1883 authorizing the purchase of the property from Washington Parke Custis Lee for $150,000. The War Department began restoration of the house in 1925, and the National Park Service took over management in 1933. In 1955, Arlington House was designated a memorial to Robert E. Lee, and in 1966 it was placed on the National Register of Historic Places. Welcoming numerous visitors, the National Park Service and the Arlington House staff continue to tell the story of the Custis and Lee families in general and of Robert E. Lee specifically. Through the use of various interpretive tools, however, the story is unfolding more expansively to include the many African Americans and their families who once labored and lived here, in slavery and in freedom.