Confederate diarist Mary Boykin Chesnut wrote on February 16, 1864, that she saw in the Confederate executive mansion "the little negro Mrs. Davis rescued yesterday from his brutal negro guardian. The child is an orphan. He was dressed up in little Joe's clothes and happy as a lord." The Confederate First Lady Varina Davis recounted the story in her 1890 memoir and claimed that the president "went to the Mayor's office and had his free papers registered to insure Jim against getting into the power of the oppressor again." The free black register and other records that could corroborate or contradict her account apparently have not survived. Nineteenth-century Virginia law did not provide for formal adoption of children. Jim's status in the Davis household seems to have been informally that of a ward or what modern Americans would call a "foster child."

An ambrotype photograph taken of Jim Limber early in 1865 and correspondence between members of the Davis family suggest that he was a close playmate of the Davis children. Late in April 1865, as the Davis family fled southward from Richmond, Varina Davis wrote to her husband: "The children are well and very happy—play all day—Billy & Jim fast friends as ever … "

Jim was separated from the Davises after their capture in May 1865. A member of the Davis party wrote in her diary that Varina Davis's "pet Negro" had been taken from her. The Davises entrusted his care to an old army friend, Union general Rufus Saxton, whom Varina Davis asked "to look after our little protégé Jim's education, in order that he might not fall under the degrading influence" of a menacing Union officer. When the child realized he was to be separated, according to Davis, he "fought like a little tiger and was thus engaged the last we saw of him. I hope he has been successful in the world for he was a fine boy, notwithstanding all that had been done to mar his childhood."

Contrary to modern renditions of Jim's biography, there is no evidence that the Davises subsequently searched widely for him. Indeed, Varina Davis's own account of their separation indicates that she understood it to be permanent.

The last recorded evidence of him comes from the 1893 memoir of Elizabeth Hyde Botume, a Boston woman who came south to teach the freedmen on the South Carolina Sea Islands. Botume recalled Jim as "about seven years old, but small for his age; he was a very light mulatto, with brown curly hair, thick lips, and a defiant nose." She quoted from memory Varina Davis's note to Saxton describing how the child had come into their home and stating her intention "to keep him until he was old enough to learn a trade." Botume confirmed that Jim had been "the constant companion and playmate of Mrs. Davis's children" and "considered himself as one of them." But he apparently transferred his affections easily to his "new protectors," the Saxtons. They, in turn, gave him to the care of teachers, who took him north for schooling. He reportedly became "well-trained in all ways, having the advantage of school, as well as a good practical education, until he was old enough to support himself."

In 2008, the Sons of Confederate Veterans (SCV) announced a campaign to finance a new life-size bronze statue depicting Jefferson Davis, along with his son Joseph Evan Davis, who died tragically in 1864 from a fall, and Jim Limber. According to a column in the SCV's magazine, Jim Limber is "a person lost in history by revisionist historians, who felt his existence would impair their contrived notions of Davis." The SCV offered the statue to the American Civil War Center at Tredegar Iron Works to balance the statue of U.S. president Abraham Lincoln and his son Tad placed at Tredegar in 2003. In August 2008, the Center's board voted to accept the statue, but when museum officials did not guarantee where or whether the statue would be displayed or explain how it might be interpreted, the SCV rescinded its offer of the statue in November 2008.