The Monroe County Historical Society began to collect local historical artifacts in 1938, housing them in a storied homestead, the Sawyer House on East Front Street, before presenting them to the Historical Commission established in 1967 by Monroe County. By 1972 through a generous gift. the commission could purchase this imposing building, formerly the U.S. Post Office erected in 1913.

It was on this site that Elizabeth Bacon Custer was born. daughter of judge Daniel S. Bacon, Her Libbie's romance with the dahsing George Armstrong Custer flourished. After the civil War, whenever his Army  duties did not station them elsewhere, she and the general made their home here.