The Doolittle house was built by Colonel Ephraim Doolittle, who first saw the area during the French and Indian War. Doolittle obtained the charter for the town in 1761 and returned there to settle after the Revolutionary War. He and his wife Sarah built the brick Federal style house in the 1790s