A group of London Quakers, including William Penn, drafted the first settlement plans for the area that includes Audubon, and the first English settlers arrived in 1682. Named for naturalist John James Audubon, the borough of Audubon obtained its incorporation charter in 1905.

The land was divided into a number of farms by early settlers , including such town notables as the Cedarcroft, Bettle, Nicholson, Willits, Sutvan, Ward, and Orston families.

Audubon is proudly home to three Medal of Honor recipients: Samuel M. Sampler, World War I; Nelson V. Brittin, World War II and the Korean War; and Edward C. Benfold, the Korean War. The U.S. Navy has honored Brittin of the U.S. Army and Benfold of the U.S. Navy, both killed in action in Korea, with naval vessels named in their memories:

The USNS Brittin (christened in 2000) and the USS Benfold (christened in 1994).