2020 census

As of the 2020 United States census, there were 722 people, 370 households, and 236 families residing in the town.

2000 census

As of the census^[4] of 2000, there were 533 people, 224 households, and 154 families residing in the town. The population density was 469.9 inhabitants per square mile (181.4/km²). There were 262 housing units at an average density of 231.0 per square mile (89.2/km²). The racial makeup of the town was 79.22% White, 12.15% African American, 0.75% Asian, 0.38% from other races, and 1.50% from two or more races. Hispanic or Latino of any race were 0.38% of the population.

There were 224 households, out of which 33.0% had children under the age of 18 living with them, 40.6% were married couples living together, 24.1% had a female householder with no husband present, and 31.3% were non-families. 29.5% of all households were made up of individuals, and 11.2% had someone living alone who was 65 years of age or older. The average household size was 2.38 and the average family size was 2.92.

In the town, the population was spread out, with 26.3% under the age of 18, 8.4% from 18 to 24, 26.3% from 25 to 44, 22.1% from 45 to 64, and 16.9% who were 65 years of age or older. The median age was 36 years. For every 100 females, there were 76.5 males. For every 100 females age 18 and over, there were 70.9 males.

The median income for a household in the town was $32,708, and the median income for a family was $39,306. Males had a median income of $33,000 versus $24,583 for females. The per capita income for the town was $16,007. About 14.6% of families and 14.5% of the population were below the poverty line, including 21.2% of those under age 18 and 11.0% of those age 65 or over. Swansea is the home of Poole Training Center.