A University of Bristol-led study found 80% failed to meet the government target of taking moderate exercise at least 12 times in a four-week period.
Better-off and better-educated adults were most likely to exercise, while the poorer and least educated were most likely to be inactive.
The study analysed exercise data for more than a million adults in England.
It found about 8% of adults who were physically able to walk had not walked for even five minutes continuously during a four-week period, while 46% had not walked for leisure for more than 30 minutes continuously.
Researchers said 88% had not been swimming, 90% had not used a gym and around 20% of people over the age of 16 had done only minimal amounts of physical activity.
They say the findings provide evidence of a direct correlation between an individual’s education, household income and local area deprivation and their level of physical activity.
Those living in areas with more sports facilities and higher local authority spending on new facilities were also less likely to be inactive.
The study also found warm weather made people more likely to exercise, while rain reduced levels of physical activity.
Carol Propper, professor of economics at the university’s Centre for Market and Public Organisation, said: “Physical inactivity is the most important modifiable health behaviour for chronic disease, so knowing who is physically inactive is important for designing cost-effective policy interventions.”
She said the findings suggested that “financial as well as cultural barriers need to be overcome to reduce the prevalence of physical inactivity”.
The NHS recommends people exercise at moderate intensity for at least two and a half hours every week. This can include cycling, fast walking, hiking and basketball. Experts also recommend muscle-strengthening activities on two or more days a week to work major muscle groups (legs, hips, back, abdomen, chest, shoulders and arms).
Tam Fry, spokesman for the National Obesity Forum and honorary chairman of the Child Growth Foundation, said: “No-one should be at all surprised by these woeful statistics.”