Adolescence is thought to be the time when children go off exercise - but a study in The British Journal of Sports Medicine suggests it happens much earlier, around the age of seven.
Sitting is replacing physical activity from the time children start school, the research suggests.
Children should get at least an hour of exercise a day.
Many of the 400 children enrolled in the study did less than this as they got older.
The experts from Glasgow and Newcastle tracked the activity levels of the children over eight years using monitors worn for a week at a time.
The amount of exercise the children did was measured at age seven and then again at age nine, 12 and 15.
On average, boys spent 75 minutes a day exercising when they were seven, falling to 51 minutes when they were 15.
The average girl spent 63 minutes per day doing moderate to strenuous physical activity when seven years old, which fell to 41 minutes age 15.
Most boys and girls in the study did moderate levels of exercise at seven, which then gradually tailed off.
But one in five of the boys bucked the trend and managed to maintain their exercise levels over the eight years.
They were the ones who started off with the highest levels of activity at the age of seven, the researchers said.
Sitting too much
Although the study cannot prove what causes the drop-off in physical activity, Prof John Reilly, study author from the University of Strathclyde, said "something is going wrong in British children" long before adolescence.
He said it coincided with the peak rate of obesity cases in children and the greatest increases in weight gain - which happen around the age of seven.
Different research on the same group of children found that the time lost to exercise was spent sitting instead.
Children aged seven spent half their day sitting, and by the age of 15 this had gone up to three-quarters of their day spent sitting.
"Activity tails off from around the time of going to school, when there's a change in lifestyle," Prof Reilly said.
"Schools should be more active environments. There should be more activity breaks to break up long periods of sitting."
But he emphasised that activities outside school also had an important role to play because children only spent half of their year at school in total.
The children who took part in the study lived in Gateshead in north-east England and were tracked between 2006 and 2015.
Eustace de Sousa, national lead for children, young people and families at Public Health England, said: "It's a major concern that one in five children leaves primary school obese.
"Most children don't do enough physical activity, which has consequences for their health now and in the future," he said.
"It's up to all of us to ensure children get their recommended one hour of physical activity a day."
Mr De Sousa said this principle was at the core of the government's childhood obesity plan, which provided extra funding for schools to get children moving and support for families to keep children active outside of school.
NHS Choices says children and young people should cut back on the time they spend watching TV, playing computer games and travelling by car.
How much exercise should children be doing?
§ at least 60 minutes of physical activity every day - this should range from moderate activity, such as cycling and playground activities, to vigorous activity, such as running and tennis
§ on three days a week, these activities should involve exercises for strong muscles, such as gymnastics, and exercises for strong bones, such as jumping and running
Source: NHS Choices
Five tips for getting your child to be more active
§ walk or cycle to school as often as you can
§ find time every weekend to do something active with your children
§ take the dog for a walk - if you haven't got one, borrow one
§ support your child in any sport, club or activity that interests them
§ take part in a fun run or a charity challenge together