Teachers tend to talk too much in sessions and often lack specialist training, the Ofsted report on primary and secondary PE adds.
It also highlights the fact that only a minority of schools play competitive sport at high level.
The government says its draft PE curriculum will put competitive sport back at the heart of school life.
However the report found some important positive points, with most schools providing at least two hours of PE a week for pupils aged five to 14 and only a fifth of primary schools did not ensure that all pupils could swim before they left.
However, it was concerns about the content and nature of some PE lessons than in some weaker lessons that chief inspector Sir Michael Wilshaw highlighted.
He said: “Generally, PE in our schools is in good health, but there are some issues the report highlights as areas for improvement. In particular, we found there often wasn’t enough physical, strenuous activity in PE lessons.
Ofsted inspectors visited 120 primaries and 110 secondaries over a four-year period.