The National Governors' Association says they are "meaningless" and staff often can't or won't pray with students.
The comments by the organisation which represents more than 300,000 school governors in England are supported by the British Humanist Association.
Emma Knight is the Chief Executive of the National Governors Association. She's been telling Premier why the organisation wants change.
Revd Ronni Lamont is a former teacher and now provides schools with assembly materials.
She told Premier's News Hour she thinks there'll always be a Christian element at school assemblies.