A survey, carried out by YouGov for the Times newspaper, showed that 55% of those asked either didn't want any worship in schools or wanted children given the right to opt out without a parent's consent.
But the same poll showed 35% of people backed an obligation on the school for religious worship.
The British Humanist Association, which wants to strip away religion from all public life, claimed the poll showed an "urgent need" for change.
Jay Harman from the BHA said: "This poll is just the latest in a long line of evidence and research demonstrating that the requirement for compulsory religious observance in UK schools is discriminatory, anachronistic, and unpopular."
"The Scottish Government, as well as the governments across the UK, should move immediately not just to ensure a full right of withdrawal for children and young people, but also to repeal the requirement for collective worship."
Currently the Humanist Society of Scotland is seeking a judicial review of the Scottish Government's decision not to allow older pupils to opt out of worship without parental consent.
A spokeswoman for the Scottish government said: "Religious observance should be sensitive to individual beliefs, whether these come from a faith or non-faith perspective.
"Parents are legally entitled to withdraw their children from religious observance in our schools.
"While there is currently no legal basis for children to remove themselves from religious observance, the flexible approach to learning and teaching afforded by Curriculum for Excellence in Scotland encourages schools to discuss options with both parents and their children, this could also include allowing pupils to withdraw from religious observance if they wish."
