The Education Policy Institute has been examining Theresa May's proposal that would scrap the existing policy that lets faith schools select pupils because of their religion just half of the time.
Under the Prime Minister's plans Christian, Jewish and Muslim free schools would no longer be forced to accept young people from another faith.
She said it would increase the number of quality school places and increase social mobility.
But in its report the Education Policy Institute warned this "is unlikely to be effective".
The Department for Education told Premier the cap placed was "ineffective".
A spokesperson said: "Our proposals to expand the choice of good school places available to parents will help more young people, irrespective of their background, have the chance to go as far as their talents will take them.
"Faith schools are a vital part of this - they are among the best schools in the country and places are in high demand.
"That's why we want to remove the ineffective faith cap to establish even more good schools, while introducing new measures to improve inclusivity and diversity."
But Jonathan Romain, chair of the Accord Coalition for inclusive education, told Premier the report was accurate.
He said: "It would be unthinkable to divide children by race, it would be laughable, it would be illegal. We'd call it discrimination and yet somehow we seem to think it's alright doing the same thing by religion."
The Catholic Church has indicated it would open free schools if the rule was scraped.
Currently the faith is reluctant to do so because it fears the cap would mean it would have to turn Catholic pupils away, something it says would be against canon law.
