The claim is made in a document released by the Department for Communities and Local Government as the government reviews the way crematoria in England are run.
Officials claims many crematorium chapels now have crosses and other symbols which are removable and can be put on display, according to the wishes of each funeral.
In a statement, the department said: "Whilst we are a Christian country, it is important that we ensure there are high quality crematoria to meet the needs of families of different faiths as well."
Graham Kissack, a pastor from Peel Street Baptist Church in Accrington, Lancashire, said Christians sometimes need to make allowances for other faiths, however, some areas are non-negotiable.
Speaking on Premier Christian Radio's News Hour programme, Graham said it was wrong for the government to interfere with affairs of the Church but having removable crosses seems like a fair compromise.
He added: "Where we've got freedom of conscious, we can negotiate. We can look at compromise but on the very strong Biblical truthes, those are areas where we won't compromise."