In an interview with the Mail on Sunday, the former Secretary of State for Wales - who was promoted after Iain Duncan Smith resigned over government plans to cut £1.2bn from disability benefits - also totally refuted claims that he has supported therapy or a cure for gay people, calling the rumours "complete c***".
He told the newspaper: "There's something unattractive about politicians who wave a flag called faith, but at a personal level it's important for me.
"The Old Testament talks about 'widows and orphans' - shorthand for the poor and vulnerable.
"Not a bad guiding star for social policy. Jesus Christ oozed compassion."
Stephen Crabb, the MP for Preseli Pembrokeshire in South Wales, currently attends St Mary' Church in his home village of Wiston with his French wife Beatrice and their two children.
He has received criticism for his links to the Christian lobby group CARE, and for voting against gay marriage in 2013.
He said: "I've never suggested being gay is a disease that requires therapy or a cure. I have great gay friends."
