Crabb quit his role as work and pensions secretary "in the best interests of my family" after the Times reported on the messages.
The married Tory MP has now spoken for the first time since the scandal saying he feels angry with himself.
"Forgiveness can come very quickly, rebuilding trust, that takes much longer," he told BBC Radio 4's PM programme.
When asked what role his faith played in getting through, he said: "Faith has been an important part of the last few months in reflecting on what's happened and how you address the consequences of that."
He said he was rebuilding at home and could not give a full and rational answer for his behaviour.
Mr Crabb insisted he did not want to return to the frontbench sooner than "two, three, four years", stressing he wants to be "more of a dad" to his children, who are currently at secondary school.
"I've been through the experience of rapid promotions, riding the motorbike at 100mph and coming off it," the Preseli Pembrokeshire MP said.
"I don't think there's anything wrong at the moment just to figure out the priorities."
Crabb is a committed Christian and once worked as an intern for the charity CARE.
"Questions about private life and all of that is excruciatingly embarrassing and painful and very, very painful for the family, and Parliament is a killer for relationships," Mr Crabb said.
"It's an absolute killer.
"And the hardest thing about being a Member of Parliament is keeping family life together.
"There's no special pleading, there's lots of people in jobs which require them to work away, spend lots of time away from spouses and partners and children.
"Very quickly all of that can go."
