The Secretary of State for the Environment, Food and Rural affairs was speaking during an interview with Elizabeth Oldfield from the Christian think tank Theos.
Mr Gove said: "One of the things I find of most people who have a personal faith is it makes you more conscious of your fallibility.
"I don't want to name names but there's a well-known thinker who is unabashedly atheist who argues with a degree of certainty and absolute assurance in the power of his logic that very very few religious people in my experience would ever dare attempt to muster."
The comments came following the 11th annual Theos lecture, which Mr Gove gave. His speech was entitled 'When will there be a harvest for the world?' and focused on issues relating to the environment.
Mr Gove argued politicians should not apologise for "drawing, and reflecting, on the insights and wisdom found in the words, actions and thoughts of religious leaders.
"I speak as a Christian, convinced that the moral teachings of the Christian Church, the example of Christ's life and Christian theology all help us in reflecting on the responsibilities we all have to others.
"For Christians, the ethical responsibility we have towards the environment is encapsulated in the concept of stewardship. Christians are called to remember their rightful place within Creation – and the vast web of life it created – and their responsibility to protect and defend it.
"If, as Christians, we believe creation is a gift we must preserve, then we also believe creativity is a gift we must use to the full. So when it comes to the agriculture of the future, we must develop new technologies which improve productivity while also respecting nature's limits. "
When Mr Gove was later asked what spiritual habits he would recommend for society, he responded, "Always weigh the impact of your words on others. While you can apologise, what's said very often cannot be unsaid.
"One of the truths of most faiths and certainly of Christianity is that it enjoins upon you the vital importance of empathy and appreciating the positions that others occupy and the difficulties they face in their life".
Speaking about a lesson he learned while Justice Secretary, Mr Gove said: "There are very few people who are truly evil. There are those who have done evil things, but they are capable of redemption. Even when people do the wrong things, an element of imaginative sympathy and understanding is necessary even as you might criticise and condemn. Because what you mustn't do is so execrate and damn so that you deny the humanity in another human being."
Stay up to date with the latest news stories from a Christian perspective. Sign up to our daily newsletter and receive more stories like this straight to your inbox every morning.