He made the comments after facing criticism from Conservative colleague Phillip Davies, who accused him of being too liberal with his justice policies.
Specifically, Mr Davies accused the Justice Secretary of taking too much advice from the Howard League of Penal Reform, and putting prisoners ahead of victims.
The Howard League of Penal Reform campaigns on various issues such as, giving the vote to prisoners, the rehabilitation of prisoners, the problem of the death penalty and the standard of prison conditions.
Speaking in the Commons, Mr Gove said: "I am not sure that MPs on the Opposite [Labour] benches would agree I've become a sandal-wearing muesli-munching vegan vaguester.
"I think they would probably say I'm the same red in tooth-and-claw blue Tory I have always been.
"It's because I am a Conservative I believe in the rule of law as the foundation stone of our civilisation.
"It's because I'm a Conservative I believe that evil must be punished. But it's also because I'm a Conservative and a Christian that I believe in redemption.
"The purpose of our prison system is to keep people safe by making people better."
