In an article for the New Statesman the clergyman said speaking ill of God was "the language of protest" and it was vital that everyone was allowed to speak out against Christianity.
"If you are forbidden to voice the hard questions, this might suggest that faith survives only by never being challenged," he wrote.
Lord Williams added: "The person who actually expresses their fury or disgust or disillusion can, at least sometimes, be demonstrating faith of a sort, confidence that, if God is real, it is possible, even necessary, to say what you feel about Him - and that, unless you can say this, the God you started with is not worth believing in.
"If God is real, these "thought experiments" are all to do with testing what we really believe."
He added that his view underpins many of the Jewish Psalms or the poems of George Herbert or Gerard Manley Hopkins.
"Blasphemy resists the conspiracy of silence about the agonising difficulties of belief, resists the stifling of a real and honest response to an unjust world," he said.
"Herbert, Job and some of the Psalms remind us that sometimes the seriousness of faith is most effectively explored precisely in the risky business of testing the limits. And without such testing, such forcefully expressed doubt, you may never know the real strength or weakness of what you claim to believe."
It's not the first time the former Archbishop has made a controversial comment. In 2008 he was forced to deny he backed Sharia law following a speech he made.