In two years, bishops from across the world will gather in London for the once a decade meeting.
It comes at a time when conservatives and liberals in the worldwide Communion remain divided over the issue of sexuality.
Some groups such as the Scottish Episcopal Church and the US Episcopal Church have moved to a position of affirmation of same sex marriages and of gay clergy.
Others fall into the GAFCON movement, a group of conservative Anglicans, who feel those church groups are moving away from what the Bible teaches.
The new blow for Justin Welby came on Tuesday via the Archbishop of Uganda who's been speaking at the GAFCON conference in Jerusalem.
Addressing the issue of the Lambeth Conference, Most Rev Stanley Ntangali suggested he and the other bishops in his country would boycott the event because the Anglican Communion hasn't shown strong leadership on the issue of sexuality.
He said: "As a House of Bishops, because things have not changed we will not attend Lambeth 2020.
"Unless godly order is restored within the Anglican Communion, we shall not attend other meetings invited by Canterbury."
He also said his stance wouldn't change unless the Anglican Communion recognised the Church groups set up in Brazil and the USA to cater for the needs of conservative Anglicans who feel they can no longer serve in the current Anglican structures.
The Archbishop's comments were met with a standing ovation.
Premier has contacted the Anglican Communion and is waiting for a response.
Stay up to date with the latest news stories from a Christian perspective. Sign up to our daily newsletter and receive more stories like this straight to your inbox every morning.