In a statement, Rt Rev Dr Stephen Croft apologised to the survivors of Peter Ball's abuse and said the Church of England's actions of failing to report Ball's actions were inexcusable.
He added: "We must now act on the recommendations put forward by Dame Moira Gibb, and all bishops must demonstrate our accountability for making sure everyone in our Church is kept safe.
He also explained the Archbishop of Canterbury has written to Lord Carey and asked him to "carefully consider his position as honorary Assistant Bishop".
Bishop Stephen continued: "As I hold responsibility for granting him a licence to enable him to carry out his duties, Archbishop Justin has asked Lord Carey to talk to me and we have agreed to meet in the coming days for that conversation.
"In the meantime he has voluntarily agreed to step back from public ministry."
