Rt Rev James Newcome has been asked to stand down from his role in two charities - the Royal British Legion and the Children's Society - while a character reference he gave about an abusive priest is looked into.
He was vice-president at the Children's Society and national chaplain at the Royal British Legion.
The decision is because of what he calls an "error of judgement" regarding the priest Rev Robert Bailey, who was sentenced to eight years and four months in prison in September for sexually assaulting two girls. The crimes were not committed during his time in the Diocese of Carlisle.
Bishop James gave a character reference for Bailey in April this year for the pre-sentence stage of the process.
Last month the Diocese of Carlisle said in a statement: "We can confirm that the submission of a character reference for Bailey by the Bishop of Carlisle, the Rt Rev James Newcome, and which formed part of a pre-sentence report, is to be investigated further by the Church of England's National Safeguarding Team."
Bishop James said: "Robert Bailey was a vicar in the Diocese of Carlisle from 1999 until his retirement to Salisbury in 2011 and is someone I have known for 18 years. In April of this year, and at his request, I agreed to provide a character reference to go before the courts.
"On reflection I considered this to have been an error of judgement on my part and asked that the reference be withdrawn.
"I'm truly sorry that the reference was submitted and understand the pain that this will have caused those who suffered as a result of Robert Bailey's crimes. I offer them my heartfelt apology.
"I will, of course, co-operate fully with the national church safeguarding team and any subsequent investigation around this matter, as I seek to learn from this situation."
A diocese spokesperson said: "It is important to note that there are no allegations of Bishop James presenting a direct risk of harm to any children or vulnerable adults."
He has been removed from any decisions involving safeguarding in the diocese.