The Rt Revd Michael Perham attended a police station in Gloucester on Tuesday but was not arrested.
Police confirmed they had spoken to a 66 year old man over allegations of offences in 1980 and 1981 relating to a child under 18 and a women over that age.
On Friday Bishop Michael announced he was going to "step back" from his duties with immediate effect.
The Diocese of Gloucester said it was a police matter and it had no comment.
The Bishop of Tewkesbury, the Rt Revd Martyn Snow, has taken over the duties of Bishop Michael.