Bishop Kieran Conry, 63, stood down from the role in the Diocese of Arundel and Brighton on Saturday, saying he had been 'unfaithful" to his promises as a Catholic priest "going back some years".
Local newspaper The Argus is reporting that he will now be forced to leave his diocese-owned gated house near Crawley.
A Sunday newspaper alleged that Rt Rev Conry had a relationship with a married woman and the Mail on Sunday claimed she had stayed with him at the home.
A stand-in diocesan administrator should be announced before the end of the week, but the search for a new bishop could take up to a month.