Baroness Butler-Sloss was due to lead a review into child sex abuse at public institution including the Church of England.
"It has become apparent over the last few days, however, that there is a widespread perception, particularly among victim and survivor groups, that I am not the right person to chair the inquiry." She said.
She had faced accusations she's wasn't the right person for the job.
In a previous investigation into abuse in the church she was accused of telling an abused choir boy she wanted to exclude some of his claims from her report so she could protect the Church of England.
She said she has never put an organisation in front of a victim.
"This is a victim-orientated inquiry and those who wish to be heard must have confidence that the members of the panel will pay proper regard to their concerns and give appropriate advice to Government.
"Nor should media attention be allowed to be diverted from the extremely important issues at stake, namely whether enough has been done to protect children from sexual abuse and hold to account those who commit these appalling crimes.
"Having listened to the concerns of victim and survivor groups and the criticisms of MPs and the media, I have come to the conclusion that I should not chair this inquiry and have so informed the Home Secretary." She said.
Downing Street said the decision was made by Baroness Butler-Sloss alone.
Home Secretary Theresa May said: "I am deeply saddened by Baroness Butler-Sloss's decision to withdraw but understand and respect her reasons.
"Baroness Butler-Sloss is a woman of the highest integrity and compassion and continues to have an enormous contribution to make to public life.
"As she has said herself, the work of this inquiry is more important than any individual and an announcement will be made on who will take over the chairmanship and membership of the panel as soon as possible so this important work can move forward."