In the latest setback for the inquiry, set up by then home secretary Theresa May, the Shirley Oaks Survivors Association, described it as a "stage-managed event which has now been contrived in such a way that it enables the guilty to wash their dirty hands, whilst the establishment pats itself on the back".
The group represents people affected by abuse at children's homes run by Lambeth council in south London. One of the abusers, ex-priest Philip Temple, has already been jailed.
The announcement comes two days after it emerged that another senior lawyer at the inquiry had resigned.
Aileen McColgan, a law professor at Kings' College London involved in the inquiry's investigations into the Anglican and Catholic Church, reportedly quit due to concerns over the inquiry's leadership.
Her departure follows the resignation of the inquiry's senior counsel, Ben Emmerson, and his junior colleague, Elizabeth Prochaska.
The inquiry has also had four different chairwomen since it was set up in 2014.
In its lengthy and highly critical statement, the association called on panel members to resign "for the sake of all those children who were abused historically", saying survivors were being re-traumatised by the perceived failings.
Peter Saunders, a Christian and the founder of the National Association for People Abused in Childhood, who sits on the inquiry's victims advisory panel, told Today: "Shirley Oaks is one group. Napac as a charity hears every year from many, many thousands - in fact probably more than 100,000 people have been in touch with Napac since our support line was launched.
"So let's not be distracted by just one group... They indicated they were probably going to walk away some time ago, but they are maximising their publicity. Whether that's good for child protection, I'm not quite so sure."
Home Secretary Amber Rudd said: "The Independent Inquiry has a vital role to play in exposing the failure of public bodies and other major organisations to prevent child sexual abuse.
"We must learn the lessons of the past and we owe it to victims and survivors to get behind the Inquiry, and its chair Alexis Jay, in its endeavour."
