Patrick Coppeard from Loughton, Essex, faces a prison sentence after pleading guilty at Chelmsford Crown Court on Monday.
The former magistrate, who worked at investment bank Merrill Lynch, promised friends at St John's Church in Buckhurst Hill huge profits if they invested money with him.
But the Ponzi-style scheme reportedly cost some people up to £250,000 after being promised swift returns on their investments by Mr Coppeard.
Last May, Mr Coppeard turned himself in to Rev Dr Ian Farley, rector of St John's Church, and they went to a police station together.
Speaking to The Telegraph, Rob Smith, a retired company owner and managing director, said he was defrauded out of tens of thousands of pounds by Mr Coppeard, who was his best friend and next-door-neighbour for a decade.
"Towards the end I suspected that something wasn't quite right.
"The day before [Mr Coppeard] turned himself in to the police, he told me had been lying to me for five years and he had lost all my money.
"Here were six of us who started investing in him as a little club. It was just for fun. I had no idea that it had escalated so much, that his tentacles had spread so far. He betrayed everybody's trust."
Mr Smith said he had forgiven Mr Coppeard, but added that many parishioners remained "extremely bitter".
"After [the sentencing] I hope we will be able to draw a line under all this. It has been hanging over us like the Sword of Damocles."
Mr Coppeard has been handed conditional bail ahead of sentencing later this month.