Speaking to the BBC, the brothers said it wasn't until they heard a story about an abusive vicar on the radio that they realised they had both been abused as children by Rev Roy Cotton.
Phil Johnson explained: "... I could just see out of the corner of my eye that Gary froze. He stopped what he was doing and it was just like the penny dropped in that moment.
"I just turned to him and I said 'did anything happen to you with Roy Cotton?'"
Phil and Gary Johnson claim they were both abused by the Rev Cotton whilst they were members of the choir at St Andrews church in Eastbourne.
Gary Johnson said he felt "time sort of froze" when his older brother asked him about what he had kept a secret since he was a child.
He said: "What was going through my mind in the moment was less about what happened in the past. It was more about the fact that what I had tried to keep secret was now out.
"And I knew there was no way back. I didn't know what was going to happen next. I didn't know how to respond, so that's why I just didn't.
"But it was a horrible sensation. It was almost like being caught red-handed."
Rev Cotton, who died in 2006, was never prosecuted, as the brothers were told there weren't enough independent witnesses to prosecute him.
However, the brothers' case led to the prosecutions of several other clergy and a national inquiry ordered by Theresa May when she was home secretary.