Eric Adams, a Democratic politician, also encouraged off-duty police officers to follow suit.
The retired cop-turned-politician told attendees of a public press conference on Sunday: "I used to carry my gun all the time when I went to church.
"If we have police officers standing in front of churches, then we can't say it's wrong for a police officer who's off duty to be inside churches with a gun.
"If they're leaving those firearms home, I now say to them, 'top leaving your firearm home. Do as I do. Bring your firearm to church'."
On Saturday, Robert Gregory Bowers killed eight men and three women inside the Tree of Life Synagogue during worship services before a tactical police team tracked him down and shot him.
Authorities say Bowers made statements about genocide and killing Jewish people.
He is being treated for gunshot wounds and is due in court on Monday.
A chief prosecutor in Pittsburgh has said they are seeking approval for the death penalty against the mass shooting suspect.
Stay up to date with the latest news stories from a Christian perspective. Sign up to our daily newsletter and receive more stories like this straight to your inbox every morning.