Video footage capturing the moment Fr Jacques Lacroix struck a two-year-old boy across the face with the palm of his hand went viral on the internet last week.
During an interview with France Info radio, the 89-year-old said: "I am finishing my ministry now, it was my last baptism, there is an end to everything."
"It was somewhere between a caress and a slap, I hoped to calm him down, I didn't know what to do."
The Bishop of Meaux - east of Paris - said Fr Jacques had been suspended from all baptism and marriage ceremonies, adding that the baby's family had accepted the apology.
Jean-Yves Nahmias said the priest had been guilty of a "loss of coolness that can be explained but not excused".
You made find the following video distressing.
He added: "This gesture is all the more regrettable because the baptism is meant to be a happy moment, but tiredness and great age clearly played a part."
Fr Jacques prompted gasps from the congregation when he slapped the baby - the exact location and date of the incident are unclear.
Footage showed his mother urging the priest to let her son go, before the father was able to wrestle the infant away.
Stay up to date with the latest news stories from a Christian perspective. Sign up to our daily newsletter and receive more stories like this straight to your inbox every morning.