The psychology professor, who has expressed divisive views on topics including gender and race, accused bosses of caving into a "inclusivity mob" after his invitation was rescinded last week.
The university declined at the time to comment. But, in a statement, Vice Chancellor Stephen Toope said Prof Peterson had made a "casual endorsement by association".
In a statement, he said: "The faculty became aware of a photograph of Professor Peterson posing with his arm around a man wearing a T-shirt that clearly bore the slogan 'I'm a proud Islamophobe'.
Statement from Vice-Chancellor Professor Stephen J Toope about the decision by the Faculty of Divinity to rescind Professor Jordan Peterson's visiting fellowship.https://t.co/BExP7G0BED
— Cambridge University (@Cambridge_Uni) March 25, 2019
"The casual endorsement by association of this message was thought to be antithetical to the work of a faculty that prides itself in the advancement of inter-faith understanding.
"Some difficult decisions will always be necessary to ensure that our universities remain places of robust, often challenging and even uncomfortable dialogue, while balancing academic freedom with respect for members of our community."
Jordan Peterson requested a visiting fellowship at the Faculty of Divinity, and an initial offer has been rescinded after a further review.
— Faculty of Divinity (@CamDivinity) March 20, 2019
The image was taken in New Zealand last month, according to The Times.
The newspaper reported that the faculty became aware of the picture early last week; at the same time Prof Peterson announced he had been invited to begin a visiting fellowship in the autumn of 2019.
Premier has contacted Jordan Peterson for a comment.
Stay up to date with the latest news stories from a Christian perspective. Sign up to our daily newsletter and receive more stories like this straight to your inbox every morning.
Tell us what you think about our news service, what you want and how Premier fits into your news landscape. Take our quick survey!