Police spokesman Ruwan Gunasekara says the remains were removed from Kalliyankadu cemetery in the presence of a judge, a medical officer and police and were taken to a hospital mortuary.
They will remain there with police protection until the government provides a new burial site.
An official last week said the remains consisted only of a head, but Mr Gunasekara said they also included some other body parts.
Hundreds of people had protested against the burial of Mohamed Azar, who attacked Zion Church in Batticaloa town, killing 27 people and wounding more than 70 others.
Stay up to date with the latest news stories from a Christian perspective. Sign up to our daily newsletter and receive more stories like this straight to your inbox every morning.
