It appeared at the Old Church of St James in Stanstead Abbotts, Hertfordshire.
Hertfordshire Police said the grave was reported to officers on Monday but may have been in situ since September 2016.
A person has been seen tending the plot, which is "well kept" and contains bones of an unascertained age, a force spokesman said.
Detective Inspector Anna Borella said: "The church is only used around four times a year, but we know that people regularly visit the graveyard.
"We are appealing to anyone who has seen someone visit the mystery grave or who has information which may help us identify it to contact us.
"The grave is situated close to the church and there are no graves directly next to it, so anyone visiting may stand out."
