A wood and perspex cover protecting the Grade II-listed "African grave" may have to be removed as it does not have permission to be there.
But the locals who made the cover and put it there to save the 200-year-old monument say action needs to be taken now to stop it eroding further.
The grave belongs to a man only identified as ID, who is said to be a native of Africa who died in Bishop's Castle on September 9 1801.
The headstone contains the words "God hath created of one blood all nations of men" from Acts chapter 17 – a biblical passage often used by those campaigning for the abolition of slavery.
But mystery surrounds who exactly ID was, who put up his headstone and how he came to be in Bishop's Castle.
Now town residents are worried the stone, included in a list of English Heritage "sites of memory" to commemorate the Abolition of the Trans-Atlantic Slave Trade Act in 1807, is deteriorating and will soon be unreadable.
Residents Graham Whatley and Bamber Hawes decided to take matters into their own hands and create a cover for the stone, which sits in the churchyard of St John the Baptist Church in Church Lane.
But they have been told they must remove it again by representatives of the Diocese of Hereford.
Mr Whatley said the red tape involved in getting some kind of official protection for the stone would need reports to multiple bodies and applications for funding which would all take months when immediate action needed.
He said: "The procedure is arcane, stupid and not fit for purpose, and while those who care struggle angrily, the gravestone deteriorates – so the system fails its objectives."
Diocese of Hereford property secretary Stephen Challenger said he was "surprised" to learn the cover had been put in place without consent. In a letter to Mr Whatley he said: "I understand your concern that this important memorial should not be lost and we have provided you with information about taking the right steps to ensure its future.
"The first step must be to have a professional assessment of the condition of the memorial and appropriate conservation proposals. The most likely source of funding for such a report is the Church Buildings Council.
"Until an agreed plan of action is in place the cover must be removed from the memorial," he added.