Town councillors said the site should be relocated from Smithfield Car Park to the Eco Park, and criticised Shropshire Council for setting it up without consulting locals.
Further questions were also raised over why the site appeared to be under-used while Ludlow residents were being sent elsewhere for tests.
Public health chiefs have proposed keeping the site at its current location for up to another 12 months, as part of plans currently being drawn up to ensure testing remains available longer-term as the county emerges from the pandemic.
But the suggestion was not met with support by Ludlow Town Council’s representational committee.
Councillor Philip Adams said: “I just think it’s astonishing for them to put the thing there and now ask for our comments.
“If they put the Covid testing site at the Eco Park it would make use of a pretty redundant Eco Park, and at the same time it would protect the county’s income from the Smithfield.
“I can’t believe they’re asking us now, it’s a bit late in the day. But I would still say move it to the Eco Park.”
His comments were echoed by Councillor Viv Parry, who said Smithfield Car Park had been “packed” since all shops and outdoor seating areas for cafes, pubs and restaurants reopened.
She said: “It (the testing site) should be moved, there’s hardly anybody going to it.
“I have had people who have been telling me that they’re being told to go to Telford or Shrewsbury or Hereford for their tests.
“I personally think it should be moved out to the Eco Park.”
Councillor Parry said the site in its current location was also causing disturbance to people living nearby due to bright lights being “left on all night”.
Councillor Rod Naysmith said: “It’s ridiculous, the size of the thing. They could test everybody from the whole of Shropshire in there.
“The Eco Park is an ideal place for it.”
Councillor Robin Pote added: “The testing site now is beginning to impinge on Ludlow’s economic recovery, and we need space for customers to the town.”
Councillor Colin Sheward, chairing the meeting, proposed writing to Shropshire Council public health to say: “We do not want this testing centre at the car park, we would prefer it to be at the Eco Park.”
This was unanimously agreed by the committee.