So far, 29 people have tested positive in Craven Arms after an outbreak at the Long Lane park. Yesterday, Shropshire Council said all of the travellers had been asked to self-isolate and not leave the site, but that some had ignored the guidance.
But some of the group say they have been told by the council that they are allowed out of quarantine.
A member of the group called James, who did not wish to give his surname, wanted to make it public knowledge that some of the travellers are allowed out, and asked for people to stop giving them "dirty looks".
He said: "I got stopped this morning by police. There's nothing but negativity towards us. People have just been giving my family dirty looks in the street and at the petrol station.
More Covid-19 coverage:
"We're not stupid. We know not to go out when we have to quarantine. We don't want to pass the virus around."
He added: "A lady in one shop said she won't serve us any more, it's losing her business.
"But there is one guy who can't do enough for us. The whole week he's gone out of his way to help.
"There has been a bit of negativity before but it's not usually too bad. When we first moved in in 2017 everyone loved us because they had problems with the previous people."
James said around half of the group have received letters to say they do not need to quarantine, and that no-one is showing Covid-19 symptoms.
Meanwhile, police are considering taking action against people who do leave the camp when they should be self-isolating.
Craven Arms councillor David Evans has insisted that the town is still "open for business" despite the outbreak, and said visitors are still welcome.
Shropshire Council has been contacted for comment.