A Government planning inspector has ruled the homes can be built in Rosemary Lane in Leintwardine, overturning a decision made by Herefordshire Council last year.
Inspector Anne Napier said developers had now agreed to a range of measures to minimise the impact on the village, which meant the council no longer objected.
Residents had claimed the development would overwhelm the village and today said they were "dismayed" at the decision, which was made following a planning inquiry that concluded last week.
Councillors will discuss their next step on Thursday at a meeting of Leintwardine Group Parish Council, which led a major campaign against the homes when they were first put forward about two years ago.
About 200 people had objected to the plans put forward by LWD Developments and 400 people signed a petition opposing the plans – almost half of the village's population of 830.
Mr Challis, who is best known for playing TV character Boycie and lives in nearby Wigmore, said he believed the decision to allow the homes would "destroy" the character of Leintwardine.
He said: "It's just ridiculous, there is absolutely no infrastructure to support it.
"The school is too small and there's hardly any bus service. Where are these people going to go? Where will these people park their cars? How are the roads around there going to handle it?
"I suppose I am being a 'nimby' but I think more people should be – this will destroy what Leintwardine is all about. It's a familiar story, it's happening all over the country."
Councillor Michael Collins, chairman of the parish council, said the decision had come as a nasty shock. He said: "The whole community is dismayed and we will be discussing what view the parish council will take at our meeting on Thursday."
Others against the homes when the plans were refused in January last year were Anne Brandrick, headteacher of Leintwardine Endowed Primary School, and North Herefordshire MP Bill Wiggin who wrote to the Secretary of State for Communities and Local Government with his concerns.
In her report, Ms Napier said: "The proposal would deliver significant social as well as economic benefits, and the development would be in a location that is within a reasonable distance of some local services and facilities.
"For the reasons given, the proposal would not result in unacceptable harm to the local landscape and visual amenities, but would have some benefits to the overall character and appearance of the area.
"As such, I conclude that any harm resulting from the proposal would be significantly and demonstrably outweighed by the overall benefits of the scheme."
A list of conditions has been made as part of the planning consent, including that 40 per cent of the housing is made up of affordable homes.