The planning permission was granted for the home in Hendomen Lane, Montgomery, in December last year.
Residents say that there were strong objections to the application and one person is now taking matters further.
Bruce Lawson has applied for a judicial review of the permission, and instructed counsel.
He is challenging the "Reserved Matters" planning application, confirmed under delegated powers, for the large, oak timber framed house.
When the application was made it led to 29 objections to the proposals from local residents.
Mr Lawson, who has lived in the area for many years, said the lane had been deemed access-only since 2000.
However he said it was an established rat run despite being only nine feet wide in places.
"The site is situated in the Vale of Montgomery – a listed National Historic Landscape of Wales. The site is clearly visible from Montgomery Castle and is just 86 metres from the 1070 Hendomen Castle and Pre-Conquest Fields."
"Outline permission was given without any local inhabitants at the settlement being properly advised. The area’s county councillor made strong objections on the basis of local needs and highways issues, but his objections were apparently mislaid by the planning department, for three months.
"They were brought to the planning committee’s attention by him, three days before they granted outline."
Mr Lawson said that the proposed house was out of keeping with others in the settlement and was, manifestly, not meeting local needs.
He said that Powys County Council suspended the ability for councillors to call in planning applications from March 19 until July 2.
"Extended time was given to local residents to make comments up till August 21, but when our councillor objected on August 14 was deemed out of time. This by just two days during the Covid crisis."
He also said that the wrong postcode had been given for the site.
"There are too many questions, which cannot go unchallenged," Mr Lawson said.
A spokesperson for Powys County Council said: “Outline planning permission establishing the principle of a dwelling at Pwll Farm, Hendomen, Montgomery was granted by the council’s planning committee in February 2018.
“A reserved matter application for the erection of a dwelling at this site, which relates to the approval of the outline planning permission that was granted in February 2018, relating to the outline planning permission for the erection of a dwelling at this site was approved last month.