Officials told the Southern Planning Committee that Golden Moments, in Broad Street, is not considered to be in the town's main shopping area where it would be protected from such a change of use.
Ludlow councillor Andy Boddington could not vote on the application at Tuesday's meeting because it is in his division.
He said the restaurant has been trading for more than 20 years, it was only 300m out of the town centre area, was "thriving" and there was no evidence that it created a nuisance.
Councillor Boddington said the town centre is "bouncing back after the pandemic but is fragile" and he said the "gentrification" of the town should be resisted.
But the councillor found himself a lonely voice, as his colleagues lined up to support the proposal.
Councillor Claire Wild said it would "improve the street scene" and Councillor Richard Marshall, who has used the restaurant said he was "100 per cent" behind plans to turn it back into a house.
And Councillor Caroline Bagnall said she recognised that Golden Moments was a "well loved and well used restaurant" but the plan also has "merits".
The site is located within Ludlow Conservation Area and on the west side of Broad Street. The building currently has living accommodation above the restaurant, and extends over three levels including a basement, with a single storey range of buildings that extends west into rear garden spaces.
The restaurant lease expired in 2020, is currently in holdover and has been given notice by the landlord.
Reacting to the decision, Councillor Boddington said: "This decision is damaging to the retail and economic health of the town centre.
"It is the first time the Southern Planning Committee has made a decision that could close a legitimate business.
"But planning committee members had no regard for the popularity of the venue, or the fact it had been there for two decades.
"They had a chocolate box view of what Broad Street should look like and Golden Moments didn’t fit in to that."
He added: "I am appalled by the decision which is not in the best interests of Ludlow and its town centre.
"Shropshire Council has a history of making bad planning decisions and this is one of the worst. I long for a council and councillors who believe like me that if businesses don't thrive, Ludlow won't thrive.
"We need to make better decisions than this."