Around 12 feet high, the image was once surrounded by flowers and lovingly tended during the summer months.
But now the brick figure, which lies on the A53 bypass near the town, is to be ripped up.
It will be removed after councillors decided it had become an eyesore that was too difficult to maintain.
The connection between gingerbread and Market Drayton dates back to 1793 when the first making of the treat was recorded.
Roland Lateward was the first recorded gingerbread baker in the town and built a bakery at the back of his cottage in Shropshire Street. Since then, the town has been known as the home of gingerbread bakers for more than 200 years.
In 1817 another bakery began in the corner of High Street and Church Street, which then became known as Billingtons Gingerbread. Here the towns secret recipe was developed and was said to include rum while many claim it should be eaten after being dipped in port.
In the early 20th century, the small town of Market Drayton had four gingerbread bakers.
The smell of the spicy little treat would have wafted around the town at the time.
Image On Food was first set up in the 1980s by Tim and Sarah Hopcroft. Today, the company still makes the sweet treat at its base in Burnside Business Park in the town and sells to Heals, Selfridges, Harvey Nichols, John Lewis and Waitrose.
Ginger is not only delicious it is good for you. It is thought to help digestion and stomach problems as well as burn blisters, heatstroke, colds and gout.
While the town's connections with the history of gingerbread have been debated over the years, many people are sad to see him go.
Mark Whittle, chair of the town's chamber of trade and commerce, said he found it hard to believe councillors had voted to rip up part of the town's unique identity.
He said: "It is a huge figure, about 12 feet. Market Drayton isn't famous for much, but it does have links to gingerbread and there is only one place now where you can still get it in the town.
"I was really shocked to find out it was being removed. When it is cleaned up and looking good it looks fantastic.
"It is such a pity it is being removed."
The town's links to gingerbread are thought to date back hundreds of years – the first recorded mention of it being made in the town was 1793.
Now, just one company, Image On Food, still makes the sweet treat in the town.
The brick symbol was created by the town's In Bloom committee about 15 years ago.
It was the centrepiece of a display that was the envy of many rival towns.
But councillor Roger Hughes said the figure had been too difficult to maintain and had become unattractive.
He said: "It has always been difficult to maintain.
"People need to drive along the bypass and see that you can barely see it. I don't think most people realise it is even there.
"We have always had problems planting flowers and getting things to grow there.
"The decision was made at a committee meeting and we are going to replace it with something.
"If anyone has ideas they are of course able to contact the town council."