A huge section of the roof of Bethel Chapel caved in on Monday morning with emergency services having to close a section of the busy New Road, through the town centre.
People in the area at the time said they heard a loud rumble like the noise of a low flying aeroplane.
The middle section of the roof collapsed, leaving the front facade and rear of the chapel standing.
There has been a chapel on the site since 1810 but the current Gothic-style building was built in 1876.
It has lain empty for many years, having been sold in 2008, and a section of the upper balcony in the building had already collapsed.
Mr Richard Lewis, a director of Hughes Architects, said it was devastating to see the collapsed building.
"I went down to have a look as soon as I heard the news.
"It is always devastating to see our architectural heritage and fabric disappearing," he said.
"This is one of the reasons why conservation architecture is so important in securing our important older buildings."
Mr Lewis said that Hughes Architects had been trying to speak to the chapel's owners for some time.
"We have been trying to start a conversation going with the owners to look at what the opportunities there could be to bring it back to life, and we shall carry on trying to have that conversation.
"Of course the collapse will make it harder but we hope that this does not lead to its demolition. Churches and chapels are lovely building and wonderful spaces.
"Bethel Chapel is an important part of the fabric of the town and there is real interest in retaining the building."
He said that the state of the derelict chapel had been an ongoing point of discussion for a long time.
"The building was exposed to the elements, there were lots of slates missing on the roof," he said.
Recently Newtown and Llanllwchaiarn Town Council raised concerns about the state of the building and had asked Powys County Council to consider making an order to tidy up the building.
However the town council was told that county planners, while acknowledging its poor state of repair at present, did not consider it warranted the serving of a Section 215 Untidy Land Notice.