But now author and historian Derek Beattie is out to present a picture of how ordinary people lived and played with his book Ludlow At Leisure.
And an underlying theme is that some of the things they got up to in days gone by were felt so unsuitable by the town's well-to-do that sustained and mostly successful efforts were made to suppress them.
"I've done a few books on Ludlow. I enjoy social history, and leisure is a major part of people's lives," said Derek, who lives in Lower Broad Street in the town.
The emerging middle classes of the 19th century, with their Victorian moral outlook, first forged their own leisure patterns, and then tried to change those of the working classes.
"They tried to get them away from the demon drink and that kind of thing, and deemed their leisure activities too violent and exuberant. They tried to stop it and channel it into getting them into football teams and teach them sportsmanship.
"And of course as football became more working class, they moved away and set up their own hockey clubs and tennis clubs which were middle class bastions safe behind a high subscription."
In terms of mass mayhem, Ludlow's tug-of-war must have taken some beating.
"Half the town was pitted against the other half of the town. There were four wards, and two wards were pitted against the other two. The mayor or leader of the Corporation had a rope and tossed it to the crowd, and the two sides had to get it down to the River Teme, down Mill Street or Old Street. It was always on a Shrove Tuesday and was very, very, violent."
Fighting even broke out among spectators, and windows would be boarded up in the town so they were not broken.
Ludlow's leaders eventually were able to ban the event, and by 1853 the town had seen the last of the tug-of-wars.
"Fighting was of course a leisure activity among the working classes and everyone came out to watch. To be honest it remained so up to the 70s and 80s in Ludlow, especially at the dance halls and things like that.
"In the 19th and early 20th centuries there were fist fights with certain manners, in which people took their jackets off and rolled their sleeves up. It was pretty widespread, and if women were fighting then the crowds really would come out.
"And boxing was very popular in Ludlow right through to the 1950s and 1960s."
Derek says there are histories which feature architecture, beautiful major buildings, and the people who ran the town, but he says: "I prefer in some ways working class history, who are the majority of the people in Ludlow, and how they filled their time.
"I wrote a book on Ludlow called How The Other Half Lived. This one is how the other half played."
Ludlow At Leisure is published by Ludlow publishing outfit Merlin Unwin and costs £14.99. Derek says it is available from the Castle Book Shop in the town and the usual online outlets.