Fiona Kyle, a volunteer guide in Coalbrookdale and a Friend of Ironbridge Gorge Museum, says the early Darbys were devout Quakers – a group which played a major role in the abolition movement.
"The whole ethos of equality is fundamental to everything the Quakers do, and it follows logically that they would find the whole concept of slavery abhorrent.
"The Quakers were at the forefront of the abolition of slavery movement. There's even an abolition paperweight in Rosehill House," said Fiona, from Bridgnorth, referring to a Darby family home in Coalbrookdale.
By successfully smelting iron using coke, rather than charcoal, Abraham Darby I was the man who effectively kickstarted the Industrial Revolution, which is why Coalbrookdale lays claim to be its birthplace.
His grandson Abraham Darby III left an enduring monument in iron – the Iron Bridge over the River Severn, dating from 1779.
Fiona said that although it was currently topical to look at slave trade links in the heritage sector, by bringing in the Darbys the report had gone too far.
"The suggested link, which is such a tenuous link, is unfair and an insult to this important Shropshire family."
The recently published Historic England report is called The Transatlantic Slave Economy and England's Built Environment.
It tells how the Goldney family from Bristol invested in the Atlantic slave economy. At the time Abraham Darby I was starting to build his reputation in the Bristol area.
The report by Historic England, which bills itself as the public body that champions and protects England's historic places, says: "Abraham Darby, with backing from Thomas Goldney, established a number of brass and copper mills along the River Avon; he managed the Baptist Mills brassworks in 1702 with surplus money from his [presumably "his" here refers to Goldney] expanding slave trade business in the area of Bristol still known as Baptist Mills."
Darby later set up at Coalbrookdale, where his innovations were to change the world.
The report also says: "Shropshire coal and products from Shropshire ironmasters were also shipped to Bristol. The Goldney and Darby families from Bristol have associations with the ironworks in Coalbrookdale."