Lee Knight, of Canterbury Road, Habberley, with an "unblemished" work record, was summarily dismissed in October 2011 after bolts from the MoD site at Donnington, Shropshire, were found in the van.
"The site was a secure site containing a great deal of MoD equipment from battle tanks downward," Judge David Richardson told the Employment Appeal Tribunal.
Mr Knight's employers – Robert Bates Wrekin Lanscapes Ltd – who had a contract to maintain the MoD site's grounds operated on the basis that its staff were fully aware of the need for tight security.
A section of Mr Knight's contract also made clear that "the employer was entitled to dismiss the employee summarily for any breach of the customer's security rules", the company argued.
He was awarded £4,000 damages after an employment tribunal ruled in 2012 that his dismissal had been wrongful and unfair.
Robert Bates Wrekin challenged that ruling before the Employment Appeal Tribunal – its lawyers arguing that Mr Knight should be held to the strict terms of his contract with which he was very familiar.
Mr Knight, said the bolts, worth about £2 scrap value, were picked up by him "while litter-picking" and that he had a long unblemished work record. He said it was unjust that he should lose his job for a "single instance of forgetfulness".
Dismissing the company's appeal, Judge Richardson ruled that the company did not have the right to dismiss Mr Knight for any breach of a security rule, "however minor or inadvertent".