The money, allocated by the Department for Business, Energy and Industrial Strategy (DBEIS), was intended to be used on improving branches.
The company is locked in a High Court battle with hundreds of former workers who claim they were wrongly blamed for financial discrepancies which were actually caused by a glitch in its computer system.
Among them are Tracy Felstead, 37, from Telford, who was jailed for six months in 2001 after being convicted of stealing £11,500 from a post office she worked at as a teenage counter clerk. She has always protested her innocence, and says the discrepancy was caused by a fault with the Post Office's Horizon computer database.
She is one of 557 former post office workers involved in a group litigation against the company.
But, in the latest development, the Post Office has been told by a senior civil servant to pay back £2.3 million it had received from the DBEIS. A request for a further £2.4 million was also withdrawn.
Alex Chisholm, permanent secretary at the DBEIS, wrote to the Post Office's chief executive Paula Vennells in January, saying that the money was not allowed to be spent on litigation costs. Miss Vennells stood down from her post office role last month to take up a job at a health trust.
Earlier this month, the Hon Mr Justice Fraser ruled in favour of the post office workers, in the first of four trials, accusing the Post Office of 'oppressive behaviour' in demanding sums of money which could not be accounted for by sub-postmasters.
But the second trial, which began on March 4, was sensationally halted last week when the Post Office accused the judge of showing bias. The Post Office has asked for trial to begin again under a different judge, and the proceedings were suspended.
There will be a hearing on April 3 to decide whether the judge should be changed. The trial will then resume unless the court finds in the Post Office's favour.
In a separate action, both Miss Felstead, of Bournside Drive, Brookside, and Rubbina Shaheen – who was jailed for 12 months in 2010 for false accounting – are seeking to get their convictions overturned through the Criminal Cases Review Commission.
Mrs Shaheen, who now lives in Worthen, kept Greenfields Post Office in Shrewsbury when £40,000 allegedly went missing.