The House of Lords could be forced to leave Parliament while extensive restoration works are carried out.
Peers wanted to use the Queen Elizabeth II Centre in Westminster – minutes away from Parliament.
But Levelling Up Secretary Michael Gove has said he will not support the plans.
His department owns the freehold and the body which runs the QEII Centre – a large conference and exhibition space with views of Big Ben and Westminster Abbey.
In a letter seen by the BBC, Mr Gove said he “cannot endorse” a plan for peers to “decamp to a temporary home a mere 200 yards from the Palace of Westminster”.
Mr Gove suggested they looked at alternative buildings in the north of England, the Midlands, the South West, Scotland or Wales.
He told the Lord Speaker Lord McFall: “It is clear to me that the House of Lords moving elsewhere, even for a temporary period, would be widely welcomed.
“I know cities and towns across the UK would be pleased to extend their hospitality to peers,” he added.
The restoration of Parliament will cost billions – but the price will go up considerably if MPs or peers stay put in the building.
The Lords had begun working on plans for a relocation to the QEII.
Any plan will be subject to a final vote by Parliament.
The idea of moving the Upper Chamber out of London first emerged in January 2020.
Downing Street said it would demonstrate its commitment to levelling up to spread decision-making outside London.
York was considered as an option – but several leading peers said it was impractical to separate the Lords from the Commons.
The then Lord Speaker, Lord Fowler, said it was “gesture politics” while Commons Speaker Sir Lindsay Hoyle said the idea was “great PR” but questioned how it would work.
By the summer of that year, the relocation of peers to York was effectively axed.