There is frustration at the role of a European court in stopping flights for asylum seekers taking off last year.
The minister told the BBC the UK was “probably” being punished for Brexit.
The government has said the UK will remain in the ECHR.
“The government has been clear that it will abide by its international treaty obligations,” a spokesperson said.
“As we’ve set out previously – we believe our Stop the Boats Bill will deliver the changes necessary to reduce the incentives for people to risk their lives through illegal crossings while remaining party to the ECHR.”
But the comments by senior figures in government are likely to reignite a debate in the Conservative Party about the ECHR – which splits opinion among Tory MPs.
The ECHR was established in 1950 by a number of countries including the UK.
The treaty, which sets out the rights and freedoms people are entitled to in the 47 signatory countries, is overseen by the European Court of Human Rights.
It is separate to the European Union – so the UK remained part of both after Brexit.
Ministers are desperate to show they can deliver on tackling illegal migration to the UK before an election – likely next year.
But so far the Rwanda scheme, which would see the UK send some asylum seekers to the east African country, has not got off the ground and the timetable is slipping.
The first flights were stopped by a European judge in a last minute intervention last year – despite being cleared by UK courts.
Since then, the scheme has become bogged down in legal action.
The UK’s Court of Appeal ruled against the plan last month, to the surprise of the Home Office.
There will now be an appeal in the Supreme Court in the autumn, but senior ministers are privately unsure about whether they can overturn the ruling.
They fear that even if the government does win, it could take many months for deportations to begin because individual legal challenges will take place too.
Alternative plans are being considered, but they would also face legal challenges and take time to deliver.
That is likely to lead to further debate about whether the UK needs to take further action to ensure the policy is delivered.
Legislation passed in recent weeks will put a legal obligation on the government to remove people who have entered the country illegally.
But without comprehensive returns agreements – and with the Rwanda policy stuck in the courts – it is unclear when ministers will trigger the implementation of that responsibility.
The government also now has the power to ignore certain ECHR interim injunctions relating to border security.
But some Conservative MPs wanted the government to go further and remove itself from other ECHR obligations.
A cabinet minister said events could make it “inevitable” the Conservatives end up backing leaving the ECHR.
Some backbenchers have said the Conservatives should campaign to leave at the next election regardless of what happens with the Rwanda scheme.
Jonathan Gullis said: “Time and again we see the quasi-legislative European Court of Human Rights continue to undermine the government’s plan to stop the boats.”
He added the electorate should be given a choice over ECHR withdrawal at the next election.
But a former cabinet minister – strongly opposed to withdrawal – described the idea as “knee jerk nonsense”.
They said those who backed quitting were a “minority” and there would be significant opposition in cabinet.
