Rishi Sunak joked as he switched on the Christmas tree outside 10 Downing Street that it would be nice to “just click a button” and have everything turn out brighter. It was rare light moment in a week in which the prime minister found himself under fire for both the buttons he pushed and those he couldn’t bring himself to press.

Sunak spent much of the week defending a snap decision to cancel a meeting with Kyriakos Mitsotakis after the Greek premier repeated calls for the British Museum to return of its prized collection of Parthenon sculptures. One minister privately described it as a massive own-goal that alienated a potential partner against illegal immigration, while another acknowledged the dispute gave Labour leader Keir Starmer his best parliamentary performance to date against Sunak.

“Never mind the British Museum — It’s the prime minister who has obviously lost his marbles,” Starmer said.

But it was a decision that Sunak hasn’t made that fueled the most acrimony among the Conservative Party benches. More than two weeks after promising to respond to a Supreme Court ruling against the party’s plan to deport asylum-seekers to Rwanda — a response he said would come “quickly and robustly” — the prime minister hasn’t yet unveiled legislation to save it.

Getting control over Britain’s borders was a key argument for the country’s pivotal 2016 to decision to leave the European Union, a move backed by Conservatives such as Sunak. While migration to and from the continent has fallen off, legal immigration and the flow of asylum-seekers across the English Channel have surged.

“My patience is wearing thin; the country’s patience is wearing thin,” Sunak told reporters en route to climate talks in Dubai. “We have to end this merry-go-round.”

Sunak has been locked a debate with his recently reshuffled Cabinet about how far they can push Britain’s international human rights commitments to carry out deportation flights before a likely general election next year, according to people familiar with the discussions. The dispute has been fanned by migration statistics last week that also confirmed a near-record influx of legal immigrants, many filling jobs in health and social care.

Without a clear response from Sunak or his new home secretary, James Cleverly, Tory lawmakers have floated a range of options from the cautious to the extreme to force through the Rwanda flights and scale back legal migration. Deputy Conservative Chairman Lee Anderson has called on Sunak to “ignore” the Supreme Court ruling, while ousted former Home Secretary Suella Braverman wants him to “dis-apply” Britain’s commitments under international pacts, such as the European Convention on Human Rights.

Others including Cleverly and Justice Secretary Alex Chalk are wary that breaking international commitments could undermine relations with Britain’s allies and trading partners, particularly since the ECHR underpins the Good Friday Agreement that helped end decades of sectarian violence in Northern Ireland. Dozens more Tories have indicated they wouldn’t support such a hardline measure.

Meanwhile, front-benchers such as Business Secretary Kemi Badenoch and Immigration Minister Robert Jenrick have publicly questioned the country’s policies allowing legal immigration. Those comments have put them at odds with Sunak, who just Monday was touting the country’s “competitive” visa policies while addressing top bankers and other executives at the Global Investment Summit at Hampton Court.

The disagreements have exposed Sunak’s struggles to control Cabinet messaging barely three weeks after a shock reshuffle that saw the populist Braverman fired and the centrist former Prime Minister David Cameron brought in as foreign secretary. Neither that shake-up nor a plan to cut payroll taxes by more than £9 billion ($11.4 billion) next year have diminished Labour’s poll lead, which a YouGov survey showed widening to 23 percentage points on Friday.

With setbacks overshadowing his successes, Sunak has increasingly expressed frustration in recent meetings, according to people familiar with the matter, including one especially tense gathering after the Supreme Court decision. Cabinet ministers have begun to share unhappiness more freely with Westminster journalists, complaining that Sunak declines requests for meetings and doesn’t respond to their communications.

A Downing Street official denied reports of tempers flaring in the wake of the Rwanda loss. The prime minister himself appeared in good spirits while traveling to Dubai for international climate talks on Thursday, bantering with reporters on the plane out.

Nevertheless, the migration debate make this holiday season potentially perilous for Sunak. He has little more than two weeks to deliver his promised migration bill — and a Rwanda treaty to go with it — before a planned Parliament recess begins Dec. 19.

Downing Street aides are watching how many letters of no-confidence Tory MPs lodge with Graham Brady, the head of the influential 1922 committee of backbench Conservatives. It would only take 54 to trigger a party confidence vote, and Brady keeps their number closely guarded.

Such votes helped sink the premierships of both Boris Johnson and Theresa May, exposing their lack of support within their own party. Although there’s little appetite among Conservatives to replace Sunak after cycling through five leaders in seven years, a Rwanda bill seen as either too hard or too soft could cause a sudden spike in dissenting MPs, breaching the threshold.

Gavin Williamson and Julian Smith — two former Tory chief whips who are close to Sunak — have been seen more frequently in Parliament in recent days, in what MPs view as an attempt by Downing Street to monitor the party mood. Another traditional measure of support — the cheering from the backbenches during prime minister’s questions — was less than resounding as Sunak sparred with Starmer on Wednesday.

With such a chill running through Westminster, the prime minister seemed to relish being outside on Thursday, as he and his wife, Akshata Murty, took in Christmas carols and brought festive lights to the early London twilight.

--With assistance from Ellen Milligan, Lucy White and Andrew Atkinson.