UK’s Conservative Party — and by extension Prime Minister Boris Johnson—  suffered a double blow on Friday after being rejected by voters in two special elections. The polls were dominated by questions about Johnson’s leadership and ethics.

The party’s chairman quit after the results early Friday, saying the party “cannot carry on with business as usual.”

The centrist Liberal Democrats overturned a big Conservative majority to win the rural southwest England seat of Tiverton and Honiton, while the main opposition Labour Party reclaimed Wakefield in northern England from Johnson’s Tories.

Also Read: UK to introduce bill overriding Brexit deal with EU, sparking fears of trade war

The contests, triggered by the resignations of Conservative lawmakers hit by sex scandals, offered voters the chance to give their verdict on the prime minister just weeks after 41% of his own MPs cast their ballots against him.

“The people of Tiverton and Honiton have spoken for Britain,” said the area’s newly elected Liberal Democrat lawmaker, Richard Foord. “They sent a loud and clear message: It’s time for Boris Johnson to go, and go now.”

Defeat in either district would have been a setback for the prime minister’s party. Losing both increases jitters among restive Conservatives who already worry the ebullient but erratic and divisive Johnson is no longer an electoral asset.

Party chairman Oliver Dowden quit, saying “our supporters are distressed and disappointed by recent events, and I share their feelings.”

“We cannot carry on with business as usual,” he said. “Somebody must take responsibility and I have concluded that, in these circumstances, it would not be right for me to remain in office.”

The prime minister was 4,000 miles (6,400 kilometers) away at a Commonwealth summit in Rwanda as the results were announced.

He told reporters on Thursday that he would not step down if the Conservatives lost both elections, replying to the suggestion with: “Are you crazy?”

“Governing parties generally do not win by-elections, particularly not in midterm,” he said. “That’s just the reality.”

The electoral tests come as Britain faces the worst cost-of-living crisis in a generation, with Russia’s war in Ukraine squeezing supplies of energy and food staples at a time of soaring consumer demand while the coronavirus pandemic recedes.

Also Read: United Kingdom PM Boris Johnson jeered at Queen Elizabeth’s Platinum Jubilee service

Even with the defeats, Johnson hold a large majority in Parliament. But his authority among his own lawmakers has been deeply eroded.

He survived a no-confidence vote by his own party this month but was left weakened after 41% of Conservative lawmakers voted to remove him. Johnson could face another rebellion in the coming months.

Losing both special elections will increase jitters among restive Conservatives who already worry the ebullient but erratic and divisive Johnson is no longer an electoral asset.