David Cameron says he is "puzzled and disappointed" after Iain Duncan Smith quit as work and pensions secretary.
Mr Duncan Smith said the latest planned cuts to disability benefits were "not defensible" in a Budget that benefited higher-earning taxpayers.
In response, Mr Cameron said they had "collectively agreed" on the proposals, before deciding on Friday to have a rethink to "get these policies right".
Mr Duncan Smith's replacement is expected to be named later on Saturday.
Before he was ousted as Conservative leader in 2003, Iain Duncan Smith was dubbed "the quiet man". But there was nothing subdued or understated about the manner of his departure last night.
Instead, a zinger of a resignation letter, designed to inflict maximum damage on Chancellor George Osborne; a relationship that had long been testy, tested to destruction by this week's Budget.
By questioning, as the Conservatives' critics have long done, the Tory slogan to justify cuts - "We are all in this together" - Mr Duncan Smith knowingly provided the government's opponents with ammunition.
For a man with ambitions to lead his party, these have been difficult days for George Osborne.
This morning David Cameron wakes to a vacancy in his cabinet and a wound inflicted on his government.
In his Budget on Wednesday, Chancellor George Osborne had said the government would be spending an extra £1bn on disability but changes to disability benefits announced a few days earlier had suggested the government would save £4.4bn by 2020-21.
They included changes to Personal Independence Payments (PIP), which will replace Disability Living Allowance (DLA) in January 2017, that were expected to save £1.3bn and sparked outcry from opposition parties and some Tory MPs.
On Friday, prior to Mr Duncan Smith's resignation, a government source said the planned changes would be "kicked into the long grass".
In his reply to Mr Duncan Smith, Mr Cameron said they had all agreed that "the increased resources being spent on disabled people should be properly managed and focused on those who need it most".
"That is why we collectively agreed - you, No 10 and the Treasury - proposals which you and your department then announced a week ago," he said.
"Today we agreed not to proceed with the policies in their current form and instead to work together to get these policies right over the coming months.
"In the light of this, I am puzzled and disappointed that you have chosen to resign."
Mr Cameron's letter also highlighted that they were "on different sides in the vital debate about the future of Britain's relations with Europe".
Sources close to Mr Duncan Smith, who wants the UK to leave the EU, say his resignation was not about Europe.
In his resignation letter, Mr Duncan Smith said that "because of the perilous public finances we inherited from the last Labour administration, difficult cuts have been necessary".
"I have found some of these cuts easier to justify than others but aware of the economic situation and determined to be a team player I have accepted their necessity.
"You are aware that I believe the cuts would have been even fairer to younger families and people of working age if we had been willing to reduce some of the benefits given to better-off pensioners but I have attempted to work within the constraints that you and the chancellor set."
But he went on: "I have for some time and rather reluctantly come to believe that the latest changes to benefits to the disabled and the context in which they've been made are a compromise too far.
"While they are defensible in narrow terms, given the continuing deficit, they are not defensible in the way they were placed within a Budget that benefits higher earning taxpayers."
He added: "I am unable to watch passively whilst certain policies are enacted in order to meet the fiscal self imposed restraints that I believe are more and more perceived as distinctly political rather than in the national economic interest."
BBC political correspondent Iain Watson said: "Tensions with the chancellor had been simmering for some time, but there was little warning of Friday's explosion.
"What seemed to be at its core were briefings by government sources which placed responsibility for the controversial changes to PIP firmly at Mr Duncan Smith's door, while he had in fact felt some pressure from the Treasury to deliver them."
Mr Duncan Smith was Conservative Party leader and Leader of the Opposition from 2001 to 2003, and had been work and pensions secretary since after the 2010 general election.
Labour leader Jeremy Corbyn and Liberal Democrat chief Tim Farron have both called for Mr Osborne to resign.
Mr Corbyn said: "The resignation of Iain Duncan Smith reveals a government in disarray and a chancellor who has lost the credibility to manage the economy in the interests of the majority of our people."