Called it "the hardest decision I've ever made," an emotional Mr Key said "I don't know what I'll do next".
Deputy PM Bill English is likely to take over until the National Party holds a caucus to choose a new PM.
Mr Key, a popular leader, is stepping down at the request of his wife Bronagh, the New Zealand Herald reports.
He won a third term for the National Party at elections in September 2014. He said he would not be contesting the 2017 election.
Mr Key made the announcement during his weekly press conference, citing family reasons for the surprise decision. He set a date of 12 December for the formal resignation.
He said his job required great sacrifices "from those who are dearest to me" and that his children had coped with "an extraordinary level of intrusion".
"All I can say is that I gave it everything I had. I have left nothing in the tank."
My Key, who was formerly at Merrill Lynch as a foreign exchange dealer, ended nine years of Labour Party rule in 2008 when he ousted Helen Clark as prime minister.
Labour leader Andrew Little tweeted: "John Key has served New Zealand generously and with dedication. I wish him and his family the best for the future."
Green Party co-leader Meteria Turei also wished him well.
"I fought every day against John's politics but always supported his right to be a dad and a husband first," she tweeted.
Known by the local media as "Teflon John" because very little controversy has stuck to him during his time in office, Mr Key is credited with steering New Zealand through the 2008 global economic crisis and out of recession.
He has sought to build closer ties with the US, taking a leading role in supporting President Barack Obama's Trans Pacific Partnership (TPP) - a 12 country trade deal covering 40% of the world economy.
However, Donald Trump's recent victory in the US has derailed that process, our correspondent says, with his announcement the US would be quitting the TPP on his first day in office in January.
When Australian Prime Minister Malcolm Turnbull learned of the news, he is reported to have sent Mr Key a text message reading "say it ain't so, bro".
Mr Turnbull said New Zealand had boxed above its weight under Mr Key's leadership.
"He will be a great loss to New Zealand and a great loss to the world," Mr Turnbull said.
Former Australian prime minister Tony Abbott played on their two countries' cricketing rivalry to say Mr Key had enjoyed a "fine innings".
"Not many pollies retire unbeaten on a double ton," he tweeted.
Australian Opposition Leader Bill Shorten said: "John Key has been a good friend to Australia. I wish him and his family all the best."