Called it "the hardest decision I've ever made," an emotional Mr Key said "I don't know what I'll do next".
Deputy PM Bill English is likely to take over until the National Party holds a caucus to choose a new PM.
Mr Key, a popular leader, is stepping down at the request of his wife Bronagh, the New Zealand Herald reports.
He won a third term for the National Party at elections in September 2014. He said he would not be contesting the 2017 election.