The All Blacks were five points behind at half-time with a man in the sin-bin as four penalties from Handre Pollard cancelled out Jerome Kaino's early try.
A Dan Carter drop-goal and a Beauden Barrett try put the champions ahead.
South Africa replied with two penalties but a Carter effort in between secured victory despite ferocious Bok defence.
It was seldom pretty but once again in this World Cup there was a gripping finale, the All Blacks holding that two-point lead for the last 12 minutes as their opponents finally tired.
No team has ever retained the Webb Ellis trophy, but the All Blacks will be favourites to do so when they meet either Australia or Argentina next Saturday after finding their way through a brutal encounter.
This was nothing like the free-flowing attacking rugby that had eviscerated France in the quarter-finals, but in its own way it was just as admirable, as South Africa refused to buckle despite having just 43% possession.
The game swung in the 20 minutes after half-time, New Zealand beginning that period five points down and with Kaino off the pitch, but ending it five points up and with Springbok wing Bryan Habana in the sin-bin instead.
And despite replacement Pat Lambie cutting the lead to less than a score, the vast experience of the champions saw them through.
Poor All Black discipline
After Pollard's early penalty, the All Blacks struck back when Kaino thundered through De Jager down the right and touched down one-handed in the corner, but two more pin-point penalties from Pollard gave his side a two-point lead with 20 minutes gone.
South Africa were having significant success at the breakdown, New Zealand shipping six penalties in short order, and the world champions looked instead for territory, kicking behind the defence and into the corners.
Carter hit the post with a long-range penalty, and despite his side dominating possession and territory the Springbok defensive wall continued to hold.
When South Africa finally made it out of their half, they struck again. Kaino was sin-binned for kicking the ball away and Pollard stroked over his fourth penalty to make it 12-7 at the interval, the green-shirted supporters in the capacity crowd starting to believe that the upset was on.
New Zealand seize control
The Springboks had not lost to the All Blacks in 21 Tests when leading or level at half-time, but despite being a man down New Zealand closed the gap to two points as a spell of carefully worked territory was capped with a Carter drop-goal.
Schalk Burger then lost the ball metres from his own line, the All Blacks went right and drove on through their forwards before Ma'a Nonu sent replacement Barrett diving into the corner.
With Habana sin-binned for knocking the ball from scrum-half Aaron Smith's hands and Carter bending over the conversion for 17-12, the match appeared to have turned.
Pollard and Carter exchanged penalties, and with replacements pouring on to the pitch and the rain hammering down the two heavyweights went toe to toe.
Pat Lambie came on for Pollard to make it 20-18 with 10 minutes to go, and an air of desperation was apparent in both sides.
Carter sprinted half the length of the pitch to clear from De Allende, South Africa failed to capitalise from an attacking line-out and New Zealand escaped into opposition territory as the seconds crawled by, running the clock down with a series of drives to celebrate with exhaustion at the end.
Man of the match
Ben Smith was faultless under the high ball and Dan Carter's decision-making once again peerless.
However, for his muscular, repeated carrying in a relentlessly physical match and the superb run and timing of pass for Beauden Barrettt's critical second-half try, Ma'a Nonu deserves the honour.
What they said
New Zealand coach Steve Hansen: "It was always going to be close, they're a great side and they showed that again. I was really proud of our guys, they kept their composure.
"We just needed to come out for the second half and start taking the game to them rather than waiting for them to take it to us. We talked about it at half-time. We talked about keeping composure and talked about winning the first 10 minutes."
South Africa coach Heyneke Meyer: "Our discipline in the second half was just not good enough. They coped better with the weather in the second half. Discipline was the most important thing, especially in these rainy conditions.
"We wanted to make our country proud but we didn't. We should have pulled this win through but all credit to the All Blacks, they are a quality side."