Her conservative CDU/CSU bloc has seen its worst result in almost 70 years but will remain the largest in parliament.
Its current coalition partner, the social democratic SPD, says it will go into opposition after historic losses.
The nationalist AfD has won its first seats and is set to be the third party, a result that sparked some protests.
Dozens of demonstrators have gathered outside the right-wing, anti-Islam party's headquarters in Berlin, some with placards saying "Refugees are welcome".
Protests have also been held in Frankfurt and Cologne.
