The Alternative fuer Deutschland (AfD) party took about 21% of the vote behind the centre-left SPD's 30%.
The German chancellor's CDU was backed by only about 19% of those who voted, according to the partial results.
The vote was seen as a key test before German parliamentary elections in 2017.
Before the vote in Mecklenburg-West Pomerania, in the former East Germany, all of Germany's other parties ruled out forming a governing coalition with the AfD.
However, the party's strong showing could weaken Mrs Merkel ahead of the national elections next