Tokyo will also halt talks with South Korea on a planned currency swap and delay high-level economic dialogue as part of its "initial" response to the statue, Japan's Chief Cabinet Secretary Yoshihide Suga told a press briefing Friday.
The statue was erected by a civil group in December and represents "comfort women," women who were forced to work as sex slaves for Japanese soldiers during World War II.
"The fact that the girls' statue was set up has an unfavorable influence on relations between Japan and the Republic of Korea, and it is extremely regrettable," Suga said.
The temporary recall involves the Japanese Ambassador to South Korea and the Consulate General of Japan in Busan.
South Korea's Foreign Ministry issued a statement in response to the recall, expressing regret over Japan's decision.
"Our government stresses once again that even if there are difficult issues between the two countries, both governments, based on trust, need to continuously develop the relationship," the statement said.
South Korea's semi-official news agency Yonhap reported the civil group first tried to install the statue on December 28, but was stopped by ward officials and police -- who confiscated the statue and kept it in the ward office.
That prompted a public outcry and two days later the officials gave into the pressure, apologizing to the group and allowing it to erect the statue on the road in front of the Japanese Consulate.
The two countries had reached an agreement on the comfort women issue in December 2015.
Japan agreed to give 1 billion yen ($8.6 million) to a fund to help survivors. South Korea's foreign minister at the time said as long as Tokyo sticks to its side of the deal, Seoul will consider the issue "irreversibly resolved."
But the deal was criticized by advocacy groups for former comfort women. One called it a "diplomatic humiliation."
It's estimated that up to 200,000 women, mainly Korean, were forced to work as sex slaves for Japanese soldiers during WWII. Other women came from China, Taiwan and Indonesia.