The 9,852 acres of land on the island of Okinawa, part of a territory officially referred to as the Northern Training Area, is in a large US military base complex on the Pacific island more than 960 miles (1,550 kilometers) southwest of Tokyo.
The US had turned most of Okinawa over to Japan in 1972 after controlling it from the end of World War II in 1945.
This is the largest return of US-occupied land since then.
To facilitate the return, the Japanese government agreed to build six helicopter landing zones as well as some access roads to allow US troops to train in the area, a senior US defense official told CNN.
The US military is expected to continue to administer the area, which was used for jungle warfare training, a US official said.
The US and Japan have been negotiating the handover since the 1990s.
Carter made the announcement while appearing alongside Japanese Prime Minister Shinzo Abe at the Kantei, his office in Tokyo.
A senior defense official told CNN that the formal handover is due to take place on December 21 and 22 at an event to be attended by US Ambassador Caroline Kennedy and the commander of US Forces Japan, Lt. Gen. Jerry Martinez.
"We share your objective of completing the transfer by December 22," Carter told Abe.
Carter arrived in Japan on Monday on a trip to shore up US-Japan military relations amid fears that President-elect Donald Trump might pivot away from the decades-long security alliance.
More than 52,000 US troops are in Japan, according to US Forces Japan. Add civilian personnel, contractors and dependents and the US presence there tops 100,000.
That presence has been most contentious on Okinawa, where about half of those US troops are based and where relations between US military personnel and residents deteriorated this year following several violent incidents related to American bases on the island.
At a June protest attended by more than 60,000 people in the Okinawan capital of Naha, the island's governor, Takeshi Onaga, said he would strongly ask the Japanese government to move all US military bases outside of Okinawa.
"The government should know that the anger of the people in Okinawa is almost reaching a limit and it is not [right] to sacrifice Okinawa people for military bases anymore," he said.