The US, UK, Australia and Canada will not send government representatives to the Games because of concerns over China's human rights record.
This includes widespread allegations of abuse against the Uyghur minority group.
France, host of the next Summer Games, said it would not join the boycott.
The Winter Olympics are set to take place in Beijing in February.
"The United States, Britain and Australia have used the Olympics platform for political manipulation," Wang Wenbin, a spokesperson at the Chinese foreign ministry, said.
Chinese state media claimed on Wednesday that Beijing "never planned to invite US and Western politicians who hype the 'boycott' topic".
The US was the first country to announce a diplomatic boycott, with Australia, Canada and the UK following suit.
The move by these countries stops short of preventing athletes from attending - something which has been welcomed by International Olympic Committee (IOC) President Thomas Bach.
"The presence of government officials is a political decision for each government so the principle of IOC neutrality applies," he said.