He pointedly failed to thank Mr Trump or praise his policies.
The former ExxonMobil chief had a series of public rifts with the White House after being appointed last year.
Mr Trump officially fired Mr Tillerson via Twitter, naming CIA Director Mike Pompeo as his replacement.
Mr Trump also named Gina Haspel to become the CIA's first female director.
Speaking to reporters at the Department of State, Mr Tillerson said good work had been done to have better relations with China, and rein in North Korea's nuclear weapons programme.
But he added: "Much work remains to respond to the troubling behaviour and actions on the part of the Russian government.
"Russia must assess carefully as to how its actions are in the best interest of the Russian people and of the world more broadly. Continuing on their current trajectory is likely to lead to greater isolation on their part, a situation which is not in anyone's interest."
Sounding breathless and looking shaken, Mr Tillerson said he had spoken to the president to ensure clarity in the days ahead.
Although Mr Tillerson will not officially leave until 31 March, his deputy John Sullivan take over running the state department pending the confirmation of Mr Pompeo.
"What is most important is to ensure an orderly and smooth transition during a time that the country continues to face significant policy and national security challenges," Mr Tillerson said.
Speaking to reporters outside the White House on Tuesday, Mr Trump said his differences with Mr Tillerson came down to personal "chemistry".
"We got along actually quite well, but we disagreed on things," said the president.
"When you look at the Iran deal, I think it's terrible. I guess he thought it was OK.
"With Mike, Mike Pompeo, we have a very similar thought process. I think it's going to go very well."
Mr Tillerson is the latest in a long list of senior officials who have either resigned or been fired since Mr Trump took office.