This was during a meeting last week with a group of Syrian civilians, according to an audio recording obtained by CNN.
But Kerry also expressed sympathy for the Syrians' demands that the United States intervene more forcefully amid Syrian and Russian airstrikes against civilians, telling the group that he "lost the argument" for using military force against the regime of Syrian President Bashar al-Assad.
Kerry's comments came at a meeting that took place at the Dutch Mission to the United Nations on the sidelines the UN General Assembly, where Kerry was going from session to session in a frenzied effort to resuscitate a ceasefire that seemed poised to collapse.
"I think you're looking at three people, four people in the administration. I lost the argument. I've argued for the use of force. I'm the guy who stood up and announced that we're going to attack Assad for the use of weapons," Kerry is heard telling the Syrian attendees, referring to internal deliberations within the administration of President Barack Obama that followed Assad's use of chemical weapons in 2013.
Kerry also faulted Congress for failing to support such a retaliatory strike, saying, "The bottom line is that Congress refused even to vote to allow that."
"We have a Congress that will not authorize our use of force," he added, explaining that a new military intervention would be difficult to bring about.
The discussion in the recording occurred only days after the United States and Russia announced a ceasefire agreement in Geneva, an accord that has since collapsed with reports of regime bombing attacks and the positioning of some 10,000 Syrian regime-aligned troops preparing to advance on Aleppo.
The recorded conversation contains a series of revelations about Kerry's view of the ongoing civil war in Syria. Several of the meeting's participants have confirmed the recording's authenticity.
The Syrian civilians repeatedly questioned Kerry about violations of the ceasefire and America's unwillingness to enforce the agreement more strongly, and lobbied for a bigger US role.
Kerry cited legal restrictions and local air defenses as reasons why a US military intervention was not feasible.
"The problem is the Russians don't care about international law and we do. And we don't have a basis, our lawyers tell us, unless we have a Security Council resolution," he said.
"They were invited in, we were not," he added, referring to Moscow's military operations in Syria.
"We don't behave like Russians. It's just a different standard," Kerry said.
"The only reason they are letting us fly is because we are going after ISIL," Kerry said, using another term for ISIS. "If we were going after Assad, we would have to take out all the air defenses and we don't have a legal justification for doing that."
Kerry added, "So far, American legal theory does not buy into the so-called right to protect."
"Nobody (is) more frustrated than me," Kerry told the gathering.
Acknowledging that Russia's military actions have "changed the equation" and made removing Assad more difficult, Kerry suggested that Syrian refugees could one day help eject Assad if given the right to vote.
Asked about the recording, State Department spokesman John Kirby told CNN, "While we will decline to comment on a private conversation, Secretary Kerry was grateful for the chance to meet with this group of Syrians, to hear their concerns firsthand and to express our continued focus on ending this civil war."
Kerry found himself on the defensive as the attendees implored the United States to help stop the violence against Syrian civilians through direct action.
"You can be mad at us, but what we are trying to do is help Syrians to fight for their own country, and we have been spending a lot of money, a lot of effort," he said.
"I am frustrated, too," Kerry said.
He later added, "A lot of Americans don't believe that we should be fighting and sending young Americans over to die in another country."