"There's a whole kind of Chucky element to them. They're a little creepy," he told People Magazine.
When First Lady Michelle joked about putting one in their bedroom Mr Obama said he would move out if he saw one there.
But that didn't stop White House staff having a go at scaring him.
Official White House photographer Pete Souza explained the prank on his Instagram page.
"For the past three weeks, there have been four snowmen on display in the Rose Garden.
"We've been joking that we should move the snowmen a few feet closer to the Oval Office every day to see if anyone noticed.
"Then we realised the snowmen were too heavy to easily lift.
"But finally this morning before the president came to the office, some helpful staff - I won't say who - moved all the snowmen so each one was peeking through a different window into the Oval."
Once the figures had been moved a picture of a snowman looking over the president with the caption "do you want to build a snowman?" appeared on the official White House Instagram account.
Despite some suggesting the image is a bit sinister, Mr Obama apparently took the joke well with Pete Souza writing that "he enjoyed the prank".