President Barack Obama said Sunday the Secret Service will thoroughly and rigorously investigate a report that members of his security detail were caught having sex with prostitutes in Colombia shortly before he arrived to attend the Summit of the Americas conference.

"If it turns out that some of the allegations that have been made in the press are confirmed, then of course I'll be angry," Obama said during a news conference in Cartagena, where the summit is being held.

The allegations assert that at least one prostitute from Cartagena refused to leave the hotel room rented by the Secret Service until she was paid, said Peter King, a Republican congressman and head of the House Homeland Security Committee.

As of Sunday, 11 Secret Service agents had been placed on administrative leave because of the incident, along five other U.S. military servicemen, according to Reuters. 

More from Reuters:

A local police source said the Americans brought a number of prostitutes back to their beachfront hotel near where Obama was due to stay, and that at least one member of the security team flashed his badge and demanded that hotel staff allow him to remain with a woman.

A U.S. source familiar with the situation said it appeared some of the agents had gone to a bar and brought back various friendly women who turned out to be asking for money when they got back to the hotelrooms.

Prostitution in Colombia is legal in "tolerance zones," but local police often turn the other cheek outside sanctioned areas.