The vote was 295-123, with Obama losing the support of 70 of his Democrats one day after Secretary of State Hillary Rodham Clinton had made a last-minute plea for the mission.
While the congressional action had no immediate effect on American involvement in the NATO-led mission, it was an embarrassment to a sitting president and certain to have reverberations in Tripoli and NATO capitals.
The vote marked the first time since 1999 that either House has voted against a military operation. The last time was over President Bill Clinton's authority in the Bosnian war.
The House planned a second vote on legislation to cut off funds for the operation.
House Republican leaders pushed for the vote, with rank-and-file members saying the president broke the law by failing to seek congressional approval for the 3-month-old war.
"The president has operated in what we now know is called the zone of twilight as to whether or not he even needs our approval," said Rep. Tom Rooney, a Republican. "So what are we left with?"
Some Democrats accused the Republicans of playing politics with national security. They said the vote would send a message to Libyan leader Moammar Gadhafi.
Rep. Adam Smith, the top Democrat on the Armed Services Committee said the vote would essentially "stop the mission in Libya and empower Moammar Gadhafi."
The defeated resolution mirrors a Senate measure sponsored by Sens. John Kerry, a Democrat and John McCain, a Republican, that Obama has indicated he would welcome. The Senate Foreign Relations Committee will consider the resolution on Tuesday.
The second House mdeasure is a bill to cut off funds that would make an exception for search and rescue efforts, intelligence, surveillance, reconnaissance, aerial refueling and operational planning to continue the NATO effort in Libya. It has no chance in the Democratic-controlled Senate.
House Republicans and Democrats are furious with Obama for failing to seek congressional authorization as required under the War Powers Resolution. The 1973 law, often ignored by Republican and Democratic presidents, says the commander in chief must seek congressional consent for military actions within 60 days. That deadline has long passed.
Obama stirred congressional unrest last week when he told lawmakers he didn't need authorization because the operation was not full-blown hostilities. NATO commands the Libya operation, but the United States still plays a significant support role that includes aerial refueling of warplanes and intelligence, surveillance and reconnaissance work as well as drone attacks and bombings.
A New York Times report that said Obama overruled some of his legal advisers further incensed members of Congress.
In a last-ditch effort Thursday, Clinton met with rank-and-file Democrats to explain the mission and discuss the implications if the House votes to cut off funds. The administration requested the closed-door meeting.
Rep. Tim Walz, a Democrat, said Clinton apologized for not coming to Congress earlier. But he said she warned about the implications of a House vote to cut off money.
"The secretary expressed her deep concern that you're probably not on the right track when Gadhafi supports your efforts," Walz said.
Rep. Howard Berman of California, the top Democrat on the House Foreign Affairs Committee, said such a vote "ensures the failure of the whole mission."
Earlier this week Clinton said lawmakers were free to raise questions, but she asked, "Are you on Gadhafi's side, or are you on the side on the aspirations of the Libyan people and the international coalition that has been bringing them support?"
In the Senate, backers of a resolution to authorize the operation wondered whether the administration had waited too long to address the concerns of House members.
"It's way late," said McCain, the top Republican on the Armed Services Committee. "This is one of the reasons why they're having this veritable uprising in the House, because of a lack of communication. And then the icing on the cake was probably for them when he (Obama) said that we're not engaged in hostilities. That obviously is foolishness."
He added, however, "That is not a reason to pass a resolution that would encourage Moammar Gadhafi to stay in power."
Earlier this month, the House voted 268-145 to rebuke Obama for failing to provide a "compelling rationale" for the Libyan mission and for launching U.S. military forces without congressional approval.