The Merseyside outfit took the lead when Glen Johnson’s impressive long-range strike left West Ham goalkeeper Jussi Jasskerlainan stranded.
However, Sam Allardyce’s men responded, with Mark Noble scoring a spot-kick after Joe Allen had handled in Pepe Reina’s area before Steven Gerrard netted an own goal just before half-time.
But Rodgers men managed to secure three points and move into the top half of the Premier League table for the first time this term courtesy of Joe Cole’s close-range finish and Collins’ error in the 79th minute.
“It was a brilliant win for us. West Ham have been terrific this year. Sam has done a great job. I thought our team were outstanding,” Rodgers told BBC Sport.
“We were unfortunate to concede the penalty, I don’t know what Joe Allen can do to avoid that. Maybe the roar of the crowd made the referee change his mind. It was a very harsh decision for us.
“We’ve threatened to score three, four and five goals through most of this season, but through not being ruthless or having bad luck we haven’t scored as many as we would have liked.”
MORE: Latest Man Utd news
MORE: Latest Arsenal news
MORE: The latest football news