The Merseyside outfit took the lead when Glen Johnson’s impressive long-range strike left West Ham goalkeeper Jussi Jasskerlainan stranded.
But Rodgers men managed to secure three points and move into the top half of the Premier League table for the first time this term courtesy of Joe Cole’s close-range finish and Collins’ error in the 79th minute.
“It was a brilliant win for us. West Ham have been terrific this year. Sam has done a great job. I thought our team were outstanding,” Rodgers told BBC Sport.
“We were unfortunate to concede the penalty, I don’t know what Joe Allen can do to avoid that. Maybe the roar of the crowd made the referee change his mind. It was a very harsh decision for us.
“We’ve threatened to score three, four and five goals through most of this season, but through not being ruthless or having bad luck we haven’t scored as many as we would have liked.”
ENHANCED ODDS: Get 3/1 on England to beat Lithuania