The Reds appeared to be coasting towards three points after Brazil international Philippe Coutinho and Liverpool striker Daniel Sturridge gave the away side a comfortable 2-0 lead.
The Merseyside outfit could have been further clear but Joe Allen had a goal ruled out for offside, while the Welshman and Sturridge missed good chances to extend the visitors’s lead.
Sadio Mane missed a second-half penalty before he halved the deficit with a good finish, which prompted Klopp to bring on Benteke in an attempt to put three points out of Southampton’s reach.
And Benteke wasted a good opportunity in the 74th minute after he dragged a finish wide with only Fraser Forster to beat, much to Klopp’s annoyance on the sidelines.
The Liverpool manager appeared to be reminding Benteke of his miss at full-time after Mane and Graziano Pelle had turned the game on its head to seal a 3-2 win for the Saints.
Benteke moved to Liverpool in a £32.5m deal from Southampton.
Get 8/1 enhanced odds on Tottenham to win or 11/1 enhanced odds on Liverpool to win at White Hart Lane on Saturday afternoon. Click here to sign up now. T&Cs apply.
MORE: Latest Man Utd news
MORE: Latest Arsenal news
MORE: The latest football news