Referee Mark Halsey pointed to the spot in the 34th minute after Jay Rodriguez went down in the box without contact from Enda Stevens and Rickie Lambert netted the eventual winner for Nigel Adkins’ men.
Villa have won just two of their last 10 league outings, with their fourth defeat in five games dropping them down behind Wigan into 18th place in the table.
Lambert was dismayed by the penalty decision and insisted his side deserved something from the game.
“We did not deserve to lose but that is the way things are going for us,” Lambert told BBC Sport. “It was never a penalty.
“We had a lot of chances again, and we are creating a lot at the moment. If the players keep that up we will start to get more wins.
“I have seen the penalty replay at half-time and unless you are 100 per cent you cannot give a penalty. It is evident it was never a penalty.
“Enda Stevens was not even close to him and that is the most hurtful thing. I think referee Mark Halsey will be embarrassed by that.”
Meanwhile, Saints boss Adkins admitted the decision was harsh on Villa.
“We could have had seven or eight penalties this season, and they say it levels out,” he said.
“For me there is no contact with the player but the referee has a decision to make. All we can do at our place is make sure the players are not diving.”
MORE: Man United latest news
MORE: Arsenal latest news
MORE: Chelsea latest news
MORE: Liverpool latest news