John McGinn put the visitors ahead in the 20th minute before Ainsley Maitland-Niles was sent off.
Nicolas Pepe pulled a goal back for the hosts from the penalty spot with his first goal for Arsenal in the 59th minute – but Wesley restored the visitors’ lead in the 60th minute.
However, Arsenal fought back and substitute Callum Chambers levelled with a goal in the 81st minute, before Pierre-Emerick Aubameyang wrapped up the three points with a winner in the 84th minute.
The result hoisted Arsenal up into fourth place in the Premier League table and left them seven points behind leaders Liverpool FC.
Celebrity Arsenal fan and former Daily Mirror editor Morgan was clearly delighted by the result after not have been so impressed with the first parts of the game.
After Villa’s second goal, Morgan tweeted: “FFS. Didn’t even have time to tweet about our goal before our defence collapsed again.”
However, his tune soon changed as the game moved in Arsenal’s favour.
After Chambers’ strike, he posted: “2-2! Brilliant goal by @CalumChambers95 – finally, some bloody commitment. 👊.”
And then after the winner, Morgan wrote: “BOOOOOM!! 3-2.. @Aubameyang7 👊👊👍👍.”
Arsenal will take on Nottingham Forest in the third round of the League Cup on Tuesday night.
MORE: Man United latest news
MORE: Arsenal latest news
MORE: Chelsea latest news
MORE: Liverpool latest news