After a tight first half, substitute Michu gave the Swansea the lead within a minute of coming on after out-pacing Laurent Koscielny.
However, Lukas Podolski restored in the 81st minute, before Kieran Gibbs’ superb effort appeared to have booked the Gunners’ place in the fourth round.
But Danny Graham salvaged a draw for the Swans in the 86th minute, when he fired a loose ball high past Arsenal goalkeeper Wojciech Szczesny from a Swansea corner.
“We should have won this game,” said Wenger. “I felt the first half was a bit locked and one-paced, as much as the second half became much more open.
“I think in the second half we created many chances and suddenly found ourselves 1-0 down. We have shown character and quality.
“I felt when we conceded the corner [that led to the second goal], we lacked a little bit [of] calmness and focus. It’s frustrating to concede a goal just like that on a corner where we couldn’t clear the ball before the corner.
“Even on the corner, we were first to the ball and didn’t win the ball. We have shown quality and heart and it’s quite good if we can repeat that.”
Wenger also admitted his frustration at his side being forced to add another fixture to their schedule for the replay at Emirates Stadium.
“I am frustrated because I wouldn’t want one, but if that’s the choice between going out or staying in the hat, I take the replay,” he added. “We play at the Emirates. They have many games as well.”
MORE: Man United latest news
MORE: Arsenal latest news
MORE: Chelsea latest news
MORE: Liverpool latest news