The defending Premier League champions have been linked with a surprise move to sign the Ivorian midfielder.
Chelsea, though, will face competition from London rivals Arsenal and Tottenham Hotspur for the winger.
Zaha has scored four times in 12 games in the Premier League this season to capture the eye of the big clubs.
However, former Chelsea defender Townsend has ruled out Zaha being allowed to leave Palace given their relegation woes.
“Not Zaha,” Townsend told Sky Sports. “Not at the moment. It is the perfect fit for Zaha. It would be utter madness.”
Zaha spent two unhappy seasons at Manchester United after he became the final signing of the Sir Alex Ferguson era.
The Red Devils sold Zaha back to Palace in 2015 during Louis van Gaal’s spell in charge of the Manchester club.
The African star made just four appearances for Manchester United during a testing spell at Old Trafford.
Chelsea signed six players in the summer transfer window in an attempt to reinforce their squad ahead of a return to the Champions League after a 12-month absence.
The west London side signed Manchester City goalkeeper Willy Caballero, AS Roma defender Antonio Rudiger, Real Madrid striker Alvaro Morata, Monaco midfielder Tiemoue Bakayoko, Torino defender Davide Zappacosta and Leicester midfielder Danny Drinkwater.
MORE: Man United latest news
MORE: Arsenal latest news
MORE: Chelsea latest news
MORE: Liverpool latest news