Excellent question. I'm not sure of the answer. I'll do the same to mine and see if I get it. Could be your door isn't closing on the firewall that seperates outside air from entering car (recirculation).
Sorry I can't help, but it's very interesting....and annoying I can relate.