There are ignorant people that listen to every kind of music. That is a dumb arguement.
Music, rap or otherwise, is aimed at ANYONE that will enjoy it. If you consider most people that listen to a certain kind of music as ignorant, then you are stereotyping. That's quite a bad stereotype to have, because there are so many un-ignorant people that happen to listen to rap.
I think it goes back to what Andrew said. Rap potrays a certain image, and that image could definitely make a single minded person think that rap listeners are ignorant.