Mine did the same thing when I removed mine. It got better when I got a intake. There will always be rasp. Some people on the boards are putting an edlebrock hi-flow resonoator or muffler in place of the stock resonator, retaining that nice sould without the rasp. I think there like $40 or something.

1998.5 Silver SVT
Brullen Cat-back
kkm
FRW
Off road Y
Aussie pipe
5% mirror
painted red calipers