I recently mystery modded & found that I have very little extra noise. I noted a touch more drone, my wife didn't notice any change - she's the one with the good hearing.
Much of the volume from headers can be dealt with by sizing & type of exhaust components. Keeping the main cat will quiet things considerably, add a nice muffler behind it & perhaps something like a resonator or another muffler at the rear & you should be in good shape.
