Well today was the day. The speakers went in, and there was a noticeable improvement right away. Even sounded better when I replaced the head unit, and best when I bypassed the amp.
This will last me a while until I feel the urge to add a sub or something, but I am really liking the new sound, and I can really turn it up without hearing distortion (and I don't have to turn the bass down either)...
Now the problem is that the car has too many parts that rattle. I don't hear them while driving, but when sitting still I can hear the doors rattling if it is really turned up. I need to do a "walk around" when it is turned up to see if the trunk rattles or anything like that which, to me, is embarassing. Nothing like someone who has spent time and money on a powerful audio system, just to have it rattle like crazy.