I suspect you might have an air leak from around the speaker basket. It won't hurt mids and highs, but bass will suffer. Get some 1'x1' pieces of dynamat and install on the door and then reattach the speaker.
If that still doesn't help, you're gonna need an amp. Your HU really puts out about 15x4 rms(continuous) at most. Most manufacturers rate their HU at peak output, which is a fraction of a second filled with clipping, distortion, crap, etc... Basically, it's marketing glitz.
If you're after bass you can really hear and feel, I'm sure you know that you'll need a sub with a dedicated amp.
See the post on "Confused about amps" for some more detailed info on power ratings if you're interested.
