You didn't state this explicitly, so I feel I have to ask: did you remove the window and power lock controls? You need to remove the door handle plastic bezel screw, then pull out unit and disconnect the switches. Its been over a year since I took off my door panels, so I'm not exactly sure what you're missing besides that.
