I used primer on my read door handles, it didn't work!
but I'd clean the surfaces really well with some rubbing alcohol, to remove any grease or built up dirt. Maybe a light sanding with a real fine grit sand-paper and then paint and let it sit till it's compleltely dry. If you don't have to use the car for a couple of days that'd be even better so the paint could cure before you reinstall it.