For the passenger side if you pull on the hub you can pull the inner CV joint apart and likely create a lot of unnecessary extra stress in your life. If this happens you may destroy the rubber boot. It is sort of like getting a disjointed shoulder joint.
