Mine is pretty bad above 50 mph, enough to take the car back home to get my other one. But only after it is driven for a while, not on short trips. Maybe it is the brakes after they heat up. Other thoughts are the control arm bushings or inner CV joint(s).
As for the vibration going away after hard braking, it might be the calipers sticking.