Should have bought the $20 ball joint/tie rod removal too1 (the one that almost everyone says is not needed).
Tie rod/ball joint removal tool
The axle end is like a spline. Just turn it a little bit if it doesn't go in the first time you push it in.
Some pictures from Autozone 
