that sounds gnarly.
i wouldn't think it would be the sway bar (nor lack there of, as it should not affect straight line stability all that much).
if your tie rod(s) are bent this could contribute, especially if you were on a rutted road.
or if your control arms are bent it could definitely wander and then go the other way when you turned the wheel to correct it.
i'd take a look at those items before you have the diff checked out.
always go cheapest to most expensive if at all possible.
best of luck!
