If rear toe was too far in on one side, but correct on the other three positions it would probably cause something like this.
Does the car pull at all when the rear tire on the opposite side hits a bump? Hitting the bump would disturb traction on the good side, allowing the bad side to track where it wants. The car would slightly pull when this happens, kind of like you are oversteering.
