So actually it's the heater vacuum control system acting up, rather than the blower fan.
As you've already determined loss of vacuum causes system to default to defrost vents. Suggest you check vacuum reservoir tank and associated check valve and hoses, located under dash, pass side next to heater blower resistors.
