Seth, a couple things.
Many cars come without a/c or get old and don't work. The defrost still works in all but the hot and humid when trying to defog the windows.
The remote start on my wife's F-150 does not turn on the a/c compressor. So, when it's parked overnight in the winter and I leave the switch on defrost and fan on high, the windos still end up clear after 10-15 minutes. I learned that the compressor doesn't come on the first time I tried to use the remote start to cool it down in the summer. Still hot, sticky, sweaty seats after 10 minutes.
Most A/C systems work fine. However our Contiques use this amazingly inefficient compressor. Bonus is the non-linearity of the power it uses too. I noticed it the first time I drove the SVT. Never have I experienced such a change in engine behaviour.
A friend of mine used to have a 97 SE V6 5-speed. He sad it wasn't too noticable in that car and almost always has the A/C on. I labeled him an A/C freak as he does this in his home as well. Then he got a '00 SVT. He couldn't stand the A/C now. So, maybe it's more hit and miss with the power suck down. Feels like about 40Hp at times.
If your car doesn't do this, you can try mine at the next SVTOA meet and feel it.
Now, what you said for the most part is true. Running the compressor in defrost and other modes accelerates the task, but I'd rather wait an extra couple minutes at times than deal with the engine.
Aaron