As far as "build quality" goes, I think that also depends on what/where your idea of build quality lies, and what you're willing to do to improve it. If it's materials, like you don't like the dash plastic or something, then that's one thing, but there is nothing about either car that can't be disassembled, have parts replaced, and bolted back together to spec (or better) if you've got the time, talent, and/or money.
Personally, I'd take the SVTC and earmark the extra $4K you'd spend on the Focus for rebuilds/upgrades on the Contour. Drop a few hundred on interior tweaks (maybe a nice new shift knob, stereo & speakers, etc.) and then use the rest on getting a 3L conversion and maybe a suspension rebuild and as long as the work is done right, you won't ever think "Wow, I wish I had a newer car."