Basically, the factory amp is the cheapest thing Ford could possibly put in there. The only reason it's there is because Ford did not have a single-DIN AM/FM/CD player with a built-in amp.
The factory amp on other Ford vehicles is much better, and on those, it might be worthwhile to use it. Not on the Contour.