The Contour is known to have a lot of driveline lash, but I usually notice it at lower RPMs. If you are unsure, I would suggest finding another one to test-drive. What you've observed may be this common trait, or it could indicate a problem with the one you are planning to buy. In any case, it doesn't hurt to be extra sure.
The Duratec has an emissions "feature" which causes the revs to not drop at a predictable rate when you let off the throttle. This is known as "throttle hang" or "TH" for short and I think it also contributes to the jerkiness when coasting in low gear. There is a way around this, but the fix causes a loss of low to mid-end power in at least some cars. The loss in power doesn't make any sense to me however I have personally observed it. You can learn more about TH here:
http://www.contour.org/FAQ/FAQ.cgi?s=probs&display=throttlehang An easy fix which hasn't been shown to hurt performance would be to get polyurethane motor mount inserts. This mod results in smoother shifting and low-gear coasting, and it also helps to prevent wheel hop on hard launches.