Well, I wasn't having much luck with a hammer, I rechecked the Ford CD and it indicates you can install the pulley with just a "special pulley installer tool" - which looks to be an M8-1.25x40 bolt.
So, I pulled the pulley and ordered a new one to the tune of $14.
As far as I can tell, the cam is ok - I certainly hope so, because we all know pulling that is a major PITA.
Find out tomorrow if installing a new pulley is that much easier.