Not only did you risk damage to the bearings and the rest of the crank, but... The most imediate damage would be to the snout of the crank, scoring it. Scoring makes it difficult to remove and install things for a little while, but it could also do something worse... like, make it easier to install a damper if enough material were removed from the crank during the tap on process. If enough material were removed, the damper would actually walk around on the end of the crank, wearing it out and making it useless in less than 20,000 miles or so.
What's it take to do it right? Less than $10 in hardware from the local hardware store. I'm not saying that I've never tapped a damper on back in high school (Trans Am crank failed less than a year later along with a cracked damper), but now that I know what I'm risking, I'll tke the cheap way out and buy the bolt, nut, and thick washer.