You are talking about the OUTSIDE handle, so that there is no keyhole, correct?
Wouldn't you remove the door panel on the inside, and then undo the outer beveled surround from within the door? May require removal of the actual handle mechanism, too, then re-install..
Ray