If the buildup is just wax try using an old toothbrush to get the crud out. Depending on what year your car is you might consider replacing the seals altogether. Its really not too hard of a job. Involves taking off the interior door panels and removing a couple bolts to get the door handle loose. I can give you some in depth details if you decide to replace them but theres no need unless it would be useful so just let me know.
As for your friends advice, ignore it, the seals most definately server a purpose and should not be left out. Not only do they seal the door handles from moisture they create a pad between the moving door handle parts and the main body of the door. If it wasn't there the paint would eventually rub off in places and then rust would probably set in.

And thats no good.
