How in the hell does a piece of very soft fabric lightly drug over your paint scratch anything. Maybe if you pressed on it, but you're supposed to just drag it across the surface.
Hell, your buffing rag is more harsh on the paint than that thing is. Not to mention the trillions of dirt/dust particles in the air bombarding your car at 80mph when you drive down the street.
The duster is the least of your worries when it comes to scratching your paint.