Originally posted by Big Jim:
The voltage regulator is built into the alternator so it is not usually serviced seperately.
However, RPM related failure at exactly the same speed is nearly always from failure of the alternator rotor. You will need to replace the alternator, or stay under 4500 RPM (just joking).
The part that is almost for sure causing your problem is #6 on your diagram, the rotor. Unless things have changed since the last time I priced individual parts for an alternator, you will be better off getting a rebuilt exchange alternator from a reliable source.
You also need to address the battery. A red light in the built in hydrometer means that the battery is near the end of it's life. You need one anyway, so replace it first and see what happens.