First off, what kind of primer? Etching primer is what you should have started with. Filler primer is what you should spray next, and that can be block sanded with 400 & 600. A sealer primer is purely optional, it'll give your top coating process come out that much better.
Second, 600 grit is fine enough to where you won't see the sanding marks when painted and clear coated.
