Neo has trouble defeating Smith precisely because he is no longer an agent. Neo and Smith are basically equals now as shown in the huge fight scene between the two. Neo can still easily defeat regular agents, like he does at the start of the film. It is just that since they are computer programs the Matrix can easily generate more with little difficulty.
