What you describe means the drive's contents are changing during the defragmentation. It has to effectively start over every time some data it had planned to rearrange changes.
See if you can pinpoint the culprit or perhaps start up in safe mode to defrag.
