I've heard of lag when folks attempt to play movies/music vids from their SD cards (I don't have any direct experience with that as I've not tried it myself)
Though I have played movies AND music vids from my iPod and can't say that it has ever happened to me. Mine are also MP4 formatted files.
Sorry, I couldn't say why that is happening, but wanted to let you there was someone else that had the same type of files that WEREN'T experiencing the same problem you are encountering.
Sorry! Hope you get in resolved!
Kev