This might seem strange (but I've checked with a magnifing glass)
but when that happens I see the pixels just moving from 1 to the very next one next to it and back, so for a split second (1 or two frames in video) the object is larger than in the frames before and after.
What I do is to dissolve out the object if it is going off to infinity.
It seems to clear about 80% of that problem.
If it dosen't go off into infinity but just stays off in the distance then I play with the dissolve in the envelope depending on the distance from the camera, (dissolve more if farther and dissolve less if closing in)
Also ya might wanna try the LOD plugin to keep the poly count down as it moves from the camera.