1) Boot up the computer, log into Gnome.
2) Try to play a video file with Totem (0.99.16, xine-backend, set to use Xv).
3) Watch the nice blue window while the sound is playing.
4) Close Totem.
5) Try to play a video file with Totem again.
6) Enjoy the video.
7) Repeat steps 4-6 an arbitrary number of times without worries.

I have reproduced this on two Debian Sid systems; one with a Matrox G450 card
(with the free XFree86 4.3.0 driver) and one with an nVIDIA TNT2 Ultra card
(with the proprietary driver).

Other video players (mplayer and, curiously, xine) don't exhibit this behaviour.
Also, running mplayer or xine before Totem doesn't "count" so as to make the
problem go away; still, on the first launch of Totem, the window displays just
the overlay color with no video, but subsequently, it works fine.