When trying to play a video after installing gstreamer-vaapi, I get the following error:

$ LANG=C GST_DEBUG=1 totem foo.ts 
Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 14: reading configurations from ~/.fonts.conf is deprecated. please move it to /home/julas/.config/fontconfig/fonts.conf manually
libva info: VA-API version 0.36.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib64/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_0_36
libva info: va_openDriver() returns 0
0:00:00.756122893  6831 0x7f689c002ed0 ERROR            vaapidecode gstvaapidecode.c:384:gst_vaapidecode_push_decoded_frame: video sink rejected the video buffer (error -4)
0:00:00.803397937  6831       0xa83270 ERROR                default totem-gst-helpers.c:61:totem_gst_message_print: message = Internal data stream error.
0:00:00.803451270  6831       0xa83270 ERROR                default totem-gst-helpers.c:63:totem_gst_message_print: domain  = 5149 (gst-stream-error-quark)
0:00:00.803472659  6831       0xa83270 ERROR                default totem-gst-helpers.c:64:totem_gst_message_print: code    = 1
0:00:00.803487760  6831       0xa83270 ERROR                default totem-gst-helpers.c:65:totem_gst_message_print: debug   = mpegtsbase.c(1358): mpegts_base_loop (): /GstPlayBin:play/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstTSDemux:tsdemux0:
stream stopped, reason not-negotiated
0:00:00.803518790  6831       0xa83270 ERROR                default totem-gst-helpers.c:66:totem_gst_message_print: source  = <tsdemux0>
0:00:00.803538191  6831       0xa83270 ERROR                default totem-gst-helpers.c:67:totem_gst_message_print: uri     = (NULL)


Packages involved:
gstreamer1-vaapi-0.5.10-1.fc21.x86_64
libva-vdpau-driver-0.7.4-10.fc21.x86_64
xorg-x11-drv-nvidia-346.35-3.fc21.x86_64