Hi.

I found huge memory leak when using vaapivp8enc. This pipeline demonstrates this:

gst-launch-1.0 videotestsrc ! video/x-raw,width=40,height=20 ! vaapivp8enc ! fakesink

gst-launch eats memory at ~150MB per second. If i replace encoder to vaapih264enc or vaapimpeg2enc then everything is fine.

valgrind shows this result:

valgrind --leak-check=full gst-launch-1.0 videotestsrc num-buffers=3000 ! video/x-raw,width=2,height=2 ! vaapivp8enc ! fakesink

==18007== 16,384 bytes in 1 blocks are definitely lost in loss record 2,251 of 2,259
==18007==    at 0x4C29F5F: malloc (vg_replace_malloc.c:296)
==18007==    by 0x544A975: g_malloc (gmem.c:94)
==18007==    by 0x544AC32: g_malloc_n (gmem.c:330)
==18007==    by 0x5454E25: g_quark_init (gquark.c:62)
==18007==    by 0x543E2CC: glib_init (glib-init.c:236)
==18007==    by 0x543E2D8: glib_init_ctor (glib-init.c:287)
==18007==    by 0x400F229: call_init.part.0 (dl-init.c:72)
==18007==    by 0x400F33A: call_init (dl-init.c:30)
==18007==    by 0x400F33A: _dl_init (dl-init.c:120)
==18007==    by 0x4000C79: ??? (in /lib64/ld-2.22.so)
==18007==    by 0x8: ???
==18007==    by 0xFFF000AA6: ???
==18007==    by 0xFFF000AB5: ???
==18007== 
==18007== 131,072 bytes in 8 blocks are possibly lost in loss record 2,258 of 2,259
==18007==    at 0x4C2C1BC: calloc (vg_replace_malloc.c:623)
==18007==    by 0xE35DF28: avc_bitstream_start (i965_encoder_utils.c:100)
==18007==    by 0xE35F116: binarize_vp8_frame_header (i965_encoder_utils.c:551)
==18007==    by 0xE301FDA: vp8_enc_frame_header_binarize (gen8_mfc.c:3582)
==18007==    by 0xE3028FE: gen8_mfc_vp8_init (gen8_mfc.c:3760)
==18007==    by 0xE304653: gen8_mfc_vp8_encode_picture (gen8_mfc.c:4406)
==18007==    by 0xE304B2E: gen8_mfc_pipeline (gen8_mfc.c:4536)
==18007==    by 0xE35D801: intel_encoder_end_picture (i965_encoder.c:780)
==18007==    by 0xE350CFE: i965_EndPicture (i965_drv_video.c:3572)
==18007==    by 0x99AD8EC: vaEndPicture (va.c:1232)
==18007==    by 0x8921748: gst_vaapi_enc_picture_encode (gstvaapiencoder_objects.c:581)
==18007==    by 0x8924B4A: gst_vaapi_encoder_vp8_encode (gstvaapiencoder_vp8.c:376)
==18007== 
==18007== 49,020,928 bytes in 2,992 blocks are definitely lost in loss record 2,259 of 2,259
==18007==    at 0x4C2C1BC: calloc (vg_replace_malloc.c:623)
==18007==    by 0xE35DF28: avc_bitstream_start (i965_encoder_utils.c:100)
==18007==    by 0xE35F116: binarize_vp8_frame_header (i965_encoder_utils.c:551)
==18007==    by 0xE301FDA: vp8_enc_frame_header_binarize (gen8_mfc.c:3582)
==18007==    by 0xE3028FE: gen8_mfc_vp8_init (gen8_mfc.c:3760)
==18007==    by 0xE304653: gen8_mfc_vp8_encode_picture (gen8_mfc.c:4406)
==18007==    by 0xE304B2E: gen8_mfc_pipeline (gen8_mfc.c:4536)
==18007==    by 0xE35D801: intel_encoder_end_picture (i965_encoder.c:780)
==18007==    by 0xE350CFE: i965_EndPicture (i965_drv_video.c:3572)
==18007==    by 0x99AD8EC: vaEndPicture (va.c:1232)
==18007==    by 0x8921748: gst_vaapi_enc_picture_encode (gstvaapiencoder_objects.c:581)
==18007==    by 0x8924B4A: gst_vaapi_encoder_vp8_encode (gstvaapiencoder_vp8.c:376)
==18007== 
==18007== LEAK SUMMARY:
==18007==    definitely lost: 49,037,344 bytes in 2,994 blocks
==18007==    indirectly lost: 1,136 bytes in 22 blocks
==18007==      possibly lost: 140,872 bytes in 162 blocks
==18007==    still reachable: 577,456 bytes in 2,504 blocks
==18007==         suppressed: 0 bytes in 0 blocks
==18007== Reachable blocks (those to which a pointer was found) are not shown.
==18007== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==18007== 
==18007== For counts of detected and suppressed errors, rerun with: -v
==18007== ERROR SUMMARY: 6692690 errors from 903 contexts (suppressed: 0 from 0)

Tested on gstreamer 1.8.2 and 1.9.1 with lateset libva/libva-intel-driver stable versions and also from git master on intel skylake hardware(celeron g3900).