GNOME Bugzilla – Bug 756422
rtpj2kpay: wrong offsets in output
Last modified: 2016-06-21 08:31:36 UTC
As described in bug 745187, streaming with rtpj2kpay and opejpegenc doesn't work. I tried using jpeg2000 codec from libav. After applying patch to ext/libav/gstavcodecmap.c, as suggested in bug 756288, pipeline just with encoder/decoder does work: gst-launch-1.0 videotestsrc ! videoconvert ! avenc_jpeg2000 ! avdec_jpeg2000 ! videoconvert ! autovideosink However, when I tried linking avenc_jpeg2000 to rtpj2kpay I got internal data flow error, because sink format from avenc_jpeg2000 is "image/x-j2c" and rtpj2kpay wants image/x-jpc" as source format. I was able to "fix" this by modifying already patched code (though I'm not sure it was right thing to do): case AV_CODEC_ID_JPEG2000: caps = gst_ff_vid_caps_new (context, NULL, codec_id, encode, "image/x-j2c", NULL); if (!encode) { gst_caps_append (caps, gst_ff_vid_caps_new (context, NULL, codec_id, encode, "image/x-jpc", NULL)); gst_caps_append (caps, gst_ff_vid_caps_new (context, NULL, codec_id, encode, "image/jp2", NULL)); } break; I just swapped "image/x-j2c" and "image/x-jpc", so now pipeline doesn't produce internal data flow error: gst-launch-1.0 videotestsrc ! videoconvert ! avenc_jpeg2000 ! rtpj2kpay ! rtpj2kdepay ! avdec_jpeg2000 ! videoconvert ! autovideosink However, it still doesn't work, nothing happens. This pipeline still works though: gst-launch-1.0 videotestsrc ! videoconvert ! avenc_jpeg2000 ! avdec_jpeg2000 ! videoconvert ! autovideosink So I'm starting to suspect that a problem lays in rtpj2kpay/depay.
image/x-j2c and image/x-jpc are different formats for jpeg2000, the avenc_jpeg2000 can only produce one type while the rtp element wants to consume a different one. Switching the formats will just fool the rtp element into thinking it is receiving one format while it is actually another. I see 2 options here: 1) use openjpegenc that claims to support the format the rtp element want 2) improve avenc_jpeg2000 to support the desired format (ffmpeg might have an API to set that, but I haven't checked so far) Anyway, trying with openjpegenc also seems to cause some issues on the received side, I see lots of: "rtpj2kdepay gstrtpj2kdepay.c:558:gst_rtp_j2k_depay_process:<rtpj2kdepay0> discard packet, no sync" and no output comes out. Some more investigation is needed here.
image/x-j2c and image/x-jpc are the same, the only difference is that image/x-j2c prepends a "jp2c" box to each frame. IMHO that one should disappear and be done by qtmux/qtdemux as needed instead... but we can only do that in 2.0.
Hm, so this could be easily done in the payloader for 1.x for now.
Created attachment 313201 [details] [review] rtpj2kpay: update fragment offset It was always being set to 0, making the resulting stream broken for the receiver
I'm afraid that doesn't seem to work with openjpeg: gst-launch-1.0 videotestsrc ! videoconvert ! openjpegenc ! rtpj2kpay ! rtpj2kdepay ! openjpegdec ! fakesink gives this several times: 0:00:11.165840987 10166 0x253c280 DEBUG default gstsegment.c:492:gst_segment_to_running_time_full: invalid position (-1) 0:00:11.165859164 10166 0x253c280 DEBUG openjpegdec gstopenjpegdec.c:973:gst_openjpeg_dec_handle_frame:<openjpegdec0> Handling frame 0:00:11.165883792 10166 0x253c280 WARN videodecoder gstvideodecoder.c:4065:_gst_video_decoder_error:<openjpegdec0> error: Failed to decode OpenJPEG stream and stops. As far as I can see, openjpegdec for some reason can't decode stream after depayer. I reverted my "fix" in gstavcodecmap.c, so avenc_jpeg2000 and rtpj2kpay do not want to link together (because of different image type). (In reply to Thiago Sousa Santos from comment #4) > Created attachment 313201 [details] [review] [review] > rtpj2kpay: update fragment offset > > It was always being set to 0, making the resulting stream broken > for the receiver
There is something wrong in opejpeg it seems. For example: gst-launch-1.0 videotestsrc num-buffers=1 pattern=black ! openjpegenc ! "image/x-jpc, colorspace=sYUV" ! openjpegdec ! fakesink is crashing at the encoder here, while adding the rtp payloader and depayloader crashes at the decoder. I compared the input and output of the payloader and depayloader and they are identical after my patch, but: /GstPipeline:pipeline0/GstRtpJ2KPay:rtpj2kpay0.GstPad:sink: caps = "image/x-jpc\,\ colorspace\=\(string\)sRGB\,\ num-components\=\(int\)4\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ framerate\=\(fraction\)30/1" and /GstPipeline:pipeline0/GstRtpJ2KDepay:rtpj2kdepay0.GstPad:src: caps = "image/x-jpc\,\ framerate\=\(fraction\)0/1\,\ fields\=\(int\)1\,\ colorspace\=\(string\)sYUV" doesn't seem right. I wonder if any extra signaling is needed to get all the info into the rtp depayloader. The RTP caps is: "application/x-rtp\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000\,\ encoding-name\=\(string\)JPEG2000\,\ payload\=\(int\)96\,\ ssrc\=\(uint\)2639853500\,\ timestamp-offset\=\(uint\)1296653318\,\ seqnum-offset\=\(uint\)26752"
Anyway the rtp fix kills one of the problems, at least: commit 539ebd0f42e715684f5547751ae6abe50c06bc2e Author: Thiago Santos <thiagoss@osg.samsung.com> Date: Tue Oct 13 12:42:56 2015 -0300 rtpj2kpay: update fragment offset It was always being set to 0, making the resulting stream broken for the receiver https://bugzilla.gnome.org/show_bug.cgi?id=756422
I tried all of the openjpeg pipelines mentioned here with the current master, and there were no crashes or abnormal behaviour. gst-launch-1.0 videotestsrc num-buffers=1 pattern=black ! openjpegenc ! "image/x-jpc, colorspace=sYUV" ! openjpegdec ! fakesink had a not-negotiated error, but I don't think that that is a bug. As for avenc_jpeg2000/avdec_jpeg2000, my impression from discussing this on the ffmpeg-devel group is that openjpeg is a much better codec right now. And it works fine with rtpj2kpay/rtpj2kdepay. So, I would encourage users to use openjpeg instead. In the end, I don't think there is a bug here.
Can this bug be closed ?