Can JPC/J2C be actually detected reliably?

Does typefind require that magic bytes be at the very beginning of the frame? If so, I think we can detect jpc and jc2 reliably.

That would be one way. You can also peek at other bytes. Basically you need some way to detect what format it is, reliably. You don't want to say JPC if it's actually an MP3 for example ;)

Right. Just to summarize the different formats:

JP2 boxes are in the format 

L T BBBBBBBBBBBBBBBBBBBBBBBBB

where :

L is a 4 byte number storing the length of the box, including L,
T is a 4 byte type, 
BBBBBBBBBBBBBBBBBBBBBBBBBb are the bytes in the box.



0) JPC is the raw jpeg 2000 code stream, starting with the SOC code stream marker : magic bytes 0xFF4FFF51


1) JP2 has the JP2 signature at the beginning of the file

  /* jp2 signature */
  if (memcmp (data, "\000\000\000\014jP  \015\012\207\012", 12) != 0)
    return;


JP2 also contains a contiguous codestream box of type T = "jp2c", and length L = 8;

It also contains a JPC file inside (raw code stream)



2) J2C format is identical to JPC except that it has a contiguous codestream box at the beginning, but unlike in JP2, the length L equals the length of the entire frame.

So, to find J2C, we can search for XXXX MAKE_FOURCC("jp2c") 0xFF4FFF51
where XXXX match any bytes.


And to find JPC, we need to just search for 0xFF4FFF51. Not sure if this is enough to be unique.

Actually, the SIZ, COD and QCD markers are mandatory for JPC, so I think it is possible.

Correction: J2C uses exactly the same contiguous codestream box as JP2.

So, JP2 format contains J2C format, which contains JPC format.

Created attachment 353646 [details] [review]
typefind: handle jpc format

This patch, unfortunately, fails on the following pipeline:

gst-launch-1.0 -v filesrc location=some.j2k ! typefind ! openjpegdec ! imagefreeze ! autovideosink

The following file can be used for testing:

https://github.com/GrokImageCompression/grok-test-data/blob/master/input/nonregression/Bretagne2.j2k

(a j2k file is the same as a jpc file)

typefind *is* detecting the file as jpc, but negotiation still fails.

Created attachment 353647 [details] [review]
support j2k file tag

(In reply to Aaron Boxer from comment #7)

> typefind *is* detecting the file as jpc, but negotiation still fails.

Does it work when adding jpeg2000parse? The caps need to include all fields that are required by openjpegdec

Review of attachment 353647 [details] [review]:

::: gst/typefind/gsttypefindfunctions.c
@@ +3205,3 @@
+
+  /* SOC marker + SIZ marker */
+  if (memcmp (data, "\377\117\377\121", 4) != 0)

If this is all you want to test, you could use the TYPE_FIND_REGISTER_START_WITH() macro :)

What you could add in addition here is for example to return a lower probability if you only find SOC/SIZ, and if you find afterwards one of the other markers (after skipping markers by size), increase the probability.


Also I don't think we need a typefinder for the JPEG2000 codestream with the MP4 box in front of it, it's nothing that should appear outside of MP4/MOV files.

Thanks! So, good, can ignore J2C.

SOC + SIZ + COD + QCD markers are mandatory, but COD and QCD could appear in any order. That wouldn't be hard to deal with. I will take a look.

Regarding failed negotiations: if jpeg2000parse is always needed, do we even need typefind support for JPC ?

It's needed to make uridecodebin/decodebin work.

Thanks.

So,  the following pipeline now works:


gst-launch-1.0 -v filesrc location-Bretagne2.j2k ! typefind ! jpeg2000parse ! openjpegdec ! imagefreeze ! autovideosink


progress!

However, this pipeline

gst-launch-1.0 -v filesrc location-Bretagne2.j2k ! decodebin ! imagefreeze ! autovideosink

fails with

Missing decoder: JPEG 2000 (image/x-jpc)

Sorry, correction, the following pipeline 


gst-launch-1.0 -v filesrc location-Bretagne2.j2k ! typefind ! jpeg2000parse ! openjpegdec ! imagefreeze ! autovideosink

does not work.

It only works with my other patch:

https://bugzilla.gnome.org/show_bug.cgi?id=783291

And with my other patch, typefind is not needed, because the parser can figure out the type from the code stream itself.

Your typefinder needs to provide caps that contain all fields needed by the next element (jpeg2000parse I guess). Then the parser needs a rank >= GST_RANK_MARGINAL to be autoplugged. And then the parser needs to provide caps that contains all fields needed by openjpegdec (which also needs a high enough rank).

Once that is given, decodebin should autoplug everything.

Thanks. So, for the pipeline

gst-launch-1.0 -v filesrc location-Bretagne2.j2k ! decodebin ! imagefreeze ! autovideosink

I don't think jpeg2000parse should be involved, just openjpegdec.

My typefind code for jpc just mimics the code for jp2. So, typefind does
identify the input as jpc.

So, I guess the problem is in openjpegdec ?

i.e. why is jpc treated differently than jp2? I searched the code for all
references to image/jp2 and I handled x-jpc exactly how jp2 is handled....

For the pipeline

gst-launch-1.0 -v filesrc location-Bretagne2.j2k ! decodebin ! imagefreeze ! autovideosink

here is output at DEBUG level 2:


Setting pipeline to PAUSED ...
0:00:00.060799798 ^[[332m 2953^[[00m      0x2126a70 ^[[33;01mWARN   ^[[00m ^[[00m             basesrc gstbasesrc.c:3480:gst_base_src_start_complete:<filesrc0>^[[00m pad not activated yet
Pipeline is PREROLLING ...
Got context from element 'autovideosink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = image/x-jpc
0:00:00.062651108 ^[[332m 2953^[[00m      0x2121d40 ^[[33;01mWARN   ^[[00m ^[[00m           decodebin gstdecodebin2.c:4600:gst_decode_bin_expose:<decodebin0>^[[00m error: no suitable plugins found:
Missing decoder: JPEG 2000 (image/x-jpc)

Missing element: JPEG 2000 decoder
0:00:00.062690968 ^[[332m 2953^[[00m      0x2121d40 ^[[33;01mWARN   ^[[00m ^[[00;32;43m            typefind gsttypefindelement.c:1228:gst_type_find_element_loop:<typefind>^[[00m error: Internal data stream error.
0:00:00.062697484 ^[[332m 2953^[[00m      0x2121d40 ^[[33;01mWARN   ^[[00m ^[[00;32;43m            typefind gsttypefindelement.c:1228:gst_type_find_element_loop:<typefind>^[[00m error: streaming stopped, reason not-linked (-1)
ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0: Your GStreamer installation is missing a plug-in.
Additional debug info:
gstdecodebin2.c(4600): gst_decode_bin_expose (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0:
no suitable plugins found:
Missing decoder: JPEG 2000 (image/x-jpc)

ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

Aha!!! 

For jp2, openjpegdec does NOT require GST_JPEG2000_SAMPLING_LIST
while jpc does. So, yes, you were right, a caps field was missing :)

And, I don't think we can supply the GST_JPEG2000_SAMPLING field reliably,
we can make a guess. So, we need the parser after all - can decodebin
require that the parser be inserted in the pipeline ?

Or should we abandon support for jpc ?

support in typefind

Or remove requirement for GST_JPEG2000_SAMPLING field in openjpegdec sink caps ?

Looking at the code in openjpegdec, the sampling field is only 
used when the decoded image has an unknown openjpeg colour space.

In this edge case, we can instead use the number of components, and the presence
of the MCT colour transform, to guess the colour space of the image.

So, we don't actually need the sampling field in the decoder.

Actually, we don't have access to presence of MCT colour transform, so we could just use the number of components to guess either RGB or GRAY. But, I think this is very much an edge case.

We can still check for the sampling field, in case there is a parser, and use it, but otherwise I think the number of components is sufficient to handle the edge case.

Correction :) For jpc, we actually don't have a colour box in the header, so this is not an edge case. Not sure what to do here.....

One solution is to separate out the parser code that calculates the sampling, and use it in both the parser and the decoder - put it in one static method, and use it from both elements.

(In reply to Aaron Boxer from comment #18)
> Thanks. So, for the pipeline
> 
> gst-launch-1.0 -v filesrc location-Bretagne2.j2k ! decodebin ! imagefreeze !
> autovideosink
> 
> I don't think jpeg2000parse should be involved, just openjpegdec.

openjpegdec wants a whole JPEG2000 frame per buffer. So jpeg2000parse should be involved.

(In reply to Aaron Boxer from comment #22)
> can decodebin require that the parser be inserted in the pipeline ?

See my earlier comment about that. Rank of the parser needs to be high enough and caps fitting, then decodebin will first put parser before decoder.

(In reply to Aaron Boxer from comment #30)
> One solution is to separate out the parser code that calculates the
> sampling, and use it in both the parser and the decoder - put it in one
> static method, and use it from both elements.

Can't we just always require the parser? This is exactly the point of a parser. It should extract information from the stream so that not every decoder has to do it again (or muxer, or ...).

And you need the parser anyway if you want to directly read from filesrc.

Thanks, that makes sense. 

Currently, parser is registered like so:

  ret |= gst_element_register (plugin, "jpeg2000parse",
      GST_RANK_PRIMARY, GST_TYPE_JPEG2000_PARSE);

But I don't see where openjpegdec is registered.

What needs to change so that the parser is ranked higher ? 

If the parser is put first, then typefind should be able to negotiate, since the parser's sink caps require only the media type and colour space.

Correction, I see it now - so they are both ranked primary.

I tried changing the rank of openjpegdec to secondary, but still does not negotiate.

So, one solution was to set the sink caps in jpeg2000parse to be simply

"image/x-jpc, image/x-j2c". 

Then the parser will get added, but errors out because
of lack of caps.  This issue is fixed in another bugzilla issue. 

https://bugzilla.gnome.org/show_bug.cgi?id=783291


So, if the parser can parse the file without relying on caps, then everything should work :)

Since this is a parser, it should be able to get all of its information from the file itself.

If the parser has a rank >= MARGINAL, it will be plugged before the decoder. Independent of the rank of the decoder. As long as the caps are supported by the parser, which seems to be the problem here, typefind producing caps that the parser can't accept.

Yes, exactly.  I didn't realize that parsers are given priority, which makes sense. So, I can confirm that with the fix from 783291, the pipeline is working fine.

Created attachment 354399 [details] [review]
typefind: handle jpc

We do a full parse of the main header up to the SOT marker. This is a pretty foolproof way of detecting a jpc frame.

Created attachment 354400 [details] [review]
typefind: handle jpc format

Created attachment 354401 [details] [review]
typefind: handle jpc format

clean up the code a bit

a gentle reminder that this patch is complete. I have tested it in my workflow - works correctly for jpc code streams, ready to be merged :)

Also, since a significant part of the jpc header is parsed, there is little chance of confusion with other file formats.

Review of attachment 354401 [details] [review]:

::: gst/typefind/gsttypefindfunctions.c
@@ +3206,3 @@
+  /* SOC marker + SIZ marker */
+  if ((data = gst_type_find_peek (tf, 0, 4)) != NULL) {
+    if (memcmp (data, "\xFF\x4F\xFF\x51", 4) != 0)

Nicer to put this into a const guint8 marker[] = {0xff, 0x4f, 0xff, 0x51};

@@ +3213,3 @@
+  }
+
+  while (!found_sot) {

This should probably not search until the end of file, but stop after 1KB or so

Thanks for the review! JPC header can actually be quite large, so I am not sure if it is a good idea to limit to 1K. If code gets to the while loop, it is most probably JPC format ?

I will add an exit to the loop if we read a marker that should not be in the header.

Created attachment 355018 [details] [review]
typefind: handle jpc

Look for specific markers in header. If we read an unrecognized marker, then exit without identifying type.

Created attachment 355022 [details] [review]
typefind: handle jpc

for jpc, only look for main header markers in typefind

commit 12c69c4cdf772e561bebf7de8d1918e01809e0a0 (HEAD -> master)
Author: Aaron Boxer <boxerab@gmail.com>
Date:   Mon Jun 12 22:57:26 2017 -0400

    typefind: Detect JPEG2000 codestreams
    
    https://bugzilla.gnome.org/show_bug.cgi?id=783625