Created attachment 327552 [details]
Log of the video-only file

Created attachment 327553 [details]
Log of the video/audio file

Problem is on line 6853 of the video/audio log.

This is unrelated to typefind. Can you also try if it works better with version 1.8.1? It seems like decodebin does not even want to try the amcviddec-omxnvidiah264decode.

Can you get the sinkpad caps from that element and provide them here?

Fair enough for typefind. You can edit the title.

I'm not 100% sure to understand what you are asking for but I guess that a log of the working audio/video will be the answer.

Created attachment 327559 [details]
Log of the video/audio file working when forcing decoder

Does it work with 1.8.1?

And please instantiate an amcviddec-omxnvidiah264decode and get the sinkpad caps, that is:
>  element = gst_element_factory_make("amcviddec-omxnvidiah264decode", NULL);
>  pad = gst_element_get_static_pad(element, "sink");
>  caps = gst_pad_query_caps(pad, NULL);
>  caps_string = gst_caps_to_string(caps);

They are unfortunately not printed in the logs.

Sorry. Here is the caps you are asking for:

video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high;

video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)main;

video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)baseline;

video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)constrained-baseline


Here is the src caps of h264parse just in case it's needed:
video/x-h264, parsed=(boolean)true, stream-format=(string){ avc, avc3, byte-stream }, alignment=(string){ au, nal }


Note that I have also tried "h264parse ! decodebin" in my application and it's not working either.

Last, I have also tried the unmodified tutorial5 which uses playbin, and the video doesn't come through. The audio plays with a black screen. I see the same behavior in my application if I use playbin (audio but no video).

For some reason h264parse does not want to produce byte-stream h264 here. Things fail because of it outputting avc and not byte-stream.

That seems like a generic problem.

But not sure why, making avdec_h264 only accept byte-stream here does not break h264 playback. Needs further debugging.

With the following pipeline:

filesrc location=/storage/emulated/0/aiTennis3D/videoaudio/videov.mp4 ! qtdemux name=q q.video_0 ! queue ! h264parse ! video/x-h264,parsed=true,stream-format=byte-stream,alignment=au ! decodebin name=forhwgl ! queue ! identity name=forhalffps ! glimagesink force-aspect-ratio=false q.audio_0 ! queue ! decodebin ! queue ! autoaudiosink

I get sound and I get the first video frame on screen. Perhaps, I could have a quick fix along that path?

Created attachment 327602 [details]
Log of the video/audio file while forcing caps

Any idea? What could I provide more to debug this issue?

Not really, it needs someone to look more into this. I can't reproduce the problem on my computer (when setting up things in the same way), so need to try it on Android to understand why exactly it happens.

I confirm that I can play the file too on my Linux Ubuntu 14.04 PC (gstreamer 1.2.4).

Did it work on Android with pre-1.8.1 versions btw? And on the desktop, you need some code changes to get into a similar situation at all... but that works.

Very latest of gstreamer, just compiled, 1.9.0.1 is actually worse!

Audio/video fails the same way:

05-13 09:29:41.851  6376  6416 W GStreamer+uridecodebin: 0:00:16.892240306 0xf321c350 gsturidecodebin.c:938:unknown_type_cb:<uridecodebin2> warning: No decoder available for type 'video/x-h264, parsed=(boolean)true, stream-format=(string)avc, alignment=(string)au, width=(int)1280, height=(int)1440, framerate=(fraction)151/3, profile=(string)baseline'.

Video only doesn't play! A regression compared to my current version which is 2-3 months old.

You really need to step in ;-)

I have also tested the binaries 1.8.1 found on the website. Audio/video fails. Video works.

So the regression mentioned in Comment #17 appears after 1.8.1.

Ah that's good to know! Can you get the sinkpad caps again with the very latest build too?

It's the same. Why would you think it would be different?



video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high

video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)main

video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)baseline

video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)constrained-baseline

Just making sure that at least that part didn't change :)

So 1.9.1 apparently has changes somewhere in playbin that makes things worse. Can you get a debug log of the new situation too? Maybe that one is more helpful, now that it fails more.

Actually not, the video is now playing with my application. So perhaps, I was mistaken with too many tests.

But certainly, the audio/video bug is still there on all versions.

The audio/video file doesn't play in Android media player because it's a mp3/h264 combination which is not supported by Android.

If I only remove the audio part (video part is unchanged) from the mp4 container, the new (video only) file plays in Android media player. It still doesn't play with Gstreamer Android. The same file plays on a PC with any stack (vlc, Gstreamer, mplayer).

So, it's definitely a problem on Gstreamer Android.

Is there any update on this bug? any other info needed?

What is the current status of this now, you added some confusing comments in between so I'm not sure anymore what the problem currently is. Maybe you can get a new debug log? Is the problem that h264parse does not want to produce the stream-format needed by the decoder, or ...?

Still very alive :-(((

I had to step back away from 1.9 because of bug 766993.

But this bug is still present across any Android gstreamer 1.7.x, 1.8.x and 1.9.x.

The audio/video file doesn't play on Android Gstreamer whatever pipeline (playbin or decodebin), my app or tutorial-5, and whatever output type (memcpy or not).

The same file plays on PC across any player (gstreamer-0.10, gstreamer-0.1, vlc, mplayer).

I can produce another log but it will be the same.

Ok, so if this happens in all those versions, then it's not really a blocker as it's not really a regression from older versions.

Without more details about what goes wrong I can't really help here. "Doesn't play" is not something you can start working on, and the information in the comments above is contradicting so I don't know what the actual problem is at this point.

Once bug 766993 is solved, please try to get a new debug log and describe the problem again.

I don't think your last comment is fair. I can not provide log on Android gst-tutorial5 because of bug 76693, but I can provide logs on my application which disables memcpy and uses the legacy decoder->nv12->upload->colorconversion

I have just cloned the very latest (June 6 morning) to test again.

The test files haven't changed: http://gentil.com/tmp/samples.zip

I upload the log files. It includes *:5 on WORKING video-only, *:5 on NON-WORKING video-audio and *:6 on NON-WORKING video-audio.

I think it's a blocker as it works everywhere except on Gstreamer Android.

Created attachment 329237 [details]
Log with latest as of June 6, 2016

With the latest comment here https://bugzilla.gnome.org/show_bug.cgi?id=766993#c19, I can now confirm that the problem exists also with the plain Android gst-tutorial-5.

The playbin pipeline handles the video-audio file but only plays the audio part. Screen is black.

To avoid another ping-pong party (it's because of Pixel C, gles, whatever... blah-blah...), I attach the level-5 log of http://gentil.com/tmp/samples.zip on:

- Android Nexus 7 2013
- plain untouched Android 5.x-
- latest plain untouched gstreamer as of June 7, 2016
- plain untouched gst-tutorial-5 running the playbin pipeline.

The audio/video file plays only the audio, the screen being black.

Can we put back the status to new, blocker, 1.9.1? Thanks.

Created attachment 329350 [details]
Log level5 Nexus7-2013

Curiously, there's a warning from h264parse that should never happen:

W/GLib+GStreamer( 1861): gstpad.c:5071:store_sticky_event:<h264parse1:src> Sticky event misordering, got 'segment' before 'caps'

But alas, the problem is:

missing-plugin, type=(string)decoder, detail=(GstCaps)"video/x-h264\,\ parsed\=\(boolean\)true\,\ stream-format\=\(string\)avc
\,\ alignment\=\(string\)au\,\ profile\=\(string\)baseline", name=(string)"H.264\ \(Baseline\ Profile\)\ decoder";

Even though you have:
amcviddec-omxqcomvideodecoderavc
amcviddec-omxgoogleh264decoder
openh264dec

Debugging further requires finding out why and where they fail creation/use.  The log doesn't really say and is cut off at the beginning which misses the GStreamer initialization.  There are also some useful log messages in the LOG(6) category so could you get the a full *:6 log from app startup to failure.

Attached log. Is it enough? It's cut off because Android has only 16MB of buffer.

Created attachment 329362 [details]
Log level6 Nexus7-2013

Created attachment 329363 [details]
Log level6 Nexus7-2013 (longer)

Above is another longer log level6 (not sure if it's complete, middle might be missing, but begin should be there)

As a reminder, the other file (video only) works as expected.

(In reply to Matthew Waters (ystreet00) from comment #33)
> Curiously, there's a warning from h264parse that should never happen:
> 
> W/GLib+GStreamer( 1861): gstpad.c:5071:store_sticky_event:<h264parse1:src>
> Sticky event misordering, got 'segment' before 'caps'
> 
> But alas, the problem is:
> 
> missing-plugin, type=(string)decoder, detail=(GstCaps)"video/x-h264\,\
> parsed\=\(boolean\)true\,\ stream-format\=\(string\)avc
> \,\ alignment\=\(string\)au\,\ profile\=\(string\)baseline",
> name=(string)"H.264\ \(Baseline\ Profile\)\ decoder";
> 
> Even though you have:
> amcviddec-omxqcomvideodecoderavc
> amcviddec-omxgoogleh264decoder
> openh264dec

From my understanding of the (older) logs, the reason is that all those decoders want byte-stream but we get avc... and for some reason h264parse and the capsfilter break things there. However this doesn't happen with a local testcase (when preventing avdec_h264 from accepting avc).

Ok, so what happens is this.

1. qtdemux finds an empty segment and decides to push a gap event.
2. This gap event prompts baseparse to negotiate *something* for caps which ends up being almost a direct copy of upstream's caps (which has stream-format=avc).
3. These negotiated caps are then used to filter the decoders which rules out the byte-stream decoders resulting in no decoders being selected/tried and resulting in the missing decoder error.

Might be related to 00f23053b1f27c6857a2153b39e1344d2d6e9eaf in qtdemux

Did we find a way to reproduce this on a desktop?

Should be reproducible with this specific file (that creates a gap event in the beginning) and removing support for stream-format=au from gst-libav, and then just playbin. If the theory is correct

(In reply to Sebastian Dröge (slomo) from comment #43)
> Should be reproducible with this specific file (that creates a gap event in
> the beginning) and removing support for stream-format=au from gst-libav, and
> then just playbin. If the theory is correct

Just a typo avove. Remove stream-format=avc and I can reproduce the bug.

Here is a new file that still doesn't work though I don't think it has the empty video initial segment.

Those audio/video files are generated with gstreamer-0.10 on an embedded device. The audio part seems to start a little bit quicker than the video part, explaining why there are two segments in the original files (http://gentil.com/tmp/samples.zip), the first segment being very short and with audio only.

I changed the code, added an audio and video probe before mp4mux and discarding audio as long as video is not ready on the embedded device. The resulting new file http://gentil.com/tmp/sample2.zip should have only one segment with audio and video - I was not able to confirm that. It still doesn't play on Android.

(In reply to Matthew Waters (ystreet00) from comment #40)
> Ok, so what happens is this.
> 
> 1. qtdemux finds an empty segment and decides to push a gap event.
> 2. This gap event prompts baseparse to negotiate *something* for caps which
> ends up being almost a direct copy of upstream's caps (which has
> stream-format=avc).
> 3. These negotiated caps are then used to filter the decoders which rules
> out the byte-stream decoders resulting in no decoders being selected/tried
> and resulting in the missing decoder error.

Any solution for this? After decodebin has ruled out the decoders, should it retry if caps have changed? Any better ideas?

(In reply to Gregoire from comment #45)
> Here is a new file that still doesn't work though I don't think it has the
> empty video initial segment.
> 
> Those audio/video files are generated with gstreamer-0.10 on an embedded
> device. The audio part seems to start a little bit quicker than the video
> part, explaining why there are two segments in the original files
> (http://gentil.com/tmp/samples.zip), the first segment being very short and
> with audio only.
> 
> I changed the code, added an audio and video probe before mp4mux and
> discarding audio as long as video is not ready on the embedded device. The
> resulting new file http://gentil.com/tmp/sample2.zip should have only one
> segment with audio and video - I was not able to confirm that. It still
> doesn't play on Android.

It still has a small gap at the beginning of one of the tracks.

I'm not surprised that it has a small gap as it doesn't play...

Is there a way to avoid the gap while encoding? My pipeline is:

v4l2src ! ducatih264enc ! queue ! h264parse name=startVideo ! queue ! m.video_0 vs1063src ! mpegaudioparse name=startAudio ! queue ! m.audio_0 mp4mux name=m ! queue ! filesink

As said, I have a buffer probe in startVideo and startAudio and I return true only when both probe have been called.

I would still prefer to have the original bug fixed but if I can improve my file so that it doesn't have weird data, that would be a plus.

(In reply to Gregoire from comment #49)
> I would still prefer to have the original bug fixed but if I can improve my
> file so that it doesn't have weird data, that would be a plus.

It is very common to have gaps at the beginning of audio and video recording as the devices usually have different startup latencies. To avoid it you would need to make sure audio and video arrive at the muxer with the same exact running times.

I would understand it's common as it's difficult to have audio and video with the exact same startup latencies.

Your comment would also comfort this bug needs to be addressed and fixed.

That being said, I thought that by having buffer probe and return true only when they have been called at least once would insure that mp4mux receives both audio and video at the same time. Is there an option in mp4mux like "wait audio and video before muxing" or is there a better proper way to achieve an exact sync so that everything arrives at the same time?

So, what's the next step? I think it's a common bug and I'm surprised that it didn't arise before because I don't see how one can have audio and video arrive at the same exact moment. Or is there a way when encoding to avoid any empty segment?

I'd say, drop 0.10 in the first place, and try this proposed element https://bugzilla.gnome.org/show_bug.cgi?id=750794

Still need to be reviewed, but works for the author.

I have made a lot of efforts on Android to be as close as possible to upstream but on this device, this is quasi impossible. I would have to port more than 5000 lines of code from gstreamer-.10 to gstreamer-1.0 with the risk of introducing bugs, performance issue and so on...

And there would be no guarantee at all that it would work better on gstreamer-1.0.

I have tried to stop feeding audio for a second or so, so that I'm sure video is fed first but it's still not working.

The solution for this bug here seems to be to make h264parse do more clever negotiation on gap events, involving its knowledge about what it can convert to. Currently baseparse just does this gap negotiation by itself without asking the subclass, that apparently has to be changed.

Shouldn't be too difficult, either needs some kind of new vfunc or direct handling of the gap event in h264parse instead of letting the subclass do it.

I'm willing to try to patch. Sebastian, Can you please guide me (babysitting me) what needs to be done exactly?

Not really, other than take a look at the h264parse code and baseparse code where it is deciding the caps... and especially the code where baseparse is handling the default negotiation for GAP events. Try to understand that code.

And then this code somehow needs to call into h264parse, either by having h264parse handle GAP events directly instead of the base class or a new vfunc. And that new code then needs to take into account h264parse's conversion abilities, like the other caps selection code that already exists in h264parse.

Sorry, it seems that already know the reason, but no result?
so it is a general problem, especially in mobile devices, AFAIK,because their HW decoders usually only support byte-stream？:)

http://gstreamer-devel.966125.n4.nabble.com/How-to-convert-avc-au-to-byte-stream-au-td4683201.html#a4683451

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/388.