the problem is that g_input_stream_read here:

https://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/ext/soup/gstsouphttpsrc.c#n1710

return 0 but we are not really at the end of the file since other implementations work as expected, could be a glib bug?

According to the docs here:

https://developer.gnome.org/gio/stable/GInputStream.html#g-input-stream-read

"""
Zero is returned on end of file (or if count is zero), but never otherwise.
"""

this not seems the case, thanks

I tested with:

- glib 2.56.1, gstreamer 1.14.0, libsoup 2.62.1
- glib 2.50.3, gstreamer 1.12.3, libsoup 2.56.0

both these combinations shows the problem

If you think this is a glib bug why did you move it to libsoup?

It's most likely the *implementation* in libsoup for GInputStream. It returns 0 bytes before the whole response body is returned (i.e. later calls would probably give the remainder).

Claudio, when using sync=true GStreamer asks to libsoup to download the file at a speed slower than the available network bandwidth, the problem does not happen when using sync=false that means download as fast as possibile,

curl or wget work as expected both at full speed than at a slower speed with the given http server (a network camera) and this exclude a server side problem

even my proof of concept GStreamer plugin based on libav (bug 788583) works as expected in both use cases

It's going to take a while for me to try to debug this; also I'm not familiar at all with how gstreamer works. Is there any chance you could write a minimal test case using libsoup's relevant input stream implementation that can reproduce this issue? It would help narrow it down.

well, I can try in my spare time, but I'm not familiar with libsoup, I'll try to copy libsoup relevant code from souphttpsrc's GStreamer plugin.

Sebastian, I think that a good test case to simulate sync=true outside GStreamer is a timer that read for example 8KB (or something lesser than the available bandwidth) from libsoup's GInputStream every seconds until g_input_stream_read returns 0 and then check if the received bytes are lesser than the expected ones, do you agree?

(In reply to Nicola from comment #0)
> so the problem is in souphttpsrc or libsoup and it is probably related to
> the missing Content-Length header

(In reply to Nicola from comment #5)
> when using sync=true GStreamer asks to libsoup to download the file
> at a speed slower than the available network bandwidth

Are you sure the server isn't just timing out on you and closing the connection?

Thanks Dan,

you are right,

please take a look at these wireshark dumps here:

http://94.177.162.225/temp/795176.7z

the video file duration is 5 minutes and 18 seconds, avsrc read from the server all the time, while souphttpsrc read for about 125 seconds and 90 seconds after the last read the server close the connection.

Sebastian, there is no bug in libsoup, I think the problem is in souphttpsrc, specifically here:

https://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/ext/soup/gstsouphttpsrc.c#n1628

I see logs such as this one

souphttpsrc gstsouphttpsrc.c:1642:gst_soup_http_src_check_update_blocksize:<souphttpsrc0> Increased blocksize to 4194304

increasing the blocksize is a problem when you read high compressed streams and you play them synced on the clock, how do you propose to solve this, a property to limit tha max blocksize? an hardcoded limit? thanks

You could use libsoup's get.c, move it to use the GInputStream, read in chunks and add a sleep in the read-loop. That should reproduce the problem from what I understand

Ah nevermind, missed the last comment. Yeah so on the GStreamer side it should not just read 0 but also get an GError that the connection was closed, or not?

For the solution, I don't really know. Limiting the block-size is an option of course, but there's always a trade-off here. Lower block-sizes have lower latency and higher overhead, and the opposite for bigger block-sizes.

A property could make sense, but ideally we could solve this without requiring any settings.

(In reply to Sebastian Dröge (slomo) from comment #11)
> Ah nevermind, missed the last comment. Yeah so on the GStreamer side it
> should not just read 0 but also get an GError that the connection was
> closed, or not?


I think the current behaviour is correct here, the server can close the connection at the end of the file too

> 
> For the solution, I don't really know. Limiting the block-size is an option
> of course, but there's always a trade-off here. Lower block-sizes have lower
> latency and higher overhead, and the opposite for bigger block-sizes.
> 
> A property could make sense, but ideally we could solve this without
> requiring any settings.

probably we should increase block size if bytes_read >= blocksize * GROW_BLOCKSIZE_LIMIT and the difference in time between the current socket read and the previous one is <= 1 second or even a smaller value. This way we increase the block size only if the pipeline is really consuming all these bytes. 

If the pipeline does not consume the readed bytes quick enough I think that increasing the blocksize is useless.

If you agree I can work on a patch that implements this but I'm really busy at the moment and I don't know when I'll have an hour to do this. 

Thanks for your support,

Nicola

Created attachment 370975 [details] [review]
proof of concept patch

with this patch the issue is fixed, you can easily test with any large enough file served using an http server on localhost, when using sync=false the blocksize is increased as before, with sync=true the blocksize remains small enough to avoid the connection to time out

The approach makes sense to me. Thiago, you added this code. What do you think?

Comment on attachment 370975 [details] [review]
proof of concept patch

Looks good to me

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/463.