How would you get the cookies from the source that requested the playlist to the GstURIDownloader? They could probably be put as GstMeta on the source's buffers.

Also what about referer and other information one might want to pass to the HTTP request?

Good points. I suppose you could attach a GstMeta containing all the headers for the original HTTP response with the buffer. That way Set-Cookies and any other headers could be extracted as needed in downstream elements.

Although that wouldn't cope with multiple requests setting different cookies, so I suppose a GstMeta with a shared cookie jar object in it might also be a way to go.

Or both to get the best of both worlds.

I mainly flagged this because firefox, using gstreamer, failed to play back some HLS content from BBC iPlayer using a new CDN. The error message on the CDN server indicated that the cookie wasn't provided, and a quick check showed that uridownloader didn't do anything with cookies. Just thought it would be a good idea to flag this issue up.

Yes, I remembered observing a similar issue with local broadcasting service in Québec Canada. Reportedly it's using The Platform framework with a Wowsa server. Basically it checks that cookies obtained when loading the web page are present when downloading the playlist (and also the fragments (though cross domain here)) as a best effort to prevent external player from playing the stream (hence preventing adds skip). I then forgot to file it, as this service has been blocked since for non Apple devices.

Ideally there would be a firefox HTTP source element that is using the firefox infrastructure, and uses that to get cookies, authentication information, etc. We did the same for webkit some years ago for this very reason.

I've been experimenting with this approach:

- source element handles a "http" context query performed by uridownloader and adaptivedemux
- the context is set on adaptivedemux
- the context somehow stores the cookies. For now I'm reusing the GstCookieJar from bug 731170
- when downloading fragments adaptivedemux fetches the cookies from the context and set them on the source element if it has a cookies property

We could store more informations inside the context, like HTTP referer, user-agent, custom headers, I suppose.

With this approach I have here WebKit fetching the main playlist with the webkitwebsrc element, and uridownloader/adaptivedemux downloading fragments/manifests with cookies in http requests managed by souphttpsrc :)

Created attachment 314301 [details] [review]
adaptivedemux: minimal HTTP context support

The uridownloader is now querying the source element for an HTTP
context, which stores session data (cookies only for now), and reusing
the data when fetching data over HTTP. Additionally the context is set
on adaptivedemux, which allows it to also properly use session data
when downloading fragments.

Created attachment 314302 [details] [review]
souphttpsrc: cookie jar and context query support

Use a volatile Cookie jar to store cookies and handle the context
query so that session data can be shared with other elements (like
adaptivedemux).

Any news about this? I think Thiago was playing around on this topic as well.

Yes, I have some WIP patches but the server I was using for testing lifted the cookies requirement so I'm trying to find an alternative to testing this.

Any other known servers?

The only one I know is private :/

But I suppose it shouldn't be too hard to build an HTTP server in Python that would check the cookies of the request before serving the video data.

There's a thing like that implemented in Perl in the WebKit layout test harness :)

Hosted a small python flask-based server that simulates a cookies requirement: https://github.com/thiagoss/adaptive-test-server

Patches already work for DASH but fail for HLS because we still need to figure out how to share the context with uridownloader.

With the 2 patches of this bug I have it working for HLS (in WebKit). If you have different changes can you please upload the patches?

Latest work is kept here: http://cgit.freedesktop.org/~thiagoss/gst-plugins-good/log/?h=cookies

I've opted to sharing the cookie-jar directly because otherwise it is too complicated to keep cookies synchronized accross multiple elements. It works for HLS and DASH because the cookies are set on the first access and then just replicated but then new cookies set aren't synchronized anymore.

A generic GstCookieJar could be written that would listen and update different cookiejar implementations but it seems to me a bit overkill as long as we don't have pipeline with multiple different types of http sources working together. Usually there is only a single http source element involved.

Ok I'll try this on my side but what about adaptivedemux/uridownloader? With your approach which element is querying the context?

Just finished the uridownloader part:
http://cgit.freedesktop.org/~thiagoss/gst-plugins-bad/log/?h=cookies

There was also a bug about sharing the same user-agent and extra-headers IIRC. It would be nice to solve all of that in one go.

The patches are not working here after implementing the need-context message support in WebKit :(

Review of attachment 314302 [details] [review]:

I think the approach is good, just one comment here (and we need to make sure the GstContext machinery works here properly)

::: ext/soup/gstsouphttpsrc.c
@@ +2039,3 @@
+          cookies[0] = cookie;
+          cookies[1] = NULL;
+          gst_structure_set (context_structure, "cookies", G_TYPE_STRV, cookies,

Is this really enough? Isn't a cookie usually stored together with the domain and where it can be used, to prevent arbitrary websites reading your cookies from elsewhere?

Comment on attachment 314301 [details] [review]
adaptivedemux: minimal HTTP context support

Shouldn't this also do something about the HAVE_CONTEXT and NEED_CONTEXT messages?

And should souphttpsrc also do those?

(In reply to Philippe Normand from comment #16)
> The patches are not working here after implementing the need-context message
> support in WebKit :(

Did you check why it fails? Do you get the need-context messages in webkit?

(In reply to Thiago Sousa Santos from comment #19)
> (In reply to Philippe Normand from comment #16)
> > The patches are not working here after implementing the need-context message
> > support in WebKit :(
> 
> Did you check why it fails? Do you get the need-context messages in webkit?

I do get and correctly handle the need-context messages in WebKit.
I haven't had time to debug this further yet but I suspect either:

- the uridownloader isn't using the cookies and/or hasn't the context set yet when requesting the fragments.
- or perhaps the cookie jar isn't storing the cookies for the right first-party domain. That would be an issue on WebKit side

(In reply to Philippe Normand from comment #20)
> (In reply to Thiago Sousa Santos from comment #19)
> > (In reply to Philippe Normand from comment #16)
> > > The patches are not working here after implementing the need-context message
> > > support in WebKit :(
> > 
> > Did you check why it fails? Do you get the need-context messages in webkit?
> 
> I do get and correctly handle the need-context messages in WebKit.
> I haven't had time to debug this further yet but I suspect either:
> 
> - the uridownloader isn't using the cookies and/or hasn't the context set
> yet when requesting the fragments.
> - or perhaps the cookie jar isn't storing the cookies for the right
> first-party domain. That would be an issue on WebKit side

Did you handle those from a sync bus message handler? The context must be set before the handler returns (in sync with the element requesting it).

Right on, Thiago :) It works now after moving the http context handling to a sync message handler.

Are the patches ready for review Thiago? It would be nice if we can have them merged before 1.8.0 :)

Have you sorted out the cookies issues? Please let me know if I can help.

I believe those patches are ready to be merged:

https://cgit.freedesktop.org/~thiagoss/gst-plugins-base/log/?h=gstcookies
https://cgit.freedesktop.org/~thiagoss/gst-plugins-good/log/?h=gst-cookies

There are quite some patches, how do you prefer to review them?

Thanks to Philippe for all the help with this.

These 2 patches as well?

https://cgit.freedesktop.org/~thiagoss/gst-plugins-bad/log/?h=cookies

I think we should close this bug as dupe of #761099 and continue the discussion there :)

Let's do that then :)

*** This bug has been marked as a duplicate of bug 761099 ***