Youtube topic: https://productforums.google.com/forum/#!topic/youtube/CEUCHAJpNHY;context-place=forum/youtube

It is possible that the problem could be in the WebKit backend; this should also be investigated.

This is a very old issue; present as long as Youtube has been offering HTML5.

It has been suggested that youtube's advertisements are the problem:
https://bbs.archlinux.org/viewtopic.php?id=137695
Why are the advertisements incompatible with Epiphany in particular?

Another theory holds that spoofing Epiphany's useragent as Chrome will work around the problem and cause Youtube to offer a compatible video:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=567311
Is this still relevant? Could we convince the Youtube devs to include recognition for Epiphany?

Another suggested workaround is to use nspluginwrapper for flash, which somehow works around improperly encoded videos on youtube:
https://bugzilla.redhat.com/show_bug.cgi?id=749223
Given that flash now works in Epiphany without nspluginwrapper, I find it unlikely that this solution is relevant. Is there still a problem with codecs? Is this what causes advertisements to fail, causing videos not to play? Why doesn't Epiphany play the advertisements with the flash plugin (flash works everywhere else) and switch to HTML5 for the videos?

nspluginwrapper is not compatible with recent versions of flashplugin.so

nspluginwrapper -i libflashplayer.so
nspluginwrapper: libflashplayer.so is not a valid NPAPI plugin

yaourt -Qo /usr/lib/mozilla/plugins/libflashplayer.so 
/usr/lib/mozilla/plugins/libflashplayer.so is owned by flashplugin 11.2.202.508-1

(In reply to Que Quotion from comment #0)
> Epiphany does not play many videos on Youtube.
> 
> It may be the case that Youtube's advertisements do something Epiphany is
> not compatible with, preventing the videos from playing.

Bingo, good guess. If you turn off the adblocker in preferences, the video plays fine. It's possible the problem is with Epiphany's adblocker, but that is not very likely. More likely, either YouTube's scripts are broken (failure to expect the adblocker would block the ad, possibly, though that seems strange as surely they would be prepared to handle adblockers?), or there is some strange problem in WebKit that is only triggered by that error case (this seems most likely to me, but I have no clue what it might be :)

> To that end I am filing reports for both Youtube and GNOME. It is important
> to recognize that Epiphany is part of the problem and not put all of the
> blame Youtube, flash player, or another piece of software. Epiphany is the
> application having a problem, both from the user perspective and Youtube's
> (the site itself, not its developers). In order to resolve this issue as
> easily and quickly as possible, I reccommend that the two development teams
> communicate and cooperate.

I'm happy to work with the YouTube folks on this if we can figure out that changes in Epiphany or WebKit are needed, but we can't debug site-specific issues ourselves, due to the complexity of modern websites, js obfuscation, etc. We do fix WebKit bugs if someone is able to provide a reduction (a simple test case that demonstrates the issue), but this is hard to do, and it's quite rare that anyone other than one of the website developers is able to do so.

(In reply to Que Quotion from comment #2)
> It is possible that the problem could be in the WebKit backend; this should
> also be investigated.

See also:

https://bugs.webkit.org/show_bug.cgi?id=135972
https://bugs.webkit.org/show_bug.cgi?id=147000

> This is a very old issue; present as long as Youtube has been offering HTML5.
> 
> It has been suggested that youtube's advertisements are the problem:
> https://bbs.archlinux.org/viewtopic.php?id=137695
> Why are the advertisements incompatible with Epiphany in particular?

No clue.

> Another theory holds that spoofing Epiphany's useragent as Chrome will work
> around the problem and cause Youtube to offer a compatible video:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=567311
> Is this still relevant? Could we convince the Youtube devs to include
> recognition for Epiphany?

I will test this after I post this comment, since it requires me to close my browser. WebKit already has infrastructure to fake the user agent on specific broken sites, for compatibility with those sites. I'm willing to add YouTube to that list in WebKit if need be. (We're not going to add any user-agent hacks at the Epiphany level.) But that bug report is quite old, and unlikely to still be relevant; I don't expect changing the user agent to make a difference. But I will test in a second and find out.

> Another suggested workaround is to use nspluginwrapper for flash, which
> somehow works around improperly encoded videos on youtube:
> https://bugzilla.redhat.com/show_bug.cgi?id=749223
> Given that flash now works in Epiphany without nspluginwrapper, I find it
> unlikely that this solution is relevant. Is there still a problem with
> codecs? Is this what causes advertisements to fail, causing videos not to
> play? Why doesn't Epiphany play the advertisements with the flash plugin
> (flash works everywhere else) and switch to HTML5 for the videos?

Another quite old bug report. The reason to use nspluginwrapper was to be able to load Flash (a GTK+2 plugin) in a GTK+3 browser (Epiphany), which is impossible due to symbol conflicts. nspluginwrapper must be running the plugin in another process. Anyway, that's not needed anymore, because nowadays we have separate a separate compatibility plugin process to support GTK+ 2 plugins.

"Is there still a problem with codecs?" Depends. YouTube has been pretty good about hosting WebM videos, for the most part. But for some reason, a small subset of their videos are H.264-only. H.264 codecs cannot be legally distributed in the US and in most of the EU (actually, in most everywhere ;) without paying a royalty to MPEG, and no Linux distros pay that royalty. Distros without corporate sponsors ship that codec anyway and hope not to get sued, which distros with corporate sponsors normally do not ship the codec. If you're using Fedora or openSUSE or Ubuntu, you don't have an H.264 codec. Otherwise, you probably do.

"Is this what causes advertisements to fail, causing videos not to play?" No clue. I don't have Flash installed, nor am I willing to install it, because it is nonfree and I will not agree with its license agreement. If it's mandatory for adverts to play on YouTube, it might be a choice of YouTube to fail the video rather than let us get away with adblock.

(In reply to Michael Catanzaro from comment #4) 
> I will test this after I post this comment, since it requires me to close my
> browser.

I tested with this user agent:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36

But the video did not play, so that's not a problem.

(In reply to Michael Catanzaro from comment #4)
> "Is this what causes advertisements to fail, causing videos not to play?" No
> clue. I don't have Flash installed, nor am I willing to install it, because
> it is nonfree and I will not agree with its license agreement. If it's
> mandatory for adverts to play on YouTube, it might be a choice of YouTube to
> fail the video rather than let us get away with adblock.

Um, well the answer of course is "no," because the ads play fine if I disable the adblocker. ;)

The one thing I forgot to check was the adblocker... (I just leave it on, like cruise control).

How does the adblocker work? Is it subscribed to EasyList? I don't recall having this kind of problem using AdBlock+ in Firefox (I'd need to reinstall all of that to test).

I'm sure Youtube sees their advertisements as madatory, in which case they should be saying something when the advertisements are blocked. Currently their player just sits there, blacked out at 00:00. It looks like something is broken; the buttons on the player respond to clicks but no video plays.

Maybe I can convince them to put up a "don't block advertisements!" in place of the black screen.

Or should Epiphany be able to block *and skip* the advertisements?

(In reply to Que Quotion from comment #7)
> The one thing I forgot to check was the adblocker... (I just leave it on,
> like cruise control).
> 
> How does the adblocker work? Is it subscribed to EasyList?

Yup.

(In reply to Que Quotion from comment #8)
> Or should Epiphany be able to block *and skip* the advertisements?

That's up to the scripts on YouTube. Epiphany is going to block them from loading their advertisement... how they choose to handle that is up to them. :)

I wonder how adblock+ (with EasyList+EasyPrivacy subscription) does this in Firefox; I recall that videos continued to play although the advertisements were blocked. Does it have a site-specific hack to tell Youtube the advertisement was "seen"? There's also the possibility that Youtube's advertising methods have changed in the months since I switched to Epiphany.

How about a "click to show" feature in Epiphany's adblocker? (ala NoScript)

The adblocker just blocks the resource from loading, as if there was a network error. It's very simple because we don't have to know anything about e.g. where the resource would be displayed on the page.

ABP does some more work, like element hiding, in order to block more stuff than ours does.

I don't have much time to work on this but a starting point would be to provide us a gst debug log:

GST_DEBUG_NO_COLOR=1 GST_DEBUG=3,webkit*:5 epiphany ... 2> gst.log

And attach the log file to this bug, please.

(In reply to Philippe Normand from comment #13)
> I don't have much time to work on this but a starting point would be to
> provide us a gst debug log:
> 
> GST_DEBUG_NO_COLOR=1 GST_DEBUG=3,webkit*:5 epiphany ... 2> gst.log
> 
> And attach the log file to this bug, please.

There's nothing in the log (except unrelated warnings about deprecated GObject properties) for the video that doesn't play, so the GStreamer code is never hit. For the video that does play properly, the log gets filled up with GStreamer debug output. This is not unexpected to me, considering we're forcing a network failure.

Marking NEEDINFO; we won't get anywhere without feedback from a YouTube developer.

CCing some GNOME folks who work for Google in case they're possibly interested in prodding about this internally, since the YouTube developers ignored the bug report.

The only thing I can think to change in Epiphany is to disable the adblocker on YouTube. Users are not going to like that, but it's better than breaking videos.

*** Bug 756022 has been marked as a duplicate of this bug. ***

For what is worth, it seems that the issue is not strictly limited to YouTube: with the AdBlocker enabled (tested with 3.18.0-11-ge76bc6717ae3) I can't load Metacafe videos[1] and Yahoo Screen[2] is broken too as clicking items in the "Must Watch" list or in the categories list on the left does nothing.

Either we find the cause of the breakage or I guess disabling the adblocker would be a safer course of action. :(

[1] http://www.metacafe.com/watch/11373845/everything_wrong_with_transformers_age_of_extinction_part_1/

[2] https://screen.yahoo.com/

>>Emanuele Aina:
>Either we find the cause of the breakage or I guess disabling the adblocker
would be a safer course of action. :(

I share in your :(

Default adblocking is a /good/ thing, something epiphany has going for it (not only for minimizing annoyances, but simplification of interface and reduction of resources). My issues with ads are disruptive clutter and bandwidth hijacking; not to mention the occasional memory-leaking flash crasher. Epiphany does most of what Firefox with Adblock+ & NoScript did after lots of configuration*, without any configuration.

Actually, I am delighted to see this much work getting done on it. I wasn't entirely surprised that Youtube didn't acknowledge my report.  Thanks to everyone for taking an interest in this!

Now that we have some other sites suffering the same problem, maybe we can triage. What do those sites ads have in common with Youtube's ads?

*Sometimes I miss ElementHidingHelper and NoScript. Come to think of it, NoScript allowed filtered blacklisting and whitelisting of scripts by domain. I usually set this to allow scripts from the site itself (*.sitedomain.*) but not from third parties (*.ajaxdomain.*). I found myself frequently fiddling with this, temporarily allowing this script and that and reloading pages. I don't miss doing that, but it made me wonder if these sites share any outside scripting hosts?

While investigating this I stumbled over what the Safari AdBlock Plus extension does: they use a new framework landed recently in WebCore, contentextensions, see  https://www.webkit.org/blog/3476/content-blockers-first-look/

It also handles hiding elements using CSS selectors and blocking cookies only and it may be interesting to look at it as an alternative route.

Ok, this seems to be due to a bug in WebKit which has been fixed recently: https://bugs.webkit.org/show_bug.cgi?id=146706

(In reply to Emanuele Aina from comment #21)
> Ok, this seems to be due to a bug in WebKit which has been fixed recently:
> https://bugs.webkit.org/show_bug.cgi?id=146706

Haha, excellent sleuthing! Not often a major bug gets fixed before we figure out what was even wrong! I will put that on the list for a backport to 2.10.

Thank you everyone for taking the time to look into this and making the effort to get to the bottom of it.

The patch for webkit sounds promising:
http://trac.webkit.org/changeset/191077

but the last comment on the bug report does not:
https://bugs.webkit.org/show_bug.cgi?id=146706#c20

Would it be sufficent to install upstream webkit (largest source code ever?) and then rebuild epiphany from source to test?

Indeed, YouTube now seems to work fine even with the adblocker enabled, but both Metacafe[1] and Yahoo Screen[2] are still broken. :(

[1] http://www.metacafe.com/watch/11373845/everything_wrong_with_transformers_age_of_extinction_part_1/

[2] https://screen.yahoo.com/

Reopening.

Created attachment 315145 [details] [review]
EphyUriTester: Ensure "||" rules are anchored at the start

Filter rules beginning with "||" mean that the following expression must match (at least partially) the domain.

This means that "||example.com/foo" should match "foo.example.com/foo" but not "fooexample.com/foo" nor "example.com/bar?redir=example.com/foo" while the current code will happily match all of them.

This fixes the adblocker on Metacafe.

Yahoo Screen is broken due to us lacking any implementation of the "@@" exclusion rules.

Review of attachment 315145 [details] [review]:

Thanks! This is good for gnome-3-18 as well as master. Please just add a comment above the regex, since it's not very readable. Also, I'm not sure if you have a GNOME account or not, so let me know if you need me to push this for you.

Created attachment 315158 [details] [review]
EphyUriTester: Ensure "||" rules are anchored at the start

Added a comment to describe the purpose of the regex. If you think that it's
clear enough I can push directly to git. :)

Review of attachment 315158 [details] [review]:

Thanks!

Would this be related to crunchyroll videos not loading when the ad blocker is enabled?

http://www.crunchyroll.com/space-brothers/episode-1-01-little-brother-hibito-and-big-brother-mutta-594787

Comment on attachment 315158 [details] [review]
EphyUriTester: Ensure "||" rules are anchored at the start

Attachment 315158 [details] pushed as 4d873f4 - EphyUriTester: Ensure "||" rules are anchored at the start

Also pushed to the gnome-3-18 branch with a minor change due to the (Ephy)UriTester renaming.

Created attachment 315201 [details] [review]
EphyUriTester: Escape '[' and ']' when converting patterns to regexps

Created attachment 315202 [details] [review]
EphyUriTester: Handle '@@' whitelisting exception rules

The AdBlockPlus filter language has some special rules that start with
'@@' and are meant to override some overly matching blocking rules by
whitelisting those which should succeed.

Created attachment 315203 [details] [review]
EphyUriTester: Drop keys/signatures logic

It seems to be a performance optimization, but it seems to be quite
premature and resulted in some rules not being handled correctly: in
particular http://screen.yahoo.com was still broken as the
exception/whitelisted rule was for some reason skipped by the code
computing signatures.

With these patches Metacafe and Yahoo Screen work even with the adblocker enabled.

Unfortunately I haven't had much luck with Crunchyroll. :(

Review of attachment 315201 [details] [review]:

Thanks

Review of attachment 315202 [details] [review]:

Thanks, please fix these nits before pushing. Up to you whether to backport to gnome-3-18 or not.

::: embed/web-extension/ephy-uri-tester.c
@@ +316,3 @@
                             const char *req_uri,
+                            const char *page_uri,
+                            gboolean whitelist)

Just line up the parameter names please (the w in whitelist under the p in page_uri)

@@ +355,3 @@
   while (g_hash_table_iter_next (&iter, &patt, &regex))
     {
+      if (ephy_uri_tester_check_rule(tester, regex, patt, req_uri, page_uri, whitelist))

Missing space before opening parentheses here. (Yeah it was already missing, but since you're touching that line, it'd be good to fix it.)

@@ +366,3 @@
                                    const char *req_uri,
+                                   const char *page_uri,
+                                   gboolean whitelist)

Same alignment problem here

@@ +424,2 @@
     {
+      g_hash_table_insert (urlcache, g_strdup (req_uri), g_strdup("1"));

Same nit about opening parentheses here...

@@ +431,2 @@
     {
+      g_hash_table_insert (urlcache, g_strdup (req_uri), g_strdup("1"));

...and here...

@@ +434,3 @@
     }
 
+  g_hash_table_insert (urlcache, g_strdup (req_uri), g_strdup("0"));

...and here.

Review of attachment 315203 [details] [review]:

Yes please!

(In reply to Que Quotion from comment #23)
> Would it be sufficent to install upstream webkit (largest source code ever?)
> and then rebuild epiphany from source to test?

Yeah, but it's a bit of a project if you haven't done it before, I wouldn't recommend it... easiest to wait until your distro has an updated package. I expect (but don't promise) that the fix will be in WebKitGTK+ 2.10.4.

Created attachment 315255 [details] [review]
EphyUriTester: Escape '[' and ']' when converting patterns to regexps

Created attachment 315256 [details] [review]
EphyUriTester: Handle '@@' whitelisting exception rules

The AdBlockPlus filter language has some special rules that start with
'@@' and are meant to override some overly matching blocking rules by
whitelisting those which should succeed.

Created attachment 315257 [details] [review]
EphyUriTester: Drop keys/signatures logic

It seems to be a performance optimization, but it seems to be quite
premature and resulted in some rules not being handled correctly: in
particular http://screen.yahoo.com was still broken as the
exception/whitelisted rule was for some reason skipped by the code
computing signatures.

Attachment 315255 [details] pushed as 399c642 - EphyUriTester: Escape '[' and ']' when converting patterns to regexps
Attachment 315256 [details] pushed as 3da9b1f - EphyUriTester: Handle '@@' whitelisting exception rules
Attachment 315257 [details] pushed as 1600a5f - EphyUriTester: Drop keys/signatures logic

Pushed to the gnome-3.18 branch as I had to backport them anyway.

Created attachment 315258 [details] [review]
EphyUriTester: Avoid g_strdup() when a boolean is enough

It bothered me too much, so I did a simple conversion to boolean values
directly stored in pointers instead of g_strdup("1") and g_strdup("0").

(In reply to Michael Catanzaro from comment #39)
> Yeah, but it's a bit of a project if you haven't done it before

Unfortunately I have :) at one point I was attempting to package webkit-svn for multilib as part of a certain project... I was lucky enough to actually download the whole thing once, but svn rarely gets through that much data without some kind of breakage.

(In reply to Emanuele Aina from comment #42)
> Created attachment 315257 [details] [review] [review]
> EphyUriTester: Drop keys/signatures logic
> 
> It seems to be a performance optimization, but it seems to be quite
> premature and resulted in some rules not being handled correctly: in
> particular http://screen.yahoo.com was still broken as the
> exception/whitelisted rule was for some reason skipped by the code
> computing signatures.

I reverted this for bug #759501, until we have time to investigate this further. :/

Any updates?

-- GitLab Migration Automatic Message --

This bug has been migrated to GNOME's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/epiphany/issues/265.