Second command should read:
chmod ugo-w ~/.gstreamer-0.10/

Sorry for the typo.

What makes your setup different from the the many users that don't have this problem?

Do you mount your home directory over NFS?

Yes, GStreamer will still work fine if it can't write the registry file, but it means that all the plugin shared libraries have to be scanned on startup, because the cached information is missing. This will make startup of your applications slower (a *lot* slower if there are many plugins), and waste memory loading information that isn't needed.

Nope, just standard ubuntu installations on regular local ext3 partitions. Mind you this is not just me: I had to "fix" this on my two computers regularly, and on two friends' computers which I had nothing to do with in daily use (anyway, they are using different ubuntu releases, so not an ubuntu problem I would expect).

The "lot more people" argument is difficult to evaluate: maybe I'm the only human out there that noticed that registry file and the fact that clearing it solves problems.
Other users would (or already did) thik "meh, totem/gst sucks" and install VLC, never to touch totem/gst again.

Lot slower you say? I have all the plugins (good/bad/ugly, with multiverse variants, with w32codecs and gst pitfdll) I can get my paws on, here is my stopwatch benchmarks launching totem:

- without write access to the gst folder (no registry): 2 seconds 30
- with write access and no existing registry: 1 second 90
- with write access and existing registry: 1 second 70
(cold boots between each trials)

Warm startup time of 1 second 55.

My machine: 1.8 ghz pentium M, 768 mb of DDR ram, 5400 rpm IDE hard drive. Ubuntu 7.10 (which is known for being slower than previous releases).

My conclusions from those rough benchmarks:
- the launch times are very low no matter what you do
- I don't even know if those numbers are "statistically significative" - don't forget they have limited precision as I need to physically start and stop my wristwatch. I did run each trial a few times (lots of reboots), and you can easily get an error margin of half a second or worse!
- do we really care about a variance of rougly 40% of a second on the cold launch of an app like totem? Wouldn't benefits of not having that registry offset gaining 400 ms of cold startup time?

Please attach a copy of the broken registry.*.xml file.

I was dumb enough to delete it days before I reported this bug, so until it happens again I cannot provide it.

I'm curious though, how is such a registry file supposed to handle cases where the user installs or removes codecs? If it can still detect that (I don't know if it does), how is it different than not having a registry at all?

Ok, this tickled me, so I went googlin' and I found these "proofs" that I am certainly not the only one who has noticed this problem and found the culprit being the registry:

http://ubuntuforums.org/archive/index.php/t-348987.html
"Might be a bit of a long shot, but it might help if you delete your gstreamer registry (it regenerates itself automatically once you use gstreamer again)"

https://lists.ubuntu.com/archives/ubuntu-users/2006-September/094501.html (I think the registry is mentionned somewhere down that thread)

http://ubuntuforums.org/showpost.php?p=1139960&postcount=2 (notice how he clearly put rm -f ~/.gstreamer-0.10/registry.i486.xml in the "installation" process!)

http://ubuntuforums.org/showpost.php?p=3662184&postcount=7
"another thing to look for is corruption of the $HOME/.gstreamer-0.10/registry.i486.xml database. It has happened to me more times I care to remember that a problem with gstreamer was solved by deleting that file"

So I would tend to reopen that bug report as, I think, the solution is to not have a registry at all (unless you can show me benchmarks that it really makes a huge performance difference, not just 0.4 seconds)

Rest assured that we wouldn't bother with the registry if there wasn't a reason for having it. It's good of you to enquire about or investigate the purpose of it all though before declaring it useless.

Yes, earlier versions of GStreamer-0.10 indeed had a bunch of issues related to registry reading and writing (mind you, at least one of those reports refers to ubuntu dapper, which is ancient).  Those should be fixed these days though, and neither our build bots nor any users have reported any mysterious-possibly-registry-related-failures for a long time.

However, I did have a look at the code and spotted a problem that may cause partial/broken registry files: we don't check the return value of close() when writing the file, yet sometimes write problems such as out-of-diskspace are only reported when the file is closed rather than during the write.  I've fixed this in CVS:

 2007-12-16  Tim-Philipp Müller  <tim at centricular dot net>

        * gst/gstregistrybinary.c: (gst_registry_binary_write_cache):
          Use g_remove() and g_rename(). Check result of g_rename(), and
          don't leak the open file descriptor if we error out when writing.

        * gst/gstregistryxml.c: (load_plugin), (gst_registry_xml_write_cache):
          Must check the return value of close() after writing out the new
          registry file.  Sometimes write problems such as out-of-diskspace
          are only reported when the file is closed and not already during
          the write.  This may have caused partial/broken registry files in
          some rare circumstances. Should fix #503675.

I hope this will fix it for you. If not, we probably need more information to go on.

If you ever do run into this problem again, please do keep the registry file so we can have a look a it. Thanks!

Lastly, there is one known problem with the registry and that is plugins wrapping other plugins (such as pitfdll [don't use that btw, it's highly unstable] or our libvisual plugin).  If wrapped plugins are added or removed (e.g win32codecs changes), GStreamer doesn't know about this and doesn't update the registry.  This should in the worst case result in error messages at run-time though, but not in 'codecs disappearing' without a reason.  Bug #350477 is about this, and it should be harmless if things are packaged correctly.  And of course the registry is updated in the normal case of codecs being updated, added or removed (e.g. files in /usr/lib/gstreamer-0.10 added/removed/changed).

ok, thank you for your thorough explanation. I hope the problem you stumbled upon was the cause of all this (if not, I will try to remember keeping the registry next time). I don't remember having my hard disks full in a long time though, is gstreamer 0.10.14 considered so "old"?

This happened again today with latest stuff from the gstreamer developers PPA for ubuntu.

Created attachment 139746 [details]
samples of the registry

The corrupt version prevented me from playing h264 and flv files.

The file looks ok at first glance. All the ffmpeg data is in the middle and not at the end, and I seem to be able to read the file just fine current git (ie. the latest core prerelease roughly):

$ GST_REGISTRY=/home/tpm/tmp/broken-gst-registry/registry\ \(corrupt\).i486.bin gst-inspect-0.10 --gst-disable-registry-update | grep ffdec_h264
ffmpeg:  ffdec_h264: FFMPEG H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 decoder

No warnings or errors while reading in the file, and valgrind shows no problems either. So I'm tempted to say the registry file is probably alright.

Indeed.  These files appear to be identical except for ordering.

If there is a bug here, it is not related to the registry.

I had yet another occurence of this problem today on my netbook. I installed all the possible codecs (good, bad, ugly, gst-ffmpeg and ffmpeg itself) and still couldn't play H.264 files. I then saved a copy of my gstreamer registry (I put it under version control!) and cleared it out. The registry was recreated and everything worked fine. Will attach my samples of "broken" registry and working registry. It irks me that the registry has to be cleared periodically for the darn thing to work.

Created attachment 149200 [details]
samples of the registry 2

Here's a question. Let's say you put those registry files in your .gstreamer-0.10 folder, and they prevent you from playing back H.264 and the like. Shouldn't those files be automatically overwritten by a clean and uncorrupt registry by gstreamer? How can they just sit there on my disk and require deletion, rather than update themselves?

Also, maybe it doesn't update when you install gst-ffmpeg and ffmpeg?

You have the registry cache under version control? Thats not a good idea. This is a cache and it is autogenerated/autoupdated as needed. The update is triggered by changes in the plugin files (new or removed ones) as well as timestamp changes.

Yes, it's under version control (bazaar to be exact), but it's just an added tool. It doesn't block the registry from auto updating or anything. The only thing I do with it is "bzr commit" when I noticed that the registry changed to something significant (ex: nonworking state) and then commit again when I reset the registry file.

It acts more like a snapshot/time machine system.

Created attachment 185909 [details]
samples of the registry 3

Happened again today. I have a fedora 15 machine with gstreamer-* installed (even nonfree), and yet "gst-inspect|grep videomixer" returns nothing; pitivi didn't start because of this. Again, this is caused by a broken registry file, as the investigation steps below demonstrate:


# First, let's try with the broken (original) registry file

jeff@boris:~/.gstreamer-0.10$ gst-inspect-0.10 /usr/lib/gstreamer-0.10/libgstvideomixer.so 
Plugin Details:
  Name:			videomixer
  Description:		Video mixer
  Filename:		/usr/lib/gstreamer-0.10/libgstvideomixer.so
  Version:		0.10.27
  License:		LGPL
  Source module:	gst-plugins-good
  Source release date:	2011-01-21
  Binary package:	Fedora gstreamer-plugins-good package
  Origin URL:		http://download.fedora.redhat.com/fedora
 
  videomixer: Video mixer
  videomixer2: Video mixer 2
 
  2 features:
  +-- 2 elements

# Yet...
jeff@boris:~/.gstreamer-0.10$ gst-inspect-0.10|grep videomixer
 
# Nothing! So let's move the broken registry file:
jeff@boris:~/.gstreamer-0.10$ mv registry.i386.bin registry.broken
 

# Regenerate the registry file
jeff@boris:~/.gstreamer-0.10$ gst-inspect-0.10|grep videomixer
 
(gst-plugin-scanner:21511): GStreamer-WARNING **: Failed to load plugin '/usr/lib/gstreamer-0.10/libgstffmpeg.so': /usr/lib/gstreamer-0.10/libgstffmpeg.so: symbol av_parser_parse, version LIBAVCODEC_52 not defined in file libavcodec.so.52 with link time reference
Traceback (most recent call last):

+ Trace 226719

videomixer:  videomixer2: Video mixer 2
videomixer:  videomixer: Video mixer
 
 
 
# Test again: it works!
jeff@boris:~/.gstreamer-0.10$ gst-inspect-0.10|grep videomixer
videomixer:  videomixer: Video mixer
videomixer:  videomixer2: Video mixer 2
 
 
 
jeff@boris:~/.gstreamer-0.10$ gst-inspect-0.10 /usr/lib/gstreamer-0.10/libgstvideomixer.so 
Plugin Details:
  Name:			videomixer
  Description:		Video mixer
  Filename:		/usr/lib/gstreamer-0.10/libgstvideomixer.so
  Version:		0.10.27
  License:		LGPL
  Source module:	gst-plugins-good
  Source release date:	2011-01-21
  Binary package:	Fedora gstreamer-plugins-good package
  Origin URL:		http://download.fedora.redhat.com/fedora
 
  videomixer: Video mixer
  videomixer2: Video mixer 2
 
  2 features:
  +-- 2 elements

The registry file isn't broken, for some reasons those plugins got blacklisted, see e.g.:

tpm@zingle:~/tmp$ hexdump -C registry-broken.i386.bin  | grep -C 10 videomixer|more
00018280  65 72 2d 30 2e 31 30 2f  6c 69 62 67 73 74 63 75  |er-0.10/libgstcu|
00018290  74 74 65 72 2e 73 6f 00  30 2e 30 2e 30 00 42 4c  |tter.so.0.0.0.BL|
000182a0  41 43 4b 4c 49 53 54 00  42 4c 41 43 4b 4c 49 53  |ACKLIST.BLACKLIS|
000182b0  54 00 42 4c 41 43 4b 4c  49 53 54 00 42 4c 41 43  |T.BLACKLIST.BLAC|
000182c0  4b 4c 49 53 54 00 00 00  64 1c 02 00 61 50 52 4d  |KLIST...d...aPRM|
000182d0  00 00 00 00 00 00 00 00  6c 69 62 67 73 74 76 69  |........libgstvi|
000182e0  64 65 6f 6d 69 78 65 72  2e 73 6f 00 50 6c 75 67  |deomixer.so.Plug|
000182f0  69 6e 20 66 6f 72 20 62  6c 61 63 6b 6c 69 73 74  |in for blacklist|
00018300  65 64 20 66 69 6c 65 00  2f 75 73 72 2f 6c 69 62  |ed file./usr/lib|
00018310  2f 67 73 74 72 65 61 6d  65 72 2d 30 2e 31 30 2f  |/gstreamer-0.10/|
00018320  6c 69 62 67 73 74 76 69  64 65 6f 6d 69 78 65 72  |libgstvideomixer|
00018330  2e 73 6f 00 30 2e 30 2e  30 00 42 4c 41 43 4b 4c  |.so.0.0.0.BLACKL|
00018340  49 53 54 00 42 4c 41 43  4b 4c 49 53 54 00 42 4c  |IST.BLACKLIST.BL|
00018350  41 43 4b 4c 49 53 54 00  42 4c 41 43 4b 4c 49 53  |ACKLIST.BLACKLIS|
00018360  54 00 00 00 e0 7f 00 00  d2 4d 3f 4d 00 00 00 00  |T........M?M....|
00018370  00 00 00 00 6c 69 62 67  73 74 61 64 64 65 72 2e  |....libgstadder.|
00018380  73 6f 00 50 6c 75 67 69  6e 20 66 6f 72 20 62 6c  |so.Plugin for bl|
00018390  61 63 6b 6c 69 73 74 65  64 20 66 69 6c 65 00 2f  |acklisted file./|
000183a0  75 73 72 2f 6c 69 62 2f  67 73 74 72 65 61 6d 65  |usr/lib/gstreame|
000183b0  72 2d 30 2e 31 30 2f 6c  69 62 67 73 74 61 64 64  |r-0.10/libgstadd|
000183c0  65 72 2e 73 6f 00 30 2e  30 2e 30 00 42 4c 41 43  |er.so.0.0.0.BLAC|
tpm@zingle:~/tmp$ md5sum registry-broken.i386.bin 
83afd4cf3701029e29044efb5fadf76a  registry-broken.i386.bin

I can't say why they get blacklisted of course, but it's usually because of some packaging / library version issue (missing library / incompatible library version / library not compiled with the rigth switches / mixing packages from multiple repositories).

Created attachment 185918 [details]
debug logs

In this file is the output of:

gst-inspect -b
ls -1 /usr/lib/gstreamer-0.10/
GST_DEBUG=*:5 gst-inspect 2>dbg.log


In this particular case, the videomixer plugin was blacklisted, preventing pitivi from starting. I think that blacklisted plugins should be re-checked when they are accessed.

I think that there were interesting ideas that came out of a discussion today, so I'm posting the cleaned up IRC log. As I don't see a particular question right now, changing from NEEDINFO to unconfirmed; feel free to request more information.

---------
<__tim> nekohayo, your registry file is full of BLACKLISTED. The problem isn't registry file writing or parsing, but that plugins get blacklisted for some reason
<nekohayo> When does blacklisting occur?
<__tim> nekohayo, when an error occurs when loading the plugin, usually a missing symbol or library or somesuch
<nekohayo> videomixer was indeed on the gst-inspect blacklist

<__tim> are you sometimes downgrading versions of core or gst-plugins-base?
<nekohayo> no, not that I know of. Downgrading was downright impossible in ubuntu, and I haven't used downgrade in fedora yet
<__tim> GStreamer installed in different prefixes?
<nekohayo> currently this machine is using the official gstreamer packages from fedora + rpmfusion

<nekohayo> I'm wondering why those blacklisted plugins are never "re-scanned" by gstreamer
<__tim> that's a fair question I guess. But when would one rescan them? You don't want to do it every time. So once an hour? once a day? once a week?

<nekohayo> what if they were rescanned when apps try to access them? like when pitivi tries to use videomixer (on top of your approach of rescanning periodically maybe)
<__tim> interesting, maybe one could do something clever there
<__tim> ideally we'd store the reason for the blacklisting as well
<ocrete> arent they rechecked if the file changes already ?
<thaytan> the plugins are rescanned if they change
<__tim> but if they don't change ...
<thaytan> but if they failed because some underlying lib was broken and then fixed, that's not detected
<__tim> right
<ocrete> hmm interesting point
<thaytan> storing the reason for the blacklisting is probably hard
<ocrete> also check if ld.so.cache has changed ?
<thaytan> since it's usually just 'failed to g_module_open() because of an unresolved symbol'
<ocrete> and ignore blacklists if ld.so.cache has changed (and rescan)
<__tim> thaytan, it doesn't mention the symbol? I think it does..
<ds> if the so fails to load, there isn't much overhead trying to load every time
<ocrete> that said, doesnt help if one use using LD_LIBRARY_PATH
<thaytan> ds: except for forking the scanner process to re-check an otherwise clean registry
<ds> oooh, maybe recheck blacklisted plugins whenever the scanner is run, but not run the scanner just for blacklisted plugins. Although, also mark potential crasher blacklist plugins

I have created a new bug (bug  #648010) about better handling of blacklisted plugins, closing this one.