I don't think it's wrong. Basically, NULL as caps on a pad template or
as a return value of _getcaps() means 'anything', but NULL as an
intersection means 'nothing' - the combination of these still means
'anything'.

This is somewhat insonsistent, but it makes sense.

No, it still doesn't make sense. :)

It is easy to generate a case where
gst_caps_intersect(a,gst_caps_intersect(b,c)) and
gst_caps_intersect(gst_caps_intersect(a,b),c) return different values.

Looking at it from a set theory perspective, we use NULL to indicate
both the empty set (curiously also called the null set), and the union
of all sets.  This plays much havoc with our set operations.  We also
use NULL to indicate the end of chained caps.

I'd like to change my recommendation to define a GST_CAPS_ANY to
indicate the union of all possible caps, which might not be NULL.  In
any case, we should probably start using GST_CAPS_ANY and
GST_CAPS_NONE instead of NULL.

Just want to voice support for Davids proposal.

So how will we define GST_CAPS_ANY? I could propose:

#define GST_CAPS_NONE NULL
#define GST_CAPS_ANY gst_caps_get_any()

GstCaps *
gst_caps_get_any (void)
{
  static GstCaps *caps = NULL;

  if (!caps) {
    caps = GST_CAPS_NEW ("gst_caps_any",
                         "*",
                           NULL);
  }

  return gst_caps_ref (caps);
}

That'd sound ok to me.

My first thought was to use ((void *)1) and ((void *)2) for ANY and
NONE, but that causes problems testing for NULL.  I don't know why
anyone would need to test for NULL, though.

GST_CAPS_NONE probably should be NULL.  It can be used as the end of
chain indicator as well.

I like the idea of making GST_CAPS_ANY a fixed number, since that
makes it easy to test, instead of needing a GST_CAPS_IS_ANY(). 
Alternatively, we could special-case gst_caps_copy() so that it won't
actually copy the ANY caps.

I'm starting with the migration.  The first step is to define
GST_CAPS_ANY and GST_CAPS_NONE (done) and identify as many places in
gstreamer and gst-plugins where NULL needs to be converted to
GST_CAPS_ANY.  (working on it)  (GST_CAPS_ANY is still defined as
NULL.)

After that is done, I'll add code to properly intersect caps with
GST_CAPS_ANY, and add an assertion to make sure that pad caps are not
set to GST_CAPS_NONE.

I *think* that this can all be done so that one can flip a #define and
get either the old behavior or new behavior.  At least, I'm going to
attempt to do it that way, to make debugging easier.

One thing that hasn't been discussed: props may need similar
treatment.  However, it should only need to be internal.  A caps
with GST_PROPS_NONE would imply GST_CAPS_NONE, so it would only need
to be used internally in the props intersection code.

Fixed in CAPS merge