Currently Gstreamer will read all of a file from disk even if certain streams are not needed and could be skipped.  This can be an issue when, for example, one wants to process only the audio stream from a file containing both audio and high-bitrate video, since performance will be bottlenecked by hard disk throughput.
If elements like qtdemux could determine which streams aren't being used, avoid reading data for unused streams, and instead only read the index and the streams which are being used; this would hugely increase performance in those use cases.

A good test uses footage from a Canon DSLR, which is commonly used for professional video.  One 12 minute file is 4GB in size, of which less than 150MB is audio.  
The following pipeline takes approximately as long as simply reading the entire file from disk. (about 40 seconds *uncached* on a Sandybridge laptop using the file below)
time gst-launch-1.0 filesrc location="$INPUT_FILE" ! qtdemux ! audioconvert ! filesink location="OUTPUT_FILE"
When I profiled it, most time was spent in functions for reading from disk (kernel -> system_call_fastpath -> sys_read), with CPU usage remaining low.

After extracting the audio only and placing it in another qt container, the time required was dramatically reduced.

Example DLSR file can be found here (about 4GB).
http://uds-o.novacut.com/GWQ5M6GPPNVEY3SAGUORKMLUAMQGD6O4.mov

We should figure out a way to signal this behavior properly for elements like qtdemux.