Easy way to reproduce the unwanted behaviour:

GST_DEBUG=*libav*:5 gst-launch-1.0 gnlurisource uri=file:///anything/with/h264 inpoint=0 duration=5000000000 caps=video/x-raw ! xvimagesink qos=false 2>&1 | grep "Received new data of size"

I've got a simplistic patch which I'll attach after, which doesn't take the rate into account, but shows what I would like to see happen (past 5 seconds, the decoder returns EOS which ends up stopping the pipeline way sooner, in my case with HD samples, decoding speed is roughly equal to playback speed, and I end up waiting way too much time for my liking :)

I suppose there might be a very good reason for my patch not to be correct (apart from the rate considerations), but I have a hard time figuring it out.