The jitter buffer does not play nicely with encoders supporting dtx.
e.g. speexenc

1) When dtx period starts lost event(s) are generated on the decoding side which
increase the expected sequence number. If now a valid packet with the old seqnum
is generated by the encoding side it is considered late and gets dropped by
the receiver. This may be acceptable depending on the use-case.

If the encoder regularly sends comfort noise frames on a lower packet rate
the jitter buffer will adapt to the new period, this may clean up the above
situation depending on the latency in jitterbuffer, latency in the network,
the time silence is present and the implementation of the de/encoder,
but leads to 2)

2) Some codecs expect to get triggered with the original packet rate to
generate comfort noise and/or do packet loss concealment. This could be solved
inside the decoder by detaching input and output but it would move a lot of
logic from the jitter buffer to the decoder.

So the dtx feature somehow conflicts with packet loss and re-transmission.

Some helpful information can be found here
https://tools.ietf.org/html/rfc3551#section-4.1
A very old discussion touching the topic a little was here
http://gstreamer-devel.966125.n4.nabble.com/comfort-noise-generation-bin-td2064259.html

As far as I understand it, there are three possible situations

a) sender sends nothing during dtx
-> not possible to distinguish between packet loss, dtx or disconnect/timeout

b) sender sends a single "event", e.g. silence insertion descriptor on dtx
-> decoder can detect the event, period of silence is marked clearly
   disconnect/timeout can't be detected

c) sender sends comfort noise frames on a lower packet rate
-> packet loss and dtx can be distinguished *after* receiving the first CN frame
   disconnect/timeout can be detected

An example pipeline showing the current behavior
(not including re-transmission):

GST_DEBUG=3,speexenc:5,multiudpsink:6 \
gst-launch-1.0 rtpbin name=rtpbin do-lost=TRUE \
pulsesrc do-timestamp=TRUE ! "audio/x-raw,channels=1,rate=8000" ! speexenc vbr=TRUE dtx=TRUE ! rtpspeexpay pt=96 ! rtpbin.send_rtp_sink_0 \
rtpbin.send_rtp_src_0 ! udpsink host=192.168.117.101 port=5000

GST_DEBUG=3,speexdec:5,udpsrc:6,rtpjitterbuffer:5 \
gst-launch-1.0 rtpbin name=rtpbin do-lost=TRUE \
udpsrc address=192.168.117.101 port=5000 caps="application/x-rtp, media=audio, clock-rate=8000, encoding-name=SPEEX, payload=96, encoding-params=1, channels=1" ! rtpbin.recv_rtp_sink_0 \
rtpbin. ! rtpspeexdepay ! speexdec plc=TRUE ! pulsesink

I use those on two hosts in a 100Mbit ethernet network

1) lost will show up by generating multiple sequences of noise and silence
2) pulsesink will generate underflows on dtx