When doing the following, tcpserversink buffers audio until the first client connects, forever maintaining the latency between starting the server and the first client connecting:
server:
gst-launch-1.0 -vmt pulsesrc device=alsa_output...stereo.monitor ! audio/x-raw,format=S16LE,rate=48000,channels=1 ! opusenc bitrate=8000 ! gdppay ! tcpserversink host=0.0.0.0
client:
gst-launch-1.0 tcpclientsrc host=... ! gdpdepay ! opusdec ! pulsesink

Steps to reproduce:
1. start server (note the use of the pulsesrc monitor device to serve a remote server's audio over the network)
2. wait 15 seconds, connect a client
3. immediately play some audio on the server
4. observe a 15 second delay before audio is played on the client

Things I have tried:
server: ... ! tcpserversink unit-format=time units-max=2000000
server: inserting a leaky=downstream queue before the tcpserversink with max-size-time=1000
server: ... ! tcpserversink unit-format=buffers units-max=1 buffers-max=1
server: leaky queue with max-size-buffers=1

Things that work:
A RTP/UDP stream from the server to the client works with <1 second latency (pulsesrc ! opusenc ! rtpopuspay ! udpsink host=$client).