It looks like perhaps the bitrate parameter should be bits per second instead of kilobits per second though it's not clear.

gst-launch-1.0 -ev videotestsrc num-buffers=1500 ! \
    video/x-raw,width=640,height=360,framerate=15/1 ! \
    timeoverlay ! vtenc_h264 bitrate=768 ! h264parse ! \
    mp4mux ! filesink location=test.mp4

The above would give 100s of video. It encodes in about 14s which is nice. The resulting file size is: 1380225 bytes which works out as 110kbps. Nowhere near 768000.

If I run it with bitrate=768000, the resulting file size is: 22373189 bytes which works out as 1.8Mbps, which is also way off.

I was curious whether the noise in the default smpte pattern was screwing the rate control when using bitrate=768000 so I tried also one of the easier patterns, 'ball'. That gave a file 837721 bytes large which works out as 67kbps.

For something possibly more realistic, I captured two minutes of what should have been 720p30 from the FaceTime HD camera in this current model 13" Retina MacBook Pro with bitrate=3000000 and I got 9Mbps. With bitrate=3000 I got 0.59Mbps with really awful quality.

Now, that camera is quite noisy and the framerate is very sensitive to dropping if you're not in really bright lighting conditions. But even so, it seems like something is wacky with the bitrate of this H264 encoder.