Use of g_signal_handlers_disconnect_by_func() needs to do more work than
necessary to find all the matching handlers. Instead, just hold on to the
signal identifier and remove it directly so we hit the fast path.

Not terribly ground breaking in terms of performance gains, but its done
enough to be worthwhile.