The current version of yelp available for testing on MacPorts is 3.8.1, and it includes the following code that prevents compilation in an environment where gtk3 has been configured to use the native quartz backend, rather than X11.  To my knowledge, it compiles and runs fine under the default X11 gtk3 configuration though.  I was able to make a patch that allows yelp to build, install and run on gtk3-quartz, and I hope that you will review it and consider integrating it into the yelp source if you are able to do so.

The first build failure is in libyelp/yelp-view.c, which specifies #include <gdk/gdkx.h>, yet simply removing the header for gtk3-quartz is sufficient to compile without further error.

The second failure is in src/yelp-application.c, which specifies the following code, and has done so since the 2.x series:

gtk_widget_show_all (GTK_WIDGET (window));

    /* Metacity no longer does anything useful with gtk_window_present */
    gdk_window = gtk_widget_get_window (GTK_WIDGET (window));
    if (gdk_window)
        gdk_x11_window_move_to_current_desktop (gdk_window);

    /* Ensure we actually present the window when invoked from the command
     * line. This is somewhat evil, but the minor evil of Yelp stealing
     * focus (after you requested it) is outweighed for me by the major
     * evil of no help window appearing when you click Help.
     */
    if (data->timestamp == 0)
        data->timestamp = gdk_x11_get_server_time (gtk_widget_get_window (GTK_WIDGET (window)));

    gtk_window_present_with_time (GTK_WINDOW (window), data->timestamp);

However, since the gtk3-quartz environment does not use X11 it fails, and does not need metacity.  The following line gets the app to compile and run on OS X:

gtk_widget_show_all (GTK_WIDGET (window));
gtk_window_present (GTK_WIDGET (window));

I have included both changes in .diff format.  I hope that you will be able to incorporate these changes quickly in a manner not disruptive to the needs of people using X11.