In HtmlHelp::initialize, if CHM_INDEX_ENCODING is undefined, it defaults to Windows-1250. The Studio 12 iconv_open builtin doesn't know this encoding name and returns -1. A core dump follows shortly after from HtmlHelp::recode during index generation. CP1250 works ok, and is listed as a synonym, but I have no idea if that's an acceptable equivalent --- I just don't have much knowledge of what's being generated here. The generated HTML looks ok in a casual inspection with Firefox.

I'm forcing the solaris-cc build environment. CC version info is Sun C++ 5.9 SunOS_sparc Patch 124863-01 2007/07/25

const char *str = Config_getString("CHM_INDEX_ENCODING");
  /*
    In the Solaris CC environment, the builtin iconv doesn't grok Windows-1250.
    It does understand the (supposedly equivalent) CP1250.

    if(!str) str = "Windows-1250";
  */
  if(!str) str = "CP1250";
  m_fromUtf8 = portable_iconv_open(str,"UTF-8");