GNOME Bugzilla – Bug 638927
Memory leak in global state when using multi-threading
Last modified: 2021-07-05 13:21:16 UTC
The following code : #include <stdlib.h> #include <stdio.h> #include <libxml/parser.h> #include <libxml/tree.h> #include <omp.h> int main(void) { xmlDoc *doc; int tn; char fname[32]; omp_set_num_threads(2); xmlInitParser(); #pragma omp parallel private(doc,tn,fname) { tn = omp_get_thread_num(); sprintf(fname,"testdoc%d.xml",tn); doc = xmlReadFile(fname,NULL,0); printf("document %s parsed on thread %d (%p)\n",fname,tn,doc); xmlFreeDoc(doc); } xmlCleanupParser(); return EXIT_SUCCESS; } generate a memory leak, although every is done to insure thread safety (http://xmlsoft.org/threads.html) Valgrind reports : HEAP SUMMARY: in use at exit: 9,000 bytes in 8 blocks total heap usage: 956 allocs, 948 frees, 184,464 bytes allocated 968 bytes in 1 blocks are definitely lost in loss record 6 of 8 at 0x1000107AF: malloc (vg_replace_malloc.c:236) by 0x1000B2590: xmlGetGlobalState (in /opt/local/lib/libxml2.2.dylib) by 0x1000B1A18: __xmlDefaultSAXHandler (in /opt/local/lib/libxml2.2.dylib) by 0x100106D18: xmlDefaultSAXHandlerInit (in /opt/local/lib/libxml2.2.dylib) by 0x100041BE7: xmlInitParserCtxt (in /opt/local/lib/libxml2.2.dylib) by 0x100042145: xmlNewParserCtxt (in /opt/local/lib/libxml2.2.dylib) by 0x10004615E: xmlCreateURLParserCtxt (in /opt/local/lib/libxml2.2.dylib) by 0x10005B56B: xmlReadFile (in /opt/local/lib/libxml2.2.dylib) by 0x100000E03: main.omp_fn.0 (in ./xtest) by 0x100028FA3: gomp_thread_start (in /opt/local/lib/gcc44/libgomp.1.dylib) by 0x1001E8535: _pthread_start (in /usr/lib/libSystem.B.dylib) by 0x1001E83E8: thread_start (in /usr/lib/libSystem.B.dylib) LEAK SUMMARY: definitely lost: 968 bytes in 1 blocks indirectly lost: 0 bytes in 0 blocks possibly lost: 0 bytes in 0 blocks still reachable: 8,032 bytes in 7 blocks suppressed: 0 bytes in 0 blocks Reachable blocks (those to which a pointer was found) are not shown. To see them, rerun with: --leak-check=full --show-reachable=yes
I got the same problem on ubuntu linux 10.10 using threads. Libxml2 version 2.7.6 Valgrind reports resumed: ==7279== 2,580 bytes in 5 blocks are definitely lost in loss record 40 of 41 ==7279== at 0x4024F20: malloc (vg_replace_malloc.c:236) ==7279== by 0x4726DF4: xmlGetGlobalState (in /usr/lib/libxml2.so.2.7.6) ==7279== by 0x4725D9C: __xmlDefaultSAXHandler (in /usr/lib/libxml2.so.2.7.6) ==7279== by 0x46B1CB9: xmlFreeParserCtxt (in /usr/lib/libxml2.so.2.7.6) ==7279== by 0x41C0392: parserXML::initParser(char const*, int, char const*, bool, int*, char const**, char const*, bool) (xmlparser.cpp:1248) ==7279== by 0x41BDF1A: parserXML::parserXML(char const*, int, char const*, bool, int*, char const**, char const*) (xmlparser.cpp:286) ==7279== by 0x41B9B78: centra_asset_type_unpack (centra_asset_types.cpp:245) ==7279== by 0x41B76B0: centra_asset_type_load (centra_conf.cpp:323) ==7279== by 0x804E707: log_ctx_init (log_ctx.c:235) ==7279== by 0x804AE78: proc_thread_reload_conf (proc_threads.c:268) ==7279== by 0x804AF1B: proc_thread_main (proc_threads.c:300) ==7279== by 0x419796D: start_thread (pthread_create.c:300) ==7279== ==7279== 2,580 bytes in 5 blocks are definitely lost in loss record 41 of 41 ==7279== at 0x4024F20: malloc (vg_replace_malloc.c:236) ==7279== by 0x4726DF4: xmlGetGlobalState (in /usr/lib/libxml2.so.2.7.6) ==7279== by 0x4725E9C: __xmlLastError (in /usr/lib/libxml2.so.2.7.6) ==7279== by 0x46AF406: xmlResetLastError (in /usr/lib/libxml2.so.2.7.6) ==7279== by 0x46B661E: xmlCleanupParser (in /usr/lib/libxml2.so.2.7.6) ==7279== by 0x41BE0CA: parserXML::~parserXML() (xmlparser.cpp:303) ==7279== by 0x41B9CF0: centra_asset_type_unpack (centra_asset_types.cpp:254) ==7279== by 0x41B76B0: centra_asset_type_load (centra_conf.cpp:323) ==7279== by 0x804E707: log_ctx_init (log_ctx.c:235) ==7279== by 0x804AE78: proc_thread_reload_conf (proc_threads.c:268) ==7279== by 0x804AF1B: proc_thread_main (proc_threads.c:300) ==7279== by 0x419796D: start_thread (pthread_create.c:300) ==7279== LEAK SUMMARY: ==7279== definitely lost: 91,460 bytes in 178 blocks ==7279== indirectly lost: 0 bytes in 0 blocks ==7279== possibly lost: 1,548 bytes in 3 blocks ==7279== still reachable: 28 bytes in 1 blocks ==7279== suppressed: 0 bytes in 0 blocks
Created attachment 309524 [details] Multiple init/getglobalstate/cleanup test case I have found this problem in my environment after upgrading existing software that repeatedly calls xmlInitParser() and xmlCleanupParser(). I isolated the problem to pthread_key handling around xmlGetGlobalState(). I found that the first call to xmlGetGlobalState will invoke xmlOnceInit and set a valid globalkey value. The globalkey value is used in the pthread_getspecific and pthread_setspecific calls and works correctly as a thread-local instance. However xmlCleanupParser calls xmlCleanupThreads which invokes pthread_key_delete on globalkey. Now that globalkey is invalid a subsequent xmlInitParser call and xmlGetGlobalState call will not re-initialise it due to the pthread_once protection around xmlOnceInit. pthread_getspecific(globalkey) now always returns null, and moreover pthread_setspecific(globalkey, tsd) always fails on the invalid key (an unchecked EINVAL is returned in my case). Thus, all subsequent calls to xmlGetGlobalState will now create and leak a new instance of xmlGlobalState. Attached a test case showing rapid growth.
I went searching to find out why the program I have been testing worked with an earlier libxml2 revision, 2.6.27. I think the problem was introduced in [1], as previously the globalkey value would remain valid even after the cleanup occurred. I think the fix should be one of the following: 1. Document clearly that it is not legal to call xmlInitParser after xmlCleanupParser 2. Revert git d4a3f2418a19b5de3d533490b992450b6f4623c2 / svn 3810 such that globalkey remains valid over multiple init/cleanup cycles 3. Figure out a way of calling the xmlOnceInit code again after a cleanup that doesn't break threading. This might be the hardest. 4. Make "I'm really done forever and will never call xmlInitParser again but just want valgrind to free absolutely everything" some other call maybe? [1] https://git.gnome.org/browse/libxml2/commit/threads.c?id=d4a3f2418a19b5de3d533490b992450b6f4623c2
GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org. As part of that, we are mass-closing older open tickets in bugzilla.gnome.org which have not seen updates for a longer time (resources are unfortunately quite limited so not every ticket can get handled). If you can still reproduce the situation described in this ticket in a recent and supported software version, then please follow https://wiki.gnome.org/GettingInTouch/BugReportingGuidelines and create a new ticket at https://gitlab.gnome.org/GNOME/libxml2/-/issues/ Thank you for your understanding and your help.