GNOME Bugzilla – Bug 677091
Embedded Python C program segfaults in subsequent call after reinitialization
Last modified: 2018-01-10 20:16:42 UTC
Created attachment 215239 [details] minimal example for problem demonstration The minimal example illustrates the problem. The output: $ ./test_embed_python running embedded python... ------------ script end ---------- ...done running embedded python... Segmentation fault The script test.py: #from gi.repository import GLib #from gi.repository import Gtk #from gi.repository import Cairo #from gi.repository import Pango from gi.repository import Gio def call_python(): print(" ------------ script end ----------") return 0 if __name__ == '__main__': call_python() --------------------------------- Once at least one package from the gi.repository is imported, the embedded python fails to run it after the reinitialization (Py_Initialize/Finalize).
Created attachment 215240 [details] test script
FYI: I'm using Debian Wheezy with python2.7 (2.7.3~rc2-2.1), pygobject (3.2.2-1). Here is the backtrace: running embedded python... ------------ script end ---------- [34268 refs] ...done running embedded python... Program received signal SIGSEGV, Segmentation fault. 0x00007ffff79dcc27 in PyObject_GC_UnTrack (op=0xb3b880) at ../Modules/gcmodule.c:1428 1428 ../Modules/gcmodule.c: No such file or directory. (gdb) bt
+ Trace 230333
I've noticed, it is enough just to import the script, not to call it at all. The updated minimal example and backtrace are attached.
Created attachment 216053 [details] An update example
Backtrace: running embedded python... [36161 refs] ...done running embedded python... Program received signal SIGSEGV, Segmentation fault. 0x00007ffff79dcc27 in PyObject_GC_UnTrack (op=0xb60da8) at ../Modules/gcmodule.c:1428 1428 ../Modules/gcmodule.c: No such file or directory. (gdb) bt
+ Trace 230334
Verified. It looks like there is lack of conformance in parts of PyGObject in regards to GC tracking. http://docs.python.org/3/c-api/gcsupport.html
Created attachment 241958 [details] [review] Remove half implemented GC in PyGIBaseInfo, PyGIStruct, and PyGIBoxed Remove half implemented GC tracking from PyGIBaseInfo as it was not needed (the implemented was also missing usage of PyObject_GC_New/Track). Ensure weakref list for PyGIBaseInfo is initialized to NULL and cleared properly. Remove invalid calls to PyObject_GC_UnTrack and PyObject_ClearWeakRefs for both PyGIStruct and PyGIBoxed as these types were not being advertised as GC aware with Py_TPFLAGS_HAVE_GC.
The previous patch gets through the segfault but then runs into another problem: Traceback (most recent call last):
+ Trace 231818
from gi.repository import GLib
dynamic_module._load()
overrides_modules = __import__('gi.overrides', fromlist=[self._namespace])
class Variant(GLib.Variant):
super(StructMeta, cls).__init__(name, bases, dict_)
'NoneType' object is not callable
Comment on attachment 241958 [details] [review] Remove half implemented GC in PyGIBaseInfo, PyGIStruct, and PyGIBoxed TBH I'm not very familiar with how the GC works. Do you think this half-implementation would actually need to be completed to avoid memleaks here? Can you compare valgrind output before/after (note you need a python compiled with valgrind support to sensibly do that). If this doesn't cause new memleaks, this seems fine to me. Thanks!
Hi Martin, GC tracking can be used if an object holds references to other Python objects to detect reference cycles (e.g. lists and tuples). Actually I had no idea how this stuff worked either until I started researching this crash and read through: http://docs.python.org/2/extending/newtypes.html#supporting-cyclic-garbage-collection In the case PyGIInfo, the object doesn't hold references to other objects and the "_base_info_traverse" function is empty. There was also no usage of PyObject_GC_New, but there was usage of PyObject_GC_UnTrack, which was causing the crashes described here. In the case of PyGIStruct and PyGIBoxed, the TP_FLAGS_HAVE_GC was not set, so the call to PyObject_GC_UnTrack was actually erroneous. These are most likely copy/paste mistakes from the PyGObject type, which does need to support GC.
Comment on attachment 241958 [details] [review] Remove half implemented GC in PyGIBaseInfo, PyGIStruct, and PyGIBoxed Ah, thanks for the additional explanation!
Comment on attachment 241958 [details] [review] Remove half implemented GC in PyGIBaseInfo, PyGIStruct, and PyGIBoxed Attachment 241958 [details] pushed as a703217 - Remove half implemented GC in PyGIBaseInfo, PyGIStruct, and PyGIBoxed
Related Python tickets: http://bugs.python.org/issue11321 http://bugs.python.org/issue11349 http://bugs.python.org/issue812369 Related mail thread: http://mail.python.org/pipermail/python-dev/2009-October/093029.html It seems like PyGObject needs to be audited in regards to module cleanup and possibly implement the GC module shutdown approach described in issue812369. In general there seems to be problems with module re-loading between successive Py_Initialize and Py_Finalize calls and ref counting of Py_Types. This sounds very much like the case of the current error: "'NoneType' object is not callable" we now get with the test case. I won't have time to revisit this for a while, so if anyone wants to take this further, have at it!
Created attachment 286106 [details] [review] tests: Setup automake parallel test harness Additionally always link with Python libraries as this will be needed for embedded interpreter testing. Rename RUN_TESTS_ENV_VARS to TESTS_ENVIRONMENT which will then be used by the parallel test harness.
Created attachment 286107 [details] [review] tests: Add test for multiple interpreter init and shutdowns Add a simple embeded interpreter test which initializes Python, imports pygi, calls a function, and shuts down the interpreter. This is run twice comparing the results of GLib.get_host_name() per-run.
-- GitLab Migration Automatic Message -- This bug has been migrated to GNOME's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/pygobject/issues/27.