After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 786948 - GstMessageType above GST_MESSAGE_EXTENDED not usable with python3 due to using signed integers for flags
GstMessageType above GST_MESSAGE_EXTENDED not usable with python3 due to usin...
Status: RESOLVED FIXED
Product: pygobject
Classification: Bindings
Component: general
Git master
Other Windows
: Normal critical
: ---
Assigned To: Nobody's working on this now (help wanted and appreciated)
Python bindings maintainers
Depends on:
Blocks:
 
 
Reported: 2017-08-28 21:55 UTC by Philippe Renon
Modified: 2017-10-24 16:13 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
fix an overflow when marshalling flags from py (1.83 KB, patch)
2017-08-31 14:41 UTC, Philippe Renon
committed Details | Review
flags: Add testcase for bug 786948 (3.53 KB, patch)
2017-10-23 10:45 UTC, Christoph Reiter (lazka)
none Details | Review
flags: Add testcase for bug 786948 (3.38 KB, patch)
2017-10-23 11:35 UTC, Christoph Reiter (lazka)
committed Details | Review

Description Philippe Renon 2017-08-28 21:55:18 UTC
This issue appears with GStreamer 1.12.2, gobject-introspection 1.52.1 and pygobject 3.24.1 (which are all relatively recent) on a MSYS2 system.
Comment 1 Philippe Renon 2017-08-28 21:55:37 UTC
GstMessageType above GST_MESSAGE_EXTENDED not usable with python3 on windows 

The following python3 code:

    import gi
    gi.require_version("Gst", "1.0")
    from gi.repository import Gst
    print(Gst.MessageType.get_name(Gst.MessageType.DEVICE_ADDED))

fails on windows 32 and 64 with this error:

    OverflowError: Python int too large to convert to C long

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
  • File "bug.py", line 4 in <module>
    print(Gst.MessageType.get_name(Gst.MessageType.DEVICE_ADDED))     SystemError: gi.FunctionInfo(get_name) returned a result with an error set

The same is true for any GstMessageType above or equal to GST_MESSAGE_EXTENDED.

My understanding of the issue is that, something, somewhere, is trying to map the Gst enum/flag to a C long.
On Windows, according to [1], a long is a signed 32 bit integer (range is –2147483648 through 2147483647).
But GST_MESSAGE_EXTENDED (and above enums) are too big to fit a Windows C long.
GST_MESSAGE_EXTENDED = (1 << 31) = 2147483648 > 2147483647.

I found a somewhat similar issue : https://bugzilla.gnome.org/show_bug.cgi?id=732633
But this issue is related to signed vs unsigned (and is fixed in 1.12.2).

Is that issue fixable ? If yes, where should I look : pyobject, gobject-introspection, ... ?

This issue appears with GStreamer 1.12.2, gobject-introspection 1.52.1 and pygobject 3.24.1 (which are all relatively recent) on a MSYS2 system.

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa383751(v=vs.85).aspx

PS : gstmessage.h has this comment : /* FIXME: 2.0: Make it NOT flags, just a regular 1,2,3,4.. enumeration */
Would be great to do ;) but would break ABI I guess...
Comment 2 Philippe Renon 2017-08-29 08:46:34 UTC
The OverflowError emanates from PyLong_AsLong(PyObject *obj) method (https://github.com/python/cpython/blob/3.6/Objects/longobject.c#L471)

But I am having a hard time finding the call path that leads to this method being called...

I should be able to build all components involved (gstreamer, gst-python, gobject-instrospection, pyobject) from source so I can try things out given some pointers or guidance.
Comment 3 Philippe Renon 2017-08-29 13:45:45 UTC
Some findings:

>>> from gi import _gi
>>> from gi.repository import Gst
>>> info = Gst.MessageType.__info__
>>> info.get_storage_type() == _gi.TypeTag.INT32
False
>>> info.get_storage_type() == _gi.TypeTag.UINT32
True

The above results are different from the same test done in 
https://bugzilla.gnome.org/show_bug.cgi?id=732633

So something was fixed since or, for a change, windows is better than linux.

Extract from Gst-1.0.gir:

      <member name="extended"
              value="2147483648"
              c:identifier="GST_MESSAGE_EXTENDED"
              glib:nick="extended">
        [...]
      </member>
      <member name="device_added"
              value="2147483649"
              c:identifier="GST_MESSAGE_DEVICE_ADDED"
              glib:nick="device-added">
        [...]
      </member>
        [...]
      <member name="any"
              value="4294967295"
              c:identifier="GST_MESSAGE_ANY"
              glib:nick="any">
        [...]
      </member>

Looks correct too.
Comment 4 Philippe Renon 2017-08-30 16:30:27 UTC
Using this test program:

  import gi
  gi.require_version("Gst", "1.0")
  from gi.repository import Gst
  print(Gst.MessageType.get_name(Gst.MessageType.DEVICE_ADDED))

I captured the back trace leading to the conversion error:

Thread 1 hit Breakpoint 2, PyLong_AsLong (obj=0x280d2f8)
    at ../Python-3.6.2/Objects/longobject.c:478
478             PyErr_SetString(PyExc_OverflowError,
(gdb) bt
  • #0 PyLong_AsLong
    at ../Python-3.6.2/Objects/longobject.c line 478
  • #1 _pygi_marshal_from_py_interface_flags
    at ../../pygobject-3.24.1/gi/pygi-enum-marshal.c line 198
  • #2 _invoke_marshal_in_args
    at ../../pygobject-3.24.1/gi/pygi-invoke.c line 533
  • #3 pygi_invoke_c_callable
    at ../../pygobject-3.24.1/gi/pygi-invoke.c line 674
  • #4 _function_cache_invoke_real
    at ../../pygobject-3.24.1/gi/pygi-cache.c line 782
  • #5 pygi_function_cache_invoke
    at ../../pygobject-3.24.1/gi/pygi-cache.c line 861
  • #6 pygi_callable_info_invoke
    at ../../pygobject-3.24.1/gi/pygi-invoke.c line 722
  • #7 _wrap_g_callable_info_invoke
    at ../../pygobject-3.24.1/gi/pygi-invoke.c line 759
  • #8 _callable_info_call
    at ../../pygobject-3.24.1/gi/pygi-info.c line 572
  • #9 _function_info_call
    at ../../pygobject-3.24.1/gi/pygi-info.c line 627
  • #10 _PyObject_FastCallDict
    at ../Python-3.6.2/Objects/abstract.c line 2316
  • #11 _PyObject_FastCallKeywords
    at ../Python-3.6.2/Objects/abstract.c line 2481
  • #12 call_function
    at ../Python-3.6.2/Python/ceval.c line 4833
  • #13 _PyEval_EvalFrameDefault
    at ../Python-3.6.2/Python/ceval.c line 3295
  • #14 PyEval_EvalFrameEx
    at ../Python-3.6.2/Python/ceval.c line 718
  • #15 _PyEval_EvalCodeWithName
    at ../Python-3.6.2/Python/ceval.c line 4139
  • #16 PyEval_EvalCodeEx
    at ../Python-3.6.2/Python/ceval.c line 4160
  • #17 PyEval_EvalCode
    at ../Python-3.6.2/Python/ceval.c line 695
  • #18 run_mod
    at ../Python-3.6.2/Python/pythonrun.c line 980
  • #19 PyRun_FileExFlags
    at ../Python-3.6.2/Python/pythonrun.c line 933
  • #20 PyRun_SimpleFileExFlags
    at ../Python-3.6.2/Python/pythonrun.c line 396
  • #21 PyRun_AnyFileExFlags
    at ../Python-3.6.2/Python/pythonrun.c line 80
  • #22 run_file
    at ../Python-3.6.2/Modules/main.c line 338
  • #23 Py_Main
    at ../Python-3.6.2/Modules/main.c line 809
  • #24 main
    at ../Python-3.6.2/Programs/python.c line 69

Comment 5 Philippe Renon 2017-08-30 16:54:01 UTC
Hacking the _pygi_marshal_from_py_interface_flags() method to make it use PYGLIB_PyLong_AsUnsignedLong() instead of PYGLIB_PyLong_AsLong() seems to fix the test case...

>>> import gi
>>> from gi.repository import Gst
>>> print(Gst.MessageType.get_name(Gst.MessageType.DEVICE_ADDED))
device-added

Not sure at all if this hack is acceptable though...
Comment 6 Sebastian Dröge (slomo) 2017-08-30 17:04:33 UTC
flags aka "bitfields" (in gobject-introspection speak) are always unsigned, enums are signed. So changing it for flags should be fine, for enums not so much.
Comment 7 Philippe Renon 2017-08-31 09:48:16 UTC
Fortunately enums and flags are marshalled/unmarshalled through separate code pathes. 

I'll submit a bug fix to pyobject.
Comment 8 Sebastian Dröge (slomo) 2017-08-31 11:27:58 UTC
Yeah, reassigned this bug to pygobject :)
Comment 9 Philippe Renon 2017-08-31 14:41:40 UTC
Created attachment 358860 [details] [review]
fix an overflow when marshalling flags from py
Comment 10 Philippe Renon 2017-09-08 06:46:47 UTC
Tried to submit a PR (https://github.com/GNOME/pygobject/pull/1) but it got closed (by a robot?).
Comment 11 Sebastian Dröge (slomo) 2017-09-08 07:49:01 UTC
Having the patch here is all that is needed. GitHub is only a mirror for making life easier for people using GitHub, but patch review and bug tracking happens here.
Comment 12 Christoph Reiter (lazka) 2017-10-23 10:45:37 UTC
Created attachment 362091 [details] [review]
flags: Add testcase for bug 786948
Comment 13 Christoph Reiter (lazka) 2017-10-23 11:35:38 UTC
Created attachment 362092 [details] [review]
flags: Add testcase for bug 786948

Apparently out marshaling and comparisons are also broken, so reducing the test to only test the fixed case.
Comment 14 Christoph Reiter (lazka) 2017-10-23 11:43:53 UTC
Review of attachment 358860 [details] [review]:

lgtm
Comment 15 Christoph Reiter (lazka) 2017-10-24 12:31:00 UTC
Thanks
Comment 16 Philippe Renon 2017-10-24 16:13:05 UTC
Thanks