After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 757184 - import fails when no version given and a file called "sys" exists in same directory
import fails when no version given and a file called "sys" exists in same dir...
Status: RESOLVED FIXED
Product: pygobject
Classification: Bindings
Component: general
Git master
Other Linux
: Normal normal
: ---
Assigned To: Nobody's working on this now (help wanted and appreciated)
Python bindings maintainers
http://bugs.python.org/issue25493
Depends on:
Blocks:
 
 
Reported: 2015-10-27 12:54 UTC by Michael Laß
Modified: 2015-11-12 16:30 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Actual bug report (3.75 KB, text/plain)
2015-10-27 12:54 UTC, Michael Laß
  Details
Fix import warnings pointing to the wrong code with CPython 3.3/3.5 (3.32 KB, patch)
2015-10-30 12:28 UTC, Christoph Reiter (lazka)
committed Details | Review

Description Michael Laß 2015-10-27 12:54:40 UTC
Created attachment 314220 [details]
Actual bug report

Bugzilla always gives me a 500 when I try posting this bug report. So the actual report is attached as a txt file, which for some reason works...
Comment 1 Michael Laß 2015-10-27 12:57:07 UTC
Maybe I can post the text as a response...

---

Various applications that import modules from gi.repository without specifying required versions before fail with the following error:

--snip--
Traceback (most recent call last):
  • File "/usr/lib/python3.5/tokenize.py", line 392 in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 24: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/redshift-gtk", line 26, in <module>
    from redshift_gtk.statusicon import run
  File "/usr/lib/python3.5/site-packages/redshift_gtk/statusicon.py", line 32, in <module>
    from gi.repository import Gtk, GLib, GObject
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/home/bevan/git/pygobject/dest/usr/lib/python3.5/site-packages/gi/importer.py", line 160, in load_module
    dynamic_module = load_overrides(introspection_module)
  File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/home/bevan/git/pygobject/dest/usr/lib/python3.5/site-packages/gi/importer.py", line 123, in _check_require_version
    PyGIWarning, stacklevel=stacklevel)
  File "/usr/lib/python3.5/warnings.py", line 18, in showwarning
    file.write(formatwarning(message, category, filename, lineno, line))
  File "/usr/lib/python3.5/warnings.py", line 26, in formatwarning
    line = linecache.getline(filename, lineno) if line is None else line
  File "/usr/lib/python3.5/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib/python3.5/linecache.py", line 47, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib/python3.5/linecache.py", line 136, in updatecache
    with tokenize.open(fullname) as fp:
  File "/usr/lib/python3.5/tokenize.py", line 456, in open
    encoding, lines = detect_encoding(buffer.readline)
  File "/usr/lib/python3.5/tokenize.py", line 433, in detect_encoding
    encoding = find_cookie(first)
  File "/usr/lib/python3.5/tokenize.py", line 397, in find_cookie
    raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for '/usr/bin/sys'
--snip--

A minimal example is the following:
--snip--
#!/usr/bin/env python
from gi.repository import Gtk
--snip--

/usr/bin/sys is a binary provided by openafs which should have no relation to python or gobject whatsoever. The error only occurs if the python script and the "sys" binary lie in the same directory (here: /usr/bin).

I did a git bisect and the result was the following:

--snip--
2048dc8d1d708abce7037f96483c6d776567d6b5 is the first bad commit
commit 2048dc8d1d708abce7037f96483c6d776567d6b5
Author: Christoph Reiter <creiter@src.gnome.org>
Date:   Mon Mar 2 20:58:04 2015 +0100

    Add gi.PyGIWarning and use it instead of PyGIDeprecationWarning in case the version to import wasn't specified.
    
    This makes the warning visible by default.
    See commit ef3bff4e570363e4f383d4cdae9cecd4073b03d8 for more info on the warning.
    
    https://bugzilla.gnome.org/show_bug.cgi?id=727379

:040000 040000 3358c42f8fb5e39f8a71870f07cb92f93cb5a826 d8b9a350bbf2cef8f44ef9f799c56a9aa979d3a1 M	gi
:040000 040000 37ed39b29345a56b671bb6658935f7bc01f7ced5 734e76787c7ef9eafcd3dc358b94bcf794d9c0d3 M	tests
--snip--

Specifying the version before import "solves" the problem. Example:
--snip--
#!/usr/bin/python
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk
--snip--

This and the bisect lead to the assumption that the error lies somewhere in the code generating the warnings.
Comment 2 Christoph Reiter (lazka) 2015-10-27 14:37:16 UTC
Looks like a Python bug.

The same happens with:

import warnings
warnings.warn("foo", Warning, stacklevel=2)

Python3 falls back to "sys" as display name when the top of the stack is reached and the bug is that it tries to open a file named "sys" in the same directory and interpret it as a Python file.

The reason that the stacklevel is wrong is that it apparently got changed in Python3.5 for the import path.

Using Python3.4 should work as a workaround.

Two things to do:

* Find out what the new stacklevel for Python3.5 is and adjust in PyGObject (or figure out a different approach)
* Search/File a Python bug about the "sys" problem.
Comment 3 Michael Laß 2015-10-27 16:27:11 UTC
For the "sys" problem there are two further observations:

* Another workaround is to add "-I" to the shebang of the application. This excludes the local directory from the search path.

* "import sys" works flawlessly. I wonder if there is some exception that tells python to ignore the local directory for sys and this exception is not respected in the case described above.

Definitely something for the python bug tracker. I had a quick look and did not find anything like this so I will add a report there.
Comment 4 Michael Laß 2015-10-27 16:49:15 UTC
New bug in the python bug tracker: http://bugs.python.org/issue25493
Comment 5 Christoph Reiter (lazka) 2015-10-30 12:28:19 UTC
Created attachment 314467 [details] [review]
Fix import warnings pointing to the wrong code with CPython 3.3/3.5

For making warnings point to the code doing the import, the stack frames
of the import system need to be skipped. The frame count number varries
between CPython versions and in 3.5 all frames of the import system are
skipped for warnings (https://bugs.python.org/issue24305).

This hardcodes the frame counts for all supported CPython versions
which fixes the import warning output for CPython 3.3 and 3.5.

This also fixes/works around a bug in CPython 3 where if a too
large stacklevel value was passed to warn(), CPython would try to
interpret a file called "sys" in the same directory of the
executed script (https://bugs.python.org/issue25493
and https://bugzilla.gnome.org/show_bug.cgi?id=757184).
Comment 6 Christoph Reiter (lazka) 2015-10-30 12:28:56 UTC
(In reply to Michael Laß from comment #4)
> New bug in the python bug tracker: http://bugs.python.org/issue25493

Thanks for filing a bug report there.