GNOME Bugzilla – Bug 566641
Python3 support
Last modified: 2010-09-15 06:38:40 UTC
It would be nice to be able to build/install/use PyGObject with Python 3. This new version of Python brings some new syntax (incompatible with Python 2.x), some changed C API... PyGObject should support both Python2.x and Python3.x.. choice should be made at build time, maybe with a new ./configure option.
I started working on a patch. Will push a git repo on github and link it here.
See http://github.com/philn/pygobject/tree/py3k This is only a first stab, still many things to do :/
As I told Philippe on irc: For the glib/gobject module I've been attempting to maintain compatibility against both 2.x and 3.x at the same time without changing/breaking the API in any way. However, when it comes to codegen it gets a little bit more complicated, as: - codegen should *run* under 2.x and 3.x - codegen should *generate* code compilable under both 2.x and 3.x There's obviously more things to add to the compat header which should be used to abstract the differences. I haven't thought enough about it, but I think it should be feasible without too much effort. The only problem is the somewhat messy code in codegen, which needs a big cleaning up effort to be really maintainable.
codegen now runs on 2.x and 3.x, gobject/glib import fine but there are issues with gio, mainly with GAppInfo, IIRC..
gio imports fine :) I'm now in the process of porting the tests: - fixed ltihooks.py using the libtoolimporter from Johan in giscanner - ported the testhelpermodule - now porting the python tests with 2to3 but there are issues about long type removal from py3k and unicode (u"something") related tests
Philippe: any progress on that?
No progress recently. I got distracted by other things and latest release of Python3 broke my current patch because they removed some APIs :/ I'll try to remotivate myself to get back to this.
So I took some time to make my branch compile again with Python 3.0.1 ;) Had to cope with removal of tp_compare slot and had to implement tp_richcompare for various types. Next step is to make the unittests pass again.
Some tests are still failing oddly like the properties stuff. But if I break with pdb and exit just after, they pass... I think I'll stop working on this port until Python3 gets better. Meanwhile I think my current branch could still be merged, compatibility with Python2 is not broken and the code compiles fine for Python3. I tested that pygtk still compiles and is still usable on Python2 with my branch. So, will someone review the code if i attach it to this bug?
Yes, please attach the patch(es) to this but. Try to split them up in as many small pieces as possible. At least codegen, pygobject, gio.
I have 6 patches to review in this order if it makes sense: - root.diff: modified files in top directory - codegen.diff - glib.diff - gobject.diff - gio.diff - tests.diff
Created attachment 133744 [details] [review] files modified in top directory
Created attachment 133745 [details] [review] Patch for codegen
Created attachment 133746 [details] [review] Patch for glib
Created attachment 133747 [details] [review] Patch for GObject
Created attachment 133748 [details] [review] Patch for gio
Created attachment 133749 [details] [review] Patc for the unittests
Maybe you can rediff all the patches as a git patch? To do so, apply in a local repository, commit (locally) and then do 'git format-patch HEAD^'.
Comment on attachment 133745 [details] [review] Patch for codegen > [...] > > codegen_PYTHON = \ >- __init__.py \ >- argtypes.py \ >- reversewrapper.py \ > code-coverage.py \ > codegen.py \ >- definitions.py \ >+ createdefs.py \ > defsconvert.py \ > defsgen.py \ >- defsparser.py \ >- docextract.py \ > docextract_to_xml.py \ > docgen.py \ > h2def.py \ >- defsgen.py \ >- createdefs.py \ >+ __init__.py \ > mergedefs.py \ > missingdefs.py \ > mkskel.py \ >- override.py \ >- scanvirtuals.py \ >- scmexpr.py >+ scanvirtuals.py I can see no reason for this change unless your actually removed the files. If this is a result of a merge, please pay attention to what you do. > [...] >-# -*- Mode: Python; py-indent-offset: 4 -*- >- >-__all__ = [ >- 'argtypes', >- 'codegen', >- 'definitions', >- 'defsparser', >- 'docextract', >- 'docgen', >- 'h2def', >- 'defsgen' >- 'mergedefs', >- 'mkskel', >- 'override', >- 'scmexpr' >-] Please don't touch such things. >+import subprocess This makes codegen depend on Python 2.5. On the other hand, we can probably just go and abandon 2.3 and 2.4 altogether. I didn't read the whole patch. To make it easier, please don't include any non-essential changes.
I did the codegen patch with git diff -find-copies-harder -B -C otherwise the diff would have been even bigger because of the moved files. I moved some files from codegen to codegen/libcodegen so these changes do make sense. Johan wanted multiple patches, not a big one, to ease review. Now if you want, you can still review the py3k branch I've put on github (link in comment #2)
Comment on attachment 133744 [details] [review] files modified in top directory This looks basically good. I'd like you to split out/undo the indentation changes though, they make the review a fair amount harder to do.
Hmm the indentations changes are important, Python 3 doesn't allow mixing tabs and whitespaces. About use of subprocess, I can find somthing more compatible with older versions of Python. So, let's split the smallest patch in 2... :)
Created attachment 133795 [details] [review] files modified in top directory
Created attachment 133796 [details] [review] Patch for pygtk.py Cleaning up white spaces / tabs
Comment on attachment 133796 [details] [review] Patch for pygtk.py This was already done in _January_. It would help a lot if you made patches against HEAD, not some old version.
(In reply to comment #25) > (From update of attachment 133796 [details] [review] [edit]) > This was already done in _January_. It would help a lot if you made patches > against HEAD, not some old version. > So ignore that patch. Of course all those patches are against HEAD.
(In reply to comment #22) > Hmm the indentations changes are important, Python 3 doesn't allow mixing tabs > and whitespaces. > > About use of subprocess, I can find somthing more compatible with older > versions of Python. > > So, let's split the smallest patch in 2... :) > No, please do it for all patches. I recommend that you first do a reindentation patch of everything and then rebase the rest on top of that. If it's too much work, just use interdiffs betwen your current set of patches and a pure reindentation patch.
(In reply to comment #27) > (In reply to comment #22) > > Hmm the indentations changes are important, Python 3 doesn't allow mixing tabs > > and whitespaces. > > > > About use of subprocess, I can find somthing more compatible with older > > versions of Python. > > > > So, let's split the smallest patch in 2... :) > > > > No, please do it for all patches. I recommend that you first do a reindentation > patch of everything and then rebase the rest on top of that. If it's too much > work, just use interdiffs betwen your current set of patches and a pure > reindentation patch. > Only one patch contained white spaces fixes and I reverted them (pygtk.py). I'm now attaching new versions of the tests (tiny white spaces reverts) and codegen (typo fix). Please review!
Created attachment 133856 [details] [review] Patch for codegen
Created attachment 133857 [details] [review] Patch for the unittests
Comment on attachment 133856 [details] [review] Patch for codegen This should be split as well, with the renames in a separate patch.
Johan, that patch duplicates what has been committed in January already. For the rest of stuff, I don't think it is a good idea to commit such huge changes for 2.18.
Comment on attachment 133857 [details] [review] Patch for the unittests >diff --git a/tests/runtests.py b/tests/runtests.py >index fc0558d..9a81c63 100644 >--- a/tests/runtests.py >+++ b/tests/runtests.py >@@ -24,13 +24,21 @@ common.importModules(buildDir=buildDir, > SKIP_FILES = ['common', 'runtests', > 'test_enum', 'test_conversion'] > >+if sys.version_info[0] > 2: >= 3 makes more sense > dir = os.path.split(os.path.abspath(__file__))[0] > os.chdir(dir) > > def gettestnames(): > files = glob.glob('*.py') >- names = map(lambda x: x[:-3], files) >- map(names.remove, SKIP_FILES) >+ names = [x[:-3] for x in files if x[:-3] not in SKIP_FILES] Avoid using more than 80 columns hers, indent properly. >diff --git a/tests/test_enum.py b/tests/test_enum.py >+ enums = [x for x in gobject.list_properties(win) if gobject.GType.is_a(x.value_type, gobject.GEnum)] >+ props = [x for x in gtk.container_class_list_child_properties(gtk.Table) if gobject.GType.is_a(x.value_type, gobject.GFlags)] Ditto. >diff --git a/tests/test_gio.py b/tests/test_gio.py >+if sys.version_info[0] > 2: >+ long = int >= 3 > class TestFile(unittest.TestCase): > def setUp(self): >@@ -169,9 +173,7 @@ class TestFile(unittest.TestCase): > for gfile in [gio.File("/"), > gio.File("file:///"), > gio.File(uri="file:///"), >- gio.File(path="/"), >- gio.File(u"/"), >- gio.File(path=u"/")]: >+ gio.File(path="/")]: Makes sense to still test these on 2.x. >diff --git a/tests/test_gtype.py b/tests/test_gtype.py >+if sys.version_info[0] > 2: >+ long = int See above. >diff --git a/tests/test_option.py b/tests/test_option.py >-from StringIO import StringIO >+try: >+ from io import StringIO >+except ImportError: >+ from StringIO import StringIO >+ >+try: >+ basestring >+except NameError: >+ basestring = str >+ >+try: >+ StandardError >+except NameError: >+ StandardError = Exception > I think that checking for the version makes more sense here. >diff --git a/tests/test_properties.py b/tests/test_properties.py >+if sys.version_info[0] > 2: >+ long = int See above. >diff --git a/tests/test_signal.py b/tests/test_signal.py >+if sys.version_info[0] > 2: >+ long = int See above and move this to be after the other imports.
Comment on attachment 133746 [details] [review] Patch for glib Looks good. A bit unfortunate that all that richcompare duplication is needed.
Comment on attachment 133747 [details] [review] Patch for GObject >diff --git a/gobject/propertyhelper.py b/gobject/propertyhelper.py >index 216ceb2..31de103 100644 >--- a/gobject/propertyhelper.py >+++ b/gobject/propertyhelper.py >@@ -37,7 +37,17 @@ from gobject.constants import \ > G_MAXULONG > > >-class property(object): >+if sys.version_info[0] > 2: >+ long = int >+ basestring = str >= 3
Sorry I didn't use the correct git diff options to generate the codegen patch. This one will be much smaller and easier to review.
Created attachment 133874 [details] [review] Patch for codegen
(In reply to comment #33) > (From update of attachment 133857 [details] [review] [edit]) > > > > class TestFile(unittest.TestCase): > > def setUp(self): > >@@ -169,9 +173,7 @@ class TestFile(unittest.TestCase): > > for gfile in [gio.File("/"), > > gio.File("file:///"), > > gio.File(uri="file:///"), > >- gio.File(path="/"), > >- gio.File(u"/"), > >- gio.File(path=u"/")]: > >+ gio.File(path="/")]: > > Makes sense to still test these on 2.x. > Applied all your suggested tweaks excepted this one because u"foo" is not valid python3 syntax. So what I could do is if test is runned under python < 3, import a module containing these tests, would that be fine? Pushed latest changes on github, will attach a new patch for the tests once this syntax-related issue is worked out.
Is somebody still working on this? I really need Python3 support for PyGtk.
(In reply to comment #39) > Is somebody still working on this? I really need Python3 support for PyGtk. Yea, i'm started to pick things up where Philippe left them and intented to continue porting.
(In reply to comment #40) > (In reply to comment #39) > > Is somebody still working on this? I really need Python3 support for PyGtk. > > Yea, i'm started to pick things up where Philippe left them and intented to > continue porting. Alex: did you get anywhere with this?
(In reply to comment #41) > (In reply to comment #40) > > (In reply to comment #39) > > > Is somebody still working on this? I really need Python3 support for PyGtk. > > > > Yea, i'm started to pick things up where Philippe left them and intented to > > continue porting. > > Alex: did you get anywhere with this? Nope, things changed here, i'm not in this anymore.
is there any ongoing work on PyGObject/PyGTK support from Python 3?
(In reply to comment #43) > is there any ongoing work on PyGObject/PyGTK support from Python 3? I'm also very interested in this. And I would like to contribute, but at the time I have no experience with git and a very little experience with PyGTK itself.
I can ditto that I'm interested in this, and I would like to contribute, but I have very little experience with pygobject, GObject, and the process of wrapping anything in C for Python. If someone knowledgeable were able to help with the process and answer questions, I'd be willing to do the work of porting it. It looks like PyQt has supported Python 3 for a while now. I firmly believe GTK+ is and will continue to be a great GUI frakework, but it won't be in the Python 3 environment if it doesn't get ported. Is anyone else wiling to collaborate on this?
I have local repositories of pyobject, pycairo, & pygtk with things almost working, though I need to get them to work and then go back and create reasonable patch sets. Nothing is all that difficult, but there are a fair amount of changes. My plan is to put copies of the repositories somewhere so people can try things out. Working on the bindings does require working in C rather than python, but they aren't all that complex (though I worked with a code generator before).
(In reply to comment #46) > I have local repositories of pyobject, pycairo, & pygtk with things almost > working, though I need to get them to work and then go back and create > reasonable patch sets. Nothing is all that difficult, but there are a fair > amount of changes. My plan is to put copies of the repositories somewhere so > people can try things out. > > Working on the bindings does require working in C rather than python, but they > aren't all that complex (though I worked with a code generator before). If you could host your branches on github or launchpad or something, I'd be happy to start testing the code. I've used C quite a bit in years past, so that's no problem, but I'm pretty new to GObject et al.
I've set up a repository at http://github.com/jpe/pygobject-py3k/commits/py3k It's currently part of the way to supporting python 3; more is to come. What I did was convert pygobject, pycairo, pygtk to the point where pygtk demo mostly works and now am going back to clean up the changes. Repositories for pycairo and pygtk will follow. Sorry it took so long, but make check would periodically segfault because of what I now believe was a buggy gio checkout. A few things to think about: * The macros for strings and ints to make things work are now prefixed with _PyUnicode, _PyLong, & _PyByteArray. I think I'll change this to PYGLIB_PyUnicode, PYGLIB_PyLong, & PYGLIB_PyByteArray so people know to look for the definitions in pyglib and not in python itself. * The distinction between what should be a string (all strings are unicode objects in python 3) and what should be a bytearray may require the most thought and the codegen probably needs to know whether a char* is a string or bytearray
(In reply to comment #48) Awesome, Great work John! I had a very quick look at the patches and they look mostly good. You might want to make sure that the whole testsuite passes on both 2.x and 3.x before committing. > A few things to think about: > > * The macros for strings and ints to make things work are now prefixed with > _PyUnicode, _PyLong, & _PyByteArray. I think I'll change this to > PYGLIB_PyUnicode, PYGLIB_PyLong, & PYGLIB_PyByteArray so people know to look > for the definitions in pyglib and not in python itself. Good idea, that's quite a bit clearer than what we do today. It'll make it easier when we decide to drop 2.x support as well. Remember to only make aliases of Python 3.x API, not the other way around > * The distinction between what should be a string (all strings are unicode > objects in python 3) and what should be a bytearray may require the most > thought and the codegen probably needs to know whether a char* is a string or > bytearray Right, I remember this being a tricky issue. Would it be possible to make as much as possible strings, only use bytearrays for things we'd use buffers for today?
(In reply to comment #49) Thanks for reviewing this Johan > > * The distinction between what should be a string (all strings are unicode > > objects in python 3) and what should be a bytearray may require the most > > thought and the codegen probably needs to know whether a char* is a string or > > bytearray > > Right, I remember this being a tricky issue. Would it be possible to make as > much as possible strings, only use bytearrays for things we'd use buffers for > today? The use case I've run into already is binary data being sent to or read from io channels, where PyStringObject's are used in python 2.x. I'm afraid we need to review things on a case by case basis, with most cases using PyUnicodeObject's. I just pushed my the py3k branch to the gnome.org repository so others can push changes to it. My next step is to implement enum support in python 3.
I've pushed more changes to the py3k branch on git.gnome.org, which is down to one test failure under python 2.5 and 3.1, but unfortunately it's a different test on each. I'll look into the failures. I did merge John Stowers win32 from master but am unable to test the merge right now. John, could you test these? I'll eventually be able to do it, but not for a few days. I've tested back to python 2.5 and am wondering if we want to make python 2.5 the minimum supported version. I don't think it's a tremendous amount of work for earlier versions, but it is more things to test.
I just pushed to the py3k branch a couple of fixes to bugs that are independent of the changes for python3 so all tests pass under both python 2.5 and python3.1. It's probably time to think about landing this on the master branch; obviously we want to avoid regressions in python2 support. What needs to be done before this lands on master?
So what needs to be done before this can land on master?
(In reply to comment #53) > So what needs to be done before this can land on master? Looks like the merge of PyGI into PyGObject will happen before (today?). Given that there are several non-trivial apps being ported to PyGI right now, a plan forward could be for someone to rebase the existing patches on top of PyGObject after the PyGI merge, then see if those apps break in any way with Python 2.x and 3.x. Sounds reasonable?
PyGI has been merged into PyGObject a while ago, so what is needed now is: - Merge http://git.gnome.org/browse/pygobject/log/?h=pygi-py3k into http://git.gnome.org/browse/pygobject/log/?h=py3k - Rebase it on top of HEAD - Check that all tests still pass - Test it with a non-trivial apps and report back Once this has been done we should do a quick review and merge ASAP before it bitrots again. Any takers?
*** This bug has been marked as a duplicate of bug 615872 ***