GNOME Bugzilla – Bug 554781
Full lambda support
Last modified: 2009-09-15 15:10:55 UTC
public void connect_async(int priority, GLib.Cancellable? cancellable, GLib.AsyncReadyCallback callback) { connection.connect_async(cancellable, (source, result) => { ((GLib.SocketConnection)source).connect_finish(result); var _res = new SimpleAsyncResult(this, callback, lambda); _res.run_in_thread((res, obj, cancel) => { try { input_stream = new GLib.DataInputStream(connection.input_stream); output_stream = new GLib.DataOutputStream(connection.output_stream); ulong length; weak string hello_string = input_stream.read_line(out length, null); int code = hello_string.to_int(); switch(code) { case 200: return true; case 201: return false; default: throw new ServerError.UNKNOWN_RESPONSE_CODE("Unnow status code: %i", code); } } catch(GLib.Error error) { res.set_from_error(error); } }, priority, null); } } This example shows a problems: - Sometimes it is need to access a variables in the outer scope. Passing them as 'global' variables (i.e. inside the main object) is a-very-bad-idea - Sometimes it is need to access an address of lambda itself - as shown. It would also make the recursion somehow possible: SomeDelegate fib = (i) => { return (i <= 1) ? i : (lambda(i - 1) + lambda(i - 2)); }; SomeDelegate some = (i) => { SomeDelegate in_some = lambda; if(i > 0) return i + something.call(() => { in_some(i - 1); }); else return 1; } Of course - it creates problems: - We do not know the scope - We do not know if the vales should be passed-by-value or passed-by-reference
"Lambda not allowed in this context". It should be: public void add_resolver_tests () { GLib.Test.add_func("/gnio/resolver/resolve", () => { try { var resolver = new Resolver (); var address = resolver.resolve ("localhost", null); assert (address.is_loopback); } catch (GLib.Error e) { assert_not_reached (); } }); GLib.Test.add_func("/gnio/resolver/resolve_async", () => { var resolver = new Resolver (); var loop = new GLib.MainLoop (null, false); resolver.resolve_async ("localhost", (resv, res) => { try { var address = resv.resolve_finish (res); assert (address.is_loopback); } catch (GLib.Error e) { assert_not_reached (); } finally { //loop.quit (); } }); loop.run (); }); }
Sorry - the second example is invalid.
I agree, this is important (accessing the scope of the calling function). I'm not a vala developer, but because C code is being generated to support delegates as external to the calling function anyways, then from what I can tell, full support to access the scope of the calling function is problematic. Alright, one could use nested function to support delegates, as gcc is being used. The problem there is when one uses a delegate in a thread launch, and then exits the calling function, something I do quite often in C#. When the calling function is exited, then it's stack frame would go away, including all it's local variables, some of which may be used by the delegate code. Those variable being accessed may end up stomping on (or being stomped on) by a subsequent function call at the same depth. I know, what I'm discussing is a matter of low level implementation details, and Vala is supposed to be a level language, but I don't really see a good way (or any way really) with generated C code to acheive to achieve this at all. Once again, since I'm not part of the vala development effort, smaybe I can't answer this as well as those who are.
*** Bug 537706 has been marked as a duplicate of this bug. ***
*** Bug 585158 has been marked as a duplicate of this bug. ***
(In reply to comment #3) > I agree, this is important (accessing the scope of the calling function). I'm > not a vala developer, but because C code is being generated to support > delegates as external to the calling function anyways, then from what I can > tell, full support to access the scope of the calling function is problematic. > > Alright, one could use nested function to support delegates, as gcc is being > used. The problem there is when one uses a delegate in a thread launch, and > then exits the calling function, something I do quite often in C#. When the > calling function is exited, then it's stack frame would go away, including all > it's local variables, some of which may be used by the delegate code. Those > variable being accessed may end up stomping on (or being stomped on) by a > subsequent function call at the same depth. > > I know, what I'm discussing is a matter of low level implementation details, > and Vala is supposed to be a level language, but I don't really see a good way > (or any way really) with generated C code to acheive to achieve this at all. > > Once again, since I'm not part of the vala development effort, smaybe I can't > answer this as well as those who are. > Alright, I've have since figured out how to safely use returned nested gcc functions as true closure functions, as have worked out the stack frame issues I mentioned above. Supported by a stock gcc with -std=gcc99 or -std=gcc89 but not with -c99 or by using g++. That being said, I do feel now that it is technically possible/feasible for Vala to use gcc nested functions for delegates, therefore acheiving full Lambda and Closure support. If someone wants details, just ask me. While I did get it working in a very minmal manner, I think it it would take a fair amount of effort to effectively work it into Vala.
There are two prerequisites for a proper closure support. The accessed variables from the parent scope must be accessible and modifiable from the closure, and they must be freed from memory only after both the parent scope is exited and the closure destroyed. I think this could be done simply by dynamically allocating the variables that are accessed by the closure. On the C level, they would be passed to it as an array of pointers. To ensure they are properly deallocated once both function is exited and closure destroyed, first member of that array would be a "reference" counter initiated with 2. Once decremented on the end of the parent method and once in the destructor of the closure. Once it's 0, the code would free the dynamically allocated memory. Maybe this approach has some flaws I didn't think out, but it shouldn't be that hard to implement IMO.
So, I think I've figured out how to (backwards-compatibly) implement a full closure support with correct memory management. Now I just need figure out, how to incorporate that into Vala properly.
One issue that needs to be taken into account is that local variables declared in a loop block do not have the same lifetime as other local variables in the method. Regarding integration into the Vala code base: It should be possible to share quite some code with the closures used for async support.
I thought it can be backwards compatible, but I was mistaken. Vala currently doesn't do anything about userdata when changing the variable. The problem with closures is that the variables accessed by it must be kept alive as long as the closure exists. Perhaps it would be better to replace current pair (function pointer + userdata) with GClosure. Then memory management would be easier. As for the variables, the best would probably be generating some structs for every scope. Each would contain a reference count and a pointer to it's parent scope, while the topmost one would have a pointer to closures parent object. That structures would be used instead of automatic variables for those which are accessed in the closure. Because of the hierarchical nature of scopes, we have to be careful when deciding when to free it. Also, as the variables can need some special attention (unrefing etc.) there would have to be a generated freeing method, pointer to which would also be part of the scope structure. Then it's just a matter of special inline ref and unref functions (which can be as simple as one line). ref would be invoked when some closure accessing the scope is created, unref when the closure is destroyed (easy with GClosure) and on the end of the scope. There would also have to be some attribute to decorate "old style" non-closure method pointers (for binding). Or we could, if we are to use this approach without breaking compatibility, instead add a [Closure] attribute to decorate variables, while leaving current non-closure delegates as default. Fairly easy would be modifying signals, as they already use GClosures IIRC.
Well, the way I did in C just as a proof of concept was with ucontext and a gcc nested function. If one is generating gcc compilable code using a language like Vala, then one would have to determine of the function could create a closure, and if so run it in a ucontext. When garbage collection determines that the closure is no longer accessible, then the ucontext data structure would be freed. Using ucontext is on way I see of creating closures in gcc via nested functions. I also did it with a sleeping pthread instead of a ucontext, but that is not as straight forward. gcc nested functions work properly used this way because they are trampoline functions that have specially wrappers around them connecting them to the owning functions stack frame.
Can you point me out where can I find something about this proof of concept of yours? The are things about your comment I don't quite understand. For one, I don't know what ucontext is. Secondly, there is no garbage collection in Vala. Thirdly, AFAIK nested functions are limited to the scope of the creating function, which is not the case with closures. You say the wrapper connects to the owning functions stack frame, but with closures there is no owning function and no stack frame.
Created attachment 137477 [details] Example for showing how one could do closures with gcc nested functions. This is only a proof of concept example of 2 closure implementations using gcc. I don't recommend the pthread method, I just tried it first before I found out about ucontext.
the function pointers of gcc nested functions can be passed to other functions. Other wise they would useless as callbacks. Nested functions have access to the stack frame of their calling function (via a trampoline) http://en.wikipedia.org/wiki/Trampoline_(computers) As long as one the stack frame of the "owning" function is active, then the function pointer of a gcc nested function can be safely passed around to other functions and used however, and the nested function always has access to the variable in the scope of the "owning" functions stack frame. However, if one returns a pointer to a nested function that accesses variables in the scope of the owning function (or arguments, whatever), then the calling via that returned pointer will cause problems, as the owning function's stack frame will no longer be live. My example shows 2 methods of keeping the owning functions stack frame alive, so that the returned function pointers can used after the call the owning function has returned. Ok, yeah, Vala does not have full garbage collection. I was referring to the reference counting that Vala uses, and since I don't know that much about Vala, I don't know if that would work in this case.
I can't say I really understand the code (low level C understanding is not one of my strong sides), but it seems to me this approach is a bit too complicated. It seems to me it could be problematic to properly release resources in the stack frame (unrefing objects etc.), but I'm probably be wrong about that. Another problem is that this method actually preserves the entire function context, even though (in real-world cases) most of it will never be needed. That could be considered a memory leak, which if I'm correct can't be fixed. The fact we would actually be restricting Vala usage to GCC is not so problematic to me, but maybe some people would have different opinions? Nevertheless, if you know of a way to address the two issues I mentioned, I would be quite interested. In the meantime, I'll try to come up with a working example of how I think closures could work.
Yes, this method does preserve the entire function context. In the case of the thread, when the closure is no longer needed, the sleeping thread would simply be canceled. The thread method not not really that great though, as it takes up a PID (albeit associated with a sleeping thread). In the case of ucontext, when the closure is no longer needed, the memory associated with the ucontext would be freed. The thing is though, for full lambda's that support closures (as you already stated) you need: 1) full access to the variables in the stack frame that in the lambda's scope. (declared before the lambda in the same scope or in outer scopes within the containing function. 2) Preservation of all the variables in the lambda's scope after the owning function has returned (if a pointer to the lambda was passed elsewhere. You are correct, this could be considered a memory leak, as in many cases there maybe variables in the owning functions scope that are not needed in the closure. It would be ideal if in Vala the programmer would not have to care how it is implemented. If the programmer wants to pass a lambda to a thread that accesses the some variables in the owning functions stack frame, then return from the owning function, leaving the thread running in the "closure", so be it. Or if the owning function returns a pointer to said lambda, so be it. This is how C# works from my experience. If you allocate shared variables that you would need to free them later just as a ucontext would need to be freed. A ucontext or set of variables would only need to allocated if a lambda that accessed variables in the scope of the owning function was returned. In the upcoming standard for lambdas in C++, one has to indicate what variables in the owning functions scope are being accessed by the lambda. For the new C++ lambdas however, one can not obtain a real function pointer from them, so in my opinion they are of limited usefulness.
Created attachment 137481 [details] Simplified example of using ucontext to form a closure I took out the use of macros to make it clearer what is going on.
Created attachment 137506 [details] Example of closures using generated structs and GClosure This is what I have in mind. This approach would only preserve the accessed variables and would work with compilers other than GCC. Although it may be better to use something custom instead of GClosure as the wrapper. I haven't used GClosure before and all that GValue and marshaller stuff seems unnecessarily complicated to me. There are two closures in the example, each in different sub-scope to illustrate the principle. The callbacks don't access all the variables passed, but that's just to keep it simpler. In reality, valac would encapsulate only the variables that would be needed in lambda.
commit 1f14688e1d3508a2ba3c62322c362b468cea6bf8 Author: Jürg Billeter <j@bitron.ch> Date: Tue Sep 15 17:09:02 2009 +0200 Add support for closures Fixes bug 554781.