GNOME Bugzilla – Bug 707157
traceback from partial fetch code
Last modified: 2013-09-02 18:53:28 UTC
Receiving objects: 6% (64/989) 294.6 kB[Thread 0x7ffff4c7a700 (LWP 4729) exited] (ostree pull:4725): GLib-GIO-CRITICAL **: g_simple_async_result_take_error: assertion `G_IS_SIMPLE_ASYNC_RESULT (simple)' failed Program received signal SIGTRAP, Trace/breakpoint trap. g_logv (log_domain=0x7ffff7683872 "GLib-GIO", log_level=G_LOG_LEVEL_CRITICAL, format=<value optimized out>, args=<value optimized out>) at gmessages.c:981 981 g_private_set (&g_log_depth, GUINT_TO_POINTER (depth)); Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 libacl-2.2.49-6.el6.x86_64 libarchive-2.8.3-4.el6_2.x86_64 libattr-2.4.44-7.el6.x86_64 libffi-3.0.5-3.2.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 openssl-1.0.1e-8.el6.x86_64 sqlite-3.6.20-1.el6.x86_64 xz-libs-4.999.9-0.3.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt
+ Trace 232438
$1 = (OstreeFetcherPendingURI *) 0xb17a30 (gdb) p *$ $2 = {refcount = 2, self = 0x63dc50, uri = 0xb16e00, state = OSTREE_FETCHER_STATE_PENDING, request = 0xb03b20, tmpfile = 0x7ffff00b7f60, filename = 0xb19a80 "295406e24543834532859ebb2014666c3cc49dfd52d6be66ccf4b4d205306b14", request_body = 0x0, out_stream = 0x0, content_length = 0, cancellable = 0x0, result = 0x0} (gdb) p pending->result $3 = (GSimpleAsyncResult *) 0x0
Created attachment 253665 [details] [review] fetcher: Clean up code to avoid intermediate files I think originally we had the .part/.done separation because we were trying to support partial downloads of files like repo/config and repo/refs. But now that the http server configuration won't give us partial results, we don't need to support caching those files between runs. And thus, there's no reason to have the .part/.done and do the dance with renaming them. When fetching objects/ and other things that use _with_async, we continue to use _append_to(), and if the returned range tells us we have all the bytes, then we hand the full file over to the caller. Don't attempt to shortcut in the case where the last run told us we already have the object; the object fetcher code will not make a request. While we're here, also clean up use of GError and consistently use the cancellable from the pending.
But this is still buggy - we have a file descriptor open per HTTP request, which for large pulls quickly causes us to abort. Need to only open the tmpfile once we actually start getting results back from the server.
Created attachment 253688 [details] [review] fetcher: Only open files when we are ready to write to them Otherwise we quickly run out of file descriptors when doing large requests.
Created attachment 253691 [details] [review] libostree: Change synchronous fetching API to return a stream There's not a good reason to write small things such as repo/config to the filesystem, only to read them back in again. Change the non-partial API to just return a stream, then read it into a memory buffer.
Looks good to me. go for it.
*** Bug 707300 has been marked as a duplicate of this bug. ***
Attachment 253665 [details] pushed as 8b5f684 - fetcher: Clean up code to avoid intermediate files Attachment 253688 [details] pushed as 95a7854 - fetcher: Only open files when we are ready to write to them Attachment 253691 [details] pushed as 597da6c - libostree: Change synchronous fetching API to return a stream