After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 638468 - Problem to serve large files
Problem to serve large files
Status: RESOLVED FIXED
Product: libsoup
Classification: Core
Component: HTTP Transport
2.34.x
Other Linux
: Normal normal
: ---
Assigned To: libsoup-maint@gnome.bugs
libsoup-maint@gnome.bugs
Depends on:
Blocks:
 
 
Reported: 2011-01-01 22:47 UTC by Jens Georg
Modified: 2011-06-29 16:18 UTC
See Also:
GNOME target: ---
GNOME version: ---

Attachments
logfile (13.29 KB, application/x-xz)
2011-05-01 15:09 UTC, Jens Georg 		  Details
Sample code (1.64 KB, text/x-csrc)
2011-05-08 14:07 UTC, Jens Georg 		  Details
Fix issue (796 bytes, patch)
2011-06-25 10:58 UTC, Jens Georg 		committed Details | Review

Description Jens Georg 2011-01-01 22:47:03 UTC 
The WD TV live requests some HD material and we serve it like
** (rygel:27857): DEBUG: rygel-http-get.vala:130: Following HTTP headers appended to response:
** (rygel:27857): DEBUG: rygel-http-get.vala:132: Date : Sat, 01 Jan 2011 20:02:19 GMT
** (rygel:27857): DEBUG: rygel-http-get.vala:132: Content-Type : video/x-matroska
** (rygel:27857): DEBUG: rygel-http-get.vala:132: Accept-Ranges : bytes
** (rygel:27857): DEBUG: rygel-http-get.vala:132: Content-Range : bytes 5678-8067343737/8067343738
** (rygel:27857): DEBUG: rygel-http-get.vala:132: Content-Length : 8067338060
** (rygel:27857): DEBUG: rygel-http-get.vala:132: contentFeatures.dlna.org : *

the duration is ~2hrs and after 1 hour rygel or libsoup freaks out and spams the log with assertions. Log will be provided later.

As this happens more or less exactly at the middle of the file, I assume that some uint32 somewhere overflows (4Gb), either within rygel or libsoup.
Comment 1 Jens Georg 2011-01-02 15:16:54 UTC 
Log is at http://jensge.org/rygel.log.bz2 (~150Mb extracted)
Comment 2 Zeeshan Ali 2011-04-19 21:04:34 UTC 
Is this reproducible after latest changes in streaming code?
Comment 3 Jens Georg 2011-04-19 21:11:02 UTC 
Have to check when I'm back home.
Comment 4 Jens Georg 2011-05-01 15:09:07 UTC 
Created attachment 186975 [details]
logfile
Comment 5 Jens Georg 2011-05-01 15:39:54 UTC 
Still there
Comment 6 Jens Georg 2011-05-08 12:24:40 UTC 
Backtrace from first critical, doesn't look very helpful :-/

+ Trace 227044

  • #0 __kernel_vsyscall
  • #1 raise
    at ../nptl/sysdeps/unix/sysv/linux/raise.c line 64
  • #2 abort
    at abort.c line 92
  • #3 g_logv
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./glib/gmessages.c line 557
  • #4 g_log
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./glib/gmessages.c line 577
  • #5 g_return_if_fail_warning
  • #6 soup_message_body_wrote_chunk
    from /usr/lib/libsoup-2.4.so.1
  • #7 ??
    from /usr/lib/libsoup-2.4.so.1
  • #8 g_cclosure_marshal_VOID__VOID
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./gobject/gmarshal.c line 79
  • #9 g_closure_invoke
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./gobject/gclosure.c line 767
  • #10 signal_emit_unlocked_R
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./gobject/gsignal.c line 3252
  • #11 g_signal_emit_valist
  • #12 g_signal_emit
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./gobject/gsignal.c line 3040
  • #13 ??
    from /usr/lib/libsoup-2.4.so.1
  • #14 pollable_source_dispatch
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./gio/gpollableinputstream.c line 233
  • #15 g_main_dispatch
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./glib/gmain.c line 2440
  • #16 g_main_context_dispatch
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./glib/gmain.c line 3013
  • #17 g_main_context_iterate
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./glib/gmain.c line 3091
  • #18 g_main_loop_run
    at /build/buildd-glib2.0_2.28.6-1-i386-A3fp41/glib2.0-2.28.6/./glib/gmain.c line 3299
  • #19 main 






Comment 7


Jens Georg





          2011-05-08 12:56:59 UTC
        



Failing part of the assertion is 
((SoupBufferPrivate *)chunk2)->use == SOUP_MEMORY_TEMPORARY:





+
Trace
    227045






    

  • 
#6
soup_message_body_wrote_chunk

    
                at soup-message-body.c
                  line
                  699

    

    • 







$1 = (SoupBuffer *) 0x851b508
(gdb) print chunk2
$2 = (SoupBuffer *) 0x865f9b0
(gdb) print ((SoupBufferPrivate *)chunk2)->use
$3 = SOUP_MEMORY_TAKE






Comment 8


Jens Georg





          2011-05-08 13:00:13 UTC
        



Ok, I suspect SoupBuffer having a gsize as length which is 32bit on 32bit systems is the culprit. Patching libsoup now for verification (Good that it only takes 1.3hrs to reproduce this bug…)






Comment 9


Zeeshan Ali





          2011-05-08 13:01:30 UTC
        



Looks like a libsoup bug to me so re-assigning..






Comment 10


Jens Georg





          2011-05-08 14:07:51 UTC
        



Created attachment 187459 [details]
Sample code

Example program. Able to serve the full 7.5 Gig on amd64, but fails on x86, roughly here (wget -O/dev/null http://localhost:12345/foo output):

53% [===============================================>                                           ] 4,295,032,832 33.8M/s  eta 67s






Comment 11


Jens Georg





          2011-06-25 10:58:43 UTC
        



Created attachment 190642 [details] [review]
Fix issue

SoupMessageIOData.write_body_offset was overflowing on 32bit since gsize is only 32bit there. This patch makes it a goffset which is 64 bit.






Comment 12


Dan Winship





          2011-06-29 16:18:04 UTC
        



fixed in git. thanks