GNOME Bugzilla – Bug 600874
Sqlite backend: Save As Example: Image data not saved
Last modified: 2021-07-05 10:52:44 UTC
Looking at http://git.gnome.org/cgit/glom/tree/examples/example_smallbusiness.glom?id=a73e2570a0ab3ba50b7ebb672bcda495f2952e0d#n132 it seems as if binary data was interpreted as Unicode sequences: \000IDATx\312Ľۮ$Ir (and many more ocurrences of arbitrary XML entities). There shouldn't be any XML entities in a BLOB. For binary encoding, we could do it like that: http://stackoverflow.com/questions/19893/how-do-you-embed-binary-data-in-xml/19939#19939 A more serious solution is probably to just use base64 for BLOBs, even if that blows up size. (I assume this was originally introduced by handling the libgda issue of returning quotes in BLOBs? See http://git.gnome.org/cgit/glom/commit/?id=c50cd3fe2b8028fda43520e34cc23f031792bd19). Checking with a current version of Glom, I found out that I could not save images into an example glom file at all.
Regarding the image saving problem: I got lots of warnings when testing it. "** (glom:8591): WARNING **: Conversions::get_pixbuf_for_gda_value(): Failed to read BLOB data"
It's not a blob, whatever that means in the context of an XML document. It's text in an XML document that is later read in and interpreted (by libgda) as binary data. > For binary encoding, we could do it like that: We already use a libgda function for this, and we already use the standard text encoding of the XML parser/builder. > I got lots of warnings when testing it. Right, so that's an actual bug, rather than an imagined one. This seems like it should have a unit test.
Well, the first question is whether storing pixbufs in the Glom document is still a required functionality. We only needed that for the example documents anyway, because the usual way is to store data in the db backend, not the document itself Then, we don't just use the gda API for conversions. There is now additional code that converts newlines and quotes. Are those conversion safely reverted (or revertable at all?) before asking gda to convert the text to a GdaBinary struct? Seeing how fixing this might affect multiple places in Glom the above question (whether we really want to support this, even it is only used in the example documents) starts to make sense, no?
(In reply to comment #3) > Well, the first question is whether storing pixbufs in the Glom document is > still a required functionality. We only needed that for the example documents > anyway, because the usual way is to store data in the db backend, not the > document itself Yes, it's still required, for example data. Why would we suddenly not want pictures in the example data? > Then, we don't just use the gda API for conversions. There is now additional > code that converts newlines and quotes. Are those conversion safely reverted > (or revertable at all?) before asking gda to convert the text to a GdaBinary > struct? Yes, I believe.
"Yes, it's still required, for example data. Why would we suddenly not want pictures in the example data?" 1. It currently doesn't work. Fixing it might be hard, looking at all the related commits from October (many potential regressions). 2. We already have a non-db data format, CSV files. Example Glom documents are only ever used to create Glom documents from them, while the data goes to the choosen db backend. Why not just have CSV example files, together with the example Glom documents? Create a document from a template, fill it with data from the CSV file. I'd rather work on improving CSV support for Glom than fixing a bug in a feature that is so rarely used. Storing real data in a Glom document also smells of duplicated effort, and we have to maintain more code this way.
If things don't work we should fix them, not just discard the feature.
And, to get you in the habit, please provide instructions for reproducing this bug. And please retitle the bug to describe the actual problem.
I created another bug @ https://bugzilla.gnome.org/show_bug.cgi?id=600977 for the <example_rows> vs. external CSV files debate. The above mentioned bug - XML entities in what is supposed to be a text representation of a gda BLOB - remains valid unless the linked debate discards the idea of <example_rows>.
What above mentioned bug? You still haven't provided instructions for reproducing any bug. You have the following incredibly annoying habits: a) Filing bug reports about imagined causes or proposed solutions rather than actual problem symptoms b) Leaving bug reports with unclear status.
This "\000IDATx\312Ľۮ$Ir" _IS_ the bug! The XML entities shouldn't be there. I even linked the commit that caused this change. This is not imagined, it is there. That at some later point the application crashes or behaves wierd because we feed invalid data to it only follows. It is really not worth to look at the symptoms unless we know what exactly causes them.
Created attachment 147146 [details] converts forth and back, using gda_binary_to_string () and gda_string_to_binary () This testcase reads from filein and shows the length before/after.
Created attachment 147147 [details] binary data represented as text, taken from here: http://git.gnome.org/cgit/glom/tree/examples/example_smallbusiness.glom?id=a73e2570a0ab3ba50b7ebb672bcda495f2952e0d#n132 Use the test programm and the test data and you'll see that gda cannot cope with this input. Which is not really surprising, as it expects the same binary-as-text representation to go into "gda_string_to_binary()" that was created with "gda_binary_to_string()" before. I hope it is now clear that this *really is* an conversion issue, and that the first thing to do is to find out where it happens. When I searched for that (even before submitting the bug) I couldn't find it, so perhaps someone else can help. That was the whole purpose of this bugreport, initially. If I had quickly found the issue myself, I wouldnt have to report it.
(In reply to comment #10) > This "\000IDATx\312Ľۮ$Ir" _IS_ the bug! The XML entities shouldn't > be there. I even linked the commit that caused this change. This is not > imagined, it is there. No, that's not an acceptable bug report. Again, please provide clear step-by-step instructions to reproduce some problem as experienced by a user. You are again talking about some aspect of a proposed cause or solution, instead of starting with an actual bug report. I am running out of patience.
Created attachment 147261 [details] An example image STEPS TO REPRODUCE THE PROBLEM: 1. Start a new project from an empty template, choosing the sqlite backend. 2. Create the first table. 3. In data mode, add an image column (Developer > Field, still in list view). 4. Switch to details view, go into layouts dialog. 5. Add the image column to the layout. 6. Back in details view, click the "no image" thumbnail to add an image (see attachment for an example image). 7. Save the document as example (data mode, as developer: File > Save as example) 8. Look at the created Glom XML file and search for the example_rows tag, it contains no data for the image. EXPECTED OUTCOME: The created example document should contain the image. ACTUAL OUTCOME: The created example document doesn't contain the image. REPRODUCIBILITY: always
Thanks. 8 should really be more user steps, because a user will not look at the XML, or care what it says. However, this is far better.
This does work when using the PostgreSQL backend. It doesn't when using a sqlite-based .glom file.
Created attachment 147386 [details] [review] glomdebug.patch Add some std::out debug output that show the problem in Field::to_file_format() and which even causes a crash.
Something is going very wrong with the GValue that should contain a GdaBinary. I guess it's a libgda problem (when using sqlite) though we'll need a simple test case to show that. With the above glomdebug.patch we see that the GType has a silly value and that there's a memory problem at that point when using valgrind: ===18989== Syscall param write(buf) points to uninitialised byte(s) ==18989== at 0x5721C0B: ??? (in /lib/tls/i686/cmov/libpthread-2.10.1.so) ==18989== by 0x535D0AD: sqlite3OsWrite (sqlite3.c:11923) ==18989== by 0x537233C: writeJournalHdr (sqlite3.c:31375) ==18989== by 0x5372430: pager_open_journal (sqlite3.c:34584) ==18989== by 0x53724C1: sqlite3PagerBegin (sqlite3.c:34647) ==18989== by 0x5385198: sqlite3BtreeBeginTrans (sqlite3.c:39341) ==18989== by 0x53A96BD: sqlite3VdbeExec (sqlite3.c:53614) ==18989== by 0x539A71B: sqlite3_step (sqlite3.c:49497) ==18989== by 0x53A66AD: sqlite3_blob_open (sqlite3.c:56737) ==18989== by 0x5347BDF: _gda_sqlite_blob_op_new (gda-sqlite-blob-op.c:134) ==18989== by 0x53571F3: fetch_next_sqlite_row (gda-sqlite-recordset.c:428) ==18989== by 0x5357A75: _gda_sqlite_recordset_new (gda-sqlite-recordset.c:160) ==18989== by 0x5354C1F: gda_sqlite_provider_statement_execute (gda-sqlite-provider.c:2484) ==18989== by 0x52C3643: gda_connection_statement_execute_v (gda-connection.c:2259) ==18989== by 0x52CAE15: gda_connection_statement_execute_select (gda-connection.c:2451) ==18989== by 0x412E57E: Gnome::Gda::Connection::statement_execute_select(Glib::RefPtr<Gnome::Gda::Statement> const&, Glib::RefPtr<Gnome::Gda::Set> const&) (connection.cc:1030) ==18989== by 0x80A2861: Glom::Base_DB::query_execute_select(Glib::ustring const&, Glib::RefPtr<Gnome::Gda::Set> const&) (base_db.cc:237) ==18989== by 0x80D889B: Glom::Frame_Glom::export_data_to_vector(std::vector<std::vector<Gnome::Gda::Value, std::allocator<Gnome::Gda::Value> >, std::allocator<std::vector<Gnome::Gda::Value, std::allocator<Gnome::Gda::Value> > > >&, Glom::FoundSet const&, std::vector<Glom::sharedptr<Glom::LayoutGroup>, std::allocator<Glom::sharedptr<Glom::LayoutGroup> > > const&) (frame_glom.cc:692) ==18989== by 0x808BBD9: Glom::App_Glom::on_menu_file_save_as_example() (application.cc:2108) ==18989== by 0x4E3FB51: Glib::SignalProxyNormal::slot0_void_callback(_GObject*, void*) (slot.h:440) ==18989== by 0x54D102D: g_cclosure_marshal_VOID__VOID (gmarshal.c:77) ==18989== by 0x54C326B: g_closure_invoke (gclosure.c:767) ==18989== by 0x54DAAA8: signal_emit_unlocked_R (gsignal.c:3317) ==18989== by 0x54DBA3A: g_signal_emit_valist (gsignal.c:2980) ==18989== by 0x54DBF7F: g_signal_emit (gsignal.c:3037) ==18989== by 0x4A8090B: _gtk_action_emit_activate (gtkaction.c:727) ==18989== by 0x4A829F6: gtk_action_activate (gtkaction.c:757) ==18989== by 0x4B7549A: gtk_real_menu_item_activate (gtkmenuitem.c:1399) ==18989== by 0x54D102D: g_cclosure_marshal_VOID__VOID (gmarshal.c:77) ==18989== by 0x54C1808: g_type_class_meta_marshal (gclosure.c:878) ==18989== Address 0x5e5a611 is 9 bytes inside a block of size 1,032 alloc'd ==18989== at 0x4024C1C: malloc (vg_replace_malloc.c:195) ==18989== by 0x536E0F4: sqlite3MemMalloc (sqlite3.c:12336) ==18989== by 0x535D695: mallocWithAlarm (sqlite3.c:15524) ==18989== by 0x535D737: sqlite3Malloc (sqlite3.c:15552) ==18989== by 0x535F05E: pcache1Alloc (sqlite3.c:29502) ==18989== by 0x535F172: sqlite3PageMalloc (sqlite3.c:29573) ==18989== by 0x5365BFC: sqlite3PagerSetPagesize (sqlite3.c:32896) ==18989== by 0x5374FF0: sqlite3BtreeFactory (sqlite3.c:33827) ==18989== by 0x537D34F: openDatabase (sqlite3.c:92569) ==18989== by 0x537D64C: sqlite3_open (sqlite3.c:92683) ==18989== by 0x5351F1C: gda_sqlite_provider_open_connection (gda-sqlite-provider.c:625) ==18989== by 0x52CC8F7: gda_connection_open (gda-connection.c:1245) ==18989== by 0x52CD0F1: gda_connection_open_from_string (gda-connection.c:1095) ==18989== by 0x412FC43: Gnome::Gda::Connection::open_from_string(Glib::ustring const&, Glib::ustring const&, Glib::ustring const&, Gnome::Gda::ConnectionOptions) (connection.cc:72) ==18989== by 0x40D316F: Glom::ConnectionPoolBackends::Sqlite::connect(Glib::ustring const&, Glib::ustring const&, Glib::ustring const&, std::auto_ptr<Glom::ExceptionConnection>&) (sqlite.cc:69) ==18989== by 0x4062120: Glom::ConnectionPool::connect() (connectionpool.cc:257) ==18989== by 0x40639EB: Glom::ConnectionPool::get_and_connect() (connectionpool.cc:182) ==18989== by 0x809BFF0: Glom::Base_DB::connect_to_server(Gtk::Window*) (base_db.cc:129) ==18989== by 0x80D96CA: Glom::Frame_Glom::connection_request_password_and_attempt(bool&, Glib::ustring, Glib::ustring const&, bool) (frame_glom.cc:2567) ==18989== by 0x80987AB: Glom::App_Glom::on_document_load() (application.cc:1111) ==18989== by 0x80F94FA: GlomBakery::App_WithDoc::open_document(Glib::ustring const&) (app_withdoc.cc:130) ==18989== by 0x808ED38: Glom::App_Glom::offer_new_or_existing() (application.cc:1359) ==18989== by 0x808F4A3: Glom::App_Glom::init(Glib::ustring const&) (application.cc:185) ==18989== by 0x80EC175: main (main.cc:615) ==18989==
(In reply to comment #11) > Created an attachment (id=147146) [details] > converts forth and back, using gda_binary_to_string () and gda_string_to_binary > () > > This testcase reads from filein and shows the length before/after. I doubt that this is relevant, but if you think this should work then please convert it to C and submit it as a libgda bug.
I do now see the image data in the saved-as-example .glom file, but it is obviously truncated. For instance: <value column="picture">\377\330\377\340\000\020JFIF\000\001\001\001\000H\000H\000\000\377\376\000\003\012\377\333\000C\000\001\001\001\001\001\001\001\001\001\001</value> The format at least seems to be correct otherwise. This is what the original example contains: <value column="picture">\377\330\377\340\000\020JFIF\000\001\001\001\000H\000H\000\000\377\333\000C\000\001\001\001\001\001\001\001\001\001\001 ... Note that a related problem was fixed in libgda a while ago in bug #663357 .
Also, though this might not be related, I notice that images don't really seem to be saved in the SQLite database after I close and reopen the Glom file (stopping and restarting Glom). When I choose a new image file, I still see it when navigating to the next record and then back, but when I restart Glom I see the original image again. I wonder if there is some flush-to-disk issue with sqlite.
GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org. As part of that, we are mass-closing older open tickets in bugzilla.gnome.org which have not seen updates for a longer time (resources are unfortunately quite limited so not every ticket can get handled). If you can still reproduce the situation described in this ticket in a recent and supported software version, then please follow https://wiki.gnome.org/GettingInTouch/BugReportingGuidelines and create a new ticket at https://gitlab.gnome.org/GNOME/glom/-/issues/ Thank you for your understanding and your help.