Microsoft Exchange and Novell GroupWise both offer SOAP-based client interfaces to fetch and send email.

Email messages (with attachments) may be *large*; we don't want to have the whole thing in memory. We *especially* don't want *multiple* copies in memory.

So we don't want to take the naïve approach of base64-encoding the whole message into a buffer in RAM, then setting that as the content of an XML node, then getting another copy of the whole message when we flatten the XML document into a buffer for transmission.

On the receive side, this is solved by feeding the data to xmlParseChunk() as it arrives, then overriding the sax2 characters() method to "steal" the contents of specific nodes and spool them to a disk file as they arrive:
http://git.gnome.org/browse/evolution-ews/tree/src/server/e-soap-message.c?id=1a2ff1aa#n137

I can't work how how to do this for transmission though. The libsoup part is solvable (bug 637845) but there seems to be no easy way to actually create the output 'request' in a streaming mode.

I looked briefly at using xmlSaveToIO() and my own iowrite() callback, in the hope that at least I'd get the flattened output "broken down" so that it was simple to match on it and detect when the 'special' node is being sent... but the buffering to MINLEN in XmlOutputBufferWrite() defeats that.

That basically means I am re-parsing the output of libxml, to find the appropriate place to insert the data. Surely there must be some better way?

(In fact, the need to post-process and reparse the output means that I might as well ditch xmlSaveToIO() and just create the flattened XML document with xmlDocDumpMemory() and re-parse *that* as I send it.)

It'd be really nice if there was some 'special' type of XML text node where the content wasn't actually present in node->content, but instead has a callback function to create it on demand. Or perhaps the XInclude support could be extended to work in this fashion. The idea is that the *full* content of the node would never need to be in memory all at once.

So the callback to create the data for this "special" text node would need *not* to return the whole of the data all at once, like the current XInclude read callback does. It would need to be able to say "here's some data; call me again when you need some more". Or perhaps we'd just pass an xmlOutputBuffer to it, and it would shovel the data straight into that?