GNOME Bugzilla – Bug 321037
synchronize notes across multiple machines
Last modified: 2008-02-27 01:30:53 UTC
The problem I'd like to have solved is being able to work in multiple instances of tomboy (desktop, notebook) on a single set of notes. Perhaps this could be linked with the HTML export somehow. The technical proposal is most likely flawed, but what about the following scenario: * The location when html exporting note set (using gnome-vfs) is stored. Export can be automated/polled periodically? * New Sync plugin is polling the location for changes. Diff is used to check for updates. Alternatively a wiki could be used instead of xml/html export.
I thought there was a tracker bug for this, as it has been proposed about 20 times already, but I can't find it. The current plan is to support using mDNS via Avahi to share notes. I have no time to work on it, so when this will happen I have no idea.
Created attachment 58592 [details] Remoting/Avahi Server/Client
Created attachment 58593 [details] Note Synchronisation Window
Created attachment 58594 [details] [review] Avahi/Networking patch This patches enable tomboy to simultaneously share notes, announce them, receive announcements and get notes from other tomboys. A textview for synchronisation has been added.
Created attachment 69532 [details] [review] All in one and update patch for Mario Manno's Avahi mods. I took Mario Manno's mods above and manually merged them into the current CVS HEAD then debugged them (fixed a few API calls etc.) I moved the public Network class into its own file Network.cs. Made a patch with the 3 new Network*.cs files contained. Unfortunately I can't get the CVS HEAD of Tomboy to start the panel applet right now (patched or otherwise). Will PM Alex about that though to see if this can be made to work. If someone else can test this in the meantime that would be great. NB. If compiled with --enable-avahi=yes (the default) you must have the Avahi daemon running or Tomboy just dies... The next version should handle this gracefully.
(In reply to comment #5) > If someone else can test this in the meantime that would be great. NB. If > compiled with --enable-avahi=yes (the default) you must have the Avahi daemon > running or Tomboy just dies... The next version should handle this gracefully. I tried your patch, and it works up to a point. It does connect and get the notes, but it puts the most recently changed remote (Note A) in $HOME/.tomboy/ and all others in $HOME/.tomboy/Remote/, so the only notes listed in the table of contents are Note A and the default Start Here. When I open A and click on a link to B, tomboy can't find it and creates a new note named B. The [Remote] notes are still in .tomboy/Remote/, but once I restart tomboy they are gone.
(In reply to comment #6) > I tried your patch, and it works up to a point. It does connect and get the > notes, but it puts the most recently changed remote (Note A) in $HOME/.tomboy/ > and all others in $HOME/.tomboy/Remote/, so the only notes listed in the table > of contents are Note A and the default Start Here. The 'Remote' directory is necessary as a temporary directory, because tomboy needs to keep every note, instantiated, in memory. > When I open A and click on a link to B, tomboy can't find it and creates a new > note named B. This is the intended behaviour if you don't copy all notes. I think the 'copy all' command was kind of buggy. The patch is not really complete, some changes were delayed until tomboy 0.4 and it was recommended to include password protection. I still think it would be great to switch the networking from mono 'Remoting' to a https server. Since there was no feedback I stopped working on this patch. Note that the google summer of code project also was about tomboy networking. Apart from that there is some talk on the tomboy mailing list about using db4o.
This has been chosen as one of the main features for Tomboy 0.8.0 (the next stable release). For a bit more information, please see the developer meeting notes: http://live.gnome.org/Tomboy/DevMeeting08Mar2007
Created attachment 90603 [details] [review] Proof-of-concept of workflow discussed in IRC Got a bad case of NIH and decided to implement a proof-of-concept of the workflow Boyd and I discussed as documented at http://live.gnome.org/Tomboy/Synchronization. Of course I'm a lazy bum so it's taken me forever to upload this. Design's pretty straightforward. You have a SyncServer for each type of central store you might want. We're looking at doing WebDAV, but I just implemented a basic FileSystemSyncServer (should be easy to swap some file manipulation methods and reuse it). You have a SyncClient to represent the status of your local instances of the notes. I just made a basic client that's based on our existing .tomboy files, but conceivably this could instead work with a Subversion working copy, etc. Then there's the SyncManager to tie it all together and actually implement the sync algorithm. There's a SyncDialog (creative, yeah?) that represents the status of the current sync, modeled after a dialog I liked in Synaptic. And then there's the SyncTitleConflictDialog, which lets you know when you're pulling a note update whose title conflicts with a note you've created since your last sync (think Start Here, NotD, etc). The dialog gives you the option to delete your local note or rename it, or just cancel sync. Major problems: * Setting Note.XmlContent just doesn't work for trying to reset the note's XML (see bug #440562). You *WILL* experience data loss. * Setting Note.Title programatically doesn't really work for renaming notes, either. * Although I've modeled the sync as if it were a transaction, it really only protects you from trying to sync simultaneously from different clients. If the network gets pulled half way through, you're left in an unpredictable state. :-( Presumably somebody smarter than me can make this better. * Totally ignoring conflicts right now, though I figure we can deal with it the same way as title conflicts addressed earlier in this comment. Other problems: * The UI needs to be prettied up, of course. Lots of missing things...not all updates are shown in SyncDialog, etc. You start sync from the Tools menu in a note window. * The general algorithm isn't very tested. * Some basic premises of this implementation may be completely wrong (ie, storing revision number in the note XML directly). * I've ignored versioning, hoping that the resolution of the Note.XmlContent could take care of that. * No support for Tasks. * Just a bad patch in general...missing Catalog.GetString calls, named variables poorly, lots of inline TODOs, etc. I just wanted to get this up before tomorrow in case it's useful. REMEMBER: THIS PATCH IS EVIL AND WILL EAT YOUR NOTES. I like testing with some dummy users in XNest sessions, personally.
Oh, quick comment: one obvious solution to the XML update problem is just to toss the old note and treat the update like a new note. However, that's no good if we want specific note metadata to not get synced (maybe note location, size, pinned, etc).
Created a branch so we can work on this collaboratively: http://svn.gnome.org/viewcvs/tomboy/branches/tomboy-sync/
(In reply to comment #9) > Major problems: > * Although I've modeled the sync as if it were a transaction, it really only > protects you from trying to sync simultaneously from different clients. If the > network gets pulled half way through, you're left in an unpredictable state. > :-( Presumably somebody smarter than me can make this better. Couldn't we: 1. Upload all the changes to a temporary location on the server 2. Once all the files are in place, move them to the proper location I guess it's still possible for that to have problems too. Maybe this is why having a real server involved is important; it could still be possible that the network be disconnected between steps 1 and 2.
Some new thoughts on transaction for client sync process (if no lock file present on server): 1. Figure out what the latest server revision is 2. Begin "work" on revision + 1 by writing lock file with revision + 1 included 3. Change note files that need to be modified by renaming the note file to *.note.old 4. Upload new/modified files as *.note.new 5. Rename all *.note.new to *.note 6. Delete *.note.old files 7. Remove lock file Other clients can know whether the previous sync cycle was successful by inspecting the lock file. If the lock file is expired, the client has some cleanup work to do. If the lock file hasn't expired, the client just waits a little longer. Cleanup process: 1. Remove *.note.new 2. Rename *.note.old *.note 3. Remove revision manifest file (specified in the lock file) 4. Begin own sync cycle
Makes sense. Seems like a clean approach. When talking to a network server (like, say, WebDAV), is a copy expensive? As in, is it equivalent to download+upload, or does the server handle it? If copy==download+upload, then this strategy uses 3X the bandwidth (though it's probably worth it for the safety involved). What if the network dies between steps 6 and 7? In that case I think the next syncing client should assume that all went well and NOT delete the new revision manifest file. It can check this by noticing that there are no *.note.old files, but the lock file still exists. But then, what if the network dies during the cleanup process after step 2? The next syncing client still sees the lock file, but there are no *.note.old files. In this case it would be wrong to assume that all went well, because the revision manifest file *should* be deleted. How can a client differentiate between these two scenarios without checking the revision number on each note and comparing it to the latest revision manifest file? If only we could trust the mtimes on each file... Anyway, I have no better ideas here.
(In reply to comment #14) > When talking to a network server (like, say, WebDAV), is a copy expensive? As > in, is it equivalent to download+upload, or does the server handle it? If > copy==download+upload, then this strategy uses 3X the bandwidth (though it's > probably worth it for the safety involved). I don't believe that we'll need to do any copying, just renaming. Hopefully our choice of protocol will do renames/moves cleanly w/o incurring download+upload. > What if the network dies between steps 6 and 7? In that case I think the next > syncing client should assume that all went well and NOT delete the new revision > manifest file. It can check this by noticing that there are no *.note.old > files, but the lock file still exists. There should be a step 6.5 that uploads a new manifest file for the revision. If the network dies between 6 & 7 and there's no manifest file, the subsequent client could assume the sync failed and needs to be cleaned up. > But then, what if the network dies during the cleanup process after step 2? > The next syncing client still sees the lock file, but there are no *.note.old > files. In this case it would be wrong to assume that all went well, because > the revision manifest file *should* be deleted. > > How can a client differentiate between these two scenarios without checking the > revision number on each note and comparing it to the latest revision manifest > file? If only we could trust the mtimes on each file... Anyway, I have no > better ideas here. We could go as far as to md5sum/checksum each note file and include the checksum in the manifest file. But perhaps that's overkill?
(In reply to comment #15) > There should be a step 6.5 that uploads a new manifest file for the revision. > If the network dies between 6 & 7 and there's no manifest file, the subsequent > client could assume the sync failed and needs to be cleaned up. But after step 6, all the notes are properly updated for the new revision. If there is a failure after step 6, there is no going back to the previous note revision. The next syncing client would have to figure out what changed and generate a new revision manifest file anyway. > We could go as far as to md5sum/checksum each note file and include the > checksum in the manifest file. But perhaps that's overkill? That's an interesting idea that may save some processing if we're already forced to download each note file anyway.
Here's a somewhat new design that will (I think) guarantee that we won't have data loss/corruption if the network connection gets hosed during mid-sync. Feedback appreciated... Everything is driven off of a /manifest.xml file off of the server. "/manifest.xml" contains a line for every note file. ---------------------- Example manifest.xml ---------------------- <sync revision="47"> <note file="abc.note" path="/0/1" /> <!-- Stored in Revision 1 --> <note file="jkl.note" path="/0/3" /> <!-- Stored in Revision 3 --> <note file="xyz.note" "/0/47" /> <!-- Stored in Revision 47 --> </sync> ---------------------- How do you know if "/manifest.xml" is valid? Since it's XML, if it parses, it's valid. If it doesn't, the client has to go look at the directories to find out what the latest revision is (which I think the code we have already does some of this). ---------------------- General Sync Process ---------------------- 1. Determine what the new revision should be (latest revision + 1) 2. Create /${modulus-dir}/revision directory on the server 3. Upload all the new/updated files to /${modulus-dir}/revision 4. Create a new local manifest.xml file with any files to be deleted not listed 5. Upload the new local manifest.xml file to /${modulus-dir}/revision 6. (optional) Rename server's /manifest.xml -> /manifest.xml.old 7. Upload the new local manifest.xml to the server in /manifest.xml 8. (somewhat optional) Delete /manifest.xml.old and older versions of files that were updated/deleted. This only conserves disk space on the storage server. ------------------------- Possible Failure Points ------------------------- The next client that attempts to synchronize will have to check for these conditions: Step 6 ------- If no manifest.xml file exists or is corrupted (not valid XML), the subsequent client will have to discover the latest revision and look for a valid /${modulus-dir}/${rev}/manifest.xml and simply restore that manifest.xml file into the root directory and begin again. Step 7 ------- Same solution as mentioned to fix Step 6. Step 8 ------- This failure can be left undetected. The only drawback would be extra files being stored on the storage server. Eventually, we could implement some sort of cleanup that would walk every revision and clean out orphaned files. ---------- Comments ---------- - Probably a lot of extra overhead - Extra space taken by a bunch of manifest.xml files (one per revision) - Easy to clean-up and revert to the previous version (i.e., almost no potential for data loss) - Large manifest.xml files when LOTS of notes are created, but probably usable?
Gonna use this format for the manifest.xml file(s): <sync revision="47"> <note id="abc" rev="1" /> <note id="jkl" rev="3" /> <note id="xyz" rev="47" /> </sync>
Created attachment 91140 [details] [review] ArchLinux paths for SetUpTools
Contents of tomboy-sync have landed on trunk now. :-) I've committed a slightly modified version of the ArchLinux patch (don't need absolute path for lsmod anymore). Please test and respond here if there are still problems.
Resolving as FIXED. Any sync issues should be new bugs at this point.
Modifying default assignee and qa contact to be tomboy-maint@gnome.bugs.