iCloud Sync

User avatar
devinganger
Posts: 1977
Joined: Sat Nov 06, 2010 1:55 pm
Platform: Mac, Win + iOS
Location: Monroe, WA 98272 (CN97au)
Contact:

Mon Apr 13, 2020 10:17 pm Post

parf wrote:Well, I still don't understand how someone dares to blame users for bad design decisions. Like a bad cook blames visitors for not having taste buds developed enough.


There is a large difference between "bad design decisions" and design constraints. Many years worth of electrons have lost their little quantum lives here in the forums talking about the designers' goals and objectives with how Scrivener is designed and how those decisions have consequences on down the line.

In any event, in all of my years in computing (back from CP/M on forward) attempts to user-proof a system have never been a substitute for user education. This notion that a user doesn't have to know anything about the constraints an application runs under and that the application will magically mitigate (when possible) and resolve (when required) all the potential error conditions under the hood really only applies when that is one of the initial design objectives and, usually, even then, falls short of the mark.

Scrivener is *not* one of those programs. One does, in fact, have to have a functional understanding of how it works in order to get the best performance from it, and realize that it has constraints. If one cannot (or does not want to) accept those constraints...well, perhaps Scrivener is not the right software for the one. Not every tool is right for every user -- if they were, there wouldn't be so many of them!
--
Devin L. Ganger, WA7DLG
Not a L&L employee; opinions are those of my cat
Winner "Best in Class", 2018 My First Supervillain Photo Shoot

Ci
Cirrocumulus
Posts: 15
Joined: Tue Sep 03, 2019 7:39 pm
Platform: Mac

Sun Jun 21, 2020 8:09 am Post

I've read KB's detailed explanation above of the hybrid document/shoebox nature of Scrivener. I've also read this article. I understand things aren't quite so simple.

What I don't understand is how the document structure of Scrivener differs from something like DEVONThink, an app which I've been using for years. DT can hold several huge databases inside file packages which can be saved anywhere (like Scrivener), has many different UI elements for showing and handling extra metadata (like Scrivener), only updates specific files on disk when modified instead of saving the entire file (like Scrivener). It also has an iOS client which can handle partial sync of data, doesn't attempt to offer feature-parity with the desktop app, can do (encrypted) sync via Dropbox, iCloud or WebDAV *and* handle sync discrepancies gracefully by creating copies of items.

This is not another attempt to say "app X does it, you should do the same". I am interesting in knowing in what sense Scrivener's database structure is so different from what I've described above. I couldn't find details about it in KB's post. Apologies if this specific question has been asked before.

Thank you.

User avatar
lunk
Posts: 3970
Joined: Wed Aug 21, 2013 4:24 pm
Platform: Mac + iOS
Location: Sweden 64° N

Sun Jun 21, 2020 8:55 am Post

Cirrocumulus wrote:I am interesting in knowing in what sense Scrivener's database structure is so different from what I've described above. I couldn't find details about it in KB's post. Apologies if this specific question has been asked before.

It has been asked and answered. Only Dropbox has provided a sufficiently detailed API to let KB write the necessary sync code.
I am a user, writing non-fiction and science, using:
* Mac Scrivener 3 on a Macbook 12”, MacBook Pro 13”, and iMac 27”, all running Mojave.
* iOS Scrivener 1 on an iPhone 11 Pro, iPad Air 9.7”, and iPad Pro 12.9”, all running the latest iOS

mb
mbbntu
Posts: 1183
Joined: Wed Aug 01, 2007 9:44 am
Platform: Mac + iOS
Location: Cambridge, UK.

Sun Jun 21, 2020 8:57 am Post

It's nothing to do with any difference between DEVONthink databases and Scrivener projects. It is that DEVONthink's developers spent a LONG time developing their own sync mechanisms. They started this some time in 2015 or 2016 (see https://www.devontechnologies.com/blog/betatestfornewsyncanddevonthinktogo20starts). There are still problems with sync in DT, as one can tell by visiting their forums, though it generally seems to work. It is not great with iCloud, which is one reason why I stopped using it with DT and switched to CloudMe.
You should judge people not by how close they get to the top, but by how far they have come from the bottom. Some people have a mountain to climb just to get to the place where others start out. (Me, 2010)

si
sidderke
Posts: 184
Joined: Thu Dec 03, 2015 11:34 pm
Platform: Mac

Sun Jun 21, 2020 8:59 am Post

lunk wrote:
Cirrocumulus wrote:I am interesting in knowing in what sense Scrivener's database structure is so different from what I've described above. I couldn't find details about it in KB's post. Apologies if this specific question has been asked before.

It has been answered. Only Dropbox has provided a sufficiently detailed API to let KB write the necessary sync code.


You didn't read his question right. He's asking out of curiosity what is the difference in how DevonThink works under the hood and Scrivener and how their database looks, where one can do it and the other can't, while (on the surface) having the same kind of features and way of handling things.

User avatar
lunk
Posts: 3970
Joined: Wed Aug 21, 2013 4:24 pm
Platform: Mac + iOS
Location: Sweden 64° N

Sun Jun 21, 2020 9:01 am Post

Well, one difference, if I understand correctly, is that Scrivener saves everything in original file formats. It’s not really a database, like Evernote or DT.
I am a user, writing non-fiction and science, using:
* Mac Scrivener 3 on a Macbook 12”, MacBook Pro 13”, and iMac 27”, all running Mojave.
* iOS Scrivener 1 on an iPhone 11 Pro, iPad Air 9.7”, and iPad Pro 12.9”, all running the latest iOS

Lo
Login
Posts: 62
Joined: Thu Apr 30, 2020 10:01 am
Platform: Mac

Sun Jun 21, 2020 2:13 pm Post

Cirrocumulus wrote:I've read KB's detailed explanation above ...


I *believe* that part of the difference is that DEVONthink is macOS and iOS only. It doesn’t have to bend to fit with the needs of Windows.

And Keith has said iCloud sync could be added, but that Scrivener would need to be rebuilt.

viewtopic.php?p=292047#p292047

If Apple does announce a shift to Arm Macs tomorrow, perhaps changes will have to be made if the underlying tech is altered in any significant way.

Wonder how telling Keith’s final sentence is in the post linked above.

mb
mbbntu
Posts: 1183
Joined: Wed Aug 01, 2007 9:44 am
Platform: Mac + iOS
Location: Cambridge, UK.

Sun Jun 21, 2020 2:20 pm Post

lunk wrote:Well, one difference, if I understand correctly, is that Scrivener saves everything in original file formats.

No, that is not correct: DEVONthink saves every file in its original format, too.
lunk wrote:It’s not really a database, like Evernote or DT.

This is incorrect, too. DEVONthink "databases" are packages, and they can be navigated using the Finder just like a Scrivener project package. All files remain in their original format, and can be accessed and copied via Finder, or any utility that functions in a similar way (Path Finder, for example).
You should judge people not by how close they get to the top, but by how far they have come from the bottom. Some people have a mountain to climb just to get to the place where others start out. (Me, 2010)

User avatar
lunk
Posts: 3970
Joined: Wed Aug 21, 2013 4:24 pm
Platform: Mac + iOS
Location: Sweden 64° N

Sun Jun 21, 2020 2:47 pm Post

Okay, @MBBNTU what is the difference then? :)
I am a user, writing non-fiction and science, using:
* Mac Scrivener 3 on a Macbook 12”, MacBook Pro 13”, and iMac 27”, all running Mojave.
* iOS Scrivener 1 on an iPhone 11 Pro, iPad Air 9.7”, and iPad Pro 12.9”, all running the latest iOS

mb
mbbntu
Posts: 1183
Joined: Wed Aug 01, 2007 9:44 am
Platform: Mac + iOS
Location: Cambridge, UK.

Sun Jun 21, 2020 2:56 pm Post

lunk wrote:Okay, @MBBNTU what is the difference then? :)

Errr! Well, one difference that I am aware of is that when you open the DT package, all files are kept in folders according to what type they are. So you get a folder for pdfs, another for rtfs, and so forth. But inside those folders are yet more folders. Like this:

Screenshot 2020-06-21 15.53.20.png
Screenshot 2020-06-21 15.53.20.png (371.06 KiB) Viewed 190 times
You should judge people not by how close they get to the top, but by how far they have come from the bottom. Some people have a mountain to climb just to get to the place where others start out. (Me, 2010)

User avatar
kewms
Posts: 6048
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sun Jun 21, 2020 5:32 pm Post

One of the key differences is the .scrivx file, which is the master index file that keeps track of where everything goes in the Binder. The most common Scrivener synchronization errors involve that file being out of sync with the actual contents of the project.

The .scrivx file is essential to what Scrivener does, because it allows you to create the structure of a novel (or whatever), by placing files in an arbitrary order.

DevonThink doesn't have anything like it. DT allows you to create groups, but the order within a group is completely out of your control: you can sort by name, creation date, etc., but you can't drag an arbitrary file to an arbitrary location and expect it to stay there.

Because DevonThink doesn't have a .scrivx file, it doesn't need to worry about the timing of uploads. In Scrivener, if an updated .scrivx file is transferred but one of the files it refers to is not, you'll get an "empty" Binder entry with no data. In the reverse case, the file will be there, but invisible, because no entry in the Binder points to it.

In DevonThink, if a file hasn't transferred -- which happens fairly regularly! -- it simply isn't there. No reference to it, no metadata, no indication that it ever existed. And either you go back and resynchronize until it appears, or you do without it until the sync algorithm catches up. Everyone willing to tolerate a missing chapter in the middle of your novel please raise your hand!

Katherine
Scrivener Support Team

Ci
Cirrocumulus
Posts: 15
Joined: Tue Sep 03, 2019 7:39 pm
Platform: Mac

Sun Jun 21, 2020 7:50 pm Post

sidderke wrote:You didn't read his question right.

Thank you.

mbbntu wrote:DEVONthink "databases" are packages, and they can be navigated using the Finder just like a Scrivener project package.

And thank you, too.

kewms wrote:DT allows you to create groups, but the order within a group is completely out of your control

Isn't this just some piece of metadata which can be synchronized like everything else?

kewms wrote:In DevonThink, if a file hasn't transferred -- which happens fairly regularly! -- it simply isn't there. No reference to it, no metadata, no indication that it ever existed. And either you go back and resynchronize until it appears, or you do without it until the sync algorithm catches up. Everyone willing to tolerate a missing chapter in the middle of your novel please raise your hand!

A file missing in DT can be just as annoying as a missing chapter in a novel, and would be a good reason to try the sync again. The worst that can happen is the file appearing twice, as "File" and "File copy", to be sorted out manually by the user. I can't understand why, if a chapter written on Device #1 doesn't appear on Device #2 because of a failed sync, the app can't simply retry the sync. If another sync doesn't work, the user will have to live with the fact that the connection is bad and the sync can't be performed. We all live with that pretty much on a daily basis.

kewms wrote:if an updated .scrivx file is transferred but one of the files it refers to is not, you'll get an "empty" Binder entry with no data. In the reverse case, the file will be there, but invisible, because no entry in the Binder points to it.

My point exactly. Live with an empty binder entry if the sync failed, or leave the binder entry at its local state if it exists already. If you edit that entry before a successful sync has been made, just duplicate the entry as per the example above and let the user choose.

I get the point about the resources and time allocated to DT development. I'm just trying to understand how big the problem really is. BTW, DT achieves all of the this *without* requiring the user to quit *any* of the apps being synchronized. You can sync in whatever order you want without quitting anything. The worst that has happened to me, after years of work with multiple-gigabyte databases including thousands of items, synchronized across two Macs and two iOS devices, is the "File copy" scenario above.

Surely all of this can't be rendered impossible because Scrivener can keep binder items in a particular order?

User avatar
kewms
Posts: 6048
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sun Jun 21, 2020 8:56 pm Post

Cirrocumulus wrote:
kewms wrote:DT allows you to create groups, but the order within a group is completely out of your control

Isn't this just some piece of metadata which can be synchronized like everything else?


No. In Scrivener, the Binder structure is maintained via a completely separate file, which is updated as needed with pointers to the component documents containing your actual writing.

I'm not sure how you would make "binder location" into "just some piece of metadata." By definition, it's an attribute of the entire project, not of the individual file. Something like, say, assigning a "location code" to individual files is meaningless unless you have a reference "grid" for that code. (Sort of like saying that a location is at 10 degrees North, 15 degrees West, but neglecting to mention that you're actually on Mars, not Earth.)

kewms wrote:In DevonThink, if a file hasn't transferred -- which happens fairly regularly! -- it simply isn't there. No reference to it, no metadata, no indication that it ever existed. And either you go back and resynchronize until it appears, or you do without it until the sync algorithm catches up. Everyone willing to tolerate a missing chapter in the middle of your novel please raise your hand!

A file missing in DT can be just as annoying as a missing chapter in a novel, and would be a good reason to try the sync again. The worst that can happen is the file appearing twice, as "File" and "File copy", to be sorted out manually by the user. I can't understand why, if a chapter written on Device #1 doesn't appear on Device #2 because of a failed sync, the app can't simply retry the sync. If another sync doesn't work, the user will have to live with the fact that the connection is bad and the sync can't be performed. We all live with that pretty much on a daily basis.


I haven't had the pleasure of reviewing the DevonThink support queue. But I can tell you that people who write to the Scrivener support address are not at all willing to "just live with" failed synchronization events. They get remarkably grumpy when they discover that entire chapters of their novels are missing or, worse, have been replaced with older versions.

(Which is another point. In my own DT usage, I don't do much editing of individual files, so there's not much risk that I'll have a version mismatch. Scrivener, on the other hand, is a "creation" application, not an "archive" application, and so the changes within component files can be enormous.)

kewms wrote:if an updated .scrivx file is transferred but one of the files it refers to is not, you'll get an "empty" Binder entry with no data. In the reverse case, the file will be there, but invisible, because no entry in the Binder points to it.

My point exactly. Live with an empty binder entry if the sync failed, or leave the binder entry at its local state if it exists already. If you edit that entry before a successful sync has been made, just duplicate the entry as per the example above and let the user choose.


Which is in fact how Scrivener handles synchronization conflicts, when it detects them.

Katherine
Scrivener Support Team

User avatar
kewms
Posts: 6048
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sun Jun 21, 2020 9:00 pm Post

Earlier in this thread, Keith discussed the differences between "shoebox" applications, like DevonThink, "document" applications, like Word, and Scrivener, which is a hybrid of both. Might be worth reading before we spend any more time beating the dead horse.
viewtopic.php?p=244405#p244405

Katherine
Scrivener Support Team

User avatar
lunk
Posts: 3970
Joined: Wed Aug 21, 2013 4:24 pm
Platform: Mac + iOS
Location: Sweden 64° N

Sun Jun 21, 2020 9:07 pm Post

I’ve tested DT in the past and my experience was that it was made to store stuff. Scrivener is primarily for writing, creating new text that doesn’t exist anywhere else or before. So if there is a serious hickup in the synchronization you might end up with several thousand freshly written words just being gone, forever. A pdf you downloaded to DT can be downloaded again, but that new text has to be written again, from scratch.

Scrivener is primarily not a database app.
I am a user, writing non-fiction and science, using:
* Mac Scrivener 3 on a Macbook 12”, MacBook Pro 13”, and iMac 27”, all running Mojave.
* iOS Scrivener 1 on an iPhone 11 Pro, iPad Air 9.7”, and iPad Pro 12.9”, all running the latest iOS