Lost writing due to faulty syncing

[kadavy]

I agree with Cirrocumulus that these syncing issues are scary, and I don't share the sentiments of others that this is some kind of user error issue, or not following the "rules."

If you understand how Scrivener's idiosyncratic syncing behavior works, that's good. But there are lots of users using Scrivener for the first time, who are used to other syncing protocols that don't have the same quirks. Them not following the "rules" is as much if not more of an experience design problem than a behavior problem.

If there's one random street in the middle of an American city on which you are supposed to drive on the left side of the road, and there's a head-on collision, whose fault is it? Those who designed the street might want to reconsider their decision.

There's an opportunity here to educate new Scrivener users so stuff like this doesn't happen.

I'm one of those idiots who didn't follow the "rules," because of a couple of things in my background:
1. I used to be a web developer/designer. When we "checked in" code, the system would check line by line for conflicts. If there was a conflict, we could see a line-by-line comparison of the differences between the files. We could then pick and choose lines and even "merge" into one final document.
2. I use Evernote heavily. It's always open on my computer. But if I'm on the go, I can jot notes into a document, and have no syncing problems.

My understanding of Scrivener synching now is that it's "dumb" – and I don't mean that in a derogatory way. What I mean is that it seems that Scrivener doesn't care about the contents of the files being synched. It only cares about the date the file was updated on Dropbox.

When I think about the technical limitations, I can see how that might be the case.

(Then again, maybe I'm wrong about it. After all, something screwy was going on with my issue. I synced with Dropbox while at the cafe. Nowhere in the Dropbox history is a file with my changes on it. Plus, even after it claimed to be synced, my iPad and Laptop showed different documents. Maybe there's something else going on under the hood. I'd love to hear a Scrivener dev's explanation of it.)

If I were designing this experience, I would force the user to check a series of boxes the first five times they synced with iOS (after five times, they could then opt to not see the warning again):
[ ] "I understand that Scrivener's syncing is 'dumb.' It does not care about the contents of my documents. It only cares about the order in which they've been updated."
[ ] "I understand that I should not update a Scrivener document on one device while it is still open on another device."

Yes, the way Scrivener syncs is strange and nonsensical to someone used to other applications. I'm sure there's a good technical reason for it. But having experience with those other applications and mistakenly expecting Scrivener to behave the same is not a failure to follow any "rules." If Scrivener's syncing is different from the state of the art, it behooves the developers to design the experience to prevent error.

[Cirrocumulus]

I would force the user to check a series of boxes the first five times they synced with iOS

I would like to emphasize that my issue is not with the idiosyncratic rules, but with the consequence of not following them. Although I'm what you may call a "new user", I am perfectly capable of learning a new piece of software and the way it functions, especially if it otherwise suits my needs. I have read the manual and so far have not had any major issues with sync.

So it's not the rules, then, but what happens if you don't follow them. I have one of those motorcycles which doesn't have a warning light when the fuel tank is almost empty. Instead, the rider relies on a valve which cuts the fuel flow when there's enough of it for another 30 miles or so. You're supposed to reset the valve's position every time you refuel. Nothing reminds you to do it. In ten years, I forgot to do it twice. The result was running out of fuel in the middle of the road. Very annoying, but not fatal.

I would have found it unacceptable if, should you fail to follow instructions and reset the valve's position, instead of running out of fuel the engine either burst into flames or blew up from underneath you.

We are dealing with software, not motorcycles or otherwise physical entities. Things should be reversible, and proper writes to project files should be triple-checked by the app, not by the user. Sure, you could say that if you opened a terminal session and entered e.g. 'sudo rm -rf /' and lost half your hard drive then it's your fault for not knowing what you're doing. But this is not a low-level terminal shell. It's a GUI-based writing package which should shield the user from ever having to worry about their data, at least not more than the usual worries of making regular backups.

Again (if it's not clear), I'm perfectly happy with the way Scrivener works, otherwise I wouldn't have bought it. I believe sync could be improved, though, that's all.

[devinganger]

I believe sync could be improved, though, that's all.

Okay, how?

And I'm not talking about vague nebulous "the software should do this, it should triple-check writes, it should be reversible" kind of statements -- because those are results.

What *specific* *changes* to the sync mechanism should be made to achieve those results?

If you aren't an expert on Dropbox or synchronization of multi-file packages, you may have some catch-up research to do, but that shouldn't keep you from being able to offer specific, useful changes in a couple of weeks or so.

[kewms]

Every so often, threads like this one sing the praises of ICloud: simple, reliable, rock solid. Scrivener should use iCloud!

Well... entirely coincidentally, someone in a thread about the Catalina beta linked to this:
https://mjtsai.com/blog/2019/09/03/iclo ... postponed/
Which appears to be a catalog of the house of horrors that is iCloud in the iOS 13/Mac OS 10.15 betas. There's a fairly coherent summary here: https://furbo.org/2019/09/04/icloud-clusterfuck/

Synchronization is hard. Synchronization of complex structures like Scrivener packages is more hard. Have a good backup strategy. (And no, Dropbox alone is not an adequate backup.)

Katherine

[Cirrocumulus]

you may have some catch-up research to do

I disagree. I am aware that there are important technical difficulties involved, but I write, I do not program. I don't *have* to do research in order to solve a software issue. I can only point to it when it becomes difficult to ignore.

I respectfully suggest you direct your first question at the developers of the various software packages I know of — I'm sure there are many more — which handle sync conflicts gracefully. They would probably know a lot more about this than I do.

As to @kewms' comment on iCloud: I don't think anyone on this thread has said that iCloud was a wonderful and hassle-free solution, but I may be wrong. EDIT: BTW — and please don't consider this as support for iCloud — it seems that both articles mention clearly that the issues involved are related to beta releases. The second article even has a subheading which says "Folks don’t understand beta".

[lunk]

I don't *have* to do research in order to solve a software issue. I can only point to it when it becomes difficult to ignore.

The problem, which you don't seem to understand, is that there are always three different computers involved in Scrivener's syncing, and three different softwares, of which L&L only control two, and each one of them (computer/software) only knows about itself. The individual computer/software has no knowledge about the others, or what the user perhaps wants the three of them to do together. On top of that, each one is fully capable of being used in isolation, by itself, without including the other two.

Your analogy with the motorcycle is not relevant at all, for the simple reason that a hiccup in the syncing is comparable to running out of fuel, not of having an explosion. You don't risk getting physically hurt or killed by losing some piece of text you've written. You simply have to re-write it.

The simplest way to avoid the risk of having syncing problems, if the risk really scares you, is not to sync. Don't work on the project from several devices!

[Cirrocumulus]

The problem, which you don't seem to understand, is that there are always three different computers involved in Scrivener's syncing, and three different softwares, of which L&L only control two, and each one of them (computer/software) only knows about itself. The individual computer/software has no knowledge about the others, or what the user perhaps wants the three of them to do together.

As I wrote earlier, I am syncing both DevonThink on macOS and DevonThink To Go (iOS) through a third computer, a custom-configured WebDAV server running on an Ubuntu-formatted old iMac. The syncing is being done in every direction flawlessly, with open documents everywhere. I am pretty positive the Ubuntu machine doesn't know a lot about either DevonThink, macOS or iOS. It doesn't even know it's running on a Mac.

You don't risk getting physically hurt or killed by losing some piece of text you've written.

I know. This is why it's an analogy.

The simplest way to avoid the risk of having syncing problems, if the risk really scares you, is not to sync. Don't work on the project from several devices!

I see. I'm holding it wrong.

Thank you. Never mind.

[Kinsey]

As I wrote earlier, I am syncing both DevonThink on macOS and DevonThink To Go (iOS) through a third computer, a custom-configured WebDAV server running on an Ubuntu-formatted old iMac. The syncing is being done in every direction flawlessly, with open documents everywhere. I am pretty positive the Ubuntu machine doesn't know a lot about either DevonThink, macOS or iOS. It doesn't even know it's running on a Mac.

Fellow user of Devonthink here, but not as a user of its sync capabilities. Were you on the forums when they launched the sync service? Lots of frustrated users registering complaints at the time, due to what seems to have been a very complicated system involving sync stores, etc. I think things have improved since then. My point is that it was not seamless, and required quite a lot of user intervention to set up, as far as I could tell.

I'm curious as to how Scrivener is different, in terms of requiring user management? Genuine question, as I have never had a problem with Scrivener sync. I just have it set up to close and back-up after a period of inactivity on my Macbook, so that if I ever open the iOS version, I'm not generating potential conflicts. Would that work for you?

You don't risk getting physically hurt or killed by losing some piece of text you've written.

I know. This is why it's an analogy.

[devinganger]

I respectfully suggest you direct your first question at the developers of the various software packages I know of — I'm sure there are many more — which handle sync conflicts gracefully. They would probably know a lot more about this than I do.

There's been a lot of discussions about the sync issues on these forums over the years, free to search. if you do, you'll find that Scrivener's document format is in no way similar to DevonThink's database. There are reasons why Scrivener's file format is the way it is (relating to characteristics that KB has decided are his priorities) and there has been a LOT of work to redo the file format to accommodate syncing to iOS in the most resilient way possible.

My point is, you can point at other software packages all you want, but you're not comparing apples to apples when you do so. Sync isn't a one-size-fits-all problem to solve. The world's foremost expert at sync in Scrivener (KB) has spent a long time defining the problem, defining the edge cases, and coming up with workable solutions that fit within the constraints of what makes Scrivener what it is. It's a bit presumptuous for ANY of us (since none of us have access to the code) to just snap out fingers and say, "Fix it."

Now, if people can show that there is an actual bug in the sync routines -- demonstrate the bug in a reliable way, so that KB can troubleshoot it -- then things can get better. But unless that happens, chances are good it any sync-related losses are due to user error of some sort. One of the whole design concepts behind Scrivener is that if something breaks, you can always go into the package and find the RTF files with your data in them without having to have fancy tools or special knowledge of the file format. You can't do that with DevonThink or any other database, for that matter.

[Cirrocumulus]

Were you on the forums when they launched the sync service? Lots of frustrated users registering complaints at the time, due to what seems to have been a very complicated system involving sync stores, etc.

I don't think I was on the DT forums when they launched the sync features, but I believe I was using DTTG on iOS from day one. Setting up the sync stores *is* somewhat complex, yes. You have to do more than click a couple of buttons, especially if you're using a custom WebDAV solution like I do. Since I set it up, however, I didn't have a single hiccup with sync in two years, whether via Wi-Fi or cellular, and I'm running 3 databases of around 20 gb each with thousands of documents. This has become my reference for quality in the matter, that's why I keep citing it. I don't know the devs.

I just have it set up to close and back-up after a period of inactivity on my Macbook, so that if I ever open the iOS version, I'm not generating potential conflicts. Would that work for you?

This is exactly what I'm doing, after having increased the number of backups in the preferences. It's working, I'm not saying it doesn't. It's just the potential loss of data if I'm not careful and the overall performance (much, much slower than DT) that I could live without, that's all.

It's a bit presumptuous for ANY of us (since none of us have access to the code) to just snap out fingers and say, "Fix it."

I wouldn't equate suggesting that a feature could be improved with snapping fingers and saying "Fix it". I don't snap fingers at people I don't know.

chances are good it any sync-related losses are due to user error of some sort.

This is the point I'm trying to make for the last four posts here: sync-related losses should *never* be the result of user error. Scrivener itself boasts — and rightly so — that you never need to save anything while you work. Data should be immune to user error, unless the user actively decides to wipe it. This is a general idea and is not limited to Scrivener. Software should keep the data intact no matter what incident occurs: power outage, network issues, alien invasion, a general election, whatever. The only loss of data should occur when a user presses Backspace, deletes a file or empties the Trash.

One of the whole design concepts behind Scrivener is that if something breaks, you can always go into the package and find the RTF files with your data in them without having to have fancy tools or special knowledge of the file format. You can't do that with DevonThink or any other database, for that matter.

Unless I misunderstood you, I believe you are mistaken. You can right-click any DT database, look inside the package and retrieve anything you want. This is a very important principle of DT. You lose some of the metadata, but you never lose the data. Your RTF or TXT or whatever is always there, on disk.

[kewms]

As I wrote earlier, I am syncing both DevonThink on macOS and DevonThink To Go (iOS) through a third computer, a custom-configured WebDAV server running on an Ubuntu-formatted old iMac. The syncing is being done in every direction flawlessly, with open documents everywhere. I am pretty positive the Ubuntu machine doesn't know a lot about either DevonThink, macOS or iOS. It doesn't even know it's running on a Mac.

The fact that you even know what a WebDAV server *is* puts you several steps ahead of the typical user. Any alternative sync solution *must* be able to work with a commercial service like Dropbox or iCloud.

As I noted upthread, I love DevonThink to Go, but I wouldn't describe its synchronization as "flawless" at all. There are also important differences between DevonThink databases and Scrivener projects, most notably the relationships between the component files.

Katherine

[kewms]

As to @kewms' comment on iCloud: I don't think anyone on this thread has said that iCloud was a wonderful and hassle-free solution, but I may be wrong. EDIT: BTW — and please don't consider this as support for iCloud — it seems that both articles mention clearly that the issues involved are related to beta releases. The second article even has a subheading which says "Folks don’t understand beta".

If you had read the links, you would know that people are experiencing unrecoverable data loss on devices that *aren't* running the beta, but are simply sharing an iCloud location with a device that is. One of the challenges of any synchronization service -- and the reason why I don't recommend relying on them as backups -- is that errors on one device can propagate very quickly.

My point was not to get into the relative merits of various services, though, but to observe that seamless, bulletproof synchronization is not nearly as common, or as easy to achieve, as these threads sometimes suggest.

Katherine

[kewms]

This is the point I'm trying to make for the last four posts here: sync-related losses should *never* be the result of user error. Scrivener itself boasts — and rightly so — that you never need to save anything while you work. Data should be immune to user error, unless the user actively decides to wipe it. This is a general idea and is not limited to Scrivener. Software should keep the data intact no matter what incident occurs: power outage, network issues, alien invasion, a general election, whatever. The only loss of data should occur when a user presses Backspace, deletes a file or empties the Trash.

The thing is, under some circumstances, Dropbox can erroneously believe that the user did exactly that. Version A of a project has a file. In Version B, the file is gone. If Dropbox believes that Version B is "current," then it will believe that the user *wanted* to delete the file.

Katherine

[Rayz]

As to @kewms' comment on iCloud: I don't think anyone on this thread has said that iCloud was a wonderful and hassle-free solution, but I may be wrong. EDIT: BTW — and please don't consider this as support for iCloud — it seems that both articles mention clearly that the issues involved are related to beta releases. The second article even has a subheading which says "Folks don’t understand beta".

If you had read the links, you would know that people are experiencing unrecoverable data loss on devices that *aren't* running the beta, but are simply sharing an iCloud location with a device that is.

Cirrocumulus said that the issues are related to beta releases
You seem to be saying that the people are experiencing unrecoverable data loss on devices sharing an iCloud location with a device running the beta release.

So what he said is correct: the issue is related to the beta release.

The chances are that most iCloud users only connect to devices that they own, and most iCloud users are not going to run the risk of running beta software. (And if they are then they should be savvy enough to back up their stuff).

[lunk]

So what he said is correct: the issue is related to the beta release

No, not the beta release but other syncing software in general. And as far as Scrivener syncing with iOS devices only Dropbox is involved, not iCloud.

So what he said was essentially that no user would have to think about what they do. The software should understand it.