Lost writing due to faulty syncing

The important thing to remember is that there are always three different computers involved in the syncing process. Each with its own HD where it stores the project:
Your Mac
The Dropbox server
Your iPad

On the Mac the Dropbox app is constantly monitoring the Dropbox folder when the Mac is awake. Any files changed on the Mac are immediately uploaded to the Dropbox server HD.

In Scrivener, every document in the Binder is its own file in the file system. As soon as you do something in a document, even if you only move the cursor, as soon as you pause for a few seconds that document’s file, inside the .scriv folder (which looks like a file in Finder but is actually a folder) is re-saved on the Mac HD. The Dropbox app notices that the file has changed and uploads it to the DB server, over-writing the corresponding file on its HD.

If you then look at the project on the iPad, it won’t now what happened if you had Scrivener open, because iOS Scriv doesn’t have that kind of background DB app because it can’t. iOS Scriv checks the DB server for changed files when you start iOS Scriv, but if it is already open you need to manually tell it to check for any changes by tapping the sync icon.

The keyword to long-term successful syncing is - patience! Being in a hurry is no good. Give all steps plenty of time.

Close the project in Mac Scriv and give the DB app a few seconds at least to sync everything, before closing the laptop.
Always tap the sync icon in iOS Scriv before opening a project.
Always tap the sync icon in iOS Scriv before changing app on the iPad and leaving iOS Scriv.
Always wait a few seconds when you have re-opened your Mac laptop before opening Scriv, and check in Finder that the project is up-to-date before opening it. If you want to be really sure, tap the DB icon in the top menu bar, and check that the latest changes have been downloaded and there is nothing waiting.

Hello. I find these kinds of reports extremely scary. IMHO there is absolutely no reason in 2019 to make users triple-check the order in which they are doing things so as not to lose work, and rely on daily backups in case they have tragically launched an app at the wrong time. I’m aware of the fact that it’s not easy to pull off good sync, but all the apps I’m using handle conflicts gracefully: not only iA Writer which I know uses simple iCloud sync, but even DevonThink which syncs multiple databases with gigabytes of items, notes, flags etc, on an encrypted WebDav server with nary a hiccup (and around ten times faster than Dropbox). All of this should really be transparent to the user IMHO. We shouldn’t be caring about app launch order in 2019.

Well, the way Scrivener handles syncing is part of what Scrivener is. If you don’t like it, don’t use it.
You don’t HAVE to use Scrivener you know. You’re free to use the apps you like.

I use DevonThink to Go quite extensively. It’s probably in a dead heat with Scrivener for my most-critical application. I wouldn’t say it syncs “without a hiccup” at all. It’s fairly common for me to open the iPad version of the database and find that files I recently added on the Mac aren’t there yet.

The key difference? They aren’t “my” files. They’re research files I downloaded from some other source, not thousands of words of my own writing. I’m planning to read them, not to change them. And so the consequences of synchronization delays are completely different.

Katherine

I am aware of that. I was pointing out a specific part of Scrivener which, in my opinion, needs to be improved. I am using it daily and appreciate its feature set very much, in case the obvious needed to be mentioned.

I use all kinds of sync software. Ironically, some of the best tools have the highest failures. Not because the tool is bad, but because the user doesn’t follow the rules.

ex: having the same item open in two different places. This breaks, google, icloud, one drive, dropbox, emv block replication (which is a hardware sync in big boy sans that we just discovered gets all kinds of angry if you edit the same block in two different locations).

The issue is almost never the underlying components once something is as far a long as scrivner. it is almost always user habits in conflict with the requirements for clean synchronization.

Apps are not the same and implementations are different, yes. But I would just like to point out that it’s possible to keep e.g. DevonThink (or other apps) open on a Mac, showing an open text note, do a quick modification on the note via iPhone or iPad, sync it on the way back to the Mac and see the modified contents on the (still open) document. I would rather have Scrivener complain and handle conflicted copies more gracefully than twitch in panic because I’m not sure if I’ve quit the app or not. Scrivener is the first app I’ve used that has an “Automatic Quit” option (which is of course almost essential in this case).

Just my 2c.

Well, you didn’t phrase it like that.

This sounds very demanding and negative and not as if you are only “pointing out” something.

There are literally thousands of users that switch between Mac/PC and iOS devices who don’t lose their work. Not by triple-checking the order in which they are doing things, but by not being sloppy about syncing. By simply closing the project whenever you leave your Mac or iPad in order to do something else.

I don’t understand how you can find the reports of problems posted in here to be “extremely scary”, when there is almost always an easy explanation for the problems.

I agree with Cirrocumulus that these syncing issues are scary, and I don’t share the sentiments of others that this is some kind of user error issue, or not following the “rules.”

If you understand how Scrivener’s idiosyncratic syncing behavior works, that’s good. But there are lots of users using Scrivener for the first time, who are used to other syncing protocols that don’t have the same quirks. Them not following the “rules” is as much if not more of an experience design problem than a behavior problem.

If there’s one random street in the middle of an American city on which you are supposed to drive on the left side of the road, and there’s a head-on collision, whose fault is it? Those who designed the street might want to reconsider their decision.

There’s an opportunity here to educate new Scrivener users so stuff like this doesn’t happen.

I’m one of those idiots who didn’t follow the “rules,” because of a couple of things in my background:
1. I used to be a web developer/designer. When we “checked in” code, the system would check line by line for conflicts. If there was a conflict, we could see a line-by-line comparison of the differences between the files. We could then pick and choose lines and even “merge” into one final document.
2. I use Evernote heavily. It’s always open on my computer. But if I’m on the go, I can jot notes into a document, and have no syncing problems.

My understanding of Scrivener synching now is that it’s “dumb” – and I don’t mean that in a derogatory way. What I mean is that it seems that Scrivener doesn’t care about the contents of the files being synched. It only cares about the date the file was updated on Dropbox.

When I think about the technical limitations, I can see how that might be the case.

(Then again, maybe I’m wrong about it. After all, something screwy was going on with my issue. I synced with Dropbox while at the cafe. Nowhere in the Dropbox history is a file with my changes on it. Plus, even after it claimed to be synced, my iPad and Laptop showed different documents. Maybe there’s something else going on under the hood. I’d love to hear a Scrivener dev’s explanation of it.)

If I were designing this experience, I would force the user to check a series of boxes the first five times they synced with iOS (after five times, they could then opt to not see the warning again):
[ ] “I understand that Scrivener’s syncing is ‘dumb.’ It does not care about the contents of my documents. It only cares about the order in which they’ve been updated.”
[ ] “I understand that I should not update a Scrivener document on one device while it is still open on another device.”

Yes, the way Scrivener syncs is strange and nonsensical to someone used to other applications. I’m sure there’s a good technical reason for it. But having experience with those other applications and mistakenly expecting Scrivener to behave the same is not a failure to follow any “rules.” If Scrivener’s syncing is different from the state of the art, it behooves the developers to design the experience to prevent error.

I would like to emphasize that my issue is not with the idiosyncratic rules, but with the consequence of not following them. Although I’m what you may call a “new user”, I am perfectly capable of learning a new piece of software and the way it functions, especially if it otherwise suits my needs. I have read the manual and so far have not had any major issues with sync.

So it’s not the rules, then, but what happens if you don’t follow them. I have one of those motorcycles which doesn’t have a warning light when the fuel tank is almost empty. Instead, the rider relies on a valve which cuts the fuel flow when there’s enough of it for another 30 miles or so. You’re supposed to reset the valve’s position every time you refuel. Nothing reminds you to do it. In ten years, I forgot to do it twice. The result was running out of fuel in the middle of the road. Very annoying, but not fatal.

I would have found it unacceptable if, should you fail to follow instructions and reset the valve’s position, instead of running out of fuel the engine either burst into flames or blew up from underneath you.

We are dealing with software, not motorcycles or otherwise physical entities. Things should be reversible, and proper writes to project files should be triple-checked by the app, not by the user. Sure, you could say that if you opened a terminal session and entered e.g. ‘sudo rm -rf /’ and lost half your hard drive then it’s your fault for not knowing what you’re doing. But this is not a low-level terminal shell. It’s a GUI-based writing package which should shield the user from ever having to worry about their data, at least not more than the usual worries of making regular backups.

Again (if it’s not clear), I’m perfectly happy with the way Scrivener works, otherwise I wouldn’t have bought it. I believe sync could be improved, though, that’s all.

Okay, how?

And I’m not talking about vague nebulous “the software should do this, it should triple-check writes, it should be reversible” kind of statements – because those are results.

What specific changes to the sync mechanism should be made to achieve those results?

If you aren’t an expert on Dropbox or synchronization of multi-file packages, you may have some catch-up research to do, but that shouldn’t keep you from being able to offer specific, useful changes in a couple of weeks or so.

Every so often, threads like this one sing the praises of ICloud: simple, reliable, rock solid. Scrivener should use iCloud!

Well… entirely coincidentally, someone in a thread about the Catalina beta linked to this:
mjtsai.com/blog/2019/09/03/iclo … postponed/
Which appears to be a catalog of the house of horrors that is iCloud in the iOS 13/Mac OS 10.15 betas. There’s a fairly coherent summary here: furbo.org/2019/09/04/icloud-clusterfuck/

Synchronization is hard. Synchronization of complex structures like Scrivener packages is more hard. Have a good backup strategy. (And no, Dropbox alone is not an adequate backup.)

Katherine

I disagree. I am aware that there are important technical difficulties involved, but I write, I do not program. I don’t have to do research in order to solve a software issue. I can only point to it when it becomes difficult to ignore.

I respectfully suggest you direct your first question at the developers of the various software packages I know of — I’m sure there are many more — which handle sync conflicts gracefully. They would probably know a lot more about this than I do.

As to @kewms’ comment on iCloud: I don’t think anyone on this thread has said that iCloud was a wonderful and hassle-free solution, but I may be wrong. EDIT: BTW — and please don’t consider this as support for iCloud — it seems that both articles mention clearly that the issues involved are related to beta releases. The second article even has a subheading which says “Folks don’t understand beta”.

The problem, which you don’t seem to understand, is that there are always three different computers involved in Scrivener’s syncing, and three different softwares, of which L&L only control two, and each one of them (computer/software) only knows about itself. The individual computer/software has no knowledge about the others, or what the user perhaps wants the three of them to do together. On top of that, each one is fully capable of being used in isolation, by itself, without including the other two.

Your analogy with the motorcycle is not relevant at all, for the simple reason that a hiccup in the syncing is comparable to running out of fuel, not of having an explosion. You don’t risk getting physically hurt or killed by losing some piece of text you’ve written. You simply have to re-write it.

The simplest way to avoid the risk of having syncing problems, if the risk really scares you, is not to sync. Don’t work on the project from several devices!

As I wrote earlier, I am syncing both DevonThink on macOS and DevonThink To Go (iOS) through a third computer, a custom-configured WebDAV server running on an Ubuntu-formatted old iMac. The syncing is being done in every direction flawlessly, with open documents everywhere. I am pretty positive the Ubuntu machine doesn’t know a lot about either DevonThink, macOS or iOS. It doesn’t even know it’s running on a Mac.

I know. This is why it’s an analogy.

I see. I’m holding it wrong.

Thank you. Never mind.

Fellow user of Devonthink here, but not as a user of its sync capabilities. Were you on the forums when they launched the sync service? Lots of frustrated users registering complaints at the time, due to what seems to have been a very complicated system involving sync stores, etc. I think things have improved since then. My point is that it was not seamless, and required quite a lot of user intervention to set up, as far as I could tell.

I’m curious as to how Scrivener is different, in terms of requiring user management? Genuine question, as I have never had a problem with Scrivener sync. I just have it set up to close and back-up after a period of inactivity on my Macbook, so that if I ever open the iOS version, I’m not generating potential conflicts. Would that work for you?

There’s been a lot of discussions about the sync issues on these forums over the years, free to search. if you do, you’ll find that Scrivener’s document format is in no way similar to DevonThink’s database. There are reasons why Scrivener’s file format is the way it is (relating to characteristics that KB has decided are his priorities) and there has been a LOT of work to redo the file format to accommodate syncing to iOS in the most resilient way possible.

My point is, you can point at other software packages all you want, but you’re not comparing apples to apples when you do so. Sync isn’t a one-size-fits-all problem to solve. The world’s foremost expert at sync in Scrivener (KB) has spent a long time defining the problem, defining the edge cases, and coming up with workable solutions that fit within the constraints of what makes Scrivener what it is. It’s a bit presumptuous for ANY of us (since none of us have access to the code) to just snap out fingers and say, “Fix it.”

Now, if people can show that there is an actual bug in the sync routines – demonstrate the bug in a reliable way, so that KB can troubleshoot it – then things can get better. But unless that happens, chances are good it any sync-related losses are due to user error of some sort. One of the whole design concepts behind Scrivener is that if something breaks, you can always go into the package and find the RTF files with your data in them without having to have fancy tools or special knowledge of the file format. You can’t do that with DevonThink or any other database, for that matter.

I don’t think I was on the DT forums when they launched the sync features, but I believe I was using DTTG on iOS from day one. Setting up the sync stores is somewhat complex, yes. You have to do more than click a couple of buttons, especially if you’re using a custom WebDAV solution like I do. Since I set it up, however, I didn’t have a single hiccup with sync in two years, whether via Wi-Fi or cellular, and I’m running 3 databases of around 20 gb each with thousands of documents. This has become my reference for quality in the matter, that’s why I keep citing it. I don’t know the devs.

This is exactly what I’m doing, after having increased the number of backups in the preferences. It’s working, I’m not saying it doesn’t. It’s just the potential loss of data if I’m not careful and the overall performance (much, much slower than DT) that I could live without, that’s all.

I wouldn’t equate suggesting that a feature could be improved with snapping fingers and saying “Fix it”. I don’t snap fingers at people I don’t know.

This is the point I’m trying to make for the last four posts here: sync-related losses should never be the result of user error. Scrivener itself boasts — and rightly so — that you never need to save anything while you work. Data should be immune to user error, unless the user actively decides to wipe it. This is a general idea and is not limited to Scrivener. Software should keep the data intact no matter what incident occurs: power outage, network issues, alien invasion, a general election, whatever. The only loss of data should occur when a user presses Backspace, deletes a file or empties the Trash.

Unless I misunderstood you, I believe you are mistaken. You can right-click any DT database, look inside the package and retrieve anything you want. This is a very important principle of DT. You lose some of the metadata, but you never lose the data. Your RTF or TXT or whatever is always there, on disk.

The fact that you even know what a WebDAV server is puts you several steps ahead of the typical user. Any alternative sync solution must be able to work with a commercial service like Dropbox or iCloud.

As I noted upthread, I love DevonThink to Go, but I wouldn’t describe its synchronization as “flawless” at all. There are also important differences between DevonThink databases and Scrivener projects, most notably the relationships between the component files.

Katherine

If you had read the links, you would know that people are experiencing unrecoverable data loss on devices that aren’t running the beta, but are simply sharing an iCloud location with a device that is. One of the challenges of any synchronization service – and the reason why I don’t recommend relying on them as backups – is that errors on one device can propagate very quickly.

My point was not to get into the relative merits of various services, though, but to observe that seamless, bulletproof synchronization is not nearly as common, or as easy to achieve, as these threads sometimes suggest.

Katherine