Lost writing due to faulty syncing

Thanks Katherine,
Now that you mention it, I believe I did do File > Sync with Mobile devices before closing Scrivener and re-opening (it’s a little bit of a blur, but I probably didn’t close until I noticed it wasn’t syncing).

At some point, there should have been a document synced that was my up-to-date one from the cafe, but unfortunately I’ve done Dropbox rewind on every single change that came through this morning, to no avail. Can you make sense of that?

So apparently one should close Scrivener every time they aren’t writing – just in case they leave the house and decide to capture a thought while on the go. That seems contradictory to the point of having a cross-platform writing solution. I guess Evernote has spoiled me.

Additionally, at one point, both devices claimed to be synced, but showed different versions of the same document. Can you explain that?

If both devices claim to be synced, but show a different version of the same document, that’s obviously a clue that something is amiss. At that point, I’d recommend making independent backups on both devices as protection against exactly the kind of error you saw.

It takes a small, but measurable amount of time for changes uploaded from one device to Dropbox to be available for download to other devices. This “processing” time – up to a few minutes, in my experience – is most likely to be a factor if you’re synchronizing devices in quick succession, and is a reason to be sure to synchronize devices as they are updated whenever possible. (In this context, synchronize the iPad using the coffee shop wifi instead of waiting until you get home, for example.)

The troubleshooting guide I linked above details a number of other situations that can cause erroneous synchronization.

The “on the go” scenario is why the auto-close option exists, so that Scrivener will shut itself down when you’re away. Scrivener -> Preferences -> Automatic Quit.

Katherine

This is exactly what I did! It was a couple more hours before I got home and had this issue. Something didn’t work correctly.

Didn’t know about automatic quit, but just turned it on. Thanks for that.

Hi kadavy,

Sorry to hear you lost writing.

Katherine’s point is worth repeating. I use this method after every iOS writing session. Before closing iOS Scriv I create a zipped backup of the project and save it to OneDrive, to the same folder where I store my Desktop zipped backups. Doing this ensures that I won’t lose the work I just did, in the event of some syncing glitch.

I strongly recommend you do the same.

Best,
Jim

The important thing to remember is that there are always three different computers involved in the syncing process. Each with its own HD where it stores the project:
Your Mac
The Dropbox server
Your iPad

On the Mac the Dropbox app is constantly monitoring the Dropbox folder when the Mac is awake. Any files changed on the Mac are immediately uploaded to the Dropbox server HD.

In Scrivener, every document in the Binder is its own file in the file system. As soon as you do something in a document, even if you only move the cursor, as soon as you pause for a few seconds that document’s file, inside the .scriv folder (which looks like a file in Finder but is actually a folder) is re-saved on the Mac HD. The Dropbox app notices that the file has changed and uploads it to the DB server, over-writing the corresponding file on its HD.

If you then look at the project on the iPad, it won’t now what happened if you had Scrivener open, because iOS Scriv doesn’t have that kind of background DB app because it can’t. iOS Scriv checks the DB server for changed files when you start iOS Scriv, but if it is already open you need to manually tell it to check for any changes by tapping the sync icon.

The keyword to long-term successful syncing is - patience! Being in a hurry is no good. Give all steps plenty of time.

Close the project in Mac Scriv and give the DB app a few seconds at least to sync everything, before closing the laptop.
Always tap the sync icon in iOS Scriv before opening a project.
Always tap the sync icon in iOS Scriv before changing app on the iPad and leaving iOS Scriv.
Always wait a few seconds when you have re-opened your Mac laptop before opening Scriv, and check in Finder that the project is up-to-date before opening it. If you want to be really sure, tap the DB icon in the top menu bar, and check that the latest changes have been downloaded and there is nothing waiting.

Hello. I find these kinds of reports extremely scary. IMHO there is absolutely no reason in 2019 to make users triple-check the order in which they are doing things so as not to lose work, and rely on daily backups in case they have tragically launched an app at the wrong time. I’m aware of the fact that it’s not easy to pull off good sync, but all the apps I’m using handle conflicts gracefully: not only iA Writer which I know uses simple iCloud sync, but even DevonThink which syncs multiple databases with gigabytes of items, notes, flags etc, on an encrypted WebDav server with nary a hiccup (and around ten times faster than Dropbox). All of this should really be transparent to the user IMHO. We shouldn’t be caring about app launch order in 2019.

Well, the way Scrivener handles syncing is part of what Scrivener is. If you don’t like it, don’t use it.
You don’t HAVE to use Scrivener you know. You’re free to use the apps you like.

I use DevonThink to Go quite extensively. It’s probably in a dead heat with Scrivener for my most-critical application. I wouldn’t say it syncs “without a hiccup” at all. It’s fairly common for me to open the iPad version of the database and find that files I recently added on the Mac aren’t there yet.

The key difference? They aren’t “my” files. They’re research files I downloaded from some other source, not thousands of words of my own writing. I’m planning to read them, not to change them. And so the consequences of synchronization delays are completely different.

Katherine

I am aware of that. I was pointing out a specific part of Scrivener which, in my opinion, needs to be improved. I am using it daily and appreciate its feature set very much, in case the obvious needed to be mentioned.

I use all kinds of sync software. Ironically, some of the best tools have the highest failures. Not because the tool is bad, but because the user doesn’t follow the rules.

ex: having the same item open in two different places. This breaks, google, icloud, one drive, dropbox, emv block replication (which is a hardware sync in big boy sans that we just discovered gets all kinds of angry if you edit the same block in two different locations).

The issue is almost never the underlying components once something is as far a long as scrivner. it is almost always user habits in conflict with the requirements for clean synchronization.

Apps are not the same and implementations are different, yes. But I would just like to point out that it’s possible to keep e.g. DevonThink (or other apps) open on a Mac, showing an open text note, do a quick modification on the note via iPhone or iPad, sync it on the way back to the Mac and see the modified contents on the (still open) document. I would rather have Scrivener complain and handle conflicted copies more gracefully than twitch in panic because I’m not sure if I’ve quit the app or not. Scrivener is the first app I’ve used that has an “Automatic Quit” option (which is of course almost essential in this case).

Just my 2c.

Well, you didn’t phrase it like that.

This sounds very demanding and negative and not as if you are only “pointing out” something.

There are literally thousands of users that switch between Mac/PC and iOS devices who don’t lose their work. Not by triple-checking the order in which they are doing things, but by not being sloppy about syncing. By simply closing the project whenever you leave your Mac or iPad in order to do something else.

I don’t understand how you can find the reports of problems posted in here to be “extremely scary”, when there is almost always an easy explanation for the problems.

I agree with Cirrocumulus that these syncing issues are scary, and I don’t share the sentiments of others that this is some kind of user error issue, or not following the “rules.”

If you understand how Scrivener’s idiosyncratic syncing behavior works, that’s good. But there are lots of users using Scrivener for the first time, who are used to other syncing protocols that don’t have the same quirks. Them not following the “rules” is as much if not more of an experience design problem than a behavior problem.

If there’s one random street in the middle of an American city on which you are supposed to drive on the left side of the road, and there’s a head-on collision, whose fault is it? Those who designed the street might want to reconsider their decision.

There’s an opportunity here to educate new Scrivener users so stuff like this doesn’t happen.

I’m one of those idiots who didn’t follow the “rules,” because of a couple of things in my background:
1. I used to be a web developer/designer. When we “checked in” code, the system would check line by line for conflicts. If there was a conflict, we could see a line-by-line comparison of the differences between the files. We could then pick and choose lines and even “merge” into one final document.
2. I use Evernote heavily. It’s always open on my computer. But if I’m on the go, I can jot notes into a document, and have no syncing problems.

My understanding of Scrivener synching now is that it’s “dumb” – and I don’t mean that in a derogatory way. What I mean is that it seems that Scrivener doesn’t care about the contents of the files being synched. It only cares about the date the file was updated on Dropbox.

When I think about the technical limitations, I can see how that might be the case.

(Then again, maybe I’m wrong about it. After all, something screwy was going on with my issue. I synced with Dropbox while at the cafe. Nowhere in the Dropbox history is a file with my changes on it. Plus, even after it claimed to be synced, my iPad and Laptop showed different documents. Maybe there’s something else going on under the hood. I’d love to hear a Scrivener dev’s explanation of it.)

If I were designing this experience, I would force the user to check a series of boxes the first five times they synced with iOS (after five times, they could then opt to not see the warning again):
[ ] “I understand that Scrivener’s syncing is ‘dumb.’ It does not care about the contents of my documents. It only cares about the order in which they’ve been updated.”
[ ] “I understand that I should not update a Scrivener document on one device while it is still open on another device.”

Yes, the way Scrivener syncs is strange and nonsensical to someone used to other applications. I’m sure there’s a good technical reason for it. But having experience with those other applications and mistakenly expecting Scrivener to behave the same is not a failure to follow any “rules.” If Scrivener’s syncing is different from the state of the art, it behooves the developers to design the experience to prevent error.

I would like to emphasize that my issue is not with the idiosyncratic rules, but with the consequence of not following them. Although I’m what you may call a “new user”, I am perfectly capable of learning a new piece of software and the way it functions, especially if it otherwise suits my needs. I have read the manual and so far have not had any major issues with sync.

So it’s not the rules, then, but what happens if you don’t follow them. I have one of those motorcycles which doesn’t have a warning light when the fuel tank is almost empty. Instead, the rider relies on a valve which cuts the fuel flow when there’s enough of it for another 30 miles or so. You’re supposed to reset the valve’s position every time you refuel. Nothing reminds you to do it. In ten years, I forgot to do it twice. The result was running out of fuel in the middle of the road. Very annoying, but not fatal.

I would have found it unacceptable if, should you fail to follow instructions and reset the valve’s position, instead of running out of fuel the engine either burst into flames or blew up from underneath you.

We are dealing with software, not motorcycles or otherwise physical entities. Things should be reversible, and proper writes to project files should be triple-checked by the app, not by the user. Sure, you could say that if you opened a terminal session and entered e.g. ‘sudo rm -rf /’ and lost half your hard drive then it’s your fault for not knowing what you’re doing. But this is not a low-level terminal shell. It’s a GUI-based writing package which should shield the user from ever having to worry about their data, at least not more than the usual worries of making regular backups.

Again (if it’s not clear), I’m perfectly happy with the way Scrivener works, otherwise I wouldn’t have bought it. I believe sync could be improved, though, that’s all.

Okay, how?

And I’m not talking about vague nebulous “the software should do this, it should triple-check writes, it should be reversible” kind of statements – because those are results.

What specific changes to the sync mechanism should be made to achieve those results?

If you aren’t an expert on Dropbox or synchronization of multi-file packages, you may have some catch-up research to do, but that shouldn’t keep you from being able to offer specific, useful changes in a couple of weeks or so.

Every so often, threads like this one sing the praises of ICloud: simple, reliable, rock solid. Scrivener should use iCloud!

Well… entirely coincidentally, someone in a thread about the Catalina beta linked to this:
mjtsai.com/blog/2019/09/03/iclo … postponed/
Which appears to be a catalog of the house of horrors that is iCloud in the iOS 13/Mac OS 10.15 betas. There’s a fairly coherent summary here: furbo.org/2019/09/04/icloud-clusterfuck/

Synchronization is hard. Synchronization of complex structures like Scrivener packages is more hard. Have a good backup strategy. (And no, Dropbox alone is not an adequate backup.)

Katherine

I disagree. I am aware that there are important technical difficulties involved, but I write, I do not program. I don’t have to do research in order to solve a software issue. I can only point to it when it becomes difficult to ignore.

I respectfully suggest you direct your first question at the developers of the various software packages I know of — I’m sure there are many more — which handle sync conflicts gracefully. They would probably know a lot more about this than I do.

As to @kewms’ comment on iCloud: I don’t think anyone on this thread has said that iCloud was a wonderful and hassle-free solution, but I may be wrong. EDIT: BTW — and please don’t consider this as support for iCloud — it seems that both articles mention clearly that the issues involved are related to beta releases. The second article even has a subheading which says “Folks don’t understand beta”.

The problem, which you don’t seem to understand, is that there are always three different computers involved in Scrivener’s syncing, and three different softwares, of which L&L only control two, and each one of them (computer/software) only knows about itself. The individual computer/software has no knowledge about the others, or what the user perhaps wants the three of them to do together. On top of that, each one is fully capable of being used in isolation, by itself, without including the other two.

Your analogy with the motorcycle is not relevant at all, for the simple reason that a hiccup in the syncing is comparable to running out of fuel, not of having an explosion. You don’t risk getting physically hurt or killed by losing some piece of text you’ve written. You simply have to re-write it.

The simplest way to avoid the risk of having syncing problems, if the risk really scares you, is not to sync. Don’t work on the project from several devices!

As I wrote earlier, I am syncing both DevonThink on macOS and DevonThink To Go (iOS) through a third computer, a custom-configured WebDAV server running on an Ubuntu-formatted old iMac. The syncing is being done in every direction flawlessly, with open documents everywhere. I am pretty positive the Ubuntu machine doesn’t know a lot about either DevonThink, macOS or iOS. It doesn’t even know it’s running on a Mac.

I know. This is why it’s an analogy.

I see. I’m holding it wrong.

Thank you. Never mind.