Lost writing due to faulty syncing

Well, you didn’t phrase it like that.

This sounds very demanding and negative and not as if you are only “pointing out” something.

There are literally thousands of users that switch between Mac/PC and iOS devices who don’t lose their work. Not by triple-checking the order in which they are doing things, but by not being sloppy about syncing. By simply closing the project whenever you leave your Mac or iPad in order to do something else.

I don’t understand how you can find the reports of problems posted in here to be “extremely scary”, when there is almost always an easy explanation for the problems.

I agree with Cirrocumulus that these syncing issues are scary, and I don’t share the sentiments of others that this is some kind of user error issue, or not following the “rules.”

If you understand how Scrivener’s idiosyncratic syncing behavior works, that’s good. But there are lots of users using Scrivener for the first time, who are used to other syncing protocols that don’t have the same quirks. Them not following the “rules” is as much if not more of an experience design problem than a behavior problem.

If there’s one random street in the middle of an American city on which you are supposed to drive on the left side of the road, and there’s a head-on collision, whose fault is it? Those who designed the street might want to reconsider their decision.

There’s an opportunity here to educate new Scrivener users so stuff like this doesn’t happen.

I’m one of those idiots who didn’t follow the “rules,” because of a couple of things in my background:
1. I used to be a web developer/designer. When we “checked in” code, the system would check line by line for conflicts. If there was a conflict, we could see a line-by-line comparison of the differences between the files. We could then pick and choose lines and even “merge” into one final document.
2. I use Evernote heavily. It’s always open on my computer. But if I’m on the go, I can jot notes into a document, and have no syncing problems.

My understanding of Scrivener synching now is that it’s “dumb” – and I don’t mean that in a derogatory way. What I mean is that it seems that Scrivener doesn’t care about the contents of the files being synched. It only cares about the date the file was updated on Dropbox.

When I think about the technical limitations, I can see how that might be the case.

(Then again, maybe I’m wrong about it. After all, something screwy was going on with my issue. I synced with Dropbox while at the cafe. Nowhere in the Dropbox history is a file with my changes on it. Plus, even after it claimed to be synced, my iPad and Laptop showed different documents. Maybe there’s something else going on under the hood. I’d love to hear a Scrivener dev’s explanation of it.)

If I were designing this experience, I would force the user to check a series of boxes the first five times they synced with iOS (after five times, they could then opt to not see the warning again):
[ ] “I understand that Scrivener’s syncing is ‘dumb.’ It does not care about the contents of my documents. It only cares about the order in which they’ve been updated.”
[ ] “I understand that I should not update a Scrivener document on one device while it is still open on another device.”

Yes, the way Scrivener syncs is strange and nonsensical to someone used to other applications. I’m sure there’s a good technical reason for it. But having experience with those other applications and mistakenly expecting Scrivener to behave the same is not a failure to follow any “rules.” If Scrivener’s syncing is different from the state of the art, it behooves the developers to design the experience to prevent error.

I would like to emphasize that my issue is not with the idiosyncratic rules, but with the consequence of not following them. Although I’m what you may call a “new user”, I am perfectly capable of learning a new piece of software and the way it functions, especially if it otherwise suits my needs. I have read the manual and so far have not had any major issues with sync.

So it’s not the rules, then, but what happens if you don’t follow them. I have one of those motorcycles which doesn’t have a warning light when the fuel tank is almost empty. Instead, the rider relies on a valve which cuts the fuel flow when there’s enough of it for another 30 miles or so. You’re supposed to reset the valve’s position every time you refuel. Nothing reminds you to do it. In ten years, I forgot to do it twice. The result was running out of fuel in the middle of the road. Very annoying, but not fatal.

I would have found it unacceptable if, should you fail to follow instructions and reset the valve’s position, instead of running out of fuel the engine either burst into flames or blew up from underneath you.

We are dealing with software, not motorcycles or otherwise physical entities. Things should be reversible, and proper writes to project files should be triple-checked by the app, not by the user. Sure, you could say that if you opened a terminal session and entered e.g. ‘sudo rm -rf /’ and lost half your hard drive then it’s your fault for not knowing what you’re doing. But this is not a low-level terminal shell. It’s a GUI-based writing package which should shield the user from ever having to worry about their data, at least not more than the usual worries of making regular backups.

Again (if it’s not clear), I’m perfectly happy with the way Scrivener works, otherwise I wouldn’t have bought it. I believe sync could be improved, though, that’s all.

Okay, how?

And I’m not talking about vague nebulous “the software should do this, it should triple-check writes, it should be reversible” kind of statements – because those are results.

What specific changes to the sync mechanism should be made to achieve those results?

If you aren’t an expert on Dropbox or synchronization of multi-file packages, you may have some catch-up research to do, but that shouldn’t keep you from being able to offer specific, useful changes in a couple of weeks or so.

Every so often, threads like this one sing the praises of ICloud: simple, reliable, rock solid. Scrivener should use iCloud!

Well… entirely coincidentally, someone in a thread about the Catalina beta linked to this:
mjtsai.com/blog/2019/09/03/iclo … postponed/
Which appears to be a catalog of the house of horrors that is iCloud in the iOS 13/Mac OS 10.15 betas. There’s a fairly coherent summary here: furbo.org/2019/09/04/icloud-clusterfuck/

Synchronization is hard. Synchronization of complex structures like Scrivener packages is more hard. Have a good backup strategy. (And no, Dropbox alone is not an adequate backup.)

Katherine

I disagree. I am aware that there are important technical difficulties involved, but I write, I do not program. I don’t have to do research in order to solve a software issue. I can only point to it when it becomes difficult to ignore.

I respectfully suggest you direct your first question at the developers of the various software packages I know of — I’m sure there are many more — which handle sync conflicts gracefully. They would probably know a lot more about this than I do.

As to @kewms’ comment on iCloud: I don’t think anyone on this thread has said that iCloud was a wonderful and hassle-free solution, but I may be wrong. EDIT: BTW — and please don’t consider this as support for iCloud — it seems that both articles mention clearly that the issues involved are related to beta releases. The second article even has a subheading which says “Folks don’t understand beta”.

The problem, which you don’t seem to understand, is that there are always three different computers involved in Scrivener’s syncing, and three different softwares, of which L&L only control two, and each one of them (computer/software) only knows about itself. The individual computer/software has no knowledge about the others, or what the user perhaps wants the three of them to do together. On top of that, each one is fully capable of being used in isolation, by itself, without including the other two.

Your analogy with the motorcycle is not relevant at all, for the simple reason that a hiccup in the syncing is comparable to running out of fuel, not of having an explosion. You don’t risk getting physically hurt or killed by losing some piece of text you’ve written. You simply have to re-write it.

The simplest way to avoid the risk of having syncing problems, if the risk really scares you, is not to sync. Don’t work on the project from several devices!

As I wrote earlier, I am syncing both DevonThink on macOS and DevonThink To Go (iOS) through a third computer, a custom-configured WebDAV server running on an Ubuntu-formatted old iMac. The syncing is being done in every direction flawlessly, with open documents everywhere. I am pretty positive the Ubuntu machine doesn’t know a lot about either DevonThink, macOS or iOS. It doesn’t even know it’s running on a Mac.

I know. This is why it’s an analogy.

I see. I’m holding it wrong.

Thank you. Never mind.

Fellow user of Devonthink here, but not as a user of its sync capabilities. Were you on the forums when they launched the sync service? Lots of frustrated users registering complaints at the time, due to what seems to have been a very complicated system involving sync stores, etc. I think things have improved since then. My point is that it was not seamless, and required quite a lot of user intervention to set up, as far as I could tell.

I’m curious as to how Scrivener is different, in terms of requiring user management? Genuine question, as I have never had a problem with Scrivener sync. I just have it set up to close and back-up after a period of inactivity on my Macbook, so that if I ever open the iOS version, I’m not generating potential conflicts. Would that work for you?

:laughing:

There’s been a lot of discussions about the sync issues on these forums over the years, free to search. if you do, you’ll find that Scrivener’s document format is in no way similar to DevonThink’s database. There are reasons why Scrivener’s file format is the way it is (relating to characteristics that KB has decided are his priorities) and there has been a LOT of work to redo the file format to accommodate syncing to iOS in the most resilient way possible.

My point is, you can point at other software packages all you want, but you’re not comparing apples to apples when you do so. Sync isn’t a one-size-fits-all problem to solve. The world’s foremost expert at sync in Scrivener (KB) has spent a long time defining the problem, defining the edge cases, and coming up with workable solutions that fit within the constraints of what makes Scrivener what it is. It’s a bit presumptuous for ANY of us (since none of us have access to the code) to just snap out fingers and say, “Fix it.”

Now, if people can show that there is an actual bug in the sync routines – demonstrate the bug in a reliable way, so that KB can troubleshoot it – then things can get better. But unless that happens, chances are good it any sync-related losses are due to user error of some sort. One of the whole design concepts behind Scrivener is that if something breaks, you can always go into the package and find the RTF files with your data in them without having to have fancy tools or special knowledge of the file format. You can’t do that with DevonThink or any other database, for that matter.

I don’t think I was on the DT forums when they launched the sync features, but I believe I was using DTTG on iOS from day one. Setting up the sync stores is somewhat complex, yes. You have to do more than click a couple of buttons, especially if you’re using a custom WebDAV solution like I do. Since I set it up, however, I didn’t have a single hiccup with sync in two years, whether via Wi-Fi or cellular, and I’m running 3 databases of around 20 gb each with thousands of documents. This has become my reference for quality in the matter, that’s why I keep citing it. I don’t know the devs.

This is exactly what I’m doing, after having increased the number of backups in the preferences. It’s working, I’m not saying it doesn’t. It’s just the potential loss of data if I’m not careful and the overall performance (much, much slower than DT) that I could live without, that’s all. :slight_smile:

I wouldn’t equate suggesting that a feature could be improved with snapping fingers and saying “Fix it”. I don’t snap fingers at people I don’t know.

This is the point I’m trying to make for the last four posts here: sync-related losses should never be the result of user error. Scrivener itself boasts — and rightly so — that you never need to save anything while you work. Data should be immune to user error, unless the user actively decides to wipe it. This is a general idea and is not limited to Scrivener. Software should keep the data intact no matter what incident occurs: power outage, network issues, alien invasion, a general election, whatever. The only loss of data should occur when a user presses Backspace, deletes a file or empties the Trash.

Unless I misunderstood you, I believe you are mistaken. You can right-click any DT database, look inside the package and retrieve anything you want. This is a very important principle of DT. You lose some of the metadata, but you never lose the data. Your RTF or TXT or whatever is always there, on disk.

The fact that you even know what a WebDAV server is puts you several steps ahead of the typical user. Any alternative sync solution must be able to work with a commercial service like Dropbox or iCloud.

As I noted upthread, I love DevonThink to Go, but I wouldn’t describe its synchronization as “flawless” at all. There are also important differences between DevonThink databases and Scrivener projects, most notably the relationships between the component files.

Katherine

If you had read the links, you would know that people are experiencing unrecoverable data loss on devices that aren’t running the beta, but are simply sharing an iCloud location with a device that is. One of the challenges of any synchronization service – and the reason why I don’t recommend relying on them as backups – is that errors on one device can propagate very quickly.

My point was not to get into the relative merits of various services, though, but to observe that seamless, bulletproof synchronization is not nearly as common, or as easy to achieve, as these threads sometimes suggest.

Katherine

The thing is, under some circumstances, Dropbox can erroneously believe that the user did exactly that. Version A of a project has a file. In Version B, the file is gone. If Dropbox believes that Version B is “current,” then it will believe that the user wanted to delete the file.

Katherine

Cirrocumulus said that the issues are related to beta releases
You seem to be saying that the people are experiencing unrecoverable data loss on devices sharing an iCloud location with a device running the beta release.

So what he said is correct: the issue is related to the beta release.

The chances are that most iCloud users only connect to devices that they own, and most iCloud users are not going to run the risk of running beta software. (And if they are then they should be savvy enough to back up their stuff).

No, not the beta release but other syncing software in general. And as far as Scrivener syncing with iOS devices only Dropbox is involved, not iCloud.

So what he said was essentially that no user would have to think about what they do. The software should understand it.

I’m afraid we’re going to have to agree to disagree on that one. I was pointing out that highlighting beta software as an indication that syncing is a massive unsolvable problem doesn’t really carry that much weight, which is what the message I quoted was stating. You’re expanding it to the wider argument that has been carrying on here for years:

Everyone here has suffered some sort of computer mishap, so everyone knows it’s important to back up your work.

I think the problem is that most people here have also experienced syncing that works without a five step process and a lingering sense of uncertainty, which is why this keeps coming up time and again.

I did not say that synchronization is an unsolvable problem. I did say that it is a hard problem and that many companies – including massive companies with much larger resources than Scrivener – sometimes have trouble getting it right.

And that, because synchronization is a hard problem, users should understand that a perfect, fire-and-forget solution does not exist, and therefore should make sure that their work is also backed up through other means. Maybe “everyone knows” that backups are important, but the number of people who actually act on that knowledge sometimes seems depressingly small.

Katherine

I have experienced syncing that is allegedly easy. And I have also experienced the chaos that results when it inevitably goes wrong. One of the core values of Scrivener is that I can always tear apart the project format, without any special tools, and salvage my writing if I really need to.

I think the problem is that a lot people expect their devices to understand what they are expected to do, and to excute this without any time-lag, and this is the reason why this keeps coming up again and again.

Syncing in Scrivener doesn’t require a “five step process”. It just requires a bit of patience on the Mac/PC side, the length depending on the WiFi speed, and a tap on an icon and a bit of patience on the iOS side, again depending on the WiFi speed.

Do you have any constructive suggestions for how the user can be forced to show this patience, and to stop them from putting their Mac/PC to sleep before the DB app has finished syncing, and to stop them from switching app on their iDevice in a way that causes the iDevice to remove iOS Scriv from working memory before they have synced with the DB server, and for this to be done in a way that still enables the user to actually do all of this if they really want to go on without complete syncing?