iCloud Sync

[wayofthefuture]

How is it that Scrivener just completely revamped its app, and in 2017 it still doesn’t support iCloud sync? Dropbox is a dying company, one that relies on unsecure root access to even work. As such, virtually every app I can think of has stopped supporting Dropbox sync, and those that still do all support iCloud as well. Scrivener seems like a relic from the past on this. What’s the logic? Scrivener would’ve had my money years ago if they had supported iCloud sync like most modern companies. And they would’ve had my money again today, but they’re stuck in the past. iCloud sync is simple, secure, and requires zero thought. It is what every app on iOS or Mac should aspire to support. Requiring people to jump through flaming hoops to get sync working, including downloading an entirely separate app (one that, again, is fast losing popularity and popular support) is just a foolish product move and a great way to diminish a customer base. Stop the Dropbox crap. Start supporting iCloud sync.

[rdale]

If iCloud supported Scrivener's package project format properly, they might have gone with that. But they don't. The only major cloud syncing product out there that won't mangle Scrivener's package format, and which provides the programming interface needed to implement the iOS version is Dropbox currently.

And don't even suggest changing the package format; it's why I can have gigabytes of data in a project without bringing a modest system to its knees.

Take a chill pill man.

[KB]

Happy release day!

[leb85711]

Stop the Dropbox crap. Start supporting iCloud sync.

Scrivener probably isn't for you. Live with it.

Dropbox valuation is still in flux, and I haven't kept up with the latest so my numbers may be somewhat out of date. But at the end of July it was still valued at $9.4B, significantly higher than its competitor Box. Based on their run rate, Dropbox is projected to pull in over $1B in revenues this year. Yes, their 9.4 multiplier is high - but this is a signal that the market sees a bright future for this product. This is borne out if you dig into their investors. All the heavyweights have placed big bets on Dropbox.

Your upset that iCloud sync is not available is understandable. Your hyperbole is unforgivable.

[sidderke]

I also would like iCloud Sync because Apple has more options than Dropbox, which only has their 10 dollar/1TB option or Free. BUT...

- Dropbox Sync has been working fine for me. And in the past, a lot quicker and more transparant than iCloud Drive.

- I *love* Scrivener's package format. LOVE IT. It's genius, and one of the best unsung features. It's quick, it makes for fast sync, and quick working on whatever system. I have projects that are 7GB's big. That's because of big Research files in there, but if I work on them, I only work on megabytes of data. So all the sync, even on the phone on 4G, goes quick. Can you imagine every time having to sync 7GB? No thanks.

I get it. People want iCloud Sync. But Keith has in detail provided an answer WHY it isn't supported. Your post shows you probably haven't read it. RDale put it perfectly: iCloud doesn't support Scrivener's format. Not the other way around. Keith has talked about it with Apple. Nothing changed as of yet. if iCloud allows it, Scrivener will allow it, I'm sure.

Happy Launch! Waiting anxiously for the Mac App Store Update

[Pavel]

I also would like iCloud Sync because Apple has more options than Dropbox, which only has their 10 dollar/1TB option or Free. BUT...

- Dropbox Sync has been working fine for me. And in the past, a lot quicker and more transparant than iCloud Drive.

- I *love* Scrivener's package format. LOVE IT. It's genius, and one of the best unsung features. It's quick, it makes for fast sync, and quick working on whatever system. I have projects that are 7GB's big. That's because of big Research files in there, but if I work on them, I only work on megabytes of data. So all the sync, even on the phone on 4G, goes quick. Can you imagine every time having to sync 7GB? No thanks.

I get it. People want iCloud Sync. But Keith has in detail provided an answer WHY it isn't supported. Your post shows you probably haven't read it. RDale put it perfectly: iCloud doesn't support Scrivener's format. Not the other way around. Keith has talked about it with Apple. Nothing changed as of yet. if iCloud allows it, Scrivener will allow it, I'm sure.

Happy Launch! Waiting anxiously for the Mac App Store Update

I've not kept up to date with all of this but for a long time I've saved my files locally, while at each quite I have a backup made to my iCloud drive. If I ever want it on my other machine I go and get the last cloud backup and copy it to the other local drive and run it. I've not had any problems with this semi-manual sort of sync. Is there anything I should be doing otherwise or that is a pending problem with my methodology?

I don't particularly want a drop-box account as I was never very happy with it's performance in years past and it seems that their data counting is suspect, or used to be, but am willing to consider trying it again if need be.

[scshrugged]

I also would like iCloud Sync because Apple has more options than Dropbox, which only has their 10 dollar/1TB option or Free. BUT...

- Dropbox Sync has been working fine for me. And in the past, a lot quicker and more transparant than iCloud Drive.

- I *love* Scrivener's package format. LOVE IT. It's genius, and one of the best unsung features. It's quick, it makes for fast sync, and quick working on whatever system. I have projects that are 7GB's big. That's because of big Research files in there, but if I work on them, I only work on megabytes of data. So all the sync, even on the phone on 4G, goes quick. Can you imagine every time having to sync 7GB? No thanks.

I get it. People want iCloud Sync. But Keith has in detail provided an answer WHY it isn't supported. Your post shows you probably haven't read it. RDale put it perfectly: iCloud doesn't support Scrivener's format. Not the other way around. Keith has talked about it with Apple. Nothing changed as of yet. if iCloud allows it, Scrivener will allow it, I'm sure.

Happy Launch! Waiting anxiously for the Mac App Store Update

I've not kept up to date with all of this but for a long time I've saved my files locally, while at each quite I have a backup made to my iCloud drive. If I ever want it on my other machine I go and get the last cloud backup and copy it to the other local drive and run it. I've not had any problems with this semi-manual sort of sync. Is there anything I should be doing otherwise or that is a pending problem with my methodology?

I don't particularly want a drop-box account as I was never very happy with it's performance in years past and it seems that their data counting is suspect, or used to be, but am willing to consider trying it again if need be.

There's nothing wrong with your method if you're keeping track of your versions and are unpacking the archive outside of iCloud Drive if your intention is to not store a live project there. The following article explains the method more comprehensively in its Alternate Method section, and describes best practices for using desktop Scrivener with cloud syncing.
https://scrivener.tenderapp.com/help/kb ... c-services

A good discussion was had here about iCloud Drive and desktop Scrivener:
viewtopic.php?f=2&t=39880

I've been experimenting with placing both archived backups and live projects in iCloud Drive, and the experience has been positive. Again, I'm talking about syncing between Macs here.

Dropbox is required for cloud syncing with iOS Scrivener. Two sections of an L&L article explain why that is (the full article is a comprehensive iOS syncing paper).
https://scrivener.tenderapp.com/help/kb ... automatic-
https://scrivener.tenderapp.com/help/kb ... solutions-

Part of a writer's job is to prepare for the inevitable failure of software or hardware. A solid backup plan is mandatory no matter the sync method.

[nevdev]

I get it. People want iCloud Sync. But Keith has in detail provided an answer WHY it isn't supported. Your post shows you probably haven't read it. RDale put it perfectly: iCloud doesn't support Scrivener's format. Not the other way around.

CloudKit (not iCloud Drive) can sync packages. Reinvented Software's Keep It is an example.

[devinganger]

It's not so much that iCloud can't sync packages so much as it is, if I understand correctly, that it does not sync package contents in such a way as to guarantee that the updates to the files within the package are done in such a way as to not corrupt the integrity of the package.

[LuckyJack]

This is truly a mystery to me. A package, such as a Scrivener project, is just a directory. Finder will show you its contents with the Show Package Contents option in the action menu, and at a shell prompt/terminal window, you can just "cd" into it like any other directory.

I think the reason Finder treats it as a single file is because Scrivener must have a plist file that identifies .scriv folders as document packages.

At a command prompt, if you rename a Scrivener project so it no longer ends in .scriv, Finder will just see it as a folder. Rename it back to whatever.scriv and poof - it's a package again, and as long as you didn't alter any of the files inside, it's a happy Scrivener project.

I don't want to swim upstream, but I'm not sure why iCloud would have any trouble syncing a folder. The user might have trouble telling when iCloud is finished syncing, I can certainly see that, and a partially synced Scrivener project would be a disaster.

For a view of a similar sort of thing, drop to a command prompt and use the unzip command against a .docx "file". It's a file, sure enough, but it's a file that might as well end in .zip instead of .docx. You can unzip a docx file and look at the parts and pieces if you want. Docx files are zip archives.

Mind your backups when playing with internal organs in living documents.

[kewms]

This is truly a mystery to me. A package, such as a Scrivener project, is just a directory. Finder will show you its contents with the Show Package Contents option in the action menu, and at a shell prompt/terminal window, you can just "cd" into it like any other directory.

Technically, this is correct. What you are missing is that the component files in a Scrivener project have metadata tying them to each other and to specific locations in the Binder. If the project.scrivx file -- the master index used to build the Binder -- gets disconnected from the actual documents in the project, all kinds of chaos can ensue. iOS Scrivener in particular puts a great deal of effort into minimizing the risk of that happening, and to do so it depends on the Dropbox API. CloudKit, the iCloud equivalent, does not make similar services available.

A .DOCX file is not equivalent, precisely because it *is* a ZIP file. All of its components are compressed in a single unit and travel together. It's not possible to email part of a DOCX file; it is trivially easy to email part of a Scrivener project.

Katherine

[devinganger]

I don't want to swim upstream, but I'm not sure why iCloud would have any trouble syncing a folder. The user might have trouble telling when iCloud is finished syncing, I can certainly see that, and a partially synced Scrivener project would be a disaster.

For a view of a similar sort of thing, drop to a command prompt and use the unzip command against a .docx "file". It's a file, sure enough, but it's a file that might as well end in .zip instead of .docx. You can unzip a docx file and look at the parts and pieces if you want. Docx files are zip archives.

To expand on what Katherine said, even though a Word file is actually a ZIP archive of an entire folder/file structure, it does *not* live on disk as a folder and file structure (the sole exception being in the user's configured TEMP directory, for transient periods of time). Every time you save it, all of the files that make up that Word document get zipped back into a single archive file that is a faithful point-in-time representation of the document as it was when it was saved, even though the internal components are pieces with a defined schema and links and metadata. That *single file* is then written to the hard drive and if you sync it with iCloud, Google, OneDrive, or any other sync engine, it is a *single file* that is completely independent and self-consistent, because Word is working with the copy in memory and the TEMP directory, not with the pristine saved copy it just wrote to disk. It only uses the .DOCX file when you open the file or when you save the file (or otherwise write to it).

A file package, however, from what I understand is *not* the same way. To your sync engine client looking at the file system, it's a folder with a bunch of files and sub-folders, linked in the very specific ways Katherine mentions. Scrivener and the Finder *treat* it like a single file -- albeit with the caveats you mention about being able to force your way into the folder structure -- but it is at no time a single file that gets unpackaged or packaged up. And the sync engine clients scouring the file system *do not* see it as a single package -- they see it as a folder full of tasty bits and bobs.

And the problem comes in with how they track and sync changes to those tasty bits and bobs.

Say that the sync engine is trying to save bandwidth and be clever. If you've just updated multiple files multiple times since the last sync, your sync engine has a choice on how it handles those changes:

1) Faithfully preserve the sequence of writes/changes across the entire portion of the file system the engine is monitoring, and sync those changes in the order they happened on your file system. This is slow and possibly transmits more data than necessary -- what if some of those changes were you writing new data into a document, saving it, then reverting those changes and saving back? However, it most faithfully preserves the sequence of changes and is the most "safe" option. It assumes the writes happened the way they did for a reason and doesn't make any assumptions about file formats or relationships. (This *seems* to be the way Dropbox works, BTW, from my limited testing and research.)

2) Faithfully preserve the sequence of writes/changes *per file*, but optimize those so all pending changes to a file happen at the same time before it moves on to the next file. This method will guarantee that your changes to each specific file are preserved, but for a (usually very small) period of time, any files that have relationships to each other (such as those inside a Scrivener package) *may not* be consistent with each other.

3) Try to optimize, in some fashion, the sequence of writes/changes by removing those that don't seem to be necessary. If I've saved to the same file three times, skip the first two saves and just go to the last version, right? I save time and bandwidth -- everybody wins! Unless, of course, those intermediate states track to other changes in other files that now I lose because I assume my files are independent of each other.

Say, for instance, those three saves were renaming the file (because I changed the title)[1], creating a new file with the old name, and adding content into it. By throwing those first two saves away, I've introduced corruption. And yes, some sync engines do try to cut corners like this.

The problem gets compounded when you add in multiple clients all trying to push/pull changes at the same time, or the fact that on a Windows system you don't HAVE a package, you really are dealing with a folder with lots of files and sub-folders, and now you have multiple copies of the Scrivener document open at the same time...all of these levels of complexity add more uncertainty to the mix and are more likely to expose and exercise the flaws in 2) and 3) to fatally corrupt your project.

[1] Yes, I know my example with renaming is no longer how Scrivener does it -- UUIDs are good -- but the point is there -- mess with writes and write order and you can introduce unintended inconsistencies in a non-trivial file system.

[devinganger]

A .DOCX file is not equivalent, precisely because it *is* a ZIP file. All of its components are compressed in a single unit and travel together. It's not possible to email part of a DOCX file; it is trivially easy to email part of a Scrivener project.

To be technically accurate, it *is* possible, you just have to unzip the .DOCX. However, it takes deliberate non-standard action, so is not *trivially* possible, so we take your point.

[KB]

Very, very quickly (because I'm snowed):

- iCloud has no problems syncing a folder or a package file or anything else. That's not exactly the problem.

- There are generally two types of app: shoebox apps such as Photos and iTunes (and Ulysses, for that matter) which have a single window that manages a single database of files, and document-based apps, which use a different window for each file, and whose files may be anywhere on your computer.

- Shoebox apps provide a window on a large amount of data and just save, for instance, a main index file and the single file affected if you make any changes, without touching the other gigabytes of data in the database.

- Document-based apps save the entire document every time there is a change. Document-based apps can use flat files or file packages. Either way, the way the technology is designed by Apple, when you make a change, the entire file or package is re-saved (although the OS uses some clever atomic saving to mitigate against data loss).

- iCloud is, for the most part, built around these paradigms, and integrated into the frameworks based on these common uses. (Which isn't to say it can't be used in other ways, but whereas almost any programmer can get iCloud working for these use cases, you would need some serious experience in sync technologies on board to bend it to other purposes and be prepared for a major project.)

- Scrivener, however, is a bit of a hybrid between these two app types. It uses macOS document-based technologies in order to present each project in its own window and manage a single .scriv file for each project. However, it overrides these technologies at quite a low level in order to support huge projects, since each project is akin go to a mini-shoebox app. It uses a file package (a folder that looks like a single file on any Mac Scrivener is installed), but it does not--unlike standard document-based apps--re-save the entire package every time you make a change to a project. It would be pretty awful, of course, if you have a gigabyte of PDF files in a project and have to wait ten minutes for all of that to be re-saved every time you make a change to a bit of text. Instead, Scrivener overrides all of the standard mechanisms so that it only saves files that have been changed inside the package and leaves everything else alone. Unlike other package/document-based apps, it also overrides all of the reading methods so that it only ever reads into memory files required right now. (The standard macOS mechanism is to read the entire file package into memory--but again, you don't want Scrivener reading a gigabyte project into memory every time it opens.) This ensures that Scrivener can run fast with any size of project, and ensures stability. However, it means that it bypasses all of the standard mechanisms for iCloud support.

- Shoebox apps can generally work fine with iCloud because if some files are synced and others are not, they can just show this and notify the user that such-and-such a file isn't downloaded yet if they try to click on it.

- Document-based apps generally work because the whole file (package or flat) is saved in one go, so the whole file (or package/folder) arrives as a single entity on other machines. The app can then simply reload the whole thing. (This is also made much simpler by usually only having a single editor showing a single piece of data such as a text document--unlike Scrivener with the many, many UI elements showing different pieces of data, such as the outline, current text, maybe another text, synopsis and so on.)

- Scrivener agains falls between these, though. A project is a single thing, and a project cannot work reliably if not all of its data is synced. For instance, suppose you make changes to text A on computer one, then move to computer two. But the internet is down and you don't notice that your changes haven't synced. So you make changes to text A there, too. At this point, because iCloud (unlike Dropbox) doesn't make conflicted copies, all the changes you make to text A on computer one are lost forever. And that's a minor example. You could end up losing entire outlining structures.

- And because Scrivener saves and loads piecemeal, that's the way the data would arrive, and Scrivener would have no idea of when all the project data is available to run safely.

This didn't turn out so quick. Anyway, that said:

1. It is perfectly safe to use iCloud Drive to share your projects between computers just as you use Dropbox. As with Dropbox, you just need to make sure that your projects are fully synced and closed on one computer before moving to another computer, and then make sure they are fully synced before opening them again. This is not as smooth or simple as the automatic iCloud sync that is possible with more standard document-based or shoebox-based apps, but it is the payoff for everything else that Scrivener can do.

2. I do have plans to make it so that our iOS version can open and desktop sync projects using iCloud Drive too, and I think I have a solution for this. However, it will take some care on the part of the user, and there is a lot of work involved on my end to get this working (I've been going back and forth with Apple over a solution for this for months, picking apart the technical details--the Apple engineer also points out how iCloud and the general frameworks aren't designed around the sort of thing Scrivener does). So, this won't see the light of day until sometime later on in 2018. It's being investigated, though.

I hope that clarifies some of the thorny issues involved.

All the best,
Keith

[lunk]

On top of all that you have this little checkbox under iCloud in System Settings in MacOS:

Screen Shot 2017-11-28 at 11.44.24.png (47.19 KiB) Viewed 26702 times

If it is ticked like this, it means that some things you add on another Mac or on an iPad, won't show up on the other Mac because OS X thinks that you don't have space enough and probably don't need it. I used a photo scanning application on my iPhone and the folder created by the app was visible on all my iDevices but not on any of my Macs, until I unchecked this box.

So iCloud is sometimes a bit too smart in its attempt to help us.