iCloud Sync

[mprimeaux]

I sincerely appreciate the time it takes to release a quality product and am also aware the iOS version is in first release as of the time of this post. Nevertheless, I'd like to request iCloud Sync be added as an alternative option to Dropbox.

I've added polling to this post in an attempt to avoid addition forum posts on the same topic.

Cheers

[lunk]

Did you read the blog post about syncing?

"I’m aware that the lack of iCloud support will be a disappointment to some users, but please rest assured that we will continue to monitor iCloud’s development. Apple are improving it all the time, and I hope that one day, when Apple enhances the frameworks and extends the documentation to more advanced uses, or perhaps if we can find a coder with expertise in this area to help me, we might be able to support it. (We’re not the only ones who have run into these problems, by the way: OmniFocus and Notebooks have had to omit iCloud sync for similar reasons.)"

There have been numerous posts in here since Wednesday morning about other sync options, and L&L have answered the same thing, over and over: it's not technically possible for the moment. There is also a special forum entry in here, at the top, explaining it.

[derick]

@mprimeaux What advantages do you expect iCloud sync would hold over the current system?

[KB]

As much as we'd love to add iCloud support, iCloud simply isn't yet compatible with the sort of data that Scrivener needs to sync. In addition to the blog post mentioned by lunk, please see the last two sections ("Why isn't syncing automatic?" and "What about other syncing solutions?") in our Knowledge Base here:

http://scrivener.tenderapp.com/help/kb/ ... g-with-ios

All the best,
Keith

[Red Claw]

Any thoughts about putting iCloud Drive support on the roadmap?

I believe iCloud can handle package files. Anything from the iWork suite saves documents as packages. Also, the iCloud Design Guide; Designing for Documents in iCloud indicates the following;

Adopting iCloud document storage makes your app’s documents available on all of a user’s devices. A document (based on the UIDocument class in iOS or the NSDocument class in OS X) is an assemblage of related data that can be written to disk as a single file or as a file package. A file package, in turn, is a directory presented to the user as a single file and made accessible to your app through an NSFileWrapper object.

[KB]

iWork doesn't save package files - it saves archives (zipped-up packages). It hasn't saved package files for several years now.

I'm well aware of the documentation - I've been through it in depth. The trouble is that iCloud and most of Apple's standard classes expect that the entirety of a file package is written to disk at once, not in parts. It requires you to do a lot of extra work to write or read a file package in parts (as Scrivener does), at which point iCloud will not work and you need to write a lot of low-level iCloud code, which is far more complex than Dropbox.

And it's not a matter of changing Scrivener's file format - Scrivener's file format is what gives each project the ability to store many files and great deals of research while maintaining a low memory footprint. It means that Scrivener does not have to load the entirety of a gigabyte project into memory at once.

At this point, anyway, iCloud confers no advantage over Dropbox: I have taken out a support incident with Apple engineering, and the iCloud developers have told me that the only way of implementing iCloud support for Scrivener is to have it work much as the Dropbox sync does already (except that the iCloud code will be a lot more difficult and tricky than the Dropbox code, because it requires twisting iCloud in some odd contortions). It would need to download files to a temporary location and then move them into place once done, with the same progress bars and exactly the same behaviour as the Dropbox code. So iCloud support in Scrivener would not get you automatic background syncing any more than Dropbox does. I have the feeling that even if I spent several months implementing that, many of the users asking for iCloud support would not be happy with it, because I think "iCloud support" is often used synonymously with "automatic background syncing that I don't have to think about". And that's just not possible with Scrivener given today's syncing technologies.

The ultimate problem is that a Scrivener project is a bunch of interdependent files. If only some of those files are synced (e.g. sync is interrupted), a project becomes unusable, and could become corrupt. Yes, I could just implement some really simple iCloud code that syncs whole packages in one go, which might work okay for projects of only a couple of megabytes as long as there is no interruption to the internet connection during sync, but this would lead to serious data loss issues for bigger projects or on unreliable internet connections. There is unfortunately no simple solution to syncing this sort of data. iCloud is wonderful for more simple files and for databases of files, but Scrivener falls between stools.

Not to mention that I have to rewrite much of the Dropbox code to move it over to the new Dropbox API 2, as API 1 is being deprecated.

[Red Claw]

iWork doesn't save package files - it saves archives (zipped-up packages). It hasn't saved package files for several years now.

I don't believe that is correct. The iWork suite does save its documents as packages. From About Bundles:

A package is any directory that the Finder presents to the user as if it were a single file.

So yes, Pages does have a zip archive in its documents, but the document is still a package.

Now just to add some more fuel to this:
Apple has released macOS Sierra. Along with that is some extra features for iCloud Drive called "Desktop & Documents". When turned on, it moves your files from both your local Desktop folder and Documents folder to the iCloud Drive. I have both a desktop MacPro and a MacBook Pro. On the desktop Mac I upgraded the OS from 10.11 to 10.12. The settings I had for iCloud Drive were respected and the extra features was added in but kept off. On the laptop however, I installed the OS from scratch via a bootable USB thumb drive and erasing the hard drive. When going through the Welcome setup, I entered my Apple ID to setup iCloud. This turned on iCloud Drive and the feature for Desktop & Documents was also turned on. I didn't catch this for a short while. As I was restoring my files from backup I noticed that my iCloud Drive was loosing space. That's when I realizes what was going on. When Desktop & Documents is turned on the local folders in my home directory are removed and all the files get saved up to iCloud. If I looked at my home directory in Terminal, the Desktop and Documents folders look like they're there. They are not symbolic links. But when I viewed my home directory in the Finder the Desktop and Documents folder were in fact missing. I even trued the terminal command chflags to turn and unhide then but was met with a "No such file or directory" error. I have since turned off the Desktop & Documents feature and moved the files back to where they should be.

With this in mind, how would Scrivener Projects be handled? Could users run into the potential problems listed with iCloud and have a Scrivener project become unusable and corrupt?

Remember, this in only a feature request, not a demand, and at this point is nothing more than an academic debate. Everyone on the Scrivener team has done an absolutely fantastic job creating this app. Keep up the good work.

[KB]

So yes, Pages does have a zip archive in its documents, but the document is still a package.

You're misinterpreting what Apple is saying there, I'm afraid:

A package is any directory that the Finder presents to the user as if it were a single file.

The key word is "directory" - this does *not* include zip files, which are not directories (even though they may compress and contain directories) but are technically single files, and not file packages (in Apple's terminology). Pages and iWork do not save their files as packages, but as zip files, and a zip file is *not* a file package. A file package is just a regular folder of files that macOS presents in the Finder *as if* it were a single file, because an app is registered with the system that recognises the extension and tells macOS to treat it as a single file. If you move a .scriv file to a Windows machine or to any Mac that does not have Scrivener installed, it will appear as a regular file in the Finder. The same is not true of a .pages file.

You can test this for yourself by Ctrl-clicking on a file in the Finder. Any file that shows the "Show Package Contents" menu item in the contextual menu when you do this is a package file (this allows you to see inside the package); any file that does not show this option is not. Try this on a .scriv file and a .pages file, and you will clearly see that Pages files are not packages where Scrivener is.

The difference is crucial because a zip file will not have problems with its components uploading and downloading separately (because any syncing system treats a zip file as a single file), whereas a file package will (because it is just a folder of different files). Zip files are perfect for many things. .pages, .docx and .epub are all zip files containing XML files, subfolders and resources. It is possible to read files separately from, and save them separately into, a zip file (although macOS doesn't have any standard libraries good for that, unfortunately), but they are prone to the same problems as any other monolithic file format: if the zip file gets corrupted, you can't access any of the data in it. That's not a problem for file packages. If a single file gets corrupted in a file package, everything else is still fine. File packages (folders of files) are perfect for Scrivener; unfortunately, they aren't so great for automatic sync solutions.

With this in mind, how would Scrivener Projects be handled? Could users run into the potential problems listed with iCloud and have a Scrivener project become unusable and corrupt?

That's a very good question, and not something I have explored properly yet. I would hope that macOS is sensible enough to treat file packages as single files that have to be kept intact here, but we need to do some extra testing on that. I certainly wouldn't risk it, though. I chose not to turn that feature on, as I like having all of my files on my machine and not hidden away on the cloud (I regularly back up files to Dropbox and to an external hard drive, and share many files in my Dropbox folder, so that all my crucial files are safe anyway).

Remember, this in only a feature request, not a demand, and at this point is nothing more than an academic debate. Everyone on the Scrivener team has done an absolutely fantastic job creating this app. Keep up the good work.

Oh, I hope I'm not coming across in any way defensive. It's a hugely thorny and complex problem, so it's often difficult to get across the technical issues involved because they are so complicated. From the outside, in part because of how Apple has always presented it, iCloud seems as though it provides "magic" syncing that is perfect for everything. And iCloud is a wonderful technology, but it has been built to work amazingly and easily for, say, 90% of apps. If you fall outside of the areas it was built for, though, it does not offer any advantages over other solutions and is actually more difficult to implement. Apple is adding to it all the time, though, and I still have my tech support incident out with them which I intend to follow up and investigate more as soon as I get time.

Syncing is the number one gripe users have with our iOS version, and I knew when creating it that this would be the case. It's hugely frustrating, because I'd love to offer syncing across many solutions and have it work in the background, propagating across platforms without any user interaction. But because of the way Scrivener works and the things it needs to do, that's not really possible at the moment. Well, it might be possible if we had the engineers of Apple or Google, of course, but sadly we don't! We're a really small team - I am the sole designer and developer of Scrivener for both macOS and iOS. There are no other programmers working on them but me. The Windows version has two developers working on it. Ultimately I'd love to find another programmer to come on board and work with me, and I'd like to find somebody who has some expertise in syncing. Finding someone is not easy, though, especially since we don't even have offices...

All the best,
Keith

[kewms]

Um, Red Claw, are you really arguing with the developer of Scrivener about what his own software needs to function properly?

Katherine

[Red Claw]

The key word is "directory" - this does *not* include zip files, which are not directories (even though they may compress and contain directories) but are technically single files, and not file packages (in Apple's terminology). Pages and iWork do not save their files as packages, but as zip files, and a zip file is *not* a file package. A file package is just a regular folder of files that macOS presents in the Finder *as if* it were a single file, because an app is registered with the system that recognises the extension and tells macOS to treat it as a single file. If you move a .scriv file to a Windows machine or to any Mac that does not have Scrivener installed, it will appear as a regular file in the Finder. The same is not true of a .pages file.

You can test this for yourself by Ctrl-clicking on a file in the Finder. Any file that shows the "Show Package Contents" menu item in the contextual menu when you do this is a package file (this allows you to see inside the package); any file that does not show this option is not. Try this on a .scriv file and a .pages file, and you will clearly see that Pages files are not packages where Scrivener is.

Umm…what? Off-topic, I know but:
I can right-click on a Pages document and show the package contents.
And inside the package are various other files and folders, including the zip archive.
Also, if I open the Pages document from my local hard drive, (not from iCloud Drive) I can switch the document type.

So I can right-click on a Pages document in the Finder and "Show Package Contents", meaning that it is a package.

Oh, I hope I'm not coming across in any way defensive.

Not at all. I think this is the whole point of a discussion forum. So people can have an insightful conversation.

[Red Claw]

Um, Red Claw, are you really arguing with the developer of Scrivener about what his own software needs to function properly?

Katherine

If you mean to have a formal discussion on a particular topic in a public meeting or legislative assembly… (sorry, too much debate)

To which point are you referring to? The iCloud Drive or the Pages documents as packages?

[AmberV]

Hmm, I’m not sure if having to remove the document from iCloud and then accessing an advanced optional feature to decompress the package is a good argument for packages working with iCloud.

If you explode the .pages archive into a folder of files, can you drag it back into iCloud and use it normally, syncing with an iPad, then return to the computer and find it is still a package? I don’t have iCloud myself, so I can’t test that, but I can explode a .pages file, fire it off to my phone with AirDrop, open it in Pages, edit it and send it back—and I get a .zip file, not a package.

At any rate, the principle argument is whether or not it is easy or feasible for third-party developers to use package technology with iCloud technology. Even if Pages does turn out to do that just fine and you get a package after a full round-trip cycle, it doesn’t prove much. Apple breaks all manner of rules that they set upon others. They use all kinds of technology nobody else has legitimate access to.

I mean after all, look at Pages and compare that text engine with the dregs they have everyone else using.

[Red Claw]

Hmm, I’m not sure if having to remove the document from iCloud and then accessing an advanced optional feature to decompress the package is a good argument for packages working with iCloud.

If you explode the .pages archive into a folder of files, can you drag it back into iCloud and use it normally, syncing with an iPad, then return to the computer and find it is still a package? I don’t have iCloud myself, so I can’t test that, but I can explode a .pages file, fire it off to my phone with AirDrop, open it in Pages, edit it and send it back—and I get a .zip file, not a package.

At any rate, the principle argument is whether or not it is easy or feasible for third-party developers to use package technology with iCloud technology. Even if Pages does turn out to do that just fine and you get a package after a full round-trip cycle, it doesn’t prove much. Apple breaks all manner of rules that they set upon others. They use all kinds of technology nobody else has legitimate access to.

I mean after all, look at Pages and compare that text engine with the dregs they have everyone else using.

Okay, so here's the fun part. On Mac desktop, Pages by default creates a "single file" document. Even if it's saved to iCloud Drive. On iOS, Pages by default creates a "package" document. If I open a "single file" Pages document on iOS, make some changes, and then save it out the document gets changed to a "package" document. And apparently if you create a Pages, Numbers or Keynote document larger than 500MB you get a dialog box asking to save the document as a package. Save documents as a package or a single file in Pages, Numbers, or Keynote

Also, on my work system I'm running MacOS 10.10. On my home computers I'm running 10.12. So that could be another point that makes a difference.

For whatever reason, I'm not able to send files between my iPad and my home laptop via Airdrop. They're both on the same wireless network and they both have Bluetooth turned on. I can't test sending files via Airdrop at work either because we have wireless isolation turned on.

Since we've gone off topic, lets just put this part to rest. Great discussion though.

[xiamenese]

What vintage are your home computers? I can't use AirDrop with my late 2010 MacBook Air but I can with my late 2011 MacBook Pro, both running 10.11.6. Seems a hardware change in 2011 made it feasible.

Just chipping in here.

Mark

[KB]

So I can right-click on a Pages document in the Finder and "Show Package Contents", meaning that it is a package.

Oh, I hope I'm not coming across in any way defensive.

Not at all. I think this is the whole point of a discussion forum. So people can have an insightful conversation.

Huh! Well you learn something new every day! Pages used to use the file package type by default and then switched to zip. I didn't realise you could still switch it back to being a package. I eat humble pie there.

The main point, though, is how the packages are used. All of Apple's document infrastructure treats file packages as though they were single files. So, when you load a Pages file, it reads the entire package into memory. When it saves, it saves the entire package back to file (well, it might do some optimisation here and only write what has changed, but the main thing is that all the resources from the file are in memory, so it could rewrite the entire file if it wanted). I know this because Pages is a document-based application and so uses Apple's NSDocument class to represent each document, and that is how NSDocument is set up.

Scrivener also uses NSDocument, but I have overridden the save/load methods at a very low level to allow for incremental loads and saves. So, when you open a Scrivener project, it doesn't load everything into memory (which would obviously cause problems for those gigabyte projects containing tonnes of research). It only loads the XML file representing the binder structure and whatever document is loaded into the editor, along with the search indexes. Then, whenever you select something in the binder to open it in the editor, only at that point does Scrivener load it into memory. This keeps things fast and efficient even for large projects with lots of research. Likewise, upon saving, Scrivener never overwrites the entire .scriv file, but just saves any changed files within it.

As I say, I had to override things at quite a low level to achieve this, because Apple's NSDocument class isn't set up out of the box for this sort of thing. It does, however, expose all of the methods (functions) necessary to implement this behaviour, as long as the developer is prepared to override them and implement them him- or herself, which is what I did. I actually had to get help from Apple on this, though. Scrivener 1.x for Mac had a rare issue where whole .scriv packages could get wiped out. It only ever happened two a two or three users out of thousands, but it was enough to make me think that there must be something going wrong in the NSDocument structure. So for 2.0, I got some help from an Apple engineer with making sure that I was overriding everything exactly as I should.

The upshot of overriding NSDocument at such a low level is that Scrivener cannot take advantage of a lot of the features that Apple adds to NSDocument "for free". For instance, a few years ago, Apple added autosave across OS X, and this was built into NSDocument. But Scrivener couldn't use it because their autosave mechanisms work at a higher point in the chain, so Scrivener's low-level overrides cut them off. Which is fine, because Scrivener implements its own overrides anyway. But another big feature added to NSDocument was iCloud support. Scapple also uses NSDocument and supports iCloud - it was a doddle to implement. But that's because Scapple just uses NSDocument's standard load and save mechanisms and doesn't override them other than in standard ways. Because Scrivener overrides the save/load mechanisms at such a low level, NSDocument's iCloud support won't work for it - I would need to build in iCloud support at a low level, too, essentially trying to replicate what Apple does in their code but for a much more complicated system than it was built for.

All of which is to say, iCloud *does* work for package files very easily where they use standard save/load mechanisms and store all data in memory. It does not work so well for the way Scrivener uses package files. Scrivener is essentially a hybrid between a document-based app (an app that opens documents in different windows such as Pages, Scrivener's "documents" being its projects) and a shoebox app (an app such as Photos in which there is only one window that allows you to browse different data, each of Scrivener's projects acting like its own shoebox). This hybrid nature allows Scrivener to do all the things it does, but it also makes it quite unusual, as most apps are either shoebox or document-based, and naturally Apple's technologies tend to assume apps are one of the other too.

Sorry, that was a bit long-winded, but I hope it makes a bit more sense of why iCloud support is so complicated for Scrivener!