Why Scrivener Doesn't Support Versions on Lion

[KB]

We've been asked by several users whether Scrivener will be supporting Versions any time soon. The short answer is "no" - the only way Scrivener could support versions is by dropping some of its core features. For those interested, though, the full explanation follows.

There are two types of window-based programs on OS X, essentially:

1) Document-based programs. These are programs in which you work on different documents in different windows, such as TextEdit, Pages, Word and so on.

2) Shoebox applications. These are programs that have a single window interface that allows you to manipulate multiple files. Examples include iTunes, iPhoto and suchlike.

Lion's new "Versions" feature works for programs of type (1). Scrivener is, technically, of type (1) - each project you work on opens in a different window. However, Scrivener is in some very important ways a bit of a mix - it also has aspects of type (2), because each project contains multiple files and the binder gives you access to them.

One of the main differences between document-based programs and shoebox applications is the way they save. Document-based applications tend to save the entire document every time you hit Save or an autosave kicks in. Every time you save a Pages, TextEdit or Keynote file, the entire file is written to disk. Shoebox applications, on the other hand, can contain a lot of data - you wouldn't want your entire iPhoto or iTunes library to get rewritten to disk every time you added or edited a photo or song. So instead, shoebox applications only ever save to disk the changes made - the particular song you've added or photo you've edited - along with any necessary meta-data.

You may have noticed that Versions only appears in document-based applications; you don't find it in shoebox applications such as iPhoto or iTunes. And this is important, because Scrivener's saving mechanism is much more like a shoebox application. In order to support the importing of research documents - PDF files, image files, movie and sound files and anything else - Scrivener uses a saving mechanism more inline with shoebox applications than document-based programs. When you load a .scriv project, all that gets loaded into memory is the binder structure, the search indexes, and any files that are open in the editor. Other files get loaded "lazily" - that is, only when needed. When you click on another file in the binder, only then is it loaded from disk and into memory. That way, even if you have a project with a gigabyte of research in it, Scrivener can load fast and won't slow to a crawl, which it would if it had to load and hold in memory a gigabyte of data. Every time you make a change to a document in a Scrivener project, Scrivener takes note of the fact you have changed that document. At the first opportunity - after the autosave delay and no activity - Scrivener saves any files it notes as having changed to disk along with the updated binder structure. All other files that weren't changed are untouched. So, if you have a project containing a 1,000 files and only change two of them, only those two will be saved to disk during the next save cycle. This keeps things very fast and reduces the risk of data corruption, and in this way, Scrivener can support large projects and not worry about how many files the user is importing, or how large they are.

And this saving mechanism is why Scrivener is incompatible with Versions; Versions and Lion's new autosave feature are inseparable, and both are built for use with document-based applications that save the entire document to disk each time there is a save or autosave.

In other words, to support Versions, Scrivener would have to load the entire contents of the .scriv file into memory on project load and save the entire contents of the .scriv file every single time you save or autosave kicks in. It would therefore need to ensure that .scriv files never grew too large, or else everything would become unusably slow; it would, essentially, need to completely drop one of its core features - the ability to import as much research as you want, of any file type.

So ultimately, the choice is between Versions and Scrivener's research capabilities, and I think the advantages of the latter by far outweigh those of the former; users would rightfully be up in arms if Scrivener no longer allowed them to import research, or limited the amount of research they were allowed in any one project. Fortunately Scrivener has its own snapshots feature, though, which goes some way to making up for the lack of Versions support (and no, it's not possible to have Versions work with individual documents within the project - Versions isn't set up that way; it's the entire project or nothing). And of course we have an automatic backups feature that can backup your entire project every time you open it, save it or close it (depending on your preferences).

I hope that explains the situation.

EDIT: I have paid for and submitted a technical support request to Apple on this issue, in order to explore all options. Hopefully an engineer will either be able to provide a solution or, in a worst-case scenario, confirm that Versions is indeed not compatible with this saving model.

All the best,
Keith

[tenkata]

Thanks for the explanation Keith. I have a better understanding of the difference now. The feature that I like most about Versions in Lion is the ability to view the different versions (sorry for the redundancy) of the file in cover flow. The user can quickly shuffle through these files and get to the desired copy. Is it possible for Snapshots in Scrivener to be displayed in cover flow?

[AmberV]

Try using the up and down arrow keys in the Snapshot table, it's quite a bit faster than the Versions interface.

[mjboyle]

I've been one of the ones bugging you guys about it in the other thread and I just wanted to post here in support and say that while I'd love some way to work it in, I understand and appreciate you taking the time to explain.

I think some of this comes from the fact that people must use Scrivener in very different ways. Personally I use it on a per-project basis. So a book I'm writing, for instance, has its own project. That has quite a lot in it, but it isn't what I'd call huge when compared to movie filled Keynote presentations I've made or an entire iPhoto library.

I do suppose though, that some people might use a single project to store all their writing on all the projects they've ever done. Personally I don't know why you would want to do this, but if that is the way someone works who am I to say they shouldn't?

I do hope that you will keep an open mind and continue to review how this has worked out in other applications that push a lot of data, but for the record, whether you implement it or not, I'll be spending most of my day writing in Scrivener for the foreseeable future.

[KB]

Thanks for the kind words. I'm definitely keeping my eye on it, and an open mind, don't worry. In fact before posting this I had spent another couple of hours today doing further investigations, trying to hack around certain limitations. I think more methods are going to need to be exposed in Cocoa's NSDocument API for this to be a real possibility in the future - with my test implementations of autosave and Versions, things were a mess. Scrivener has lots of custom code it uses to test whether certain files are present and suchlike when opening a project, and when you try to open a Version it just chokes, because much of that data just isn't there. There are various other issues, too. In contrast, I was able to add autosave support and Versions to another app I've slowly been building (the sort-of-mind-mapping one with the temporary name TheBoard, over in Other L&L Software) in just a few lines of code. The thing is that as long as the program saves the entire document to file at once, it really is very, very easy to add autosave and Versions support; but custom save routines such as Scrivener's are a can of worms.

As for different ways of using Scrivener, you wouldn't believe the number of users who had problems updating projects of a gigabyte or more when trying to update to 2.0 (before I fixed bugs that thus became apparent) - there are clearly a fair few users with massive projects.

Thanks again and all the best,
Keith

[KB]

Just for the record, I today paid for and submitted a technical support incident to Apple, providing them with a sample project using Scrivener's save/load code, and asking whether it is possible for such a set-up to support Versions, and if so, for help in implementing it. I'll let you know what I hear back (it will probably be some weeks before I get anything solid).

All the best,
Keith

[pigfender]

...For those interested, though, the full explanation follows...

Fascinating!

[Mąřĉėłő]

As a side remark, ForeverSave will get an update to make it fully compatible with Lion, as its developer has announced. The software's future is still uncertain but until there I believe several programs will not provide the new features for some coding difficulties that Apple ends up omitting.

[tenkata]

Keith,

Thanks for your attention to these posts. This is why I continue to advocate to students and professors (and any other writers I encounter) engaged in writing to use Scrivener. I am in the process of converting my son who will edit two student publications (newspaper and magazine) at his university this coming academic year. This type of support gives me confidence that the final product will be thoughtful and in the best interest of the users. As a new Scrivener user, I have been so impressed with the level of interaction between you and the other users, and the quality of feedback to make the product even better. I've converted all my dissertation writing to Scrivener. I don't see using any other product in the future for my writing needs.

Ten

[KB]

Thank you, Ten, much appreciated - and many thanks for recommending it to students and professors (and to your son!).

All the best,
Keith

[jgindi]

Yes, thanks Keith for your engagement with your users. And, now that I read this thread, feel free to ignore my post about versions at the level of rtf docs not .scriv files in the wish list section. Sorry to bother you. Keep up the wonderful work.

[KB]

It wasn't ignorant at all! The trouble always is that Apple demonstrate all of these things and bandy around phrases such as, "It just works," leaving the rest of us to work through the finer distinctions. It certainly wouldn't be obvious to an end-user why you can't just launch individual files into Versions. Basically, though - to add to what I said in my other reply - Versions and autosave are inextricably tied, so Versions can only show all the data that is autosaved, and the main document/project window as its interface. We can hope that Apple expand this in the future though...

Thanks,
Keith

[Broken Thought]

I'm just going to throw in my vote to say I'm anti-Versions in Scrivener.

Scrivener does fine the way it handles the saving of projects already. I would rather not meddle around with it.

[robertdguthrie]

One other nail in the Versions-for-Scrivener coffin I've found is that (according to an Ars Technica article I just read), the older "versions" of the file you're working on are stored in a directory at the root of your boot drive. So even if Scrivener is somehow able to implement this feature, the older copies of the documents would not be contained within the Project.

Edit: here's the link where Versions storage is explained: http://arstechnica.com/apple/reviews/20 ... -internals

[KB]

Yes, that article is entirely correct - Versions don't travel with files if you send them to someone (which makes sense seeing as you probably don't want other people to see your earlier drafts) or move them to another computer. They are all stored in a hidden directory that is very difficult to access on your hard drive.