The Vapour Trail of Scrivener for iOS

We’re over halfway through 2014, so I wanted to give our users a quick update on the progress of the iOS version of Scrivener. We receive emails and forum posts daily asking how it is coming along, and because of how long it seems to have been in development, we've even had users assume that it has been abandoned (we've even been accused of lying about it being developed at all, but I promise that’s not the case, even if we have continually underestimated the time involved).

The short version - the "tl;dr" version as the cool kids say - is that we still expect it to be finished this year, but now very much doubt it will be released until next year (2015) because of the amount of testing we need to put it through before letting it loose on the world. Don't misunderstand - we are closing in on the final stages of development right now. But these final stages can still take a while, and I'd rather tell users now, in the middle of August, that we might be pushing into 2015, rather than keep my fingers crossed that we can squeeze it into 2014 and then only announce a delay at the last minute when we realise we aren't going to make our target. Thorough testing of the iOS version is crucial, because it is designed to sync with desktop projects, and syncing can be a dangerous process - we want to ensure that any bugs that could cause data loss or corruption are truly ironed out before putting it into users' hands.

I'm aware - if only because we've had it pointed out a lot! - that we have been saying "later this year", then "next year" for two years now. Naturally, users wonder how so many delays can be possible. For a start, there’s a fairly obvious rule of software development: software development takes time (and developing good software takes a good amount of time). And then there’s the corollary to this rule: software programmers always underestimate how much time it will take (myself included).

Take the sync code, for example. Most iOS apps can take quite a simple approach to sync, using the built-in features that Apple provides for iCloud. Or they can use the Dropbox API to manage uploads and downloads. Nothing is so simple for Scrivener. One of Scrivener's main features is that it allows you to bring in research files - photos, PDF files and so on - so that you can view them next to your text and keep everything together in a single project. In order to achieve this, Scrivener requires a file format quite different from most writing software. Scrivener, in fact, is something of a hybrid between a document-based program and a shoebox app.

Most writing software packages are document-based programs. A document-based program allows you to create, open and save documents, showing different documents in different windows. They use single, flat files that contain all the text and any graphical data for each document. These files are loaded entirely into memory when opened, and written out in their entirety as completely new files when saved.

A "shoebox app", in contrast, usually has a single window that allows you to interact with many different files, opening them inside the one window. There will be a single folder on disk storing all of these files, but you usually won't touch that folder at all. iTunes and iPhoto are examples of shoebox apps.

Scrivener is a bit of both. As a user, you interact with your Scrivener projects as though they were documents - each project is opened in a different window, and you have a .scriv file on disk for each project. But those .scriv files are really folders containing many different files - they are not flat files at all - and you navigate between them in the project window via the binder. This means that Scrivener doesn't have to read these files entirely into memory on opening them, or save the whole file when saving. This is very important for projects that have a lot of research in them - you don't want Scrivener storing a gigabyte of PDF files in memory while you are typing, or having to write all of this data to disk every time auto-save kicks in. This file format allows Scrivener to load into memory only the data that the user needs for a particular session, and it only ever needs to save what has changed (which also reduces the chances of catastrophic data loss caused by file system save errors).

The only downside to this file format is that whenever Apple introduces new technologies - Versions, iCloud, Handoff - it only provides out-of-the-box support for simple document-based apps. All of which to say: we have had to put together, from scratch, our own syncing solution so that our iOS version syncs with our desktop version. (We will use the Dropbox API, and hopefully other syncing APIs as time goes on, but all the conflict resolution handling has to be done by us.) Over the past couple of years, we've spent a great deal of time discussing how this could work, hashing out various ideas and solutions that could deal with the conflicts that will inevitably arise if someone works on both the iOS and desktop versions while offline so that the changes become out-of-sync. (Everything has to be considered: what happens if the user edits scene A on the mobile version but then also edits it on the desktop before there has been a chance for them to sync? What if the user's kid opens up the app on the iPad and hammers away at their precious novel? How can we tell the difference between a scene that's been deleted from one device and one that's been added on another? And so on.)

Once most of this had finally been hashed out, it then took me a month to write out the design document that covers how the code will work (a design document is necessary for our Windows team to follow as well as for me and our iOS developer). And then yet another month to write the code itself, cross-platform for Mac and iOS (our Windows team will need to write this code separately). This has been what I’ve been doing for the past couple of months, and there is still work to do on getting all the sync components that have been written into place on the Mac and on iOS and then thoroughly tested (then testing on Windows).

So, that's between three and four months on a single (but integral) feature. (And while I’ve been doing this, Tammy, our iOS developer, has been adding various “smaller” features that had to wait until late in development - printing, basic compiling, import and export, statistics, split and merge, find and replace… The sort of features that it’s easy for users to forget when suggesting that Scrivener for iOS should just be simple and only have the basics, but without which, Scrivener for iOS would feel crippled. And even some of these smaller features can take a couple of weeks each to implement, of course.)

The trouble is that it’s not only non-programmers who (understandably) don’t appreciate how long seemingly simple things can take to code - we do it ourselves. We’ll often think that a feature will only take us a day or two to complete, then find ourselves growling at the screen, desperate to have done with it, two weeks later. (This is no doubt why software and games are so notorious for being delayed throughout the industry.)

In a program as complex as Scrivener (even on iOS it requires a degree of internal complexity if it is to work with the desktop version), where various features can take a month or more to implement, the time creeps up on you and suddenly two years have passed... There are certainly programs out there that can be written in three or four months, but sadly, Scrivener isn't one of them.

Not that "development takes time" is the whole picture. Something like Scrivener can easily take a couple of years to develop properly, but it has now been two and a half years since we started work on it; we had hoped it would be finished by now as much as anyone else - we don't enjoy frustrating those of our users who write on mobile devices. The other big factor for us has been scaling up. Scrivener started out as a hobby - the software I wanted for my own writing. I wanted it so much that I taught myself how to code in order to create it. For the first couple of years, L&L was a one-man operation, and we are still a very small team. I work alone on the Mac version, two programmers work as a discrete unit on the Windows version, we have a couple of other full-time team members who write and maintain the manual and tutorials and handle all QC testing and some support, and we have a couple of part-time support staff. We have no central offices but are scattered across the world (the UK, US, Australia and Bulgaria). None of which is unusual for shareware, “artisan” software such as Scrivener.

The wonders of the information age make it easy for geographically-scattered teams to stay in constant touch and work together, but as a small company with such a setup, it meant that it was not possible for us just to hire some programmers and stand over them until the iOS version was finished. I was always adamant that I would not write the iOS version myself, because I’ve seen too much Mac software suffer as its developer has spent all his or her time working on iOS, and that would have been unfair to our existing user base. (Moreover, the Mac version is where I live for my own writing, and Scrivener is always going to be predominantly a desktop program.) So it was key to find someone self-motivated, with a good understanding of Scrivener, who could develop Scrivener for iOS and keep on developing it into the future, beyond version 1.0.

As I announced last year, unfortunately our first developer had to step away from the project because of family health issues, which had also stalled development progress throughout the eighteen months she had been working on it. We were very fortunate to find Tammy Coron, an experienced iOS developer and Scrivener user, to take over development. When she took over, she made the painful decision to rewrite much of the existing code, in part because of new technologies that had been introduced with iOS 7.0 (a rich text system at last!). Which means that while our iOS version has been in various stages of development since the end of 2011, the form it is in now has only been in development for just over a year.

We are really looking forward to releasing the iOS version - in many ways it’s become the proverbial albatross around our necks.* But it is getting there, and we hope the subsection of Scrivener users who are eager for an iOS version will be patient for a little while longer. We are genuinely grateful for your enthusiasm. I'm not known for great reserves of patience myself, so I do understand the frustration of some users who are more vocal in their eagerness, but to those I would say, please do try to remember that we want to get our iOS version into your hands as much as you want it. We're a tiny company but we have put a lot of energy and resources into the iOS version (and for all we know it could be a loss-making venture), and we think Scrivener users will be happy with the final product.

I’ll post later in the year more on what sort of features to expect, but suffice to say that it has full rich text editing, a corkboard, an outliner, an inspector and, on the iPad version, the ability to view research alongside your text as you type. It is, essentially, Scrivener on iOS.

P.S. For anyone interested, I’ve uploaded a sort of developer diary for the iOS version here:

A Scrivener for iOS Developer Diary

If nothing else, it should show at least that the iOS version has been in continual development, hitting snags along the way, even if progress has seemed glacial to the outside world.