Seeing Dropbox right...

[danwdoo]

For those only wanting to use Drop Box as an automatic backup service, it might be worth considering not saving to a Drop Box folder directly, but instead using a synchronization tool to perform a one way copy of files to a folder which is then synced with Drop Box. This would prevent any file sharing or other issues while still providing one of the basic benefits, offline backups. There are many good free sync tools that can be used, so it may be something people may want to consider given some of the potential sync problems.

[narrsd]

Dan, the thing is, doing this would unfortunately then reinvent the problem.

Whenever the files in a Scrivener project are separated, then Dropbox can and will create the differently-named 'conflict files', every time two or more Dropbox contents are changed.

Then portions of text, and also whole scrivenings, depenwill be apparently lost.

That's the problem we're trying to assure won't happen in future, and the only way that logically will do it, as far as I can see, is:

- have the Scrivener projects in single files

- do a manual merge of the enclosed projects, whenever Dropbox indicates that these single files have accumulated mutual differences from the original, which it will do by making its renamed copies.

Regards,
Clive

[danwdoo]

Dan, the thing is, doing this would unfortunately then reinvent the problem.

Whenever the files in a Scrivener project are separated, then Dropbox can and will create the differently-named 'conflict files', every time two or more Dropbox contents are changed.

Then portions of text, and also whole scrivenings, depenwill be apparently lost.

That's the problem we're trying to assure won't happen in future, and the only way that logically will do it, as far as I can see, is:

- have the Scrivener projects in single files

- do a manual merge of the enclosed projects, whenever Dropbox indicates that these single files have accumulated mutual differences from the original, which it will do by making its renamed copies.

Regards,
Clive

I'm not sure I completely follow what you mean by files being separated. Using this method, the files are copied to another directory that should contain all the files, not just the changed ones. It should be an exact replica of the actual scrivener data folder. It might be just be that I'm not really sure how the scrivener file structure works. Is scrivener reusing/renaming filenames? If so, this would certainly cause this. I will say, based on what I've seen, it would be tricky to piece together a project based on the files with the information I have found. I would like a better understanding of the files in case I did ever have to attempt to recover a project manually.

[narrsd]

Dan, there are so many issues here, actually. I have background both in the general problem and in some specific things I was asked to do on a consultancy once for the BBC.

There are two fundamentals:

- one is the need for what is called 'atomicity' in replications. This means you deliver nothing but a fully integrated and consistent result: the total state of a multi-component Scrivener project at the point _you_ saved it. This is not something Dropbox or the like can possibly provide, the way they are designed, unless the Scrivener project is archived into one single file for Dropbox to deal with.

- the other is the need for intelligent merging. Only a person, and somewhat an informed person at that, can successfully compare and merge two versions of a Scrivener project. It takes knowledge, after you have the RTF and structure editors that Scrivener provides. And you only get the chance to do it if the first condition is fulfilled.

Without these, you will at any mysterious point get text that can't be found, text that appears to be lost, even though at other times the Dropbox seems to work.

The simplest case for it is when one project-internal RTF file, which holds a scrivening, gets modified at both ends after the naive project directory is put into Dropbox (or closed from Scrivener there). Then Dropbox will create a re-named version of that RTF file, which you will never see in Scrivener. You could only manually find such files, and then, guess what, manually edit them in -- which you can't do from Scrivener itself.

There are all kinds of variations on the problem, and in proposing a solution, I found in reflection this morning that I'm still depending on Dropbox getting certain potentially tricky things right. Probably they do -- as it's a well-recommended program for many persons.

If you need another example from the field, that would be what happened when Dropbox-like programs first appeared 15 years ago, and the first thing that smarty people thought to do with them was run their Quickbooks accounting through files shared this way. Instant unreliability, for similar reasons, that a database also requires the 'atomic = all at once or refuse changes' approach. To actually have a database do this over distance is extraordinarily demanding, as someone's husband above has mentioned, and costs it from Oracle or others.

Why do people think Dropbox just ought to work? Well, naive Dropbox and Scrivener can work, indeed -- so long as their users are absolutely not naive.

If one or more persons clearly understand and absolutely accurately follow the sorts of absolute limitations with close, consistent communications Keith has outlined, then you can avoid the areas where trouble can come.

One mistake in that regime, and then trouble is perched on your shoulder. As in, two persons writing in the same area of a document -- or one person doing so from two locations without remembering to have closed Scrivener down fully before they left the one where they're not. Or additional scrivenings added, same fault circumstances.

In the end, choices are often a matter of trading off apparent convenience with risk. I've hoped the discussion here would end up with a way to take the risk out, but still leave Dropbox useful. That we've done that and raised Google Docs as the way to do true 'writing together, seems worth the candle.

Best to each,
Clive

[danwdoo]

Ah, I think you missed a key statement in my original post. I was only referring to situations where drop box is to be used as a backup method, meaning the files are not accessed on multiple machines and syncing is only one way. With this, there is the option to even capture the entire scrivener project and drop it into the drop box folder. This can even be done as a zip file. I was not intending this be used in any way to access Scrivener on multiple machines. This is simply an easy way to achieve offline storage for the project in case a laptop gets lost or stolen but you don't want to save to the actual dropbox folder.

[narrsd]

Dan, you're quite right - I did miss that crucial point. Hence going through the thoughts again, possibly not a complete waste .

If you wanted to do automatic offsite or backup storage this way (not a bad idea for cases), then would think doing it from Scrivener backups would be a nice way to go. You can use your drop-folder to re-drop into Dropbox, and get the benefit of the unique names, so that you not only back up, you back up over time.

So that's a nice idea, and sorry I mistook it. I do think it's better than simply backing up to the Dropbox, because you have the local copy which won't be affected by anything done at the other end of the Dropbox, hence real diversity. Diversity is good, in backups and time-saves, my experience.

Best,
Clive

[danwdoo]

I like the thought of knowing nothing will be messing with my files, but having an offline backup is crucial. Ultimately, it would be nice to have an automatic 'backup on Scrivener exit' feature so it drops a backup of the project into the drop box folder anytime I close the program (or at other scheduleable intervals). The downside would be a brief (hopefully) pause on exit while this is done. It's doable with other tools, so it's certainly not a mission critical feature but it would be nice.

[kewms]

I like the thought of knowing nothing will be messing with my files, but having an offline backup is crucial. Ultimately, it would be nice to have an automatic 'backup on Scrivener exit' feature so it drops a backup of the project into the drop box folder anytime I close the program (or at other scheduleable intervals). The downside would be a brief (hopefully) pause on exit while this is done. It's doable with other tools, so it's certainly not a mission critical feature but it would be nice.

Already exists in the Mac version. (See the Backup option under Preferences.) I don't know about the Windows version, but if it doesn't have it now I assume it's coming.

Katherine

[danwdoo]

Great! Then I'm sure it will eventually come to the Windows side as well

[devinganger]

This all seems to more a job for .tar, than .zip. Compression is nice when you need to preserve space, so it is great for backups, but for a working format---I guess I don't get the point of compression, either. That's a ton of processing overhead, and it would effectively cripple auto-save for even a small-ish project of 20mb. Tar is a lot faster, and it's pretty easy to extract and update individual component elements on the fly. It would still be slower than a file-system by a long shot, but in most scenarios the slow-down would be barely noticeable---and presumably anyone using this method would be aware of the fact that one huge file is not going to be as much of a performance demon as files and folders.

Sorry for the late hit on this, but I'm just finally getting caught up on forum posts.

The modern (.docx, .xlsx, .pptx, etc.) Word file formats are doing this exact trick with using the outer file as a ZIP-format wrapper. There were several reasons that Microsoft decided to use this format:

1) Office docs are often bulky, so the same document saved in .doc and .docx often shows a substantial space savings in the newer format.
2) Windows includes the ability to open/parse ZIP files in Explorer natively, so the appropriate routines are in place from Windows XP forward.
3) The new Office docs are in fact just collections of files -- mostly XML, but things like imported pictures, etc. are simply imported into the ZIP archive in the appropriate place and linked via XML. As a result, new APIs could be created to manage contents of the new file formats without requiring the use of Office COM objects.

The decision to use the ZIP-format wrapper has certainly turned out to be a good one for Office 2007 and 2010. It's actually reasonably performant (although, granted, Word's not auto-saving everything as you go). Even for netbooks, the ZIP compression algorithm doesn't require a ton of CPU. It also reduces the number of active filehandles required, and I believe I remember hearing one of the Office developers talking about how that actually coding for file I/O much simpler.

It would certainly be nice to see this as a (non-default) option in some later version of the program.

[morvenwestfield]

I skimmed over this, so apologies if I missed something, but it seems that using Dropbox is a problem only when collaborating.

I'm writing a novel and I will be the only one touching the files, ever. (When it goes to edit, the editor will get a Word version and I will stop writing/editing at my end.)

So, for single-person use, is Dropbox sufficient for day-to-day writing?

[AmberV]

There is actually very little difference between using it to collaborate and using it all by yourself. "Collaboration" just means more than one individual loads the .scriv file, and must follow all of the same rules you must follow, such as not opening it twice on different computers, allowing it to fully upload and download at the end and beginning of each session, etc. Collaboration just requires more communication to coordinate---which is of course nothing new.

[erehwon]

Having read all the way through this thread, I think we're arguing in circles. Naive tools cannot solve issues of changes by multiple people. That's the sort of issue source control systems were designed to handle. However, they are not really for naive users. Any source control system will ask the user to merge and resolve conflicts when there are changes made by another user.

Putting a scrivener project and all the rtf files etc into a distributed source system like GIT does work, since the files are almost all "text" files, and the merge process will show the individual "lines" to be merged. For RTF these will be paragraphs or control strings, and the problems will occur when there are changes to both versions of a paragraph or control string. The user will need to edit manually. And you need to avoid "binary" formats that source control tools usually cannot merge. At least with a distributed source control system, you can back track to find changes that collided and deleted someone's work - each person's delivery (in GIT a commit) is atomic. You can find the deleted text again, and re-apply it.

I don't use dropbox, I do use GIT with GitHub, but then I'm usually the only person changing my scrivenings. I also work in the software industry, and use source control tools tdaily. If you are the only user, working from different location, then drop box should work fine.

I don't think you can expect code to solve all your collaboration issues - if you are getting into the realm of "professionals" then you need to move from naive user to work and think like a professional project team.

For colaborations, you still need to impose discipline and work practices, like portioning out work. That way, only one person is changing a file or files. Break your colaboration into several scrivener projects, one per collaborator. Then one of you acts as project manager/librarian, and handles the merging of changes from contributors into one master.

Or export chapters to individuals, and then deal with importing their changes.

David

[AmberV]

For colaborations, you still need to impose discipline and work practices, like portioning out work. That way, only one person is changing a file or files. Break your colaboration into several scrivener projects, one per collaborator. Then one of you acts as project manager/librarian, and handles the merging of changes from contributors into one master.

Another way of doing this is to have everyone share a .scriv project on Dropbox and have a strict schedule on when you can open it; or an effectively way of communicating if it is currently open. Scrivener will try and protect you from opening an opened project, but its better to have a system. Then a method of using labels or some other obvious meta-data flag for "check-out" is important. You check out pieces of the outline and mark them with your name; drag them out of the project and into the binder of your offline working project. You work on them and when you are done, you drag the pieces back into the collaborative project and remove the check-out. Labels are pretty good for this, since you can tint icons and such with the colour. Give each person their own label and now you know when something is out, and who has it, so they if they forgot to check it back in or never got around to editing it, you can ask and clear it. It's not strict, and it requires cooperation and communication, but it can work well.

[erehwon]

There is one issue that perhaps Lee could address in the future. Suppose you and I are collaborating on a project, and we download the same .scriv project, and then each of us proceeds to create a new chapter. Scrivener's node numbering scheme will assign each of our new chapters the same number - the next in sequence. When we try to sync or upload, there WILL be a clash. Scrivener could use some form of UUID for each node/file so that we both will have a unique string assigned. Until then I think collaborations really need to have seperate projects for each collaborator.

Your "checkout" method will work provided only one person can create new files.