Seeing Dropbox right...

Treading softly, but it did occur to me on waking this morning that maybe a good solution for several issues would be to make a zipped project the actual default Scrivener format.

Now ducking, but here are some thoughts behind this.

• it would solve the Dropbox problem as well as it can be solved for now.

• it would remove an issue I haven’t talked about that comes in Windows 7, that if you are using Libraries as is the default, and viewing by date, your Documents library view (think My Documents) becomes filled with small bits of recent Scrivening.

• It’s actually reasonably common for file formats to be containers (mov, wmv, etc.), and often enough using zip as the enclosing format (lost for example, but know it’s so).

• new-format Scrivener projects would get their own tag; for example, MyProject.scrv

• Long term safety of texts would not be affected. You would always be able to open the scrv file with a zip utility, or rename it to MyProj.zip for convenience in this. The RTFs, notes, etc. would be right there.

Well, there it is. Shoot away, please.

Hoping not to have been rude last night, also, in a moment’s reaction about directions of the discussion.

Am sure the right answer is rather that we talk about anything, and then a new Sticky posting is made later by Scrivener HQ when they decide what further they want to do, if anything, towards official Dropbox use or recommendations.

Guess we are all still learning, about the ‘new communcations’.

Regards,
Clive

To allow DropBox to be a viable tool to Scrivener users who aren’t collaborating, the zip container idea would have to be optional. The reason being is that a Scrivener project can get quite large, and even the tiniest of changes to your project would mean a sync of every byte of every file in the entire project. The sync would take a long time for even a modestly sized project with just a few pictures and webarchives.

As it stands, individual files that change in a project require only a few seconds to sync up if they are small files, which makes DropBox a very handy and unobtrusive utility.

Hmm. Very good point indeed.

I like the idea of a configurable option – best to please the most people and their ways of working, and makes clear the older ways are always preserved.

I had meant to suggest anyway that Scrivener would still open unzipped projects; just prefer by default the new arrangement.

Also, though, Zip archives can be written uncompressed, used just as a container, which is enough to solve the problem we want to solve here about avoiding inconsistencies.

Then, as far as I know, Dropbox does incremental syncing – transfers just the portions of the file that’ve changed. If the Zip archive isn’t compressed, that should mean only the changed areas transfer, preserving the quick response you’d like.

Incremental syncing is also a quite long-known tactic, and I imagine any alternatives to Dropbox would also work that way.

Good thoughts to spark…

C.

I had been under the mistaken impression that the whole file would sync with even the slightest change in any part of it. Turns out I’m wrong.* So moving to a zip archive would actually be a viable option, if it can be done for both the Windows and Mac version. I’m not seeing a lot of down-side to this, but I would think that Keith and the gang would have considered this as an option and discarded it for some good reason; the longer I’m here, the more I notice that hardly any idea presented by a user is a new one on Keith.

• I am taking the DB website at it’s word… dropbox.com/features : “Dropbox transfers just the parts of a file that change (not the whole thing).”

Good man.

Just to say, you were righter than you were wronger originally, as ordinary default zip files would indeed be completely sent, as the compression would make their entire content different even for the smallest changes.

It’s the choice to use uncompressed, available both in zip utilities and programming libraries that solves the puzzle – and can be easily tried out.

Agree the proprieters are dependably pretty complete and cogent in their thinking – just that angles do turn up, and make life interesting .

Cheers, robertdguthrie, and cheers to any ‘gruff and quirky’,
Clive

I would suggest that if zipped archives are desired as a format for project storage, the default be left as is and the zipped format be selectable as an option. Changing the default format to zipping probably would not be a good idea. I think the majority of users would prefer to keep things text-only. I don’t use Dropbox for collaboration, just cloud-based backup, so I prefer text.

But making compression a project-based option sounds like a reasonable thing to allow for the subset of users that would prefer it.

Throwing this out there, but more ‘advanced’ users will find ways and other apps that allow them to work with Scriveners better, at least as they are currently designed. Zip is/could be nice, but certainly aren’t the only means.

I for one, put sensitive projects into a TrueCrypt Volume that I toss into Dropbox and then mount the volume as needed. This would also work perfectly for Scriveners’ files, but more as a solution if the current format of Scriveners aren’t altered rather than as something that Scrivener could integrate as a solution.

TrueCrypt with Dropbox won’t allow for simultaneous editing at all, but Dropbox isn’t the tool for that sort of thing anyway. Toss the scriveners into a TrueCrypt volume, mount it, alter it, unmount it, then the changes will re-sync and you’re good to go and secure to boot and I believe that other users would be locked out while you have it mounted (though I’m not positive on this count). Slight issues might be long upload on the front end and/or the possibility of incorrectly judging the size you need for the Volume and thus either wasting space in your Dropbox or running over within your Volume itself.

I guess the end of this is that I thoroughly plan on putting my Scriveners into something that makes Dropbox think it is a single file anyway, but it could be nice to have it treated as a single file straight out of the box.

The thing is, we’re not talking about compressed, just “combined”. On the mac, a scrivener project’s individual files are mostly hidden from users because the project’s directory is named in such a way that OS X treats it and it’s contents as one file. Making use of zip for the default file format will in no way lock you out of seeing the individual files (which you aren’t supposed to touch under normal circumstances). If you need to get to them, rename the all-in-one scrivener project to scrivproj.zip, and use a free/built-in zip utility to extract the bundle into individual files & directories. Anyone who really wants to see those files would be capable of getting to them with minimal effort, while the vast majority of people who are used to dealing with a single Word file will never be faced with the baffling array of files like 1.rtf, 2.rtf proj.scrivx settings/ snapshots/ …

This isn’t like changing the format into something that can’t be accessed by mere mortals, like the MS Office file formats, it’s just a handy way of keeping all of a project’s files together on any platform where Scrivener works.

This all seems to more a job for .tar, than .zip. Compression is nice when you need to preserve space, so it is great for backups, but for a working format—I guess I don’t get the point of compression, either. That’s a ton of processing overhead, and it would effectively cripple auto-save for even a small-ish project of 20mb. Tar is a lot faster, and it’s pretty easy to extract and update individual component elements on the fly. It would still be slower than a file-system by a long shot, but in most scenarios the slow-down would be barely noticeable—and presumably anyone using this method would be aware of the fact that one huge file is not going to be as much of a performance demon as files and folders.

As far as collaboration in a group or enterprise environment. Why reinvent the wheel? It seems like distributed source control solutions, such as Git and Mercurial, have already solved this. Why not simply open up the the file format and have the source control repository take care of the heavy lifting? By using a DVCS enables me to save and version locally. I have tag and branch certain ideas I might have and then merge them locally. If I want to share them with someone I can use something like GitHub.com to share by thoughts or changes with the rest of the world. You could even comment and collaborate using this type of tool.

Just my thoughts but I am a software engineer by trade, so this may be overkill for the average user but having said that you could abstract the average user from the complexity of source control tool by supplying the same constructs through the main UI of Scrivener.

Food for thought

Joe, I think we went over this ground earlier in the thread.

• The first thing is, the basis of Scrivener is formatted text. Thus a source control system isn’t going to work for this, unless you know of one that does edit/merge for RTF.

• The second thing is that Scrivener also has another level of structuring – the scrivenings in binders. A special-purpose editor would have to be written to merge the XML files that control this – and it would need to be integrated sufficiently with the formatted text edit/merge that complex edits on a project would be fully visible and manipulable.

It would be instructive to dream something like this up, but a great deal of effort to make it complete and workable. A look at Word’s document compare/merge will give an idea of the level of complexity just for the RTF merge aspect.

I haven’t so much more to say here, except that the idea has been to hew to what non-technical persons are comfortable with and practiced at using, so as to not get in the way of anyone’s writing, and thus not losing track of Scrivener’s purposes. I think that means Dropbox and Zip files, doesn’t it?

I also appreciate what Ioa is saying above about tar, etc., but really, a non-compressed zip just would do the job, wouldn’t it, including high-efficiency Dropbox synchronizing?

Adaptation always seems the actually sophisticated thing; at least in experience.

Regards to each,
Clive

Viable for what? If your goal is collaboration, use of a zip archive doesn’t even begin to address the issues. It helps avoid out and out file corruption, but does nothing to help with versioning and change management. And let’s not even talk about trying to do simultaneous editing in real time.

Katherine

The point I’ve been convinced of is that .zip can be used just like a virtual disk without compression , but on Windows, Linux, as well as Mac; Am I wrong in my understanding that newer versions of Windows come with a built-in Zip utility, but not tar? Using zip without compression on the project keeps all of your project’s files looking like just one file, so Drop Box no longer mucks with the internals of the project when there’s a conflict. Instead, it creates an entirely new “conflicted” project, which is much more noticable than a 324(conflicted).rtfd file appearing in your project. But again, without the compression, it shouldn’t slow down saving (you can tell a zip utility to replace individual files), and drop box will only transmit the new bytes, not the entire archive.

Viable for using as a “file format”… the discussion has drifted a bit (though it’s incredibly on topic for these forums). It’s also helpful in that if something gets conflicted, you get a new copy of the project (equivalent to a blaring siren), instead of files within your project that can remain unnoticed for days, weeks, or even months.

As for simultaneous editing; no, zip archives are not the answer. The answer is for Keith to hire about 10 more Mac programmers (bringing the total to 11), and devote a goodly chunk of their time to shoe-horning in the ability to simultaneously edit a project. Until Keith wins the lottery, or Scrivener rivals Angry Birds in number of sales, I think you’re going to have to give up on simultaneous editing of Scrivener projects. If that’s your primary concern, Google Docs seems the way to go from what I’ve read in this discussion.

Aye, Google Docs. For rapid collaboration.

Great thoughts from everyone; this is one of the most conflicted challenges in computing (the collaboration part), and it probably takes a Google to get the software right. Even they falter on the user view (e.g. Waves).

Hope I’ve taken right tones through this for everyone. Writing is my interest too, among others in that space. Technology one may understand able to be of certain help, as Scrivener; beyond such a point, often there be dragons. As we have met.

Regards to each,
Clive

For those only wanting to use Drop Box as an automatic backup service, it might be worth considering not saving to a Drop Box folder directly, but instead using a synchronization tool to perform a one way copy of files to a folder which is then synced with Drop Box. This would prevent any file sharing or other issues while still providing one of the basic benefits, offline backups. There are many good free sync tools that can be used, so it may be something people may want to consider given some of the potential sync problems.

Dan, the thing is, doing this would unfortunately then reinvent the problem.

Whenever the files in a Scrivener project are separated, then Dropbox can and will create the differently-named ‘conflict files’, every time two or more Dropbox contents are changed.

Then portions of text, and also whole scrivenings, depenwill be apparently lost.

That’s the problem we’re trying to assure won’t happen in future, and the only way that logically will do it, as far as I can see, is:

• have the Scrivener projects in single files

• do a manual merge of the enclosed projects, whenever Dropbox indicates that these single files have accumulated mutual differences from the original, which it will do by making its renamed copies.

Regards,
Clive

I’m not sure I completely follow what you mean by files being separated. Using this method, the files are copied to another directory that should contain all the files, not just the changed ones. It should be an exact replica of the actual scrivener data folder. It might be just be that I’m not really sure how the scrivener file structure works. Is scrivener reusing/renaming filenames? If so, this would certainly cause this. I will say, based on what I’ve seen, it would be tricky to piece together a project based on the files with the information I have found. I would like a better understanding of the files in case I did ever have to attempt to recover a project manually.

Dan, there are so many issues here, actually. I have background both in the general problem and in some specific things I was asked to do on a consultancy once for the BBC.

There are two fundamentals:

• one is the need for what is called ‘atomicity’ in replications. This means you deliver nothing but a fully integrated and consistent result: the total state of a multi-component Scrivener project at the point you saved it. This is not something Dropbox or the like can possibly provide, the way they are designed, unless the Scrivener project is archived into one single file for Dropbox to deal with.

• the other is the need for intelligent merging. Only a person, and somewhat an informed person at that, can successfully compare and merge two versions of a Scrivener project. It takes knowledge, after you have the RTF and structure editors that Scrivener provides. And you only get the chance to do it if the first condition is fulfilled.

Without these, you will at any mysterious point get text that can’t be found, text that appears to be lost, even though at other times the Dropbox seems to work.

The simplest case for it is when one project-internal RTF file, which holds a scrivening, gets modified at both ends after the naive project directory is put into Dropbox (or closed from Scrivener there). Then Dropbox will create a re-named version of that RTF file, which you will never see in Scrivener. You could only manually find such files, and then, guess what, manually edit them in – which you can’t do from Scrivener itself.

There are all kinds of variations on the problem, and in proposing a solution, I found in reflection this morning that I’m still depending on Dropbox getting certain potentially tricky things right. Probably they do – as it’s a well-recommended program for many persons.

If you need another example from the field, that would be what happened when Dropbox-like programs first appeared 15 years ago, and the first thing that smarty people thought to do with them was run their Quickbooks accounting through files shared this way. Instant unreliability, for similar reasons, that a database also requires the ‘atomic = all at once or refuse changes’ approach. To actually have a database do this over distance is extraordinarily demanding, as someone’s husband above has mentioned, and costs it from Oracle or others.

Why do people think Dropbox just ought to work? Well, naive Dropbox and Scrivener can work, indeed – so long as their users are absolutely not naive.

If one or more persons clearly understand and absolutely accurately follow the sorts of absolute limitations with close, consistent communications Keith has outlined, then you can avoid the areas where trouble can come.

One mistake in that regime, and then trouble is perched on your shoulder. As in, two persons writing in the same area of a document – or one person doing so from two locations without remembering to have closed Scrivener down fully before they left the one where they’re not. Or additional scrivenings added, same fault circumstances.

In the end, choices are often a matter of trading off apparent convenience with risk. I’ve hoped the discussion here would end up with a way to take the risk out, but still leave Dropbox useful. That we’ve done that and raised Google Docs as the way to do true 'writing together, seems worth the candle.

Best to each,
Clive

Ah, I think you missed a key statement in my original post. I was only referring to situations where drop box is to be used as a backup method, meaning the files are not accessed on multiple machines and syncing is only one way. With this, there is the option to even capture the entire scrivener project and drop it into the drop box folder. This can even be done as a zip file. I was not intending this be used in any way to access Scrivener on multiple machines. This is simply an easy way to achieve offline storage for the project in case a laptop gets lost or stolen but you don’t want to save to the actual dropbox folder.