Seeing Dropbox right...

User avatar
narrsd
Posts: 507
Joined: Wed Sep 22, 2010 8:34 pm
Platform: Win + iOS

Wed May 11, 2011 5:49 pm Post

Treading softly, but it did occur to me on waking this morning that maybe a good solution for several issues would be to make a zipped project the actual default Scrivener format.

Now ducking, but here are some thoughts behind this.

- it would solve the Dropbox problem as well as it can be solved for now.

- it would remove an issue I haven't talked about that comes in Windows 7, that if you are using Libraries as is the default, and viewing by date, your Documents library view (think My Documents) becomes filled with small bits of recent Scrivening.

- It's actually reasonably common for file formats to be containers (mov, wmv, etc.), and often enough using zip as the enclosing format (lost for example, but know it's so).

- new-format Scrivener projects would get their own tag; for example, MyProject.scrv

- Long term safety of texts would not be affected. You would always be able to open the scrv file with a zip utility, or rename it to MyProj.zip for convenience in this. The RTFs, notes, etc. would be right there.

Well, there it is. Shoot away, please.

Hoping not to have been rude last night, also, in a moment's reaction about directions of the discussion.

Am sure the right answer is rather that we talk about anything, and then a new Sticky posting is made later by Scrivener HQ when they decide what further they want to do, if anything, towards official Dropbox use or recommendations.

Guess we are all still learning, about the 'new communcations'.

Regards,
Clive
Last edited by narrsd on Wed May 11, 2011 6:39 pm, edited 1 time in total.

User avatar
robertdguthrie
Posts: 3075
Joined: Mon Nov 09, 2009 10:06 pm
Platform: Mac
Location: St. Louis, MO, USA
Contact:

Wed May 11, 2011 6:08 pm Post

To allow DropBox to be a viable tool to Scrivener users who aren't collaborating, the zip container idea would have to be optional. The reason being is that a Scrivener project can get quite large, and even the tiniest of changes to your project would mean a sync of every byte of every file in the entire project. The sync would take a long time for even a modestly sized project with just a few pictures and webarchives.

As it stands, individual files that change in a project require only a few seconds to sync up if they are small files, which makes DropBox a very handy and unobtrusive utility.
Often wrong, rarely in doubt.
Time for a change... I'm now rdale; same dog-avatar, same dog... channel?

User avatar
narrsd
Posts: 507
Joined: Wed Sep 22, 2010 8:34 pm
Platform: Win + iOS

Wed May 11, 2011 6:33 pm Post

Hmm. Very good point indeed.

I like the idea of a configurable option -- best to please the most people and their ways of working, and makes clear the older ways are always preserved.

I had meant to suggest anyway that Scrivener would still open unzipped projects; just prefer by default the new arrangement.

Also, though, Zip archives can be written uncompressed, used just as a container, which is enough to solve the problem we want to solve here about avoiding inconsistencies.

Then, as far as I know, Dropbox does incremental syncing -- transfers just the portions of the file that've changed. If the Zip archive isn't compressed, that should mean only the changed areas transfer, preserving the quick response you'd like.

Incremental syncing is also a quite long-known tactic, and I imagine any alternatives to Dropbox would also work that way.

Good thoughts to spark...

C.

User avatar
robertdguthrie
Posts: 3075
Joined: Mon Nov 09, 2009 10:06 pm
Platform: Mac
Location: St. Louis, MO, USA
Contact:

Wed May 11, 2011 6:54 pm Post

I had been under the mistaken impression that the whole file would sync with even the slightest change in any part of it. Turns out I'm wrong.* So moving to a zip archive would actually be a viable option, if it can be done for both the Windows and Mac version. I'm not seeing a lot of down-side to this, but I would think that Keith and the gang would have considered this as an option and discarded it for some good reason; the longer I'm here, the more I notice that hardly any idea presented by a user is a new one on Keith.

* I am taking the DB website at it's word... https://www.dropbox.com/features : "Dropbox transfers just the parts of a file that change (not the whole thing)."
Often wrong, rarely in doubt.
Time for a change... I'm now rdale; same dog-avatar, same dog... channel?

User avatar
narrsd
Posts: 507
Joined: Wed Sep 22, 2010 8:34 pm
Platform: Win + iOS

Wed May 11, 2011 7:08 pm Post

Good man.

Just to say, you were righter than you were wronger originally, as ordinary default zip files would indeed be completely sent, as the compression would make their entire content different even for the smallest changes.

It's the choice to use uncompressed, available both in zip utilities and programming libraries that solves the puzzle -- and can be easily tried out.

Agree the proprieters are dependably pretty complete and cogent in their thinking -- just that angles do turn up, and make life interesting ;).

Cheers, robertdguthrie, and cheers to any 'gruff and quirky',
Clive

jr
jravan
Posts: 212
Joined: Fri Apr 08, 2011 2:15 pm
Platform: Windows

Fri May 13, 2011 3:27 am Post

I would suggest that if zipped archives are desired as a format for project storage, the default be left as is and the zipped format be selectable as an option. Changing the default format to zipping probably would not be a good idea. I think the majority of users would prefer to keep things text-only. I don't use Dropbox for collaboration, just cloud-based backup, so I prefer text.

But making compression a project-based option sounds like a reasonable thing to allow for the subset of users that would prefer it.

ye
yeaforme
Posts: 10
Joined: Wed May 04, 2011 6:10 am
Platform: Windows
Location: NYC

Mon May 16, 2011 7:21 pm Post

Throwing this out there, but more 'advanced' users will find ways and other apps that allow them to work with Scriveners better, at least as they are currently designed. Zip is/could be nice, but certainly aren't the only means.

I for one, put sensitive projects into a TrueCrypt Volume that I toss into Dropbox and then mount the volume as needed. This would also work perfectly for Scriveners' files, but more as a solution if the current format of Scriveners aren't altered rather than as something that Scrivener could integrate as a solution.

TrueCrypt with Dropbox won't allow for simultaneous editing at all, but Dropbox isn't the tool for that sort of thing anyway. Toss the scriveners into a TrueCrypt volume, mount it, alter it, unmount it, then the changes will re-sync and you're good to go and secure to boot and I believe that other users would be locked out while you have it mounted (though I'm not positive on this count). Slight issues might be long upload on the front end and/or the possibility of incorrectly judging the size you need for the Volume and thus either wasting space in your Dropbox or running over within your Volume itself.

I guess the end of this is that I thoroughly plan on putting my Scriveners into something that makes Dropbox think it is a single file anyway, but it could be nice to have it treated as a single file straight out of the box.
Windows Vista & 7 (32-bit)
Scrivener 024

User avatar
robertdguthrie
Posts: 3075
Joined: Mon Nov 09, 2009 10:06 pm
Platform: Mac
Location: St. Louis, MO, USA
Contact:

Mon May 16, 2011 8:16 pm Post

jravan wrote:I would suggest that if zipped archives are desired as a format for project storage, the default be left as is and the zipped format be selectable as an option. Changing the default format to zipping probably would not be a good idea. I think the majority of users would prefer to keep things text-only. I don't use Dropbox for collaboration, just cloud-based backup, so I prefer text.

But making compression a project-based option sounds like a reasonable thing to allow for the subset of users that would prefer it.


The thing is, we're not talking about compressed, just "combined". On the mac, a scrivener project's individual files are mostly hidden from users because the project's directory is named in such a way that OS X treats it and it's contents as one file. Making use of zip for the default file format will in no way lock you out of seeing the individual files (which you aren't supposed to touch under normal circumstances). If you need to get to them, rename the all-in-one scrivener project to scrivproj.zip, and use a free/built-in zip utility to extract the bundle into individual files & directories. Anyone who really wants to see those files would be capable of getting to them with minimal effort, while the vast majority of people who are used to dealing with a single Word file will never be faced with the baffling array of files like 1.rtf, 2.rtf proj.scrivx settings/ snapshots/ ...

This isn't like changing the format into something that can't be accessed by mere mortals, like the MS Office file formats, it's just a handy way of keeping all of a project's files together on any platform where Scrivener works.
Often wrong, rarely in doubt.
Time for a change... I'm now rdale; same dog-avatar, same dog... channel?

User avatar
AmberV
Posts: 20613
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Tue May 17, 2011 3:09 am Post

This all seems to more a job for .tar, than .zip. Compression is nice when you need to preserve space, so it is great for backups, but for a working format---I guess I don't get the point of compression, either. That's a ton of processing overhead, and it would effectively cripple auto-save for even a small-ish project of 20mb. Tar is a lot faster, and it's pretty easy to extract and update individual component elements on the fly. It would still be slower than a file-system by a long shot, but in most scenarios the slow-down would be barely noticeable---and presumably anyone using this method would be aware of the fact that one huge file is not going to be as much of a performance demon as files and folders.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

ag
agilejoe
Posts: 1
Joined: Tue May 17, 2011 11:01 pm
Platform: Mac

Tue May 17, 2011 11:09 pm Post

As far as collaboration in a group or enterprise environment. Why reinvent the wheel? It seems like distributed source control solutions, such as Git and Mercurial, have already solved this. Why not simply open up the the file format and have the source control repository take care of the heavy lifting? By using a DVCS enables me to save and version locally. I have tag and branch certain ideas I might have and then merge them locally. If I want to share them with someone I can use something like GitHub.com to share by thoughts or changes with the rest of the world. You could even comment and collaborate using this type of tool.

Just my thoughts but I am a software engineer by trade, so this may be overkill for the average user but having said that you could abstract the average user from the complexity of source control tool by supplying the same constructs through the main UI of Scrivener.

Food for thought :)

User avatar
narrsd
Posts: 507
Joined: Wed Sep 22, 2010 8:34 pm
Platform: Win + iOS

Wed May 18, 2011 12:35 am Post

Joe, I think we went over this ground earlier in the thread.

- The first thing is, the basis of Scrivener is _formatted_ text. Thus a source control system isn't going to work for this, unless you know of one that does edit/merge for RTF.

- The second thing is that Scrivener also has another level of structuring -- the scrivenings in binders. A special-purpose editor would have to be written to merge the XML files that control this -- and it would need to be integrated sufficiently with the formatted text edit/merge that complex edits on a project would be fully visible and manipulable.

It would be instructive to dream something like this up, but a great deal of effort to make it complete and workable. A look at Word's document compare/merge will give an idea of the level of complexity just for the RTF merge aspect.

I haven't so much more to say here, except that the idea has been to hew to what non-technical persons are comfortable with and practiced at using, so as to not get in the way of anyone's writing, and thus not losing track of Scrivener's purposes. I think that means Dropbox and Zip files, doesn't it?

I also appreciate what Ioa is saying above about tar, etc., but really, a non-compressed zip just would do the job, wouldn't it, including high-efficiency Dropbox synchronizing?

Adaptation always seems the actually sophisticated thing; at least in experience.

Regards to each,
Clive

User avatar
kewms
Posts: 3245
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Wed May 18, 2011 2:36 am Post

robertdguthrie wrote:I had been under the mistaken impression that the whole file would sync with even the slightest change in any part of it. Turns out I'm wrong.* So moving to a zip archive would actually be a viable option, if it can be done for both the Windows and Mac version.


Viable for what? If your goal is collaboration, use of a zip archive doesn't even begin to address the issues. It helps avoid out and out file corruption, but does nothing to help with versioning and change management. And let's not even talk about trying to do simultaneous editing in real time.

Katherine
Scrivener Support Team

User avatar
robertdguthrie
Posts: 3075
Joined: Mon Nov 09, 2009 10:06 pm
Platform: Mac
Location: St. Louis, MO, USA
Contact:

Wed May 18, 2011 2:12 pm Post

AmberV wrote:This all seems to more a job for .tar, than .zip. Compression is nice...


The point I've been convinced of is that .zip can be used just like a virtual disk without compression , but on Windows, Linux, as well as Mac; Am I wrong in my understanding that newer versions of Windows come with a built-in Zip utility, but not tar? Using zip without compression on the project keeps all of your project's files looking like just one file, so Drop Box no longer mucks with the internals of the project when there's a conflict. Instead, it creates an entirely new "conflicted" project, which is much more noticable than a 324(conflicted).rtfd file appearing in your project. But again, without the compression, it shouldn't slow down saving (you can tell a zip utility to replace individual files), and drop box will only transmit the new bytes, not the entire archive.
Often wrong, rarely in doubt.
Time for a change... I'm now rdale; same dog-avatar, same dog... channel?

User avatar
robertdguthrie
Posts: 3075
Joined: Mon Nov 09, 2009 10:06 pm
Platform: Mac
Location: St. Louis, MO, USA
Contact:

Wed May 18, 2011 2:18 pm Post

kewms wrote:
robertdguthrie wrote:I had been under the mistaken impression that the whole file would sync with even the slightest change in any part of it. Turns out I'm wrong.* So moving to a zip archive would actually be a viable option, if it can be done for both the Windows and Mac version.


Viable for what? If your goal is collaboration, use of a zip archive doesn't even begin to address the issues. It helps avoid out and out file corruption, but does nothing to help with versioning and change management. And let's not even talk about trying to do simultaneous editing in real time.

Katherine


Viable for using as a "file format"... the discussion has drifted a bit (though it's incredibly on topic for these forums). It's also helpful in that if something gets conflicted, you get a new copy of the project (equivalent to a blaring siren), instead of files within your project that can remain unnoticed for days, weeks, or even months.

As for simultaneous editing; no, zip archives are not the answer. The answer is for Keith to hire about 10 more Mac programmers (bringing the total to 11), and devote a goodly chunk of their time to shoe-horning in the ability to simultaneously edit a project. Until Keith wins the lottery, or Scrivener rivals Angry Birds in number of sales, I think you're going to have to give up on simultaneous editing of Scrivener projects. If that's your primary concern, Google Docs seems the way to go from what I've read in this discussion.
Often wrong, rarely in doubt.
Time for a change... I'm now rdale; same dog-avatar, same dog... channel?

User avatar
narrsd
Posts: 507
Joined: Wed Sep 22, 2010 8:34 pm
Platform: Win + iOS

Wed May 18, 2011 4:52 pm Post

Aye, Google Docs. For rapid collaboration. ;)

Great thoughts from everyone; this is one of the most conflicted challenges in computing (the collaboration part), and it probably takes a Google to get the software right. Even they falter on the user view (e.g. Waves).

Hope I've taken right tones through this for everyone. Writing is my interest too, among others in that space. Technology one may understand able to be of certain help, as Scrivener; beyond such a point, often there be dragons. As we have met.

Regards to each,
Clive