Scrivener as Dedicated Research Database

re
refusion
Posts: 54
Joined: Fri Mar 20, 2009 1:14 am
Location: Shanghai, China
Contact:

Tue Apr 14, 2009 4:01 am Post

Let's say you write four books and have research for those books in each individual Scrivener file. You might want to access research data for those books later. To do so, you would have locate the file and open it up.

So, why not create a Scrivener file which is dedicated to storing only your research? Scrivener's flexibility allows you to set up a template anyway you like. When you're done with whatever it is you're writing, simply drag and drop the research folders over into your dedicated Scrivener research file. It'll be really easy for you later if you need to go back and find something again for another project or if your WIP gets deleted or corrupted. Research can be almost as expensive as the actual writing.
Old Lady: 'The universe rests on the back of a turtle!'
Scientist: 'Ah, but what does the turtle rest on?'
Old Lady: 'Young man, you can't fool me! It's turtles, turtles, turtles all the way down!'

dr
druid
Posts: 1721
Joined: Fri Jun 22, 2007 2:29 pm
Platform: Mac, Win + Linux
Location: Princeton NJ, USA

Tue Apr 14, 2009 4:39 am Post

You'd get the same effect, with far more powerful software, by creating research data folders in DevonThink Pro. As Keith recently stated, Scrivener is not optimized to be a research database.

User avatar
AmberV
Posts: 23324
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Tue Apr 14, 2009 4:37 pm Post

While true to an extent, it does happen to be one of the stated goals of Scrivener to allow a writer to assemble their research and their manuscript all in one place. Not everyone needs a huge database software whose primary unique features only start working after 10,000 entry additions. I would say that the best course of action for any writer using Scrivener to take is to simply user Scrivener and its research folder. If it starts to get unwieldy then it is isn't difficult to upgrade to EagleFiler or DTP or Boswell or whatever suits them.

The nice thing about sticking with Scrivener as long as you can, and refusion's method, is that you have full advantage of Scrivener's meta-data, split screens, and other features. Dragging items to a research project retains a large majority of the hard work that goes into filing and organising things. Dragging stuff between Scrivener and other software packages resets the meta-data to zero every time.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

ba
bashosfrog
Posts: 123
Joined: Mon Sep 18, 2006 11:14 am
Location: New England, Australia

Wed Apr 15, 2009 12:11 am Post

In my view, one failing of Scrivener as a research database is that it doesn't retain the source URL with web clips or archives. It's like having a whole lot of photocopies and no idea what books they came from. (Yes, I could do it manually, but I've now got the expectation that it will be done for me).
Scrivener's ability to handle email messages is also limited, which I think contradicts the "binder" philosophy - my real manilla folders are stuffed with correspondence, but I can't drag a Mail message to the relevant folder in Scrivener's binder.

Devonthink handles these things very deftly, which is why my 134 Scrivener projects (according to a quick check with Spotlight) all have empty Research folders.

I'd take another look at Scrivener for research, but Keith would have to consider that these features are worth the coding overhead they would create for him. I don't see a chorus of complaint, so they probably aren't for most users.

Besides, with DT databases containing six million words and more, my heavy-duty journalist's research needs are probably better met with Devonthink anyway.

User avatar
AmberV
Posts: 23324
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Wed Apr 15, 2009 12:42 am Post

Yeah, for me it's really not an issue even though I do a lot of web research, and really for two reasons. 1) If I really need the original URL, I import it normally and it gets one. Just drag the URL icon into the Binder and that's it. 2) I always, always, always make my own backup of the page by using SiteSucker or wget. The web is transitory and things come and go. I have web research from eight, nine years ago that I can still go back and find with ease because I stored the "hard copy", and if I go to the original URL it is no longer there. The site is gone, or the site changed the way they do URLs, whatever. For stuff that only need for three months or even six months, it is probably okay to rely upon the Internet to sustain your research, but otherwise there is no reason to expect it to do so.

Beyond losing the web source, the data itself is kind of fickle. You may or may not get to keep the magically imported URL 15 years from now when DEVONthink hypothetically no longer exists and you need to upgrade to the next big thing. Losing meta-data like this has caused me to, over the years, move everything into the data area and keep that annotated likewise. Then even a laserjet print-out has the full copy and citation.

Incidentally I keep my long-term archives in Boswell, not Scrivener, which is even less razzle dazzle in those terms. An entry is a text file with three meta-data fields. Period. Simple, but it works perfectly for me since I hard-copy everything and cross link it if necessary. But that is for stuff that is history; no longer being worked on. I archive the Scrivener project too, and cross-link that in Boswell as well. :)

My motto has long been, in the terms of long-term archival, the less razzle dazzle the better.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

As
AsafKeller
Posts: 40
Joined: Mon Sep 03, 2007 12:44 pm
Platform: Mac

Wed Apr 15, 2009 3:28 am Post

Amber: What does "move everything into the data area and keep that annotated likewise" mean, precisely? Do you mean within Scrivener's Research folder, or using a different application?

re
refusion
Posts: 54
Joined: Fri Mar 20, 2009 1:14 am
Location: Shanghai, China
Contact:

Wed Apr 15, 2009 1:59 pm Post

You guys got me thinking about DEVONthink again. I went today and downloaded it and Devon Agent. Man, I don't how in the world I can live with those now! Bummer, more money to spend. I was able to find exactly what I was looking in a few seconds with DEVONagent so I know I will be buying that. I'm torn now because you can get both in a $99.00 bundle deal. Hmm...
Old Lady: 'The universe rests on the back of a turtle!'
Scientist: 'Ah, but what does the turtle rest on?'
Old Lady: 'Young man, you can't fool me! It's turtles, turtles, turtles all the way down!'

User avatar
AmberV
Posts: 23324
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Wed Apr 15, 2009 5:23 pm Post

AsafKeller wrote:Amber: What does "move everything into the data area and keep that annotated likewise" mean, precisely? Do you mean within Scrivener's Research folder, or using a different application?


What I mean is, when I'm done with something and it is ready to be archived, I don't rely upon any database meta-data for data storage. Everything crucial to filing goes into the text area. Since I use MultiMarkdown, much of this is pretty easy to do in the header area of the document. For example:

Code: Select all

Title: Name of the article 
Date: 09105,763 
Keywords: some,fancy,topic


Et cetera. I average about 7-12 lines per document, but only about four of them are interesting to me as a human. Most are helpers for future automation routines. I have a pretty simple but effective filing method that is very loosely based on the sort of decimal systems that libraries use, vastly trimmed down to a scope that is meaningful to me---stuff like that. For instance, this post would be filed as:

Code: Select all

{M2.1.Scrivener}


Which I can tell at a glance means coMmunications-2(Public)-1(Forums&Wiki)-Scrivener(topical). Your response, if I chose to file it for future reference, would simply be {m2.1.Scrivener}. Letter case denotes internal or external origin. {M1.1.Scrivener}, on the other hand, would be 1(Private)-1(Email&Letter), thus most likely a bug report or something to Keith. It is compact, machine-friendly, and operates very nicely with searching mechanisms. I can search for StartsWith `{R1.1.` to get all Records-1(external)-1(observations), of any topicality, or on the other hand, I can search EndsWith `.Scrivener}` to perform a cross-axis crawl for Scrivener related documentation, correspondence, theories, and so on, across all the major branches. I've never needed a.i. or anything in fact that kind of stuff gets in the way because my system cuts through huge amounts with surgical ease. Add chronology to these topic tokens and you can find a precise article out of 8,000 in a few seconds.

So while I do make use of database meta-data, since it is there, I don't trust anything to remember that. I always assume that some day I'll have to do an emergency restoration and dump everything out as raw information. The more I can fit into the document area as annotations to that, the better.

This system has served me well. I've probably switched archival applications five or six times and every time I was able to fairly rapidly have a fully organised kit, even though thousands of files are involved---because each file is aware of its own placement in the greater system. It assembles itself.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

dr
druid
Posts: 1721
Joined: Fri Jun 22, 2007 2:29 pm
Platform: Mac, Win + Linux
Location: Princeton NJ, USA

Wed Apr 15, 2009 5:29 pm Post

refusion wrote:You guys got me thinking about DEVONthink again. . . I'm torn now because you can get both in a $99.00 bundle deal. Hmm...


If you are student or faculty at a school, you may apply for an educator discount on any DT product. That might allow you to upgrade to DT Pro.

Ve
Vermonter17032
Posts: 265
Joined: Tue Apr 15, 2008 2:24 pm
Location: Vermont, USA
Contact:

Thu Apr 16, 2009 1:07 pm Post

Amber (or anyone else who can answer this question),

Would you recommend a resource for learning more about MultiMarkdown. I checked the forum on this site, but all the links seem to be no longer functioning.

I guess I'm looking first for a quick overview and then, hopefully, for a primer of some kind.

Thank you.

User avatar
AmberV
Posts: 23324
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Thu Apr 16, 2009 3:08 pm Post

I have a rough draft of a step-by-step guide to learning MultiMarkdown which I am about a week away from posting everywhere. It's something I've been wanting to do for a while, and had a bit of spare time. Ultimately, I want to create a complete guide that goes from very basic "my first MMD hello world" type exercises to creating your own modifications to the export engine and even your own file formats. But I intend to release the basic part first and join it up with the more advanced stuff later. I'll put you on the list of people to alert.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

si
signinstranger
Posts: 268
Joined: Sun Oct 21, 2007 4:15 pm
Platform: Mac
Location: Germany

Thu Apr 16, 2009 3:34 pm Post

Vermonter17032 wrote:Would you recommend a resource for learning more about MultiMarkdown. I checked the forum on this site, but all the links seem to be no longer functioning.


I've added a list of links related to MultiMarkdown (some more interesting than others) to the Scrivener Wiki. There's currently no tutorial, but you could start with the Markdown Web Dingus:

http://daringfireball.net/projects/markdown/dingus

This is Markdown, not MMD, but the concept is basically the same.

do
douger
Posts: 19
Joined: Sun Mar 01, 2009 9:42 pm
Contact:

Thu Apr 16, 2009 7:21 pm Post

Amber,

- Now I know what N and R sand for, but I’m dying to know, what other taxonomy do you use?
- How many codes are there in your system?
- Did you start with a few and have they expanded over time?

I started down the data management journey using Journler. Like Boswell, Journler was an excellent tool for letting me organize my data. After a while Journler and I parted ways and I converted everything through Together to DEVONThink Pro. I posted a lot about this on my blog, and don’t want to repeat it here, but I’ve dumped DTP and I’m now basing all my archiving and research management on the basic file system functionality of OS X because its data retrieval and organization capabilities seem to work much more elegantly than DTP (or the other shoebox applications), and they seem to be less prone to obsolescence.

Alex Payne wrote a great web post at al3x.net (Case Against Everything Buckets.) (and the followup) about the value of using single purpose apps (of which Scrivener is one of the best) along with the file system to achieve robust research management capabilities. When I read his work it really change my outlook on these types of applications.

What Scrivener lets me do is put just the right amount of research in with my writing project, while the mother-load of the library stays out in file system. I fear that if Scrivener tried to recreate a research management tool or duplicate the file system, it would loose some of what makes it work so well as a single purpose application.

I agree with you completely, Amber, that the only valuable unique functions in DTP kick in at huge data sets - all that AI stuff. I did find value however in using DTP (and Journler, and Together, and to a lesser extend EagleFiler...) as tools to organize my data in the first place. I have thousands of notes in rtf/d files, along with a significant library of pdfs. After looking at all the lit manager program (papers et al) I came to the same conclusion that without a text based coding system you’re just setting yourself up for a future conversion project or worse, date loss problem (I once used Commence so know about that well)

Going it alone in the file system however does require one to make your own internal text based tagging system, and I am intrigued by yours, hence my question.

Doug

br
brett
Posts: 528
Joined: Tue Jul 11, 2006 3:17 am
Location: yet another Portlander

Thu Apr 16, 2009 10:43 pm Post

After using DevonNote for a couple years, I came to the same conclusion when I got Scrivener. It's just easier for me to use the file system (basically, Finder + Spotlight + TextEdit) to store my research for various projects past, present and prospective. Then use Scrivener as the respository / organizer for the projects I'm actually working on at the moment. That system is even working well so far for my book in progress.

The file system is about as future proof as we can hope for at this point, not that I expect Devon or EagleFiler or the rest to go bellyup. (And I guess you could always export everything to rtf or even text if necessary.) I have a preference for minimalism -- just the tools I need, less clutter -- and this system means just one less app (at least) to have running and consuming space and RAM and my time to RTFM and scale the learning curve. I fully appreciate the usefulness of Devon et al for research intensive projects. But for my needs as a working journalist and book writer, the file system + Scrivener combo seems to be as much as I need and no more.
Last edited by brett on Fri Apr 17, 2009 5:07 am, edited 1 time in total.

re
refusion
Posts: 54
Joined: Fri Mar 20, 2009 1:14 am
Location: Shanghai, China
Contact:

Fri Apr 17, 2009 1:37 am Post

The arguments for and against bucket software are compelling. However, the basic OSX file system doesn't always cut it for me. On the one hand, I'll rely on a basic file folder system on the hard drive for storing my music projects (seems strange as those are quite huge, I know) but then on the other hand, I'm a huge news/info. junkie and need to try to merge my email and internet data together as seamlessly as possible.

I decided that my Scrivener research folders were getting a bit cumbersome so I think DEVONthink is going to be the best solution since I'll be using agent for searches from now on anyway. I went ahead and bought DEVONthink and agent yesterday. I'm happy I'm able get searches done so quickly and thoroughly with agent and store my 'keepers' in a database in think. The other thing that sold me was the fact that I can have RSS feeds and archive emails from Entourage in DEVONthink. I'm currently using Times for my RSS newsreader but if I need to archive a story that I want to keep as research, there's no quick way of getting it out of Times and into Scrivener. And with DEVONthink, archiving and accessing email from Entourage is easy as I frequently send emails to news articles to my Gmail account (I do this at work during the day on my haggish corporate Windows setup). Now, I can simply drag/export/etc... the link over from Entourage into DEVONthink for later use in Scrivener.

I would be hesitant to use DEVONthink for databasing any of my other stuff however. I like to keep my other 'twiddly' bits in actual folders on OSX in Documents. I don't like the idea of certain things being locked up in a proprietary file system I may not be able to access later. iPhoto comes to mind on that point. I like to backup my photos to DVD in standard OSX folder structure.

I have Circus Ponies notebook but I stopped using it as I realized the proprietary save files it generates could get corrupted or if you have one of those notebooks password-protected and forget what it is, you'd be in real trouble. Also, notebook for me is too nebulous in workflow as far as structuring things goes. I wind with more clutter inside the notebooks than what I had outside to begin with. Then the other problem is should I save the info. somewhere else that just saved in Notebook as a 'just in case'? Sort of kills the whole fun of it. Notebook is a neat idea but I've never found any real use for it. Just another bucket I wasted money on...

DEVONthink and agent seem a bit more reliable and logical but then again, should I save all that stuff somewhere else...'just in case'? The debate rages on.

All of this data we save means nothing really. In the end, the software we buy to GTD means nothing either as the formats will forever mutate and leaves us saddened by the loss of those tiny digital 'nothings' we worked so hard on and spent so much time and money trying to preserve.

With that in mind, I hope to make DEVONthink and agent worth the money spent and actually use them to get all the data collected with them committed to text in the form of this book I'm working on. If there's a tangible and lasting end result that justifies the means, then I say it's worth the time and money spent on bucket software.

End of mindless rant but I hope it helps others in their decision on how to best organize and 'future-proof' their data.

End Analysis:

1) Circus Ponies Notebook- Pretty but not very useful. Too 'noodly-doodly'. Proprietary file system.
2) DEVONthink- Can integrate email, RSS feeds and searches from agent together in spot. Still a proprietary database system though.
3) When in doubt, do 'raw' backups of everything and frequently using OSX file structure. I use Time Machine.
Old Lady: 'The universe rests on the back of a turtle!'
Scientist: 'Ah, but what does the turtle rest on?'
Old Lady: 'Young man, you can't fool me! It's turtles, turtles, turtles all the way down!'