A friend has recently encouraged me to look at Scrivener as a possible alternative to Info Select, a Personal Information Manager (PIM) that I have been using for nigh on fifteen years, and in which I have invested very heavily (by this I mean that I have enormous amounts of information entered in Info Select, 100mg or so of text in approximately 34,000 individuals notes. The 'notes' in IS are equivalent to the 'text' entries in Scrivener).
What I am wondering is whether anyone here has experience of Info Select and is also conversant with Scrivener.
In particular I am interested in how Scrivener might cope with large amounts of data. I am a historian who works with large amounts of evidence, much of it in the form of snippets of information. My subject is the society of Renaissance Italy. Perhaps the best comparison would be with the archaeologist of an ancient culture, who accumulates hundreds of small pieces of data, and who then combines them painstakingly to create a portrait of the society s/he studies. Info Select has been a great boon to me for many years because of its lightning fast searches. Apparently like Scrivener, it doesn't need tags: you can search for any character string and it will immediately extract all occurrences of that string, presenting them much in the way that Scrivener uses the Binder. The hits in the search are highlighted in the text wherever they occur.
I like the look of Scrivener very much, but my first impression is perhaps that, because Scrivener is based on individual projects, that it's not really designed to be a massive bin into which one drops information for later retrieval (I often collect information without knowing exactly how I am going to use it. The context emerges in the later writing and thinking - this is why the powerful search function is so important to me).
Can Scrivener handle really large amounts of information?
How powerful is its search function?
Will it dig down deep into sub-folders of sub-folders to extract search strings from text that might have entered into the programme years before (Info Select does this brilliantly)? I am very interested in any feedback Scrivener uses might have.
Why am I looking at Scrivener if Info Select is so good? The project-writing dimension of Scrivener is clearly superior to IS. The manufacturers of IS seem not to be supporting their product as well as they once did, and they have in recent times crammed bells and whistles into the programme that somewhat obscure its original simplicity.
I would be grateful for comments - thank you in advance.
Comparing Scrivener with Info Select
8 posts
Page 1 of 1
-
AmberV
- Posts: 24669
- Joined: Sun Jun 18, 2006 4:30 am
- Platform: Mac + Linux
- Location: Ourense, Galiza
- Contact:
Let me just say off the top that Scrivener isn’t designed to be a PIM. It has features that make it possible to use as a general notepad or research assistant, yes, but that isn’t its design goal. So with that disclaimer aside…
It would be difficult to speak of capabilities at this point in time with the Windows beta. I can certainly speak of the established Mac platform, and from that can deduce probable capabilities, but until we get closer to release, high level performance is difficult to speculate on, so take this with a grain of salt.
Speaking of the existing stable version on the Mac, a 100mb project is not unheard of, and would in fact probably be an upper-middling average. I would guess the average book probably has a 30mb file, which Scrivener handles with ease. 100mbs, I have a few of those, and again, no problems with performance. The way it works is in delegating bulk to many small components. Each “note” or “text” entry in Scrivener is a separate file, and these are all addressed by a master list, the .scrivx file. This file is trim: it says, “This note exists in this spot, it’s ID is such-and-such, and it has this meta-data”. Consequently this master file loads quickly even in a huge project. Because the component files are not all loaded at once, you can have thousands of them without any impact on performance—and especially so if they are well organised. You’d probably run into UI slow-downs if you tried to put 5,000 text files into one folder and view it as a corkboard, just because of the sheer amount of assembly that would required to draw a corkboard of that scale—but if used well, organised deeply, the overall scale of the project would not harm the interface’s responsiveness.
To put this into scale, we have people on the Mac running 1, 2 even 4 or 5 gigabyte projects. In every day use, this is not a problem. Where it can become a problem is when these massive projects need to be backed up or upgraded to a new version, but by and large the format can handle massive scale quite well. It is, in the end, just files and folders. If your system can handle it, so can Scrivener; as a rule of thumb.
Now, I myself am not an advocate of accumulating that much information into a project, primarily because I am an advocate of frequently (at least daily if not more) backups, and with huge storage deposits like that, it’s unwieldy to back up consistently. For a database it’s one thing, but for a book you are actively working on—well, I’d rather have daily backups. If that much research is required, I advocate having two projects: one for research and another for the manuscript.
Outside of these extremes: 100mb is not a problem in either usage or backing up, in my opinion, especially once you factor in zip compression which can reduce 100mb of text to roughly 1/10th.
Talk of sheer scale aside: I will say many of our Mac users who rely upon tons of research do use an external database in conjunction with Scrivener. Unfortunately the favourite is not available on the Mac, but it sounds like you already have a suitable program you are comfortable with. What I’m saying is: many of us who work with lots of research material have no problem coupling Scrivener’s writing power with another program that handles data power.
Will Scrivener find scraps of text in 10 deep folders you wrote five years ago? Yes. Does it have strong meta-data capabilities for increasing the width of each entry in a search result? Yes. Does Scrivener do some of these things better than most database style programs? Subjectively, I would also answer yes.
Does that make it a better massive info dump? Maybe… but probably not in most cases.
It’s weakest area is probably the search. The search tool is great for what a writer needs. It’s super speedy and can have its scope adjusted easily—but it is only single axis. Most database styles need more than one axis, and it requires a little fiddling to do this in Scrivener. It’s a great one-shot search tool for string matching, but for complex Boolean structures based off of three or four different fields—the search isn’t really designed for that.
What it does have as an advantage is flexibility. I have many projects where I don’t really use the draft and compile. They are mainly info dumps of one variety or another. I do break things down into “topics” by project. A project for tracking Scrivener bugs; a project for philosophy research; etc. I don’t try to jam everything into one project because I feel that dilutes its meta-data powers. I can keep the label and status markers relevant to the data, and the keywords list does not grow unwieldy in size. Because it is so easy to drag items from one project to another (which the Windows version will allow eventually if not already), and it is easy to have multiple projects open at once, this isn’t a huge impediment. I myself do not have a need for an outboard database for this type of stuff. Scrivener pretty much covers it for what I need.
For reference, if I combined it all together into one lump chunk, I’d probably have about 100mb of raw plain-text as well.
…by this I mean that I have enormous amounts of information entered in Info Select, 100mg or so of text in approximately 34,000 individuals notes…
It would be difficult to speak of capabilities at this point in time with the Windows beta. I can certainly speak of the established Mac platform, and from that can deduce probable capabilities, but until we get closer to release, high level performance is difficult to speculate on, so take this with a grain of salt.
Speaking of the existing stable version on the Mac, a 100mb project is not unheard of, and would in fact probably be an upper-middling average. I would guess the average book probably has a 30mb file, which Scrivener handles with ease. 100mbs, I have a few of those, and again, no problems with performance. The way it works is in delegating bulk to many small components. Each “note” or “text” entry in Scrivener is a separate file, and these are all addressed by a master list, the .scrivx file. This file is trim: it says, “This note exists in this spot, it’s ID is such-and-such, and it has this meta-data”. Consequently this master file loads quickly even in a huge project. Because the component files are not all loaded at once, you can have thousands of them without any impact on performance—and especially so if they are well organised. You’d probably run into UI slow-downs if you tried to put 5,000 text files into one folder and view it as a corkboard, just because of the sheer amount of assembly that would required to draw a corkboard of that scale—but if used well, organised deeply, the overall scale of the project would not harm the interface’s responsiveness.
To put this into scale, we have people on the Mac running 1, 2 even 4 or 5 gigabyte projects. In every day use, this is not a problem. Where it can become a problem is when these massive projects need to be backed up or upgraded to a new version, but by and large the format can handle massive scale quite well. It is, in the end, just files and folders. If your system can handle it, so can Scrivener; as a rule of thumb.
Now, I myself am not an advocate of accumulating that much information into a project, primarily because I am an advocate of frequently (at least daily if not more) backups, and with huge storage deposits like that, it’s unwieldy to back up consistently. For a database it’s one thing, but for a book you are actively working on—well, I’d rather have daily backups. If that much research is required, I advocate having two projects: one for research and another for the manuscript.
Outside of these extremes: 100mb is not a problem in either usage or backing up, in my opinion, especially once you factor in zip compression which can reduce 100mb of text to roughly 1/10th.
Talk of sheer scale aside: I will say many of our Mac users who rely upon tons of research do use an external database in conjunction with Scrivener. Unfortunately the favourite is not available on the Mac, but it sounds like you already have a suitable program you are comfortable with. What I’m saying is: many of us who work with lots of research material have no problem coupling Scrivener’s writing power with another program that handles data power.
Will Scrivener find scraps of text in 10 deep folders you wrote five years ago? Yes. Does it have strong meta-data capabilities for increasing the width of each entry in a search result? Yes. Does Scrivener do some of these things better than most database style programs? Subjectively, I would also answer yes.
Does that make it a better massive info dump? Maybe… but probably not in most cases.
It’s weakest area is probably the search. The search tool is great for what a writer needs. It’s super speedy and can have its scope adjusted easily—but it is only single axis. Most database styles need more than one axis, and it requires a little fiddling to do this in Scrivener. It’s a great one-shot search tool for string matching, but for complex Boolean structures based off of three or four different fields—the search isn’t really designed for that.
What it does have as an advantage is flexibility. I have many projects where I don’t really use the draft and compile. They are mainly info dumps of one variety or another. I do break things down into “topics” by project. A project for tracking Scrivener bugs; a project for philosophy research; etc. I don’t try to jam everything into one project because I feel that dilutes its meta-data powers. I can keep the label and status markers relevant to the data, and the keywords list does not grow unwieldy in size. Because it is so easy to drag items from one project to another (which the Windows version will allow eventually if not already), and it is easy to have multiple projects open at once, this isn’t a huge impediment. I myself do not have a need for an outboard database for this type of stuff. Scrivener pretty much covers it for what I need.
For reference, if I combined it all together into one lump chunk, I’d probably have about 100mb of raw plain-text as well.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles
-
pietrangolopi
- Posts: 5
- Joined: Tue Nov 30, 2010 2:07 am
- Platform: Windows
Dear AmberV,
Thanks very much indeed for all that detail! It's very useful indeed: interestingly, the modus operandi you describe for your own work is rather similar to my own approach. I described Info Select as a data-base - that's gilding the lily somewhat, to be honest. It's manufacturers describe it as like a "flat-file database". Basically that means it is not a sophisticated relational database. It does useful BOOLEAN searches and a couple of other things, but that's it. I don't need sophisticated searches and so this is fine.
One of the things Info Select also does is allow you to put anything anywhere, following the logic that since you can search for any term, it's better to search than to organise. Even so, given that I've accumulated so much detail over the years (quite a bit of which I could get rid of, I'm sure), I prefer to arrange things in a logical way so that I can, as it were 'see' where I am. Separating research from individual writing projects, therefore, would be something I'd do as a matter of course.
One of the attractive things about Scrivener is the much more sophisticated format for writing - that really looks good. Currently I use Info Select for data collection and management - I do aspects of the "corkboard" phase in IS, but at a certain point I move over to Word. Being able to work in the one programme is a nice idea.
Thank you again for your enlightening explanation - much obliged, and you've given me much to think about.
Cheers,
Nick.
Thanks very much indeed for all that detail! It's very useful indeed: interestingly, the modus operandi you describe for your own work is rather similar to my own approach. I described Info Select as a data-base - that's gilding the lily somewhat, to be honest. It's manufacturers describe it as like a "flat-file database". Basically that means it is not a sophisticated relational database. It does useful BOOLEAN searches and a couple of other things, but that's it. I don't need sophisticated searches and so this is fine.
One of the things Info Select also does is allow you to put anything anywhere, following the logic that since you can search for any term, it's better to search than to organise. Even so, given that I've accumulated so much detail over the years (quite a bit of which I could get rid of, I'm sure), I prefer to arrange things in a logical way so that I can, as it were 'see' where I am. Separating research from individual writing projects, therefore, would be something I'd do as a matter of course.
One of the attractive things about Scrivener is the much more sophisticated format for writing - that really looks good. Currently I use Info Select for data collection and management - I do aspects of the "corkboard" phase in IS, but at a certain point I move over to Word. Being able to work in the one programme is a nice idea.
Thank you again for your enlightening explanation - much obliged, and you've given me much to think about.
Cheers,
Nick.
-
AmberV
- Posts: 24669
- Joined: Sun Jun 18, 2006 4:30 am
- Platform: Mac + Linux
- Location: Ourense, Galiza
- Contact:
That sounds similar to a program I use for the Mac for my end-product archival. It's a bit idiosyncratic in that it won't let you delete or edit once you've archived, which I find to be an enormous relief of burden as I know nothing I or the software can do will alter records I've committed to it. It has the same sort of "Don't think too much about it" approach to organisation. It uses Gmail style "labels" for organisation---if you aren't familiar, it means a record can exist in several "locations" simultaneously, or even none at all. On top of that, you can program simple string matching filters into the archival function so when you archive something for permanent record, it will be searched by filters and each matching filter will automatically assign it to these "labels". A single button press can file an item into a dozen useful places without you having to do anything other than compose it. That probably sounds more complicated than it is. It's like creating a thing called "scrivener" that will search for that word in everything you archive, and whenever it finds it, will stash that thing into whatever labels might be relevant, such as "Software Notes", "Scrivener" etc. Any other filters that match it will place it in other labels as well. I think I'm in the same place you are; even though the software encourages pure "Just file it and forget about it", I still spent a lot of time setting up a good complete set of filters which handle 99.9%, so even though my filing is more meticulous, I probably spend as much time as someone who does not filing at all.
I would rate Scrivener as requiring you to be more proactive than this in that it is a folder-based system and so by definition requires more organisation than punching a button, but especially if your way of finding things is via meta-data. If, and this is what it sounds like, your way of finding things is via text strings---well then Scrivener does that very well---and the way it presents search results in an outliner or corkboard is very good for quickly winnowing down the results if necessary. The Synopsis is a powerful feature for visualising your book at multiple levels of scope, but it's also powerful if you use it to amplify the title. Corkboard is highly visual, but that can be an enormous asset. If your secondary identification mechanism is the colour label, a search result with several hundred cards can be easily skimmed through if you know the pin colour you are looking for. So even though you can't search for "text string" AND [pin:red], the presentation of the results tap into the human strengths of pattern recognition in a way few database programs do---which is what I was getting at about it being better in some ways, than even a relational database for certain uses. For a writer looking for "text string" in red pin items or a succinct stamp across the front, it's easier to quickly type in "text string" and scroll through a bunch of cards quickly skimming all the red flashes as they go by, than elaborately construct an SQL query and spend 10 minutes debugging it.
Oh, and Scrivener does have a few simple text based on/off constructs. "Exact phrase", OR, and AND. So you can do a few Boolean type searches, you just can't do "(this OR that) AND this". Just "this OR that OR this", "this AND that AND this", or "this that this". Since the default search scope is All meta-data and text, if you use unique token-style phrases in keywords, you can do some semi-complex queries that gather intersections. You can also do compound search sequences with the "Search binder selection" option. In other words, perform (this OR that) for search A; select everything, set to selection search and search for the final this. Rinse and repeat as necessary. Unfortunately compound searches cannot be saved for future use as binder selection style searches cannot be saved since the selection data is fallible across sessions.
I would rate Scrivener as requiring you to be more proactive than this in that it is a folder-based system and so by definition requires more organisation than punching a button, but especially if your way of finding things is via meta-data. If, and this is what it sounds like, your way of finding things is via text strings---well then Scrivener does that very well---and the way it presents search results in an outliner or corkboard is very good for quickly winnowing down the results if necessary. The Synopsis is a powerful feature for visualising your book at multiple levels of scope, but it's also powerful if you use it to amplify the title. Corkboard is highly visual, but that can be an enormous asset. If your secondary identification mechanism is the colour label, a search result with several hundred cards can be easily skimmed through if you know the pin colour you are looking for. So even though you can't search for "text string" AND [pin:red], the presentation of the results tap into the human strengths of pattern recognition in a way few database programs do---which is what I was getting at about it being better in some ways, than even a relational database for certain uses. For a writer looking for "text string" in red pin items or a succinct stamp across the front, it's easier to quickly type in "text string" and scroll through a bunch of cards quickly skimming all the red flashes as they go by, than elaborately construct an SQL query and spend 10 minutes debugging it.
Oh, and Scrivener does have a few simple text based on/off constructs. "Exact phrase", OR, and AND. So you can do a few Boolean type searches, you just can't do "(this OR that) AND this". Just "this OR that OR this", "this AND that AND this", or "this that this". Since the default search scope is All meta-data and text, if you use unique token-style phrases in keywords, you can do some semi-complex queries that gather intersections. You can also do compound search sequences with the "Search binder selection" option. In other words, perform (this OR that) for search A; select everything, set to selection search and search for the final this. Rinse and repeat as necessary. Unfortunately compound searches cannot be saved for future use as binder selection style searches cannot be saved since the selection data is fallible across sessions.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles
-
pietrangolopi
- Posts: 5
- Joined: Tue Nov 30, 2010 2:07 am
- Platform: Windows
More food for thought there! What's the name of that Mac programme you use, by the way? Sounds interesting. Thanks again for the very interesting feedback.
-
historydollhi
- Posts: 9
- Joined: Thu Dec 16, 2010 5:05 pm
- Platform: Mac
I have just started using Scrivener in a serious way, and I *love* it (have been trying it out for a few days, and went ahead and purchased it this morning). I wish I had discovered it earlier in the dissertation-writing process, and not while I was working on the final chapter 
. So this is in no way meant to be a slam at Scrivener. It's just that what you are describing seems different from what Scrivener is designed to do, and there is a Mac-only program that sounds like exactly what you are describing: DevonThink.
http://www.devon-technologies.com/produ ... index.html
It's an information gatherer with a very good AI built in, so it can not only sort and find with amazing accuracy, it can actually suggest other relevant items that seem to go with what you're doing. I find it invaluable for research. It comes in a couple of different strengths, and there's an educational discount. (And thanks, Scrivener, for the education discount which you offer!).
. So this is in no way meant to be a slam at Scrivener. It's just that what you are describing seems different from what Scrivener is designed to do, and there is a Mac-only program that sounds like exactly what you are describing: DevonThink.http://www.devon-technologies.com/produ ... index.html
It's an information gatherer with a very good AI built in, so it can not only sort and find with amazing accuracy, it can actually suggest other relevant items that seem to go with what you're doing. I find it invaluable for research. It comes in a couple of different strengths, and there's an educational discount. (And thanks, Scrivener, for the education discount which you offer!).
-
AmberV
- Posts: 24669
- Joined: Sun Jun 18, 2006 4:30 am
- Platform: Mac + Linux
- Location: Ourense, Galiza
- Contact:
The program I use for this is called Boswell. If you search for that on the forum here, you'll find my long rantings on its uses. It doesn't approach information storage in the traditional models (files/folders/tags), so comes across as a bit strange, but once you get why it doesn't use those models, it makes quite a lot of sense.
It doesn't approach information storage in the traditional models (files/folders/tags), so comes across as a bit strange, but once you get why it doesn't use those models, it makes quite a lot of sense..:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles
Nick [Pietrangolo?],
I find your drafting and research process in 2010 — even your data concerns — much like mine today. As I now look to Scrivener as a likely replacement for Info Select v9 (2007) [Infoselect] in a bid to step up my (amateur) composition game, I will add re data that, starting with Info Select about the same time as you did, I've used it regularly about five years longer, probably now carry more info than you did then.
I have three works in stages beyond what would be 'collections' in Scrivener, sized half a MB, 1.3 MB, 3 MB: Each of these is in a separate real file and seen in Info Select as a high-level folder; In my use of Scrivener, each of these will likely be Draft subfolders, one level in from Draft.
And I have various 'Research' files of thousands and 10s of thousands of notes (which like those in the works above, range from one-liners to 10-100k extended thoughts), and annotated outside sourced texts: These four research files are sized from 7 TO 85 MBs of 'notes' (and I'd like to import/link to Scrivener ~ 2 GBs of PDFs currently handled separately).
(I have Info Select set to automatically backup the changed files every five minutes: As you likely found, I rarely notice it.)
Being more a fox than a hedgehog, I do appreciate Info Select for the serendipity it’s searches allow, which uncovers and facilitates connections despite my data's being in separate project files: I do enjoy the synergy that comes from talking about Medinet Habu when I write about Jason's Argonauts or about the physics of black holes when I write about socially-sanctioned homicide.
(I’ll never admit to having thoughts and notes that could be tossed! But I do accept that some of the information I have and perhaps more besides could go into a separate Scrivener 'research' project.)
'Search' may be the one practical hit I'll take. I probably use "(this OR that) AND this" searches about as often as I do text strings so Scrivener's limits here will likely be the thing I'll actually have to work around — until I either find a way to use synopsis and keywords and colors (colours!) to my advantage … or figure out how to construct some favorite Info Select custom search keystrokes.
And while I will use gratefully Scrivener’s inter-project dragging, I’ll have to sense that there is something ‘over there’ to grab: The (current) inability of Scrivener searches to reach across projects by default has the potential to limit the sight of my mind’s eye to what I believe (or remember) is relevant or better, what is rhetorically evocative. (Of course, continuing with Info Select as a separate research library would still require separate searches and would blinker me as much; AND would require more than mere click-drag integration of whatever I found — besides whatever my imminent move to Win10 holds in store for its operability.)
So, when you have the opportunity (and understandably, the motivation!), I would appreciate what you as a (transitioned?) long-time Info Select user has experienced with Scrivener. (Which does look good to me as I move toward more composited completions of my essays and collected thoughts.)
And as an aside, I see that in 2014, you had a concern about footnotes: Has that been resolved for you?
George.
I find your drafting and research process in 2010 — even your data concerns — much like mine today. As I now look to Scrivener as a likely replacement for Info Select v9 (2007) [Infoselect] in a bid to step up my (amateur) composition game, I will add re data that, starting with Info Select about the same time as you did, I've used it regularly about five years longer, probably now carry more info than you did then.
I have three works in stages beyond what would be 'collections' in Scrivener, sized half a MB, 1.3 MB, 3 MB: Each of these is in a separate real file and seen in Info Select as a high-level folder; In my use of Scrivener, each of these will likely be Draft subfolders, one level in from Draft.
And I have various 'Research' files of thousands and 10s of thousands of notes (which like those in the works above, range from one-liners to 10-100k extended thoughts), and annotated outside sourced texts: These four research files are sized from 7 TO 85 MBs of 'notes' (and I'd like to import/link to Scrivener ~ 2 GBs of PDFs currently handled separately).
(I have Info Select set to automatically backup the changed files every five minutes: As you likely found, I rarely notice it.)
Being more a fox than a hedgehog, I do appreciate Info Select for the serendipity it’s searches allow, which uncovers and facilitates connections despite my data's being in separate project files: I do enjoy the synergy that comes from talking about Medinet Habu when I write about Jason's Argonauts or about the physics of black holes when I write about socially-sanctioned homicide.
(I’ll never admit to having thoughts and notes that could be tossed! But I do accept that some of the information I have and perhaps more besides could go into a separate Scrivener 'research' project.)
'Search' may be the one practical hit I'll take. I probably use "(this OR that) AND this" searches about as often as I do text strings so Scrivener's limits here will likely be the thing I'll actually have to work around — until I either find a way to use synopsis and keywords and colors (colours!) to my advantage … or figure out how to construct some favorite Info Select custom search keystrokes.
And while I will use gratefully Scrivener’s inter-project dragging, I’ll have to sense that there is something ‘over there’ to grab: The (current) inability of Scrivener searches to reach across projects by default has the potential to limit the sight of my mind’s eye to what I believe (or remember) is relevant or better, what is rhetorically evocative. (Of course, continuing with Info Select as a separate research library would still require separate searches and would blinker me as much; AND would require more than mere click-drag integration of whatever I found — besides whatever my imminent move to Win10 holds in store for its operability.)
So, when you have the opportunity (and understandably, the motivation!), I would appreciate what you as a (transitioned?) long-time Info Select user has experienced with Scrivener. (Which does look good to me as I move toward more composited completions of my essays and collected thoughts.)
And as an aside, I see that in 2014, you had a concern about footnotes: Has that been resolved for you?
George.
Page 1 of 1