Some changes I'd like to see

As I’ve been using Scrivener more and more regularly, I’ve run across annoyances and other issues that cause problems, but which could be different. It could be that I’m missing something, but I’ve looked for all of these and either found them too limited or non-existent. Here are the ideas I have for making Scrivener better:

Importing Bookmarks - often when I’m researching online, I’ll come across 20-30 web sites relevant to my research, I’ll cull them and then bookmark the set so I can review them in more detail later. I’d like to be able to have Scrivener import all of these pages at once. It could be by pointing to a bookmark folder or some other means. I use Firefox, so it would have to be something that would allow me to get them from there.
Large Documents - when I’m working with large documents (200 or so pages) in my research, I’m finding that Scrivener goes into a mode which I assume is some sort of file conversion (I usually am working with PDFs of this size). It would be nice if this could be done in a low priority background process. As it is, the hard disk keeps spinning up faster and faster slowing down everything. Scrivener can’t be used at all and other applications are limited because Scrivener is hogging all of the resources
HTML editing - when I import a web page, EVERYTHING comes in. It would be nice if I could edit the HTML directly to eliminate the junk ads, links, tracking stuff, and generally anything that isn’t what I really want from the web page. I’ve tried using the ‘external editor’, but it brings up web pages in Safari. I’d use TextMate to do it, but I can’t find a way to configure it to make it the choice for external editor. I’ve been looking at the low level plists within Scrivener and I can’t find a way to change the external editor. I haven’t tried changing my default on the system for webarchives, but I don’t want to reset my global options just for scrivener.
HTML Import - it would be nice if I could select a portion of a web page and have an import that would work. I know I can do it with Safari, but I don’t use Safari. All my work is in Firefox. I need some better way to do this that will preserve the web link information
Types of Documents - The work I do is primarily non-fiction. My need for research material is to support non-fiction references. I need web page information and other sort of bibliographic information. I can store it free format on cards, which is what I do now, but the more I can organize this the better. It would be nice if there were document formats that supported this type of research. You have a link to a citations manager, but I don’t need anything that complex. I need something simpler, integrated into Scrivener so I can generate a simple list of references. I’m not doing scientific papers, just non-fictional pieces. The existing references list bothers me because even when it knows the source of the material (for example a page imported from the web) it doesn’t show up in the document references. As a minimum, I’d like to see EVERY document source listed in project references and each document at least listed in it’s own reference list. The assumption seems to be that all references have URLs, but many of mine are books, magazine articles, and so forth.
Document Formats - I’d like to be able to connect ANY document format within a project and then be able to open it using the application I designate (or that’s default on my computer for that document type). The existing documents that are automatically handled could have default settings to use the existing Scrivener tools, but ANY document could be designated to be opened by it’s original application without affecting other documents. For example, I could designated a specific, large PDF to be opened by the PDF application instead of Scrivener when I click on it. My work-around for this is to create a dummy document to which I can attach document references which will then be opened by the application, but I’d like to have the document in the Research section open directly without going through the references link.
Extensible Tools - I find it very handy to have extensible tools available on the menu. For example, iTunes scripts menu which lets me add Applescripts which I can use to add additional features I want to the application. Any sort of ability to allow me to do specialized things with the application makes it easier for me to accomplish what I need to. Ideal for me would be to hand a document to an external script and get it back after the script has modified it. I often work in Ruby or Python, both languages give me powerful capabilities that I’d love to be able to integrate. For example, I wrote a Ruby processor for web pages that gets rid of embedded ads and such, but if I do that now, I lose the reference to the page in Scrivener. I want it all :astonished:)
Project References - I’d like to see ALL references on the Project Reference list, even those defined by an individual document. Perhaps, a third option, say ‘global references’ could list all document and project references so that nothing is lost from what is there now.
Opening External Documents - When I select a document from the document references, it opens with the default application, but sometimes I’d like to be able to open it with an application of my choice. Finder handles this with the pop up menu which gives me the ‘Open With’ option. I’d like the same thing here. I occurs to me that this may be why Safari comes up when asking for an external editor on HTML documents because of a default selection without a way around it.

Some remarks from my own experience with using Scrivener.

Importing Documents + HTML editing
Depending on what you need from the site, you might wish to consider using text import rather than web archive (General section of Scrivener preferences). I prefer the text only, myself, as this results in an ordinary RTF file that can be edited and cleaned up easily.

But, if you need the layout and more than just text, you would probably want to approach this differently. In that case, I would suggest saving the page out from Firefox using the “Web page, complete” option. This will create an HTML file and a folder with a bunch of whatever it needs to render that page. At this point you can clean up the HTML file in TextMate before importing it into Scrivener as a WebArchive. Once you do that though, it isn’t really easy to edit.

Incidentally, this is why they are loading in Safari. WebArchives are a package format with most of the web page content all included in the bundle. There is an archival application that allows editing of WebArchives in a minimal fashion—not that buying another $40 application is much of a solution. Editing before importing is really the best, if you need layout at all.

I hear what you are saying about a more rapid method of processing (at least on the human side of things). I think saving from Firefox might be just the trick as that is a very rapid process. I’d just set up a folder for this purpose and dump HTML files there as I research, then drag them all over to Scrivener at once.

This trick works because Firefox changes all of the media and resource links within the HTML to point to the downloaded folder, rather than the Internet.

Opening external documents:

If this is something you wish to change permanently (for example, always opening PDFs in Skim instead of Preview, and for whatever reason they are opening in Preview), you can manually change these files inside the project bundle. I’m assuming, since you mentioned snooping around in plists that you know all about getting there. Your PDF and other media files should all be in that folder, and you can just use the Finder to set up the “Open with” flag for these files. This is all that Scrivener uses to open in external editor.

But the HTML issue, as already mentioned, has to do with the fact that HTML files don’t really exist in Scrivener. When they are dragged into a project, they get converted into WebArchives or RTF files which you can then edit. TextMate wouldn’t know what to do with a WebArchive anyway. You can technically open one with it, but it is a binary file so you’ll see a bunch of junk characters scattered everywhere.

I’d encourage you to think of Scrivener as an outlining and drafting tool, and to use either DevonThink Pro or DevonNote for research and data analysis.

Importing Bookmarks: Direct import of existing URLs, drag and drop others, and rename their labels for your purposes.

Large Documents: ditto, plus you get auto-indexing, concordance, and other forms of text analysis.

HTML Editing: ditto, the Services menu lets you add web page text in RTF or plain-text.

HTML Import: ditto, select, drag and drop web text into DTP/DN or import via Services menu.

As for all your other points, you really need DTP, plus a bibliography or reference manager. I’ve been trying out the latest version of Papers and think it’s quite impressive.

DevonTechnologies: devon-technologies.com/

Papers: mekentosj.com/papers/

The developers say Papers is mainly for science research, but it scans Google Books, Google Scholar, JSTOR, and Project Muse; all good for humanists and general nonfiction research.

I agree with druid. I suspect you would be much happier using DevonThink Pro as a research manager, and Scrivener as a writing tool. That’s what I do, and I’ve found they complement each other very nicely.

Katherine

Thanks for your comments

I’ve been trying the Devon tools … so far I don’t find them terribly useful. It could be that I just haven’t unlearned old habits, but I don’t know that I’ll be far enough along before my trial period is up. If I can’t find a way to make them work before my trial period is up, I’ll just dump them. Sorry.

I understand that WebArchive format is not compatible with TextMate. My idea was that it would be passed to TextMate as HTML for editing and returned to WebArchive format for storage. My poking around inside Scrivener was to learn how it’s set up and what sort of things might be configurable and what sort of things were hard coded. I originally thought your ‘external editor’ might be a configuration, obviously that wasn’t true.

I already download pages to files, but then I lose the original page reference in Scrivener which causes other problems. I like being able to import a page directly into Scrivener in order to get the reference link. It saves me some time in keeping things organized. Text import is not an option in most cases. The ONLY reason I want to edit the pages is to get rid of junk which clutters up the display. I wrote a program in Ruby which does that for many pages, for others I can do it with TextMate and it’s pattern matching routines. Between the two, I can quickly dump the tons of advertising which often gets inserted into pages and leave only the text and graphics that are the subject of the page. I can download a page to a file, do my cleanup, and import to Scrivener easily. It just would have been nice to simply the routing and not have to manual insert links to the original web page.

The work I’m doing is not academic, but can be quite technical. I’m presently revising an online course in Radio Frequency Propagation. There are references to catalog, graphics to capture, and a whole range of things to do. I’ve been very impressed with Scrivener as a base of operations, but it still leaves me with some manual work I had hoped could be automated. Even just an ability to execute Applescript would make it possible for me to build the extensions I’m looking for myself.

I’ve got work-arounds for everything I suggested and I’m looking at other tools, like the Devon Tools for alternates for some things, but I thought I’d vent my frustrations and hope that at least some of the ideas would work their way into future releases of Scrivener.

Terry

Scrivener’s open in editor feature is very simple. All it does is pass the file through Finder’s “Open with” association. It is something you can configure, you just have to do it at the Finder level; optimally before you import it into Scrivener. It doesn’t attempt to parse out special formats and deliver them to alternate editors (like extracting HTML from WebArchives and then injecting only the edits back into the binary file—that would probably involve a large amount of development time; as the only program I know of that allows this is Together; and it has a crappy implementation in that the editor is WYSIWYG only).

But yes, for web pages, the way things exist mean if you want an easy workflow and full page layout display, you have to live with the way the page looks down to every last ad. If you want any sort of flexibility, you lose the automatic routing and URL capturing. This is largely due to the limitations in Apple’s WebArchive format, and this problem is similar across the board for nearly every application that depends upon it. Developers either have a choice of supporting the format and having easy web page import and display for their users, or having to write their own code.

Since the suggestion was made to use DevonAgent & DevonThink, how is it that people are able to integrate this with Scrivener … or do they integrate it? I’ve been playing with the Devon apps on a trial, trying to decide if I want to spend the money for them, but I’m not sure yet how they fit into a workflow for me. I’m slowly figuring out how to get good searches done, but I’m still not sure I see how this makes things easier with Scrivener except that I get better searches.

Terry

DA and DTPro can do everything on your original wish list, plus quite a bit more. That was the reason for my suggestion. DevonTechnologies has a pretty active forum, which you might visit for lots of tips and usage scenarios. I think they also offer extended trials–they recognize that their tools are difficult to evaluate until you’ve used them for a while.

As for integration with Scrivener, well, what do you mean by “integration?” I’ll sometimes drag copies of individual files from my large DTP database into Scrivener. Sometimes I’ll just create a citation in Scrivener, knowing that I can find the file in DTP if I need it. Other times, I’ll just have Scrivener and DTP windows open side by side. For small projects, I might store all of the materials in Scrivener, but for large projects and topics I revisit frequently, DTP is a more robust tool.

Katherine

I use EagleFiler, myself. I never much cared for the DEVONthink line of products.The thing I like about EagleFiler (and Together, for that matter), is that it’s more like an optimised Finder specifically designed for archival. They both store things in ordinary files and folders, and augment that with a database file (but store everything redundantly in forms that would allow me to access all annotations and tagging even if EagleFiler completely exploded and never worked again). They also both make capturing web pages simple. While browsing in Firefox, I can just hit Cmd-F1 and keep browsing, I don’t even have to think about it. After having collected articles for whatever I am researching, I’ll go over to EagleFiler and clean them up, tag them, and if I want them in Scrivener I’ll just drag and drop between the apps. Nothing complicated about it.

I tend to keep my archival separate from my writing projects though, so integration isn’t something that concerns me. Neither is keeping things synchronised because I am a purist when it comes to archival. If something changes, the original stays behind as a version; I never edit in place. I will occasionally put things in Scrivener temporarily so that I can take advantage of split views and so forth. If those things change, then in the end they go back to EagleFiler as a successive draft.

None of this is really specific to EagleFiler. Pretty much all of these sorts of programs have their own variations of rapid capture (from many different sources, not just browsers), hierarchy and keyword sorting, and searching. Finding the right one is just a matter of trying them all out and finding the one that fits best. You might like Together, since as I said before, it does allow one to clean up WebArchives without too much fuss. But DEVONthink isn’t to be underestimated. Some people swear by it, and if you get enough data in it, those heuristic document matching algorithms probably come in handy. For massive quantities of data, there is probably nothing better for the Mac. We have lots to choose from in this particular software niche, though.

I recommend reading this article. It goes over a number of these applications (though it is a little out of date; most notably, Together and EagleFiler have moved quite a bit beyond the limitations described). Amusingly, the author decided that Scrivener made the best archival application in the end (and you’ll notice he mirrors your complaint; namely that when it comes to capturing it isn’t quite as robust as some of these other applications).

Also of interest are our very own discussions here. When it comes to Mac writers looking for good programs, this board is one of the best resources out there. Check out the Software By Others section, and specifically this thread.

I am a fairly recent (last April) convert to Mac OS. My first purchase (after Scrivener) was DevonThink Pro, because I’d read so many good things about it. But after a few weeks, I gave up on it, as I found it less than intuitive to use. Also, to make best use of its AI function, you need to break up your data into smaller chunks, including creating multiple databases… The problem with this is that DTP does not provide a cross-database search function. For the supposedly most sophisticated data manager on the market, this is a serious flaw – which, reportedly, will be corrected in the upcoming version 2.0.

In the meantime, I went in the opposite direction – that is, toward simplicity – and got Yojimbo. This works fine, but is hardly inspiring. So now I am am trialing Together and I must say I am very impressed. It is easy to get data in and out, and the tagging feature seems quicker and more intuitive than other applications I’ve looked at. I may wait and see what version 2.0 of DTP looks like before I actually buy a license for Together… although I suspect the upgrade price for DTP 2.0 will be more than the full purchase price of Together.