Academic writing: Scrivener, Zotero, Pandoc, Marked 2

This is also available on github where it is probably easier to read.

I just wanted to post a very robust setup for using Scrivener and Zotero, since it’s something that took me a while to work out, and it might save people the hassle! This is a very detailed description of how to set it up, so don’t be put off by the length. It should take no more than 15-20 minutes, less for someone familiar with the tools used. There is another way of doing this documented here, but I think that the method outlined below is less likely to break on updates and ultimately simpler.

0. This setup assumes that you will write in Pandoc flavoured markdown - this is by far the best markdown for academic writing, mainly because of its excellent citation support. This is for OSX or Linux, but I’m sure somebody could contribute a windows version if so minded.

0b. I like to set up Scrivener to look more like a plain text editor, to remind me that I’m writing in plain text. It’s also sensible to set up a compile preset. I have attached my own presets for people to use as they wish, but I’ve made no effort to check that they work on anybody else’s computer (and they definitely won’t work on Windows), so buyer beware!

Pandoc Preset.txt (32.4 KB)
Plain Text.prefs (28.2 KB)

1. Set up zotero with the Better BibTex plugin. This is a plugin which a) makes your bibtex keys saner; and b) can auto-export to a bibtex file every time you update the database. Export a collection or the whole database in ‘Better Biblatex’ style to somewhere sensible (e.g. ~/.zotero/YOUR_BIBLIOGRAPHY.bib) and make sure you tick the ‘Keep Updated’ box.

It also includes a Pandoc citation style so that you can quick cite directly from Zotero by dragging and dropping into Scrivener.

2. Install Pandoc and download the appropriate csl file for your preferred style to somewhere sensible. Pandoc will check ~/.csl/ by default.

3. Write your work in pandoc-flavoured markdown. For citations, you can either:
a. Copy and paste from Zotero using quick cite and the pandoc citation style.
b. Use my zotpick-pandoc applet. For installation and usage instructions follow the link.
4.a (optional, OSX only) It’s possible (quite easy) to set up the excellent Marked 2 app with pandoc as a processor (you can choose the processor in Marked’s preferences). You need to put the path to your pandoc installation in the ‘Path:’ box – probably /usr/local/bin/pandoc and any arguments you need in the ‘Args:’ box – probably -r markdown -w html -s -S --normalize --bibliography ~/.pandoc/YOUR_BIBLIOGRAPHY.bib --csl ~/.csl/YOUR_CITATION_STYLE.csl

This, along with marked’s ability to watch a scrivener file, gives you a live preview of your document. I have created a clear, academic-ish style for marked (called, unimaginatively, Academia.css) which you can download here: https://github.com/ttscoff/MarkedCustomStyles

4. Export your scrivener document as plain text markdown (MYPROJECT.txt) –- i.e. not using any of the markdown conversion options etc. since we’re already writing everything in markdown anyway (you can use my preset to do this if you can make it work) -– and use a pandoc command to convert it into a filetype of your choice, eg.

pandoc -s -S --normalize --bibliography ~/.pandoc/YOUR_BIBLIOGRAPHY.bib --csl ~/.csl/YOUR_CITATION_STYLE.csl -f markdown -t docx -o MYPROJECT.docx MYPROJECT.txt 

This command would create a full document including any headers, footers or markup needed ( -s ), apply smart typography such as curly quotes ( -S ), conflate any consecutive formatting elements etc. to provide neater output ( --normalize ), use the bibliography ( --bibliography ) and citation style ( --csl ) specified, and convert from ( -f ) markdown to ( -t ) docx, with the output as a file ( -o ) called MYPROJECT.docx

It’s worth noting that if you want to use pandoc’s PDF output, you need to install LaTeX – the BasicTex package will do just fine. There are loads of options for pandoc output, which you can read about in the extremely thorough documentation.

Edited to update info on Zotero plugins

I have said this before: these forums need a ‘like’ button!

Hi,

Thanks for this great tutorial.

I wrote this in another thread, but I thought I’d ask your opinion just in case, since you seem to know a bit about citations:

I have two questions.

1- I am using an MLZ Client for my zotero because we broke it to hold our specific law format. Will this work around allow me to export from Scrivener into Word WITH the footnotes in tact and recognise them as Zotero footnotes.

2- if you believe it will, can you give simpler instructions to achieve what you’ve done- I only understood about 15% of what you said in the first post of instructions. Sorry, I’m new to Mac, a law student, and computers baffle me in the way that Hayek baffles my mechanic.

Really need some solution, so if this isn’t it, I may revert to my old ways, bag Scrivener and write in word. I hate to do that, because I love what Scrivener can achieve, but ultimately, I will have to produce a word document to edit I think. I need some way to create a giant bibilography for the 100,000 words I am writing, which contains about 400,000 citations. I cannot CANNOT do that manually.

Help?

Give you have so many(!) citations, then see if your university/college provides a licence for EndNote. If it does, then switch. You will find (a) that because it is supported by the university many of your colleagues use it and (b) if you run into difficulties you will have lots of support. This support will be both professional (from your IT department for installation issues and your library for usage issues) and collegial (from staff and students who use it).

Two final points so you know where I’m coming from:

1. I don’t like EndNote (I actually prefer Papers), but EndNote’s power and flexibility is unequalled. I used it for my thesis and if I was starting again tomorrow would do so again.
2. In the unlikely event that I had to choose between using Word with a citation manager or Scrivener without one, I would still choose Scrivener. Word simply cannot do what Scrivener can do and, although I would dearly miss a citation manager, a semi-automated reference list is no substitute for the benefits that Scrivener brings to planning, structuring, writing, editing and reviewing my work. That’s not even mentioning Scrivener’s research capabilities, speed and stability. Although Word has improved in recent years, I have learnt not to trust it with long-form documents. Hmm, I also didn’t mention snapshots, backups, folder sync or cloud sync capabilities. Word is great for the final polish, but not for constructing the beast.

An average of four citations per word seems incredible. Are you not allowed to use composite footnotes?

endnotes and references

New here and haven’t been able to find a question as simple as mine; hope that means there’s an easy answer! I need to have one endnotes section and one reference section–they can’t be combined. How can I do it? As far as I can tell, what is called “references” in Scrivener has more to do with web-linking than with citing references in the traditional sense. Is it necessary to have a separate app? I’ll be compiling to a Word doc.

Thanks!

What does your “reference” section contain? Is it just a bibliography, or does it link back to specific locations in the text?

Similarly, are your “endnotes” citations, or informational?

Scrivener supports multiple note streams, so what you want should be doable, but I need a little information to help you figure out the best way to accomplish your goal.

Katherine

My assumption from the above is that “endnotes section” refers to annotations of your text and that “references sections” lists your citations. Is that correct?

Bump for various updates which make this an even better workflow, in particular much better citation support.

OK, wow! I have just got my head around Davepwsmith’s original post. Or some of it. May I translate for other ordinary mortals?

1. Please do click the github link, it is easier to follow
2. The part I haven’t got my head around is writing in Pandoc flavoured Markdown, but at least now I see why it might be a Good Thing - the perennial issue is that Scriv doesn’t work with styles so on compile to Libre Office, Word etc you need to do a bit of work to set them up. The Markdown route gets around that, though for my money, just hoping that Scriv 3 does styles and staying with the ODT/RTF Scannable Cite patch for Zotero.
3. The part of Davepwsmith’s solution that is great good news for people like me is his Applescript applet for zotpick-scannable-cite. This is clearer on the Github link than it is on this forum. If you want to go the whole hog on Pandoc (Pandora??) you need the zotpick-pandoc applet. Either way, you do need to install the Better BibTex plugin for Zotero, and Davepwsmith’s link takes you straight where you need to go for that. Download the addin and instal it in Zotero exactly as per his instructions.
4. The Applescript text (I have elsewhere denigrated it as algebra … sorry) just needs to be copied and pasted into a Script Editor window. This is where I had myself a stiff drink and took it slow … in the Finder, go to the Applications folder, within that, the Utilities folder and open the script editor. Paste in the zotpick-scannable-cite text from Davepwsmith’s github site, then go File-Save - make sure you select the option Application and save.
5. Davepwsmith tells you to allocate this application as Scrivener’s Bibliography Manager. Here is where I got the collywobbles. Go to Scrivener preferences, General, look down near the bottom and you see Bibliography Manager. Mine had Evernote by default. Click Choose, then a Finder-style window opens up. At first I couldn’t find my zotpick applet, then I discovered I had saved it into the iCloud folder by default. I dragged it into the Applications folder using Finder - then it showed up OK for Scrivener so I could select zotpick-scannable-cite as my Bibliography Manager.
6. You need to make sure you have the Zotero RTF/ODF Scan addon installed in Zotero, and have selected Scannable Cite as the default output in (Zotero) prefs.
7. Then you just cite n write - it is way worth the effort! ie - in your Scrivener Editor window just press Command-y and the Zotero picker window opens up for you to start typing the author or title name in exactly the same way as it works with the Word or LO addon. The citation is added in the RTF/ODF Scannable Cite format with any prefix or suffix you have entered.
8. None of this changes your workflow for compiling to odt and running the scan from Zotero to turn the temporary citations ‘live’. Unless of course you take another stiff drink and get your head around Davepwsmith’s Pandoc stuff.

With apologies to Davepwsmith for anything I have misrepresented or misunderstood …

OK and a few more stiff drinks later I have implemented the other part of Davepwsmith’s workflow, ie. writing in Pandoc-flavoured Markdown in Scrivener, using the zotpick-pandoc-applet for citations, compiling as a text file and using pandoc in the terminal window to convert to a formatted Word file. Given I already knew my way around basic Markdown this is not too difficult, and the instructions are perfectly clear. A very useful site is pandoc.org especially the getting Started guide. I was very impressed with the output, with citations perfectly represented in my Chicago 16 full notes style, including the Bibliography. Also pleased to see the resulting Word document had correctly allocated styles eg headers mapped to header styles, the body text as ‘first paragraph’ style, footnotes as footnote style and reference list as Bibliography style.
Given that in my writing (theology) I am using only fairly basic formatting - headers, emphasis, the occasional table plus of course tons of citations and other footnotes - this work flow would suit me perfectly … with one fairly major BUT at present, which is internal cross-referencing. eg “As I argued in chapter 5 (see p. 115) …”. I see that internal cross-reference links are supported, which is sufficient if we are publishing online, but actual printed paper needs cross-references with page and/or section numbers. Scrivener supports this with place-holder tags <$hn> and <$p> but presumably these will not survive compiling to .txt.
Does anybody know of anything in the Markdown/pandoc conversion process that would support this?

Cross-referencing with Pandoc can be done with this plug-in available here:

https://hackage.haskell.org/package/pandoc-crossref