Academic writing: Scrivener, Zotero, Pandoc, Marked 2

da
davepwsmith
Posts: 8
Joined: Sun Sep 04, 2011 10:11 pm
Platform: Mac

Tue Feb 04, 2014 12:09 pm Post

This is also available on github where it is probably easier to read.

I just wanted to post a very robust setup for using Scrivener and Zotero, since it's something that took me a while to work out, and it might save people the hassle! This is a very detailed description of how to set it up, so don't be put off by the length. It should take no more than 15-20 minutes, less for someone familiar with the tools used. There is another way of doing this documented here, but I think that the method outlined below is less likely to break on updates and ultimately simpler.

0. This setup assumes that you will write in Pandoc flavoured markdown - this is by far the best markdown for academic writing, mainly because of its excellent citation support. This is for OSX or Linux, but I'm sure somebody could contribute a windows version if so minded.

0b. I like to set up Scrivener to look more like a plain text editor, to remind me that I'm writing in plain text. It's also sensible to set up a compile preset. I have attached my own presets for people to use as they wish, but I've made no effort to check that they work on anybody else's computer (and they definitely won't work on Windows), so buyer beware!

Pandoc Preset.txt
Compile preset for exporting to pandoc/plain text. You will have to rename this to Pandoc Preset.plist, but the forum wouldn't let me upload a .plist file.
(32.36 KiB) Downloaded 292 times

Plain Text.prefs
Editor Preferences for working with plain text.
(28.18 KiB) Downloaded 249 times


1. Set up zotero with the Better BibTex plugin. This is a plugin which a) makes your bibtex keys saner; and b) can auto-export to a bibtex file every time you update the database. Export a collection or the whole database in 'Better Biblatex' style to somewhere sensible (e.g. ~/.zotero/YOUR_BIBLIOGRAPHY.bib) and make sure you tick the 'Keep Updated' box.

It also includes a Pandoc citation style so that you can quick cite directly from Zotero by dragging and dropping into Scrivener.

2. Install Pandoc and download the appropriate csl file for your preferred style to somewhere sensible. Pandoc will check ~/.csl/ by default.

3. Write your work in pandoc-flavoured markdown. For citations, you can either:
    a. Copy and paste from Zotero using quick cite and the pandoc citation style.
    b. Use my zotpick-pandoc applet. For installation and usage instructions follow the link.
4.a (optional, OSX only) It's possible (quite easy) to set up the excellent Marked 2 app with pandoc as a processor (you can choose the processor in Marked's preferences). You need to put the path to your pandoc installation in the 'Path:' box -- probably

Code: Select all

/usr/local/bin/pandoc
and any arguments you need in the 'Args:' box -- probably

Code: Select all

-r markdown -w html -s -S --normalize --bibliography ~/.pandoc/YOUR_BIBLIOGRAPHY.bib --csl ~/.csl/YOUR_CITATION_STYLE.csl


This, along with marked's ability to watch a scrivener file, gives you a live preview of your document. I have created a clear, academic-ish style for marked (called, unimaginatively, Academia.css) which you can download here: https://github.com/ttscoff/MarkedCustomStyles

4. Export your scrivener document as plain text markdown (MYPROJECT.txt) –- i.e. not using any of the markdown conversion options etc. since we're already writing everything in markdown anyway (you can use my preset to do this if you can make it work) -– and use a pandoc command to convert it into a filetype of your choice, eg.

Code: Select all

pandoc -s -S --normalize --bibliography ~/.pandoc/YOUR_BIBLIOGRAPHY.bib --csl ~/.csl/YOUR_CITATION_STYLE.csl -f markdown -t docx -o MYPROJECT.docx MYPROJECT.txt

This command would create a full document including any headers, footers or markup needed ( -s ), apply smart typography such as curly quotes ( -S ), conflate any consecutive formatting elements etc. to provide neater output ( --normalize ), use the bibliography ( --bibliography ) and citation style ( --csl ) specified, and convert from ( -f ) markdown to ( -t ) docx, with the output as a file ( -o ) called MYPROJECT.docx

It's worth noting that if you want to use pandoc's PDF output, you need to install LaTeX -- the BasicTex package will do just fine. There are loads of options for pandoc output, which you can read about in the extremely thorough documentation.

*Edited to update info on Zotero plugins*
Last edited by davepwsmith on Wed Sep 02, 2015 12:50 pm, edited 12 times in total.

ma
marcoiac
Posts: 446
Joined: Mon Aug 29, 2011 12:12 am
Platform: Mac

Wed Feb 05, 2014 2:52 pm Post

I have said this before: these forums need a 'like' button! :D

el
elischiff
Posts: 13
Joined: Sat Feb 22, 2014 9:07 am
Platform: Mac

Sat Feb 22, 2014 10:39 am Post

Hi,

Thanks for this great tutorial.

I wrote this in another thread, but I thought I'd ask your opinion just in case, since you seem to know a bit about citations:

Hi, I'm writing for a thesis project for school and the output is going to be a Wordpress site. The plan is to learn MMD so that it is simpler to go to HTML and I'll just have css styles ready to apply on Wordpress. I am wondering if anyone knows the best way to handle footnotes is with MMD and Scrivener. My goal is to have it work something like the attached image.

TL;DR: If Im using MMD to write blog posts with footnoted citations, and I want to have a bibliography page, should I use Scrivener's own "Footnote" feature, the Scivener "Reference" feature, Multi-Markdown's "Footnote" syntax, or something else entirely? Is there a way to make all footnote links point to one bibliography page?
Attachments
Screen Shot 2014-02-22 at 5.34.42 AM.png
Screen Shot 2014-02-22 at 5.34.42 AM.png (73.12 KiB) Viewed 7905 times

el
els_phd_nz
Posts: 3
Joined: Mon Mar 17, 2014 2:25 am
Platform: Windows

Mon Mar 31, 2014 10:04 pm Post

I have two questions.

1- I am using an MLZ Client for my zotero because we broke it to hold our specific law format. Will this work around allow me to export from Scrivener into Word WITH the footnotes in tact and recognise them as Zotero footnotes.

2- if you believe it will, can you give simpler instructions to achieve what you've done- I only understood about 15% of what you said in the first post of instructions. Sorry, I'm new to Mac, a law student, and computers baffle me in the way that Hayek baffles my mechanic.

Really need some solution, so if this isn't it, I may revert to my old ways, bag Scrivener and write in word. I hate to do that, because I love what Scrivener can achieve, but ultimately, I will have to produce a word document to edit I think. I need some way to create a giant bibilography for the 100,000 words I am writing, which contains about 400,000 citations. I cannot CANNOT do that manually.

Help?

User avatar
nom
Posts: 1921
Joined: Sun Aug 31, 2008 12:02 am
Platform: Mac + iOS
Location: Melbourne, Australia
Contact:

Tue Apr 01, 2014 11:14 am Post

els_phd_nz wrote:I have two questions.

1- I am using an MLZ Client for my zotero because we broke it to hold our specific law format. Will this work around allow me to export from Scrivener into Word WITH the footnotes in tact and recognise them as Zotero footnotes.

2- if you believe it will, can you give simpler instructions to achieve what you've done- I only understood about 15% of what you said in the first post of instructions. Sorry, I'm new to Mac, a law student, and computers baffle me in the way that Hayek baffles my mechanic.

Really need some solution, so if this isn't it, I may revert to my old ways, bag Scrivener and write in word. I hate to do that, because I love what Scrivener can achieve, but ultimately, I will have to produce a word document to edit I think. I need some way to create a giant bibilography for the 100,000 words I am writing, which contains about 400,000 citations. I cannot CANNOT do that manually.

Help?


Give you have so many(!) citations, then see if your university/college provides a licence for EndNote. If it does, then switch. You will find (a) that because it is supported by the university many of your colleagues use it and (b) if you run into difficulties you will have lots of support. This support will be both professional (from your IT department for installation issues and your library for usage issues) and collegial (from staff and students who use it).

Two final points so you know where I'm coming from:
1. I don't like EndNote (I actually prefer Papers), but EndNote's power and flexibility is unequalled. I used it for my thesis and if I was starting again tomorrow would do so again.
2. In the unlikely event that I had to choose between using Word with a citation manager or Scrivener without one, I would still choose Scrivener. Word simply cannot do what Scrivener can do and, although I would dearly miss a citation manager, a semi-automated reference list is no substitute for the benefits that Scrivener brings to planning, structuring, writing, editing and reviewing my work. That's not even mentioning Scrivener's research capabilities, speed and stability. Although Word has improved in recent years, I have learnt not to trust it with long-form documents. Hmm, I also didn't mention snapshots, backups, folder sync or cloud sync capabilities. Word is great for the final polish, but not for constructing the beast.
Complete and utter NOMsense.
Image

Ki
Kinsey
Posts: 418
Joined: Mon Apr 09, 2012 6:06 pm
Platform: Mac + Windows

Thu Apr 03, 2014 2:20 am Post

An average of four citations per word seems incredible. Are you not allowed to use composite footnotes?

dl
dljnobile
Posts: 3
Joined: Fri Sep 12, 2014 5:56 am
Platform: Mac

Fri Sep 12, 2014 6:20 am Post

endnotes and references

New here and haven't been able to find a question as simple as mine; hope that means there's an easy answer! I need to have one endnotes section and one reference section--they can't be combined. How can I do it? As far as I can tell, what is called "references" in Scrivener has more to do with web-linking than with citing references in the traditional sense. Is it necessary to have a separate app? I'll be compiling to a Word doc.

Thanks!

User avatar
kewms
Posts: 6412
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Wed Sep 17, 2014 11:00 pm Post

What does your "reference" section contain? Is it just a bibliography, or does it link back to specific locations in the text?

Similarly, are your "endnotes" citations, or informational?

Scrivener supports multiple note streams, so what you want should be doable, but I need a little information to help you figure out the best way to accomplish your goal.

Katherine
Scrivener Support Team

User avatar
nom
Posts: 1921
Joined: Sun Aug 31, 2008 12:02 am
Platform: Mac + iOS
Location: Melbourne, Australia
Contact:

Fri Sep 19, 2014 10:58 am Post

dljnobile wrote:endnotes and references

New here and haven't been able to find a question as simple as mine; hope that means there's an easy answer! I need to have one endnotes section and one reference section--they can't be combined. How can I do it? As far as I can tell, what is called "references" in Scrivener has more to do with web-linking than with citing references in the traditional sense. Is it necessary to have a separate app? I'll be compiling to a Word doc.

Thanks!


My assumption from the above is that "endnotes section" refers to annotations of your text and that "references sections" lists your citations. Is that correct?
Complete and utter NOMsense.
Image

da
davepwsmith
Posts: 8
Joined: Sun Sep 04, 2011 10:11 pm
Platform: Mac

Wed Sep 02, 2015 11:04 am Post

*Bump* for various updates which make this an even better workflow, in particular much better citation support.

ep
epederick
Posts: 20
Joined: Mon May 13, 2013 5:14 am
Platform: Mac

Tue Mar 08, 2016 8:26 am Post

OK, wow! I have just got my head around Davepwsmith's original post. Or some of it. May I translate for other ordinary mortals?
1. Please do click the github link, it is easier to follow
2. The part I haven't got my head around is writing in Pandoc flavoured Markdown, but at least now I see why it might be a Good Thing - the perennial issue is that Scriv doesn't work with styles so on compile to Libre Office, Word etc you need to do a bit of work to set them up. The Markdown route gets around that, though for my money, just hoping that Scriv 3 does styles and staying with the ODT/RTF Scannable Cite patch for Zotero.
3. The part of Davepwsmith's solution that is great good news for people like me is his Applescript applet for zotpick-scannable-cite. This is clearer on the Github link than it is on this forum. If you want to go the whole hog on Pandoc (Pandora??) you need the zotpick-pandoc applet. Either way, you do need to install the Better BibTex plugin for Zotero, and Davepwsmith's link takes you straight where you need to go for that. Download the addin and instal it in Zotero exactly as per his instructions.
4. The Applescript text (I have elsewhere denigrated it as algebra ... sorry) just needs to be copied and pasted into a Script Editor window. This is where I had myself a stiff drink and took it slow ... in the Finder, go to the Applications folder, within that, the Utilities folder and open the script editor. Paste in the zotpick-scannable-cite text from Davepwsmith's github site, then go File-Save - make sure you select the option Application and save.
5. Davepwsmith tells you to allocate this application as Scrivener's Bibliography Manager. Here is where I got the collywobbles. Go to Scrivener preferences, General, look down near the bottom and you see Bibliography Manager. Mine had Evernote by default. Click Choose, then a Finder-style window opens up. At first I couldn't find my zotpick applet, then I discovered I had saved it into the iCloud folder by default. I dragged it into the Applications folder using Finder - then it showed up OK for Scrivener so I could select zotpick-scannable-cite as my Bibliography Manager.
6. You need to make sure you have the Zotero RTF/ODF Scan addon installed in Zotero, and have selected Scannable Cite as the default output in (Zotero) prefs.
7. Then you just cite n write - it is way worth the effort! ie - in your Scrivener Editor window just press Command-y and the Zotero picker window opens up for you to start typing the author or title name in exactly the same way as it works with the Word or LO addon. The citation is added in the RTF/ODF Scannable Cite format with any prefix or suffix you have entered.
8. None of this changes your workflow for compiling to odt and running the scan from Zotero to turn the temporary citations 'live'. Unless of course you take another stiff drink and get your head around Davepwsmith's Pandoc stuff.

With apologies to Davepwsmith for anything I have misrepresented or misunderstood ...

ep
epederick
Posts: 20
Joined: Mon May 13, 2013 5:14 am
Platform: Mac

Thu Mar 10, 2016 1:25 am Post

OK and a few more stiff drinks later I have implemented the other part of Davepwsmith's workflow, ie. writing in Pandoc-flavoured Markdown in Scrivener, using the zotpick-pandoc-applet for citations, compiling as a text file and using pandoc in the terminal window to convert to a formatted Word file. Given I already knew my way around basic Markdown this is not too difficult, and the instructions are perfectly clear. A very useful site is pandoc.org especially the getting Started guide. I was very impressed with the output, with citations perfectly represented in my Chicago 16 full notes style, including the Bibliography. Also pleased to see the resulting Word document had correctly allocated styles eg headers mapped to header styles, the body text as 'first paragraph' style, footnotes as footnote style and reference list as Bibliography style.
Given that in my writing (theology) I am using only fairly basic formatting - headers, emphasis, the occasional table plus of course tons of citations and other footnotes - this work flow would suit me perfectly ... with one fairly major BUT at present, which is internal cross-referencing. eg "As I argued in chapter 5 (see p. 115) ...". I see that internal cross-reference links are supported, which is sufficient if we are publishing online, but actual printed paper needs cross-references with page and/or section numbers. Scrivener supports this with place-holder tags <$hn> and <$p> but presumably these will not survive compiling to .txt.
Does anybody know of anything in the Markdown/pandoc conversion process that would support this?

li
linksvexier
Posts: 2
Joined: Tue Mar 15, 2016 3:27 am
Platform: Mac

Tue Mar 15, 2016 3:50 am Post

Cross-referencing with Pandoc can be done with this plug-in available here:

https://hackage.haskell.org/package/pandoc-crossref


epederick wrote:OK and a few more stiff drinks later I have implemented the other part of Davepwsmith's workflow, ie. writing in Pandoc-flavoured Markdown in Scrivener, using the zotpick-pandoc-applet for citations, compiling as a text file and using pandoc in the terminal window to convert to a formatted Word file. Given I already knew my way around basic Markdown this is not too difficult, and the instructions are perfectly clear. A very useful site is pandoc.org especially the getting Started guide. I was very impressed with the output, with citations perfectly represented in my Chicago 16 full notes style, including the Bibliography. Also pleased to see the resulting Word document had correctly allocated styles eg headers mapped to header styles, the body text as 'first paragraph' style, footnotes as footnote style and reference list as Bibliography style.
Given that in my writing (theology) I am using only fairly basic formatting - headers, emphasis, the occasional table plus of course tons of citations and other footnotes - this work flow would suit me perfectly ... with one fairly major BUT at present, which is internal cross-referencing. eg "As I argued in chapter 5 (see p. 115) ...". I see that internal cross-reference links are supported, which is sufficient if we are publishing online, but actual printed paper needs cross-references with page and/or section numbers. Scrivener supports this with place-holder tags <$hn> and <$p> but presumably these will not survive compiling to .txt.
Does anybody know of anything in the Markdown/pandoc conversion process that would support this?