Updated MMD > latex guide / Export to multiple .tex files?

ja
jandavid
Posts: 20
Joined: Wed Apr 06, 2016 8:41 am
Platform: Mac

Tue Aug 23, 2016 10:21 pm Post

I am writing my dissertation and slowly approaching the stage where I'd like to compile first drafts for which I have started reading a few of the Scrivener > MMD > LaTeX guides.

Now my first question, in order to avoid trying to implement outdated workflows, what would be the most up to date version of the multiple guides out there? All I found is from around 3 or 4 years ago and both Scrivener and MMD have been updated since.

Then, a more specific question, which I have not found any information on whether that is possible at all:

I have a LaTeX setup where I have my main document with all relevant settings but no text, which I call with \include{Chapter1}, etc.
Is there a way to automatically export each chapter as a separate .tex file from Scrivener?

Thinking about this then gave me another idea, although that might be rather something for the future (=wish list).

Would it be possible to configure Scivener in a way that it keeps a folder in sync with the main document, containing mmd-created .tex-versions of all subdocuments of Draft? Maybe that folder could even be contained within the Scivener project package, since it's easy to point LaTeX to it. Would that make sense?

Alternatively, if that's too complicated, I don't know to what extent Scrivener is Apple-scriptable, but maybe one could come up with an Applescript that exports the chapters as .tex files and then assign a keyboard shortcut to it. That would basically accomplish the same thing.

Any hints highly appreciated!

User avatar
nontroppo
Posts: 755
Joined: Mon Mar 05, 2007 5:22 pm
Platform: Mac
Location: Airstrip One

Wed Aug 24, 2016 3:10 pm Post

Have a look at File > Sync > With External Folder... (p461 of the user manual). This can be configured to spit out plain text files for each document in your Draft. Those plain text files could then easily be converted to TeX and compiled together using a simple script and MMD.

Also have a look at Pandoc rather than MMD for conversion to LaTeX. I very rarely use LaTeX but I found Pandoc more flexible — it has a great meta-data system where you can use variables to modify your LaTex template. It is slightly more actively developed than MMD and more feature rich.

ja
jandavid
Posts: 20
Joined: Wed Apr 06, 2016 8:41 am
Platform: Mac

Wed Aug 24, 2016 7:15 pm Post

Oh wow, that looks indeed promising!
The only problem is that if I sync plain text files they lose all the formatting whatsoever (and I'm not writing directly LaTeX code).
So there's no markup that I could manipulate.
Any ideas?
It also shows me an option "Fountain" which does seem to include some markup. But I don't exactly know what that option can and cannot do...
Is there maybe a way to tell the sync mechanism to use a custom setup as it is available on compile?
Any further hints would be greatly appreciated.

PS: Thinking about this a bit further it's probably not too hard to turn the sync folder into a watched folder and then attach a script to it that whenever something is modified syncs the contents to a second folder which then converts the files to .tex files via MMD or Pandoc or some other mechanism. Or is there an easier method?

User avatar
Silverdragon
Posts: 330
Joined: Mon Jul 29, 2013 2:52 pm
Platform: Mac + iOS
Contact:

Wed Aug 24, 2016 8:49 pm Post

jandavid wrote:Oh wow, that looks indeed promising!
The only problem is that if I sync plain text files they lose all the formatting whatsoever (and I'm not writing directly LaTeX code).
So there's no markup that I could manipulate.
Any ideas?
It also shows me an option "Fountain" which does seem to include some markup. But I don't exactly know what that option can and cannot do...
Is there maybe a way to tell the sync mechanism to use a custom setup as it is available on compile?
Any further hints would be greatly appreciated.

PS: Thinking about this a bit further it's probably not too hard to turn the sync folder into a watched folder and then attach a script to it that whenever something is modified syncs the contents to a second folder which then converts the files to .tex files via MMD or Pandoc or some other mechanism. Or is there an easier method?

As I understand it, Fountain is for scriptwriting.

I used external folder sync extensively in the days Before iOS Scrivener. I don't know how applicable it is to your situation, but here's how I worked it:

1. I used MD markup in Scrivener for italics and bold (there's a handy one-way conversion command.)
2. I used three options in external sync: I set format for external Draft files to plain text and the extension to .md. I set convert plain text paragraph spacing to On.

The net results in the .md files produced were:

1. Bold and italics happened where I wanted them on the .md side. On the Scrivener side they appeared as markup, not previewed.
2. Paragraph breaks were converted to two new lines on the.md side and converted back to paragraph breaks on the Scrivener side.
3. Inline annotations appeared inside ((double parens)) and inline footnotes inside {{double braces}}. These were converted back to the appropriate formatting inside Scrivener.

I believe this is all the markup processing that Scrivener will do via external folder sync.

Hope this helps!
So you know where I'm coming from:
• I'm a user, not an L&L employee.
• Mac Scrivener 2.9 & 3.0.0, MacBook Air 11, MacOS 10.12.6 (Sierra)
• iOS Scrivener 1.1.5, iPhone 6s, iPad Air 2, iOS 11.2

User avatar
nontroppo
Posts: 755
Joined: Mon Mar 05, 2007 5:22 pm
Platform: Mac
Location: Airstrip One

Thu Aug 25, 2016 2:19 am Post

Yes, folder actions / automator should enable you to trigger a conversion whenever a .md file is added to a folder. My experience is that folder actions can sometimes be a bit fiddly to set up (and are poorly documented) but once setup work great.

You want to make sure you do not modify the .md files in the sync folder so they don't get synced back to Scrivener (though note that auto-generated snapshots can recover you if this happens if I remember correctly). The folder action takes any .md files and runs Pandoc/MMD on them to generate a series of .latex files which you can then source into your LaTeX template

To go back to the multiple files again though. With Pandoc you can create a custom LaTeX template and this gets prepended automatically to your conversion. This means you don't need to use your \include{Chapter1} inclusion method if all you want is to use a customised LaTeX template. Also it means you can use Scrivener meta-data to toggle settings in the final latex, so for example you can use `twocolumn` and let it toggle one or two column output depending on whether it is 1 or 0 etc.

ja
jandavid
Posts: 20
Joined: Wed Apr 06, 2016 8:41 am
Platform: Mac

Thu Aug 25, 2016 3:01 pm Post

Thanks @nontroppo and @Silverdragon for your suggestions.

I once set up a watched folder via fswatch and launchd which does some syncing of files when they change – I'll have to dig up how I did that but that shouldn't be too hard to adapt. Then I could execute mmd or pandoc upon any changes in the folder in order to produce my .tex files. I won't need to sync back to Scrivener because no changes will be made to the files outside of Scrivener, so I just turn that option off in the sync to not run in to problems.

The only tricky thing is to have Scivener produce a markup document. The closest I could get was indeed the Fountain option but as far as I could tell the only things it does is **bold** and *italic* markup, and footnotes are placed in [[double square brackets]], which is neither mmd nor pandoc format. I don't need a lot but titles (ideally generated from the name of the subdocument) would be nice. The citations I'd have to enter directly in the markup format but even when compiling in Scrivener it's that way (I read a little bit about pandoc BTW and I liked their citation format because, unlike MMD, it has the page numbers after the citekey in the same set of square brackets and not before).
@Silverdragon's option to first convert bold and italic to mmd and then sync as plain text might work but it involves an extra step and also doesn't resolve the problem with the rest of the markdown. Fountain at least already does the bold and italic for me.

So essentially: is there a way to manipulate the output in the sync'ed folder from Scrivener?

And if not, what do people think would be the best approach:
Using Fountain and marking up all the other stuff directly (and somehow scripting proper [[footnote]] conversion)?
Or keeping .rtf files in the folder and trying to go directly from .rtf to LaTeX (either via mmd or via pandoc)?

User avatar
nontroppo
Posts: 755
Joined: Mon Mar 05, 2007 5:22 pm
Platform: Mac
Location: Airstrip One

Fri Aug 26, 2016 1:37 pm Post

IIRC You can pass multiple files to MMD or Pandoc and it will join them all together for you. Each document contains a different heading I assume so that gets you your structure. Scrivener converts footnotes to MMD format without issue (well, did when I last used it). If you use MMD format for figures, then you even get properly structured [Figure|Caption] blocks etc.

I don't understand the need to use Fountain, just stick to plain text and MMD, and then I suppose my point is: what isn't working? 8)

EDIT: Ah just tested it and yes, not all the conversions that occur at compile time occur when using external sync (e.g. I use Scrivener links for image files and they don't get converted to the filename etc.)

So then I still wonder why you really need multiple files...

ja
jandavid
Posts: 20
Joined: Wed Apr 06, 2016 8:41 am
Platform: Mac

Fri Aug 26, 2016 2:09 pm Post

Yeah, so I tested it again too.
Plain text converts Footnotes correctly indeed but not **bold** and *italic*. And you said it doesn't do links either.
So that's why I thought Fountain might be an alternative.

But in any case, if I use pandoc and not MMD then I need to go an extra step anyways from either [[Fountain]] or {{MMD}} formatted notes to ^[Pandoc] format. So in that case maybe just keeping it as .rtf and trying to set up pandoc to convert from .rtf to .tex in a separate folder might be the easiest way. I still need to read deeper into the pandoc manual though, to figure out how to do that.

About the need for multiple files:
In LaTeX when I \include{} each chapter separately, then in the preamble I can \includeonly{} specific chapters, when I want to send these to certain people (experts on a particular topic). So that's the main advantage.
I believe there are other ways to do it, I still haven't figured out exactly what the best way would be.
It'd be ideal if one could configure Scrivener to sync that folder, i.e., in terms of hierarchy of documents etc. But I guess that's not possible (yet?).

User avatar
nontroppo
Posts: 755
Joined: Mon Mar 05, 2007 5:22 pm
Platform: Mac
Location: Airstrip One

Sat Aug 27, 2016 4:40 am Post

Pandoc has a specialised MMD compatibility mode:

http://pandoc.org/MANUAL.html#markdown-variants

I thought MMD inline notes used [^this format], whereas Pandoc uses ^[this one] (I may put a feature request in to allow the MMD inline footnote variant). I've only ever used footnotes which Scrivener converts and works without issue (both MMD and Pandoc use the same [^footnote] syntax).

The {{ … }} format in MMD is for including external files, not for comments/notes (and for instructing the placement of a {{TOC}}). I've never used Fountain so can't comment, but compile+MMD has to be more flexible/powerful.

As for specific "versions" of a document, why not just use the Compile feature in Scrivener itself, where you have detailed control of inclusion/exclusion per document in your manuscript. I use this for scientific articles where I have the "vanilla" manuscript for a resubmission, then reviewer replies, TODO, figure variants and other notes for my co-authors. I use a document based approach to this and it gives me great flexibility all from one Scrivener project...

ja
jandavid
Posts: 20
Joined: Wed Apr 06, 2016 8:41 am
Platform: Mac

Sat Aug 27, 2016 6:49 am Post

You're right about [^footnote] syntax, I think I got confused with the other post.
I might end up just using compile, as you said (I wish there was a Scrivener Apple Script library).
Initially I was intrigued by the sync option since I hadn't heard of it and since it could export to multiple files, but it doesn't seem to be feature-rich enough out of the box, and it'll probably be too cumbersome to set it up.

The only thing where now that I've I prefer Pandoc over MMD is citations, where Pandoc [see @Citekey, pp. 1--5] seems to be much more flexible and also I like that the page number comes after the citation as opposed to MMD [pp. 1--5][#Citekey].

Do you know if there is a way in the Scrivener MMD > LaTeX compile to convert cite keys from pandoc format?
I could of course also set up my own arbitrary Replacement patters upon compile, maybe a regex, but that might be error-prone.

User avatar
nontroppo
Posts: 755
Joined: Mon Mar 05, 2007 5:22 pm
Platform: Mac
Location: Airstrip One

Sun Aug 28, 2016 5:37 am Post

I think Pandoc citations like [@smith04; @doe99] will pass through just fine to the output (no need for conversion, just write them in pandoc format in the text). The important point for Pandoc is there are two ways to deal with citations, `pandoc-citeproc` is the system you've described, where Pandoc uses your .BIB file and CSL styles to generate the bibliography before it ever reaches LaTeX or it simply enables BibTex or BibLaTeX to generate the bibliography[1]:

https://github.com/jgm/pandoc-templates ... latex#L105

But the MMD>LaTeX compile path doesn't let Pandoc easily replace MMD (yet, AmberV has hinted Scrivener 3 may have more flexible compile functionality), so the easiest way to use it is to compile to MMD and then triggger Pandoc. Note that you can use YAML metadata for Pandoc from Scrivener, but it needs to go into the text itself, so for example my front-matter contains:

Code: Select all

--- 
author: <$fullname> 
title: <$projecttitle> 
documentclass: article 
mainfont: "Dolly Pro" 
sansfont: "Graublau Sans" 
monofont: "Fira Code" 
CJKmainfont: "Arial Unicode MS" 
fontsize: 15pt 
linestretch: 1.4 
twocolumn: 
toc: 
titlesec: 
maketitle: 
lang: en-GB 
papersize: A4 
geometry: 
    - margin=2cm 
--- 

# <$Projecttitle> #



I delete any MMD meta–data in Compile... because at present Scrivener doesn't generate the correct formatting for front-matter compatible with both MMD and Pandoc.

I then use Quicksilver to trigger Pandoc with custom settings, I select the .md file in Quicksilver and run scripts for either DOCX/ODT or LaTeX. I have custom templates for DOCX (a .DOTX word file with modified styles), or a modified default.latex and default metadata.yaml file. Pandoc gives higher priority to document meta–data. You could use Folder Actions to run this kind of script.

For completeness here is my ZSH script:

Code: Select all

#!/bin/zsh
export PATH=/Library/TeX/texbin:/usr/local/bin:$PATH
cd "$( dirname "$1" )"
filename=$(basename "$1")

/usr/local/bin/pandoc -s --dpi=300 --template=$HOME/.pandoc/default.latex --latex-engine=xelatex -o "$filename.tex" "$filename" $HOME/.pandoc/metadata.yaml
/usr/local/bin/pandoc -s --dpi=300 --template=$HOME/.pandoc/default.latex --latex-engine=xelatex -o "$filename.pdf" "$filename" $HOME/.pandoc/metadata.yaml


----
[1]: I don't use either, but use Bookends to scan DOCX/ODT/MD files. I am tempted to switch to pandoc-citeproc but haven't got round to sorting out converting my current papers.

ja
jandavid
Posts: 20
Joined: Wed Apr 06, 2016 8:41 am
Platform: Mac

Wed Oct 05, 2016 9:27 am Post

For the record, here's my solution to these problems.
Thanks to @nontroppo for all the pandoc hints!

(1) First step I use the fountain format to get everything out of Scrivener. I do this since this seems the only way to properly translate bold and italic formatting to **bold** and *italic*. (I prefer to keep my document in Scivener with rtf formatted italic and bold, from what I read elsewhere on the forum Scrivener 3 seems to be going to offer some further options here ... looking forward)

(2) Then I run a regex thru perl that replaces tabs with > and [[fountain script notes]] with ^[markdown footnotes]

(3) And now I run pandoc to convert markdown to latex. And voila I got my latex input (which I can then \include{} in my project).

Oh, yes and I defaulted to writing my citations in pandoc format directly.

I have opted against using the "sync" folder because it would not have any information about header and subheader levels, as it exports every document as one file. It would have been great if one could define some options that would sync documents by "chapter" or the whole draft into one document while also respecting the header levels (a compile-sync so to speak), but maybe a solution for that comes in Scrivener 3 ...
For now, since compiling to fountain has the option to respect the header levels that's the easiest and cleanest way to do it. So that's now for me the solution that gives me the best of both worlds (or rather all three worlds, rft, markdown, and latex).