Footnote formatting in MMD compile

Ro
RobertG
Posts: 15
Joined: Fri May 26, 2017 3:30 am
Platform: Windows

Sat May 27, 2017 5:39 pm Post

I have just begun using Scrivener. I have a simple test document that contains a single footnote. When I compile to .mmd, the bold and italic formatting in the footnote is stripped out. I assume this is a bug - because when I compile to docx or any other format, the formatting in the footnote is retained.

Ro
RobertG
Posts: 15
Joined: Fri May 26, 2017 3:30 am
Platform: Windows

Mon May 29, 2017 4:46 pm Post

Just tried out on a Mac - same MMD footnote bug there.

User avatar
nontroppo
Posts: 755
Joined: Mon Mar 05, 2007 5:22 pm
Platform: Mac
Location: Airstrip One

Tue May 30, 2017 7:59 am Post

I'm travelling so no access to my Scrivener to test, but can you give the exact steps; do you mean **physical** markup or RTF?

Ro
RobertG
Posts: 15
Joined: Fri May 26, 2017 3:30 am
Platform: Windows

Tue May 30, 2017 12:36 pm Post

nontroppo wrote:I'm travelling so no access to my Scrivener to test, but can you give the exact steps; do you mean **physical** markup or RTF?


What I mean is this: if I enter italics or bold in a Scrivener footnote (using ctrl-I and ctrl-B), and then compile to any MMD based format, the text formatting in the footnotes is not carried through to the compiled document. That is the case whether one uses inline or inspector footnotes. However, if I compile the same text to a Word document, the formatting is properly preserved, showing that the bug is specifically in the Scrivener to MMD translation.

User avatar
nontroppo
Posts: 755
Joined: Mon Mar 05, 2007 5:22 pm
Platform: Mac
Location: Airstrip One

Tue May 30, 2017 2:09 pm Post

This is not a bug. When you compile to MMD, the inline RTF formatting for bold, italic, superscript etc. is currently stripped out. I use RTF formatting for visual identification, but don't use it for format conversion. There is a one-shot Format > Convert > Bold and Italics to MMD option in the menu (well on the Mac at least), but that indeed doesn't seem to apply to footnotes. I always use *italic* **bold** and ~sub~ or ^superscript^ etc. when working in Scrivener.

I think several people have asked for a better way of dealing with this at compile-time. As far as I've read, Scrivener 3 will have a proper styles system (hence a named style for strong, emphasis would be possible) with greatly extended compile features, so this seems like something we may expect.

Ro
RobertG
Posts: 15
Joined: Fri May 26, 2017 3:30 am
Platform: Windows

Tue May 30, 2017 3:23 pm Post

nontroppo wrote:This is not a bug. When you compile to MMD, the inline RTF formatting for bold, italic, superscript etc. is currently stripped out. I use RTF formatting for visual identification, but don't use it for format conversion. There is a one-shot Format > Convert > Bold and Italics to MMD option in the menu (well on the Mac at least), but that indeed doesn't seem to apply to footnotes. I always use *italic* **bold** and ~sub~ or ^superscript^ etc. when working in Scrivener.

I think several people have asked for a better way of dealing with this at compile-time. As far as I've read, Scrivener 3 will have a proper styles system (hence a named style for strong, emphasis would be possible) with greatly extended compile features, so this seems like something we may expect.


OK - so it just gets stripped out in footnotes and other kinds of annotations? Because it seems to pass through just fine from the main text.

As long as I know that's the expected, stable behavior, I'll adjust my writing to fit it!

BTW: while still on footnotes, I found another bit of odd behavior. When I compile my two-document Scrivener file (now with markdown added in the footnotes) to .mmd, I get:

Code: Select all

**This** is some text. And *this* has a footnote.[^fn1]

This is the second document.[^fn2]
[^fn1]: **This is** the first and *only* footnote.

[^fn2]: This is **the** footnote.


This gets turned into the following TeX by multimarkdown (latest version):

Code: Select all

\textbf{This} is some text. And \emph{this} has a footnote.\footnote{fn1}

This is the second document.\footnote{This is \textbf{the} footnote.}
\footnote{fn1}: \textbf{This is} the first and \emph{only} footnote.


You notice that the footnotes have gotten messed up, because Scrivener does not add a blank line before the footnotes. When I add it manually, and then process with multimarkdown, I get the correct output:

Code: Select all

\textbf{This} is some text. And \emph{this} has a footnote.\footnote{\textbf{This is} the first and \emph{only} footnote.}

This is the second document.\footnote{This is \textbf{the} footnote.}


Is there some setting I need in Scrivener, to ensure that it adds an extra line before the footnotes?

User avatar
nontroppo
Posts: 755
Joined: Mon Mar 05, 2007 5:22 pm
Platform: Mac
Location: Airstrip One

Wed May 31, 2017 12:55 am Post

RobertG wrote:OK - so it just gets stripped out in footnotes and other kinds of annotations? Because it seems to pass through just fine from the main text.


Hm, that I would say is a Windows bug! As far as I know on the Mac, all RTF formatting is stripped out for documents, footnotes and everything else. This is what the manual says:

"When exporting via MultiMarkdown using any of the above methods, all rich text formatting is completely cleaned from your draft, as it passes through a plain-text engine" — 22.4. MMD AND SCRIVENER


So I always use markdown in my main documents *and* footnotes. I do use RTF styling but this is just for visual comfort during writing: viewtopic.php?f=2&t=37303&p=228717#p228717

RobertG wrote:You notice that the footnotes have gotten messed up … Is there some setting I need in Scrivener, to ensure that it adds an extra line before the footnotes?


Actually, this is an important tip for using MMD (and Pandoc, and I suspect the original markdown), and applies to a number of issues, not just this footnote one. You should always finish your Scrivener documents with a clear whitespace {end}. I'm used to pandoc so I always use [space][space][return][return] at the end of every document. This ensures that there are no potential ambiguities as separate documents are combined together. Visualising whitespace in documents for writing markdown in Scrivener is very useful for this:

Screen Shot 2017-05-31 at 08.51.35_SMALL.png
Screen Shot 2017-05-31 at 08.51.35_SMALL.png (31.42 KiB) Viewed 733 times


There should be a way to also force this using Compile > Separators, but I prefer that this is part of the documents themselves...

Ro
RobertG
Posts: 15
Joined: Fri May 26, 2017 3:30 am
Platform: Windows

Wed May 31, 2017 3:49 am Post

I wonder if there is some ambiguity about the term "formatting" here? That section of the manual is specifically talking about tabs and whitespaces (horizontal and vertical) - which are naturally stripped out in the MMD output. But I would be surprised if the intention were to strip out all text formatting. If that were the case, then MMD compile is completely misnamed, and should be called simply "plain text," since it will be MMD without any asterisks to denote font emphasis, or hashtags for section headings etc.

What I find is that, in the text of documents, italics are compiled into *italics*, bold into **bold**; numbered and unnumbered lists into markdown lists, titles into # section headings, and so on. And footnotes are turned into MMD-style footnotes. Surely that is what is to be expected? Otherwise, the rich text in Scrivener could be turned into rich text in Word, but only plain, unadorned text in LaTeX via MMS, etc.

I have not tested it, but I would be very surprised if anything different happened in the Mac version.

As I've said, though, the text formatting in footnotes is not translated into Markdown. And that's fine, so long as I know that that is the intended behavior; I'm already writing my footnotes today with literal Markdown. But the difference in MMD translation between text and footnotes is nowhere specified in the documentation. I think that it would make sense to state that clearly somewhere in the manual.

Thanks for the tip about adding spacing at the end of documents!

User avatar
nontroppo
Posts: 755
Joined: Mon Mar 05, 2007 5:22 pm
Platform: Mac
Location: Airstrip One

Wed May 31, 2017 4:38 am Post

On macOS I am sure RTF bold, italics, sub/superscript and lists are definitely not compiled to MMD markup:

Screen Shot 2017-05-31 at 12.14.23_SMALL.png
Screen Shot 2017-05-31 at 12.14.23_SMALL.png (20.98 KiB) Viewed 724 times


Note that the lists are compiled to text with the tabs preserved, this actually breaks the markdown list as indented text is considered a code block! Markdown only supports unordered and ordered lists, and the fact that the latin numerals and ◆ are exported show that on macOS RTF⇨MD lists are not supported (you can also see ** ** and * * did not make it to the markdown file). Pandoc has very nice list support and it would be difficult to imagine how to translate simple RTF lists to Pandoc. I always use markdown markup for lists, and ignore RTF list tools in Scrivener.

The manual specifically mentions bold and italic as not being compiled: "Additionally, while Bold and italics cannot be compiled, but if you prefer to write using them, they can later be converted to MMD asterisks with the Format  Convert  Bold and Italics to MultiMarkdown Syntax menu command. ." — p.323

MMD compile is not a plain text compile because other changes are made (again from the manual, consistent with some of your "what-you-expect" list):

  1. Any inline or linked images will be converted to MMD syntax, and exported ...
  2. Images that have been linked to, using Scrivener Links, will have the linked text replaced with the image name, and the image itself exported to the compile folder...
  3. Scrivener’s notation features (both inline and linked) can be used to generate MMD footnotes, and optionally its annotations and comments can be inserted into the document as well...
  4. Document titles will be converted to MMD syntax...
  5. Scrivener also supports MMD metadata during compile.

There are a few extra tweaks Scrivener makes too: for example if you resize a placed image, it will put in the MMD syntax for the sizing. So MMD compile does lots of stuff in the background that is not just a plain text compile; BUT the expectation is not to convert as much RTF formatting as possible. Best practice I think is to use MMD/Pandoc markdown except when you know that there is robust support (images, footnotes, metadata[1], titles etc.).

Again, this may change substantially when Scrivener 3 is released.

----
[1] metadata is not robustly supported for Pandoc, which uses --- to denote a YAML metadata block. MMD also supports --- but Scrivener does not use it during compile and thus metadata is broken for Pandoc. For this reason, I use a front matter document with the YAML metadata text for all my work, which compiles fine in Scrivener and works in both MMD and Pandoc...

User avatar
AmberV
Posts: 20612
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Wed May 31, 2017 5:42 am Post

RobertG wrote:OK - so it just gets stripped out in footnotes and other kinds of annotations? Because it seems to pass through just fine from the main text.

That is indeed the Windows bug right there. It shouldn’t be converting any RTF formatting at all except for the things listed in the above discussion, and documented in the manual.

We’ll be adding a more comprehensive system in the future, but it will mostly certainly be something you can control in the compile settings and not enabled by default.

As for the Windows version not properly spacing out footnotes at the bottom of the document, alas that is another bug as well—sorry for these. The basic idea is that if Scrivener is generating the MMD syntax it should be doing so in a manner that doesn’t create syntax errors, as best as possible. In those places where it is up to you—like between paragraphs in a text file—it doesn’t mess with your text, but if it is inserting something like a hashed heading then it should be formatted properly.

Regarding the spacing of documents in general: you could also use the Separators compile option pane to insert empty lines in between everything. That’s what I do, and I’ve never bothered with keeping content spaced out within the editor itself.

nontroppo wrote:For this reason, I use a front matter document with the YAML metadata text for all my work, which compiles fine in Scrivener and works in both MMD and Pandoc…

Have you tried using the “Meta-Data” file approach? If the first file in the compile group is called that, it gets automatically treated “as-is” and spaced properly with any content following. You should be able to put the YAML block in there, but I haven’t tested the idea.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

Ro
RobertG
Posts: 15
Joined: Fri May 26, 2017 3:30 am
Platform: Windows

Wed May 31, 2017 11:08 am Post

After a little more testing, I find that the only visual RTF formatting compiled into MMD is italics and bold. Not underlining/strikeout; and also not tables, lists etc. Footnotes are turned into MMD footnotes (as is documented in the manual).

User avatar
nontroppo
Posts: 755
Joined: Mon Mar 05, 2007 5:22 pm
Platform: Mac
Location: Airstrip One

Wed May 31, 2017 2:34 pm Post

AmberV wrote:Have you tried using the “Meta-Data” file approach?


I used to use this, but I like to have different named metadata documents for different compile groups so I find selecting front matter a more visually logical system.

User avatar
AmberV
Posts: 20612
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Wed May 31, 2017 3:30 pm Post

After a little more testing, I find that the only visual RTF formatting compiled into MMD is italics and bold. Not underlining/strikeout; and also not tables, lists etc. Footnotes are turned into MMD footnotes (as is documented in the manual).

Yeah, it’s not intentional so that’s why it is illogical and doesn’t work everywhere, plus you can’t turn it off! I have some projects where compiling on Windows would turn them into a huge mess because I use italics and bold as aesthetic embellishment or strictly editorial reference, not to intimate some kind of semantic assignment to the text. As I say though, expect more from the idea in the future as an option.

I used to use this, but I like to have different named metadata documents for different compile groups so I find selecting front matter a more visually logical system.

Ah well if you have different preamble sets that’s another matter. I’ve done that as well, but I’ve combined the two concepts for myself—the first file in each nicely named front matter folder is “Meta-Data”. :) The whole concept is less useful if you’re supplying the entire block though. The Meta-Data file is more useful when combining with global settings in compile—and then you’re back to it not being YAML.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

Ro
RobertG
Posts: 15
Joined: Fri May 26, 2017 3:30 am
Platform: Windows

Sat Jun 03, 2017 4:50 pm Post

Thanks, everyone, for helping me understand Scrivener's use of MMD. I came in with what I now see was a mistaken grasp of the MMD compile: that it was intended to be a translation into MMD of the RTF file. Now I see that it is a way of handling files that have been written in MMD from the beginning. I've started applying formatting styles to quotes etc. in my Markdown files, which I find very useful visually -- and can now see why it is important that all formatting is stripped out in the compile. And why the fact that the Windows compiler does translate bold and italic into MMD markup is a bug, not a feature!

User avatar
AmberV
Posts: 20612
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Sun Jun 04, 2017 4:28 pm Post

Yup! That’s the idea: Scrivener is a tool for handling a full MMD workflow, more as though it were a plain-text program. The specific exceptions listed in the user manual are all features that I felt are either burdensome to handle manually and that also happen to be easier for us to generate syntax for:

  1. Images: no need to handle them manually and refer to them with absolute or relative links.
  2. Footnotes: no need to manually come up with anchor names.
  3. Annotations: MMD has no true commenting feature (it uses HTML comments for LaTeX), so these can be exported as styled HTML.
  4. Headings: the compiler can handle all of hash marks for you. No need to change a bunch of hashes if a section gets bumped up or down in the outline.
  5. Preserve formatting = code blocks: no need to tab every line.
Specifically to points 2 and 5, MMD itself has made these easier over the years. You can now use fence quotes instead of tabbing each code line, and footnotes can be expressed inline. [^ Like so.]
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles