A better way to force sequential ordering of numbered items

User avatar
Jangari
Posts: 127
Joined: Mon Jul 23, 2012 5:03 am
Platform: Mac
Location: Melbourne

Wed Mar 06, 2013 11:48 pm Post

The title of this doesn't really convey what I mean, and there's quite a lengthy desciption of the behaviour, so I apologise for a long post, but essentially, I would like to be able to cross-reference numbered items that are forward in the text without causing the sequential numbering to go haywire.

I'm writing a linguistics masters thesis, and in linguistics we use lots and lots of example sentences to illustrate the phenomena we are describing, but this feature request would equally suit anyone who works with more than a few tables, figures or any sort of numbered item.

To number items sequentially, the tag used is <$n>; each time Scrivener comes to a <$n> tag, it will replace it with a number, in sequential order, from the start of the project to the end. It is also possible to have multiple independent numbering systems by specifying the type, in using the tag <$n:[type]>, i.e., <$n:table>, <$n:figure>, <$n:eg>. If this case, Scrivener will maintain three independent sequential orderings for the three types of numbered items, tables, figures and egs.

Cross referencing is also supported, so that you can link to a particular numbered item anywhere in the text. The way to do that is a third field in the tag, which should be a unique key in that list. So for example, <$n:table:x>, <$n:table:y>, <$n:table:z>. To link to them from somewhere else in the document, you just throw the whole tag into the text. So if x is the first table in a paper, it will get assigned number 1, and then later in the document, throwing the tag <$n:table:x> into the text will result in it being replaced with "1".

Great system. However, Scrivener it seems, cannot distinguish between what is the numbered item, and what is a reference; it doesn't know what is actually pointing to what. As far as it's concerned, it will look down through the document finding each sequential number tag of the same label, and replace it with the sequential number. This mean that if you cross-reference numbered items that haven't actually occurred yet, then you could get some unexpected behaviour. Take this paragraph, for instance:

Blah blah, examples below.

<$n:eg:first> Example sentence
<$n:eg:second> Another example sentence

Blah blah, some text, look specifically at examples <$n:eg:first> and <$n:eg:second> above, and compare them with <$n:eg:third> to <$n:eg:fifth> below.

<$n:eg:third> Yet another,
<$n:eg:fourth> And another,
<$n:eg:fifth> Last one.


Obviously the intention here is to number the example sentences sequentially all the way through, from 1 to 5, and the cross references should refer to them. So the idea is that the middle sentence there should render as
Blah blah, some text, look specifically at examples 1 and 2 above, and compare them with 3 to 5 below.


However, what actually happens is that, after numbering the first and second, and cross referencing to them in the text, the next unique codes that Scrivener gets to are the third then the fifth; the fourth comes later. So it will actually all come out looking like this:

Blah blah, examples below.

1) Example sentence
2) Another example sentence

Blah blah, some text, look specifically at examples 1 and 2 above, and compare them with 3 to 4 below.

3) Yet another,
5) And another,
4) Last one.


So, my feature request would be something to ensure that the actual numbered items are interpreted as such, and the cross references as cross references, rather than treating them as all identical entities and numbering them all sequentially, as it can have this behaviour. My workaround for the time being will be to insert something into the forward-looking cross-reference, and to go through in Word post-compile and manually insert the correct number. But for a document in which someone might be numbering thousands of items and referencing back and forth quite a lot, this would be a messy scenario.

PS. If anyone happens to arrive at this post trying to figure out how to do numbering, there is a bug that you should be aware of. If you are going to cross-reference to numbered items, then you have to have unique keys in all of your tags, otherwise Scrivener will, oddly, start an independent numbering sequence for, say, <$n:table> codes that don't have a unique key. So give them all keys even if you don't intend to cross-reference that item in particular.

Again, apologies for the lengthy post.

User avatar
AmberV
Posts: 22848
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Thu Mar 07, 2013 12:17 am Post

Yup, this is a known limitation of the current system. We might look into something more robust for 3.0. The problem is precisely what you identified: there is nothing to distinguish the pointer from the marker as both use the same code. It’s the first instance of a unique value that increments the counter, meaning references forward will increment the counter prematurely, making only backwards references valid.

So, if one wanted to stick with purely Scrivener for this, they would have to adjust their writing style to avoid forward references—something that may very well not be possible, or use placeholder text as you’ve adopted.

But another problem with these in general is that they just transform into characters when you compile. There is no actual bookmark or cross-ref in the output RTF, which may limit how flexible the document is after compilation.

Another thing to consider is MultiMarkdown -> LibreOffice -> Word. MMD cross-referencing is done using its own syntax, and uses actual marker/bookmark codes and is thus forward and backward reference capable and resilient to post-compilation editing that might shift the order of the markers, just as a word processor would be. A side advantage of this workflow is that the ultimate output result is fully wired up to stylesheets. So the output is quite flexible, whereas vanilla RTF output from Scrivener uses baked in formatting. In short it is ideal for academic and technical work (the user manual PDF for Scrivener is created using MMD, which is how I can forward-ref sections, figures and tables). The main drawback is that one types using a simple marking syntax (if you’ve ever seen Markdown, it’s about the same) rather than the prettier rich text, and some consider that to be a little too “1985”. :)
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

User avatar
Jangari
Posts: 127
Joined: Mon Jul 23, 2012 5:03 am
Platform: Mac
Location: Melbourne

Thu Mar 07, 2013 12:28 am Post

It’s the first instance of a unique value that increments the counter, meaning references forward will increment the counter prematurely, making only backwards references valid.


Wow, you said in a single sentence what took me five paragraphs or so.

MMD -> Libre -> Word, hmm, does it have to go through LibreOffice or can I just use MMD -> RTF? Do you have a link to a howto or somesuch that might explain the workflow?

Thanks!

User avatar
AmberV
Posts: 22848
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Thu Mar 07, 2013 1:11 am Post

Pretty much, as Flat ODT (.fodt) files cannot be opened by most software. LibreOffice and Nisus Writer Pro are the two I'm aware of on the Mac, though if your aim is Word that makes NWP an $80 converter, and NWP cannot handle relatively linked figures, so any work with illustrations will not open correctly. There might be a free online conversion service that can handle .fodt, but since the format is fundamentally plain-text, figures will be compiled as native graphic files alongside of it---much like an HTML page would do, and most online document conversion services are not equipped to handle multi-file scenarios such as these.

As for documentation, there is a chapter on MMD in the Scrivener user manual which strictly covers integration. For MMD usage itself you should consult the MultiMarkdown homeage. The official documentation is not a tutorial however. If you want something that goes over MMD basics in a tutorial fashion, written for Scrivener users, then check out MrGruff's beginner guide on our forums.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

User avatar
Jangari
Posts: 127
Joined: Mon Jul 23, 2012 5:03 am
Platform: Mac
Location: Melbourne

Thu Mar 07, 2013 1:53 am Post

Thanks for the help!

User avatar
KB
Site Admin
Posts: 20688
Joined: Tue Jun 13, 2006 11:23 pm
Platform: Mac
Location: Truro, Cornwall
Contact:

Thu Mar 07, 2013 10:58 am Post

Hmm, I wonder if something like this would work (not implemented, I'm just thinking of how it *could* be implemented):

<$n:ref:eg:third>
<$n:num:eg:third>

That is, two tags, one telling Scrivener it is the number that should be incremented and another telling it that it is a reference to that number. Or, it could just use a different separator between number and tags rather than a separate word:

<$n:eg:third>
<$n^eg:third>

Something like that should, in theory, be possible.

Incidentally:

PS. If anyone happens to arrive at this post trying to figure out how to do numbering, there is a bug that you should be aware of. If you are going to cross-reference to numbered items, then you have to have unique keys in all of your tags, otherwise Scrivener will, oddly, start an independent numbering sequence for, say, <$n:table> codes that don't have a unique key. So give them all keys even if you don't intend to cross-reference that item in particular.


This isn't a bug at all. If you just have a single keyword with no other tag, then it is an entirely different type of numbering stream.

All the best,
Keith
"You can't waltz in here, use my toaster, and start spouting universal truths without qualification."

User avatar
KB
Site Admin
Posts: 20688
Joined: Tue Jun 13, 2006 11:23 pm
Platform: Mac
Location: Truro, Cornwall
Contact:

Thu Mar 07, 2013 11:45 am Post

Actually, having written the above post, it occurred to me that it would take very little code to implement this - all I'd need to do was save the number tags then go through the text looking for the reference tags afterwards. So here's how it works:

Whenever you want the actual number to appear, you carry on as before:

<$n:eg:blah>

When you want to forward-reference a number, use:

<$n#eg:blah>

In fact, it is probably best to use <$n#eg:blah> for all your references to a number, as you know that these tags won't cause the numbering to increment at all. But where forward-references aren't needed, users can still use the old method, so nothing gets broken.

Your example thus becomes:

Code: Select all

Blah blah, examples below.

<$n:eg:first> Example sentence
<$n:eg:second> Another example sentence

Blah blah, some text, look specifically at examples <$n#eg:first> and <$n#eg:second> above, and compare them with <$n#eg:third> to <$n#eg:fifth> below.

<$n:eg:third> Yet another,
<$n:eg:fourth> And another,
<$n:eg:fifth> Last one.


I have uploaded a special build of 2.4 that incorporates this code here:

http://www.literatureandlatte.com/dlbet ... efTest.zip

I'd be grateful if you could test it and let me know if it works as expected.

All the best,
Keith
"You can't waltz in here, use my toaster, and start spouting universal truths without qualification."

User avatar
Jangari
Posts: 127
Joined: Mon Jul 23, 2012 5:03 am
Platform: Mac
Location: Melbourne

Sat Mar 09, 2013 3:46 am Post

KB wrote:I have uploaded a special build of 2.4 that incorporates this code here:

http://www.literatureandlatte.com/dlbet ... efTest.zip

I'd be grateful if you could test it and let me know if it works as expected.


Far out, you guys are quick! I just opened scriv a few minutes ago to find a new version was available, including this in the list of changes:

Forward references are now supported in autonumbering, using e.g. <$n#keyword:tag> to refer to numbers generated using <$n:keyword:tag>


Thanks! Testing now.

User avatar
Jangari
Posts: 127
Joined: Mon Jul 23, 2012 5:03 am
Platform: Mac
Location: Melbourne

Sat Mar 09, 2013 4:00 am Post

Works beautifully.

Only noticed now that you did this back on Thursday; sorry for not seeing the forum messages!

User avatar
AmberV
Posts: 22848
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Sat Mar 09, 2013 4:13 am Post

:)
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

Cr
Crosbie
Posts: 2
Joined: Sat Mar 09, 2013 10:50 am
Platform: Mac

Sat Mar 09, 2013 10:59 am Post

Wow. That is fantastic.

I hit this problem when numbering images and wanting to forward-refer. My workaround was the kludgy option of inserting a white-on-white reference list in an earlier page to get the numbering to happen correctly, and I figured that it would be fixed by a 'list of illustrations' at the head of the document ultimately.

But this solution is elegant, simple and works beautifully; I've only had to change that one-off symbol for an exceptional advance reference, which now is numbered as intended. Is this method now a keeper?

Thanks for this eye-opening, mind-expanding app. :)

What are the options for autonumbering -- or checking autonumbering -- without having to compile?

ma
matsgz
Posts: 312
Joined: Mon Feb 11, 2013 7:56 am
Platform: Mac + iOS

Sat Mar 09, 2013 1:31 pm Post

Wow!

I'm not using advance referencing, I'm just procrastinating, cruising the forum instead of thrashing out scenes and dialogue, as I should. But wow!

Heres a guy presenting a very specific problem, and wham! the support team of Scrivener, who are also the creators, rewrites the code and have an update out the same afternoon.

There is no software company that can compete – imagining telling MS that there is something you want to do that you can't do in Word, and before the sun sets a new version of Word is ready for download.

Just fucking great. I love this program!

Cheers,
Mats
Scribo ergo sum

User avatar
AmberV
Posts: 22848
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Sat Mar 09, 2013 10:15 pm Post

Crosbie wrote:What are the options for autonumbering -- or checking autonumbering -- without having to compile?


There are none, but I might not be understanding why you would want to do this without compiling, in the first place. Wouldn't the process of checking numbers be directly integrated with needing all of your work together in one document? Or to phrase it another way, wouldn't the portion of the proofing process, where checking numbers is important, be something you'd need the final look of the document prepared for?
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

Cr
Crosbie
Posts: 2
Joined: Sat Mar 09, 2013 10:50 am
Platform: Mac

Mon Mar 11, 2013 6:46 pm Post

AmberV wrote:
Crosbie wrote:What are the options for autonumbering -- or checking autonumbering -- without having to compile?


There are none, but I might not be understanding why you would want to do this without compiling, in the first place. Wouldn't the process of checking numbers be directly integrated with needing all of your work together in one document? Or to phrase it another way, wouldn't the portion of the proofing process, where checking numbers is important, be something you'd need the final look of the document prepared for?


A couple of clarifications: Firstly, I found I had to do a lot of checking and rechecking when tackling this misnumbering problem. I wondered whether there could be an easier way to check that numbering was working as expected. But perhaps you're right, and this is so dependent on compile settings that it just has to do the lot. Seems wasteful when all you really want is to run the numeric substitutions as a one-off...

Secondly, It'd be useful to force a numbering run on certain document headings or titles. As with the autonumbering of cards in corkboard view, it'd be useful to be able to choose this in other views.

I've been experiemnting with using Scrivener for course design -- as have others -- and being able to get numbering generated in the titles of a given folder, say, would make it a lot easier to re-order things whilst keeping the weeks or lessons tidily numbered. Perhaps this is possible already...?

User avatar
AmberV
Posts: 22848
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Santiago de Compostela, Galiza
Contact:

Mon Mar 11, 2013 7:21 pm Post

Seems wasteful when all you really want is to run the numeric substitutions as a one-off…


I’m an obsessive re-compiler, perhaps it is something that comes with a growing familiarity with the software, but compiling isn’t expensive, and doesn’t waste bytes. You might as well do it as many times as you need to get a particular job done. I’ve even set up a macro using Keyboard Maestro to compile and overwrite the previous output file without any further input from me. I just hit Shift-Opt-Cmd-E and a few moments later I have an updated PDF/TeX/etc file.

But, it would take nearly as much processing to preview token substitutions anyway. Remember that placeholders can appear in the compile settings themselves (into section header prefix or suffix fields are the most common, but Replacements would be a second common location, and more rarely in Separators). While it is true that most of the counter types being used in the compile settings would probably not be manually typed into the draft text as well, this cannot be assumed. Then you’ve got all of the myriad ways in which a document can be included or excluded from the final content, which if there are counter tokens within these excluded documents, would impact the number of the other documents around them.

I’ve been experiemnting with using Scrivener for course design – as have others – and being able to get numbering generated in the titles of a given folder, say, would make it a lot easier to re-order things whilst keeping the weeks or lessons tidily numbered.


Have a look at the `View/Corkboard Options/Show Card Numbers` option. We would like to look into broadening this to the Outliner as well—but again the whole problem of compile exclusions makes this problem particularly difficult. There are certainly straightforward compile settings that would produce predictable results in the binder, but what about cases where you are filtering items out of the compile via the assignment of a keyword? Should that item be numbered? Probably not, but using the CPU to constantly do all of this background calculation would drain batteries and just generally be an uncool citizen on the computer.

I do better understand what you mean now, thanks for the clarification. Hopefully this response better explains some of the reasoning behind the limitations. It’s just a side-effect of working in a variable state multi-file outliner system.

I guess the moral of the story is: don’t be shy about compiling. It doesn’t hurt anything to compile, and you keep a fresh copy of the output as a reference as a result.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles