Add lots of images, some are distorted on compile

[Michael Shapiro]

I have a Scrivener file with a large number of PNG images throughout. (It's the integrated script/score for a musical, and I've imported score pages as images, one per page.) The resulting file is large, 185mb right now.

On compile, most of the images look fine, but others come out as black blocks. In the scrivener file, all the images look as they should. So it seems that something is going wrong in the compile process. I wonder if it's a memory issue. I tried compiling on two different machines and got the same result.

Can anyone suggest a fix or workaround? Rehearsals are upon me and it'd be nice to be able to print out the entire thing.

[DavidWSnow]

If any of these PNG images have transparent background AND you are compiling for Kindle. (MOBI) you can get what you are discribing. Kindle doesn't support images with transparent backgrounds.

[Michael Shapiro]

No, just standard PDF viewed on a Mac desktop.

[nontroppo]

What if you compile to something like docx, then print to PDF (use LibreOffice if you don't have Word)? Also if you compile to multimarkdown (which exports the images seperately to a folder), and then check the individual PNG in the folder do they also look broken?

[AmberV]

Another thing that might help is sharing one of the images that comes out black. Feel free to contact support if you can’t do so in a public thread like this. If only some of the images are acting this way, I bet it’s a problem with how they were saved initially and simply saving them in a different format and dragging them back in would solve it.

[Michael Shapiro]

Thanks for your thoughts everyone. I tried deleting half of the image files from the Scrivener document, and found that everything exported fine.

I then tried re-adding 26 PNG images to a single text document — this is how I've been adding images so far, several to one scene. (Each image is a page of a score.) When I tried exporting, some of the images that had previously come out okay were distorted. Here's an example of what I'm seeing. This shows the boundary in the resulting PDF file of an image page that came out fine and a distorted one.

ScrivenerProblemOutput1.png (220.49 KiB) Viewed 1553 times

Interestingly, when I tried outputting to DocX, Scrivener got stuck. The Compile window showed "Converting file format..." but the file was never generated.

Further updates: compiling to OpenOffice causes a crash. Compiling to RTF with attachments produces non-distorted images - but of course the formatting is all wrong, so I can't use that.

[AmberV]

I wonder if the compiler is running out of memory. The crashing and hang you experienced when using the docx and odt converters is something we might see if there is too much information to be processed at once. The distortion is a new one to me though, I haven’t seen that happen before.

You did mention RTFD, but have you tried plain RTF? That’s the native format and thus requires the smallest amount of processing in compile—basically Scrivener makes an RTF file and then feeds it to a Java-based converted that makes a .docx or .odt file. You can thus bypass that stage if it is giving you problems by using RTF, and then loading that in something that could then turn it into whatever format you need—LibreOffice is usually good for that.

RTFD is a proprietary Mac format with limited usefulness these days given its formatting problems and non-standard image usage. It's mainly useful for working with other Mac programs (like TextEdit) that don't handle RTF well.

[Michael Shapiro]

I also suspected it was a memory issue, though switching to a machine with 24gb RAM didn't seem to help.

RTF seems to work but doesn't include images. Is that expected behavior?

[AmberV]

I should have mentioned that the memory issues you might encounter will have little to do with how much RAM the machine has, as the amount of memory available to the processing components in the compiler seem limited by some other aspect of the framework—and where it comes to post-compile conversion to .docx and such, there are additional limitations in the Java utility that does this. In my experience the Java converter collapses first, but even pure RTF can collapse eventually (300mb projects for example).

We’re hoping, and initial tests do seem to indicate a positive change, that once the whole program is running in 64-bit space that this kind of problem will be gone. But in the meanwhile, people who run into this have had good results compiling in chunks and then loading those into a word processor and pasting it all back together.

Another approach—one that might suit you very well in fact and what I would myself prefer, is better optimisation. These images seem to be primarily black on white illustrations: consider using 8bit PNG instead of 24bit, with a palette limitation of 16 levels of grey. That will nearly always be enough to keep curves smooth, and the file size will be a tiny fraction of the original.

I have attached an example project demonstrating this. I was basing this off of your screenshot, so I bet when working from the source files it will be even cleaner (these were a little blurry, which adds file complexity to the indexing algorithm), but as you can see there is no perceptual difference between the image using a vast colour storage space vs an image using just sixteen shades of grey (not even fifty!).

It’s worth a test anyway, especially to make sure they work with any other tools you use. You might also consider linking these images instead of fully embedding them. That will make further optimisations easier since you can just batch process a folder full of images instead of extracting, processing and re-embedding into Scrivener.

RTF seems to work but doesn’t include images. Is that expected behavior?

Not unless the RTF is opened in an editor that can’t read RTF images. TextEdit for example will fail to read embedded graphics in an RTF—being limited programs based on the Mac text engine are the main reason you’d need to use RTFD in fact.

[Michael Shapiro]

Thanks Amber. The low-memory theory seems to be substantiated. While I'm limited in my graphic exporting options to what Sibelius can provide, I found that reducing the dpi to 150 made significantly smaller files. Using them seems to circumvent the distortion problem.

I also found that I can export to PDF, split the resulting PDFs into many single page documents, and drag them into a Scrivener page just as I've been doing with graphics. Whereas Scrivener seems to dislike multi-page PDFs, it seems to deal with collections of single-page PDF just fine.

Thanks again for everyone's help!