Help with arithmetic pls

I am compiling from Scrivener 3 to ePub 2. I am looking for some advice on how to reduce the size of the compiled ePub file. Here are some statistics.

Case A: Text + Diagrams
My book contains 83,000 words (540,000 characters including spaces) and 52 diagrams (all in PNG format). When I compile the book, the resulting ePub is 36MB in size.

Case B: Text Only, No Diagrams.
The total size of the 52 PNG files (as shown in Mac Finder - before dragging and dropping them into my Scrivener manuscript) is 12MB. If I remove all 52 diagrams from the book, then compile it, the resulting ePub file is 0.5MB. So the sum of the two is 12MB + 0.5MB = 12.5MB.

I’m trying to understand why the Case A number (36MB) is so much larger than the Case B number (12.5). Any advice how to make the 36MB file smaller? Ideally, I’d like to get it down to 20MB, which is the maximum size for publishing to Nook.

FYI that I tried using JPG files rather than PNG files, but the result was the same. Please see attached image for more info, Thanks.

Without seeing the structure of the final EPub it is hard to know what is going on, but something seems wrong somewhere. You can examine the EPub directly if you rename mybook.epub to mybook.zip you can open it in a zip program and explore the files to see if something got duplicated (BetterZip has a file explorer and is recommended but not free). You can also use Calibre to open the EPub in its editor.

In terms of optimisation, I can strongly recommend a free tool ImageOptim, that employs several different PNG/JPG optimisers. Allowing it to do lossy optimisation of PNG can reduce file sizes by up to 70%, and quality is close to the source. It can batch convert and is very easy to use.

imageoptim.com/mac

I optimise PNGs before I add them to the Binder.

Thank you nontroppo. Your response is very helpful. I downloaded BetterZip, which appears to be free if you just want to examine a file. The attached image is from BetterZip. I see that:

A is my total ePub file size of 35.7MB, which tallies to what I see in Finder.

B is the total file size of my 52 PNG images, 35.5MB. So the problem lies with the PNG files.

C is the diagram I cited in my initial post. I notice that it has a file size of 220KB in Finder, but a size of 631KB in BetterZip, nearly 3 times as large.

Not sure what’s happening here, but the act of dragging-and-dropping my PNG files into Scrivener seems to cause them to increase 3-fold in size. Or the Scrivener Compile process is causing the increase, Or something else is happening.

If you (or anyone) has thoughts on this, pls post to this thread. In the meantime I will try the PNG optimisation tool and see what happens, then will post back.

Thanks again!

Some more info. I used the Image Optimisation tool to shrink my PNG file sizes. The tool has four optimisation settings: Fast, Normal, Extra and Insane. I tested the optimisation on 4 of my diagrams, and experienced a 23% to 27% reduction in file size. See attachment. This is a good result, but will not overcome the 3-fold increase in file size that occurs when I drag-and-drop images into Scrivener, So I guess I still need to find out what’s causing the 3-fold increase.

Screen Shot 2018-04-23 at 14.02.44.png

You should try to enable lossy compression (⌘L in ImageOptim) and see what your figures look like, but it really does provide much greater compression (~70%).

I tried to create a PNG, import it into scrivener, then create an EPub and the file stays the same size, so I can’t reproduce your problem, but perhaps it depends on the particular PNG format (alpha/DPI etc)? I vaguely remember Keith said there was a recent issue with 10.13 that it strips colorspace info, and perhaps it is doing some reencoding of the PNG somehow depending on the format? Can you try to create a PNG in the same way as your figures, then import it and then generate an EPub, if you can confirm the bug, then we can test with the same image?

OK, think I know what is going on possibly. Attached is a test.png — the original PNG was 1MB which compresses down to 125KB with ImageOptim — if I import it into Scrivener it stays the same size (open document bundle, and file is 125KB). But on compile to EPub the file is now 422KB. Checking in Preview and indeed the colourspace has been changed:

If I open the original in Preview and change the DPI (from 72 to 300 which only edits the metadata), I see a size increase, so this is an OS function that is doing this. It is probably adding additional metadata. Not sure if Keith thinks this is intentional, or linked to the bug previously mentioned (unwanted colourspace conversion).

So here is one workaround.

  1. Make sure you use the Compile option “Save source files in a folder with exported EPub”
  2. Run ImageOptim on the images folder first.
  3. Run the make EPub script .

You should now have an optimised EPub.

Thank you nontroppo. I am learning a lot from you! I’ve done some more testing too. I’ll post the results in two pieces. Here is the first piece, which relates to my PNG test file, not to Scrivener.

Attached is the PNG file I am using for my tests. I am attaching it to give you an idea of the complexity of the image. I use a project named Affinity Designer to create my diagrams. All are text-like diagrams, not photographs or complex images. And I only use four colors in the diagrams - white, blue, grey, black. When I export the diagrams from Affinity Designer to PNG image files, I can control the pixel size and DPI setting. I always use 300 DPI (I once read Amazon Kindle guidelines saying that 300 is the recommended size).

My test PNG file is 245KB is size. I played with ImageOptim some more. The results are attached. I don’t see the level of compression that you are able to achieve, perhaps because my images are less complex.

More coming in a moment …

Here is the continuation of the previous post. I created a test SCIRV manuscript, then dragged-and-dropped the 245KB PNG test file into the manuscript. I then compiled the manuscript to create the ePub file. This is with the “Save source files in a folder with exported EPub” unchecked. When I look at the resulting ePub in BetterZip, the PNG file has increased to 811KB (as expected).

I then compiled a second time, now with the “Save source files in a folder with exported ePub” checked. When I look inside the created source files folder and do Get Info on the PNG, I see it also has grown to 811KB. When I look at the ePUB in BetterZip, I see the same thing. It’s possible I’ve done something wrong, of course, but ticking the “Save source files in a folder with exported ePub” box does not seem to retain the original PNG file size. See attached image. I also attached my SCRIV test file.

You wil note I did not use an optimised PNG file in my test. I do understand it would make my ePub file smaller in size. But even if I could obtain a 70% reduction from optimisation, I would still end up with a large ePub file (though not as large as currently) due to the 3-fold increase that is taking place.

I think you might be right there is an underlying macOS function at play here, but I don’t understand colour spaces well enough to understand what it might be. Let’s see if Keith responds in this thread.

And again, I can’t tell you how much I appreciate your help on this.


Test - Image.scriv.zip (372 KB)

Try to lossy compress the version that Scrivener creates. ImageOptim should strip the metadata away and you should be back to the original size at least?

Also I only recommended “Save source files in a folder with exported ePub” as this makes it quicker to see what is happening, no need to open the EPub and you can try to recompress the PNG directly.

Unfortunately there is no way around this. When you import an image into Scrivener’s text editor, that image will become a TIFF representation inside the text (this is just how Apple’s text engine handles images). Scrivener then saves out the data as either JPG or PNG information internally, depending on the format of the original image. But Scrivener never has the original image data available. When you import an image into Scrivener’s text editor, Apple’s code inserts the image as TIFF data and Scrivener only has that to work with. So the compression you apply to your images before inserting them into Scrivener won’t have any effect on what is output from Scrivener - Scrivener can only output standard (by Apple’s standards) PNG files.

I would concentrate on reducing the size and DPI of your images rather than compressing them. 1500x images at 300DPI are quite large for ebooks. The only other option is to hand-edit the .epub file and manually replace the PNG files with your compressed versions.

All the best,
Keith

Are the original images using indexed colour palettes by chance? Even if they are not at the moment, I noticed that they have artefacts that make it look like they were at some point—you can see fields of dots all throughout the graphic, indicative of dithering. When this is used as part of an algorithm to optimise the size of the file with a minimal colour palette that’s one thing—but if it ends up part of the picture itself that is then compressed using another algorithm, all of these little tiny dots represent compression complexity that bloat the file size beyond what seems reasonable for how simple it looks.

I’ve had mixed luck putting indexed PNGs into Scrivener—the text engine doesn’t seem to be aware of that or capable of saving a file using that scheme, so the PNG ends up 24-bit full colour and that naturally bloats the file size a lot. It shouldn’t be bloating this much, I’ll grant, but I can get your diagram file down to around 200kb with a 64 colour palette (and that’s counting the huge mess of baked-in dithering—I bet with the original source files I could get that down to 10 or 20kb), and about 450kb as a full colour PNG, with Photoshop.

But what was interesting is that I tried dropping that 64-colour indexed PNG into Scrivener and then right-clicked and saved it back out—and guess what: it was around 800kb.

As a complete aside: have you considered using code for these diagrams instead of graphics? The circular disc behind the numbers aside (which could be an extremely tiny graphic even with the bloating), the entire design here could be easily done with HTML/CSS and a table. I can’t really think of any advantage to using graphics for these unless you’re targeting an environment that cannot handled rudimentary CSS (and then we wouldn’t be considering ePub 3 most likely in that case).

Keith: Thank you for confirming that this is not a Scrivener issue. Knowing where the problem does not lie helps to concentrate my efforts.

AmberV: Thank you for putting me on to a possible solution. I have a conceptual understanding of dithering, but not a technical understanding. So I did another test.

I exported my test diagram (the same one used earlier in this thread) from Affinity Designer two times: first as PNG-8 Dithered, then again as PNG-24 Undithered. I then dragged-and-dropped the resulting PNG files into Scrivener, then compiled to ePub 2.

The PNG-8 Dithered image was 245KB in size prior to dragging it into Scrivener. Inside the ePub, that same image grew to 811KB. The PNG-24 Undithered image started at 87KB in size, but grew only to 115KB inside the ePUB. So I think you are correct, that dithering might be the culprit.

I viewed the resulting ePub file in iBooks on my iPhone, and in Kindle Previewer for Mac. I really couldn’t tell the difference between the two PNG images, so I think it’s safe to go with the undithered image. Kindle Previewer for Mac has no “night mode”, however, so I wasn’t able to test how the image appears against a dark background. To my eyes, I tend to notice greater differences in night mode.

I guess I now need to rebuild all 52 of my PNG diagrams as undithered PNG images (I have been using PNG-8 Dithered as the default export format), do some more tests and see what happens. I won’t get to that in the near future, but I’ll post back when I do.

FYI to AmberV, The attachment shows the standard export settings (not the custom settings) available to me in Affinity Designer. If you see something that you think is important, pls post on this thread.

Once again, thank you to nontroppo, Keith and AmberV for all your help. This has been a good puzzle - leading me from Scrivener to ePUB to macOS to Affinity Designer - learning a number of new things along the way. Thank you!

I’ve done some more testing on this and have some (sort of) good news. It seems that it’s no longer true that the image data in a text system image is always TIFF data - that used to be the case, but I have a note-to-self in my code relating to some different image manipulation methods, where I used to have to specifically convert to PNG or JPG data but no longer do, that this is no longer the case.

That being the case, it means that the original PNG data should be stored in the text, so I was curious about why it was inflating in file size. What I found was that if you keep an eye on the underlying RTF file inside the .scriv package, it is relatively small when you first add the image and save the project. If you close and reopen the project and then edit the text, however, the file size inflates. This was a clue that opening the project was changing the data somehow.

In fact, opening the project can affect image data in one circumstance: if the image is not of the size it is supposed to be displayed at (because the text system cannot scale images for display only; it needs the underlying image data scaled). In this case, Scrivener resizes the image (maintaining the resolution), which will cause new PNG or JPG data to be generated from the image, using Apple’s standard methods - and thus inflating the image size if the original image was packed small.

However, it seems that at the moment, for some images, Scrivener is always resizing (and thus amending the data of) the image upon open simply because of some rounding errors. For instance, in the case of the image you provided, according to RTF scaling the image should be 481.39999 pt wide, whereas the image is actually 481.44 pt wide. But we hardly care about that fraction of a point - this is just down to rounding differences. So, I have fixed the code to account for rounding errors, so that images will no longer be resized every time they are opened. In the case of your sample image, this means that it now comes out as 225kb in the exported .epub file (don’t ask me why it is now smaller:slight_smile: ).

The bad news is that any images you have already imported will already be inflated, of course, so this fix will only affect images inserted from 3.0.3 on. Also, if you resize an image using Scrivener’s resizing sliders, the file size will inflate because at that point Scrivener has to manipulate the data and can only save using Apple’s standard PNG or JPG saving routines.

All the best,
Keith

Thank you Keith. I appreciate your efforts on this topic. I don’t use the Scrivener resizing sliders, so I’m ok on that count.

I look forward to trying out the rounding error code fix in a future Scrivener update.

FYI that I re-tested the “image size inflation” bug discussed in this thread. I performed the re-test using Scrivener 3.0.3, compiling to ePub 2. Here are the results:

  1. Size of image file in Mac Finder, prior to dragging-and-dropping into Scrivener Editor: 117 KB
  2. Size of image file after dragging-and-dropping into Scrivener Editor (per RTF file size): 235 KB
  3. Closed Scrivener, then re-opened, then checked image size again (per RTF file size): 235 KB
  4. Size of image file in compiled ePub 2 file (in OPS > Images folder): 128 KB

No tripling of image size occurred. Will do some more testing in the coming days, but looks like PROBLEM SOLVED!

Thank you Keith and team!

(If anyone is interested, attached are the test image and test SCRIV file)


Test - Images.scriv.zip (234 KB)