Tuesday, December 23, 2014

Kindle Publishing Guidelines 2014.3

The latest update to the Kindle Publishing Guidelines is out, with a lengthy change log that includes a few important changes. While consisting primarily of editorial revisions to specify which Kindle device a given formatting recommendation applies to, there are two specific changes to Kindle content production that are significant, these being sections 4.3.7 and 4.3.9 as given in the Revision Notes shown above.

4.3.7 Recommendation #7: Do Not Include an HTML Front Cover

The first of these reverses a long-standing error in Amazon's Kindle production policy. Until now this section heading read Include an HTML Front Cover, while it now emphatically says not to (as I have long advocated). Including one has always resulted in two cover images appearing, causing reader confusion due to the apparent unresponsiveness of the first page turn. Moreover, there has never been a good reason to include an HTML cover page, since all Kindle devices and apps render the jpeg cover image correctly, as well as using it for the bookshelf image.

As the Kindle Publishing Guidelines itself now states:
While Amazon previously recommended an HTML front cover page for fixed-format books, this is no longer necessary.
Kindle books should only have one visible JPEG cover. This cover should be a high-resolution JPEG image that has the same level of quality as the subsequent pages. Any instances of HTML cover pages should be deleted to avoid a repetition of the cover image.
Presumably the HTML cover page was originally included in order for the Guide to point to the cover as the first page the reader sees, if this was so desired (since it could not point directly to an image at that point, as it now can). But since the implementation has been inconsistent, opening to the jpeg image in many cases, as well as the reader being able to swipe back to the cover image anyway, this has caused confusion and frustration for many readers.

The recommendation to include a high resolution cover image is now all the more important, since it will be rendered as a full page image on first opening.

4.3.9 Recommendation #9: Do Not Include Start Reading Location

This is an entirely new addition which addresses an ongoing issue with the first page to which a Kindle ebook opens on first reading. This has been erratic and seemingly random, since the same ebook would open to different pages on different Kindle iterations, depending on a number of confusing factors (including, among other things, the appearance or absence of a "toc" entry in the Guide, as detailed on pages 74-5 of my Kindle formatting manual).

While Amazon has repaired several of these errant instances, some have continued to persist (such as the perplexing inability of the "Go to Beginning" entry to open at Page 1). This issue is now eliminated with the newest Guideline recommendation:
In Kindle fixed-format books, the OPF file should not include the start reading location (”Go to Beginning”) guide item. Amazon now sets this guide item to the JPEG cover for Kindle fixed-format books.
All fixed layout Kindle ebooks will now open to the cover image (which should now only be a jpeg image), rather than to the title page, the table of contents, the first page after the table of contents, or any other random location in the publication. This provides for a consistent user experience across devices and platforms, as should have been the case all along.

Note that Section 3.5.1 on "Recommended Guide Items" still lists the "Go To Beginning" entry as a valid item. This is likely an editorial oversight, but since the removal of the start reading Guide item is only a recommendation, the entry is technically still valid, though not advised.

Note also that you can still include a "bodymatter" element for the start reading location in the Landmarks section of a nav doc, though this will not affect the page to which the book first opens, but only create a linked menu entry to the chosen page. Using KindleGen

Among the other changes made in this edition is the inclusion of Dutch as an optional language in KindleGen (using the locale option nl during conversion).

3.6.3 Image Guideline #3: Use Color Images

Removed the line describing the difference between eInk and color tablet devices, and added a statement that photographs must be in the jpeg format. Specifically, the line removed stated that:
The Kindle e Ink devices currently have a black and white screen, but color is available on the Kindle Fire, Kindle for iPhone, and Kindle for PC.
This is curious, since the removal of the reference to a greyscale display hints at the coming of a color eInk screen. Although this has long been awaited, to date there is no evidence that a reflective color display is forthcoming, and we have already missed this year's pre-holiday release window, so presumably it won't appear for at least another year.

The appended statement that photos must be formatted as jpegs is followed by further image clarifications in the next section.

3.6.5 Image Guideline #5: Use GIF or PNG for Line-Art and Text

This entry has been extensively revised to make it adamantly clear that line-art and text images should not be formatted as jpegs, which blur sharp edges when adding compression, but should instead be embedded as either gifs or png files, which preserve crisp edges - and even enhance them, in the case of gifs, by reducing color values and grayscale contrast (as the added line "including black-and-white drawings" makes clear).

One line has been removed here that bears comment:
The automatic conversions applied by KindleGen are best avoided.
This statement was always inaccurate in this context, since all supported image formats are converted to jpegs during the KindleGen conversion, and therefore cannot be avoided. What was meant instead was that the cleanest possible image should be input in order for KindleGen to apply the best conversion possible. If an already compressed jpeg is embedded, its quality will only get further reduced by the additional compression applied by KindleGen when converting to the low Mobi 7 image standards.

This requirement is emphatically repeatedly in this section several times, culminating in the conclusion that
Amazon insists on GIF or PNG file formats for line-art.
Finally, a lengthy section has been added to the description of the MINIMUM size requirements for lines of text contained in images (which is 6 pixels for a lowercase "a"), including the addition of a new example image:
The added text states that the image should be taller than the 6 pixel text itself, such that
the image should be at least 45 pixels in height so that it displays proportional to surrounding text content.
This assumes, of course, that the surrounding text content has not been modified by the user's font size and line height settings, but the intention is clear: make your text images big enough that they can be clearly seen, and leave some margin space around them so they preserve the line height and don't encroach upon surrounding text.

6 Audio and Video Guidelines

Several additions have been included to reiterate the fact that, no, audio and video is not supported on eInk devices, and KDP does not accept Kindle Editions with audio/video content included. Still.

This is simply a shortcoming of the slow refresh rates of eInk screens (and one of the primary reasons reflective color displays have not been adopted), although why audio is forbidden is beyond me. Probably for the same reason Amazon stopped bothering to include speakers, or even a headphone jack, on the eInk devices. Or perhaps because of it.

9 Kindle Best Practices

Here again some lines have been removed, and the terms "e Ink" and "tablet" inserted to more clearly differentiate between the two. There is also a revision of the formats supported for viewing on devices and in Previewer, via the removal of two lines.
The Kindle Fire device view displays the content in Kindle Format 8.
This applies to Previewer, and was probably removed due to the fact that it is no longer relevant, since virtually all Kindle devices now support KF8, and Previewer displays content as such.
You can test Mobi 7 content on a Kindle e Ink device and on Kindle applications for PC/Mac/Android.
This line was made more or less redundant by the fact that Mobi 7 content is now only sent to the oldest of the old Kindle devices, and is very likely soon to be eliminated. Moreover, there is no way to actually chose which version of the converted file is being tested, since the device reads whichever one it has support for. Again, virtually all Kindle devices now read KF8, so the distinction is no longer relevant.

The subsequent line that stated KF8 could only be tested on the Kindle Fire has also been appended to include the eInk devices, but interestingly not the apps. This implies that you can no longer use (or rely on using) the desktop and mobile apps for testing files, which is sound advice, since they are the least consistent in rendering content correctly, or supporting features. Best Practice is, and has always been, to test on an actual Kindle device, followed by Previewer, and only as a last resort to use a desktop or mobile app (although the Android app is better than the other apps by far, as far as feature support goes).

All in all, these updates pave the way for future moves away from the outdated Mobi 7 format and into feature-rich KF8 support across the board (more or less), and address some outstanding issues with the format itself in order to make the reading experience more consistent.

Saturday, December 13, 2014

iBooks Image Size Increased

Although they have not yet updated their iBooks Asset Guide to reflect the change, Apple has just announced an increase to the total pixel limit allowed for interior images in ebook files published through iTunes Connect, upgrading it from 3.2 to 4 million pixels.

This follows the increase in August of last year from 2 to 3.2 million pixels, and at long last brings the overall image size allowed more in line with the actual size of the device on which they are intended to be viewed.

The increase to 3.2 million from 2 million pixels was itself, of course, a long overdue attempt to address a contradiction in Apple's recommended image size as given in the iBooks Asset Guide. With a resolution of 2048 x 1536 since the 3rd generation iteration of the full-size iPad (giving it a total pixel count of 3,145,728), the long-standing limit of 2 million pixels was nowhere near enough to fill the screen with a full bleed image in portrait orientation, let alone to zoom in for greater detail, as the Asset Guide somewhat ironically recommends:
Apple recommends providing images that are at least 1.5 times the intended viewing size up to a maximum of 3.2 million pixels.
A quick bit of math will show that 1.5 times the standard iPad display resolution is 4,718,592 pixels. So while the new 4 million pixel limit does not provide for quite that much leeway, it gets us very nearly there, and at least allows for full bleed images that can be zoomed to some degree without completely losing fidelity. The official announcement for the newest update reads as follows:
Increased Pixel Limit for Book Images
The pixel limit for all images within a book has been increased to 4 million pixels. This limit does not apply to images delivered separately from the book, such as cover art or screenshots.
Amazon, of course, has also recently increased the allowed image size in Kindle ebooks, although they use an overall image file size in megabytes rather than a pixel dimension limit. This allows for a much higher image resolution to fill the bigger 2560 x 1600 HDX 8.9 device, which boasts over 4 million pixels, before zooming (4,096,000 to be precise). By comparison, that would require an image containing 6,144,000 pixels in order to zoom to 1.5 times without interpolation! This is why the individual image limit in Kindle files has been increased to 5 MB. Converted mobi files for upload to KDP, however, are limited to an overall file size of 650 MB, which puts something of a practical limitation on the included image files (130 5Mb files to be precise, which is admittedly more than ample for most projects).

Conversely, by employing a pixel limit, Apple are restricting the total image dimensions, but not the image quality itself, which can be embedded with no compression whatsoever, if so desired, up to a total ebook file size limit of 2 GB (which is a limit imposed by the zip format itself). So while the new 4 million pixel limit does not quite allow for full 1.5 times zooming with pixel-perfect accuracy, it does come very close, and the absence of a file size limit for individual images allows for the highest quality setting possible, which might be of great benefit for art books and photography collections (in case you want to see those brush strokes on the Mona Lisa, for example).

By the way, for those wanting to know, 2309 x 1732 would be the largest dimension image you can now insert into an iBooks file at the standard 4:3 aspect ratio of the iPad display. That gives you an image containing 3,999,188 pixels. Just one pixel bigger at 2310 x 1733 puts you over at 4,003,230 pixels. A nice round numbered 2300 x 1725 gives you an image with 3,967,500 pixels.

Saturday, October 11, 2014

Why I Don't Use eBook Creation Apps

Many years ago I built my first website, using Dreamweaver. It was Version 8, I believe, back when it was still by Macromedia. The website was very basic, the program very complex. It suited my needs at the time, and had plenty of potential for expansion... if I wanted to learn how to code a complex website, that is. I didn't.

So instead, when it came time to add a store with shopping options for buying print and ebooks to my site I switched to Yahoo SiteBuilder, since Yahoo were the top web hosting service at the time, and offered a simple add-on shopping cart package (for a fee, of course). But since Yahoo used their own proprietary code for web page layout and functionality, I had to rebuild the site from scratch, being unable to import the current site and build on that. However, after much tedious effort (not to mention learning an entirely new software interface) it resulted in a nice website that filled my current needs...for a while.

But web technology moved on and Yahoo did not. Eventually I wanted to upgrade my site to include some of the nifty new features made available by HTML5 and CSS3. But that could not be done in Yahoo. So I moved my site to another 3rd party web platform...and built it all again from scratch, since Yahoo's super secret code matrix could not be transferred to another platform.

That new 3rd party service went out of business within a year (Yahoo has fared only slightly better treading water). So once again I started over and rebuilt the site from scratch (that's four for those keeping track). This time I decided to use the popular "open" platform Wordpress, with its highly customizable theme-based structure, thinking it would be more "universal," since there were seemingly endless theme and expansion packages available for it.

But Wordpress proved to be slow and balky, crashing frequently and losing data with nearly every update. And because all but the basic layout functions are handled by adding third party plug-ins for each new feature, these tended to conflict with one other more often than not (not being tested for compatibility with anything but the base platform), causing features or entire sections of the site to malfunction (or not function at all), and crashing the site when any one of them were updated, even locking it up in a perpetual feedback loop on more than one occasion - a cycle which could only be broken by deleting one or more of the expansions, which naturally took all of my data with it.

Moreover, due to the very nature of this "plug-in" methodology, the site was once again not exportable to any other platform. So yet again I was faced with the seemingly inevitable task of rebuilding my site from scratch.

So after building the same site five times, I am now back to where I started, using Dreamweaver to build all my site content with universally recognized web code based on standard HTML and CSS. It is infinitely expandable, limited only by my time and willingness to learn, and can be transferred from one hosting service to another as I see fit. And it's the best site I've built yet. More importantly, the underlying code can be read and edited using any standard text editor. I can fix it if it breaks, and add new features as I learn to implement new code. If I see something I like on another website I can view the source code in my browser and see how it was done. I'm not limited by what the software can do, but what I can do.

You might be asking yourself at this point what this has to do with ebooks. An ebook is, in essence, simply a portable website, designed as fixed or responsive page layouts, and based on a subset of the very same HTML and CSS that websites use. In general, if you know how to make a basic web page you can make an ebook too. You just need to learn the specific bits of code that makes the ebook work, and the rest is left to your imagination. And best of all, the only thing you really need is a text editor.

The very same issues that plagued my web building experience for so many years also apply to building ebooks, so take heed. Programs that build your ebook for you do so by using their own proprietary code that often can't be understood by mere mortals such as we (unless you have the patience of a saint, or the wisdom of a god, which I most definitely do not). For example, they tend to change the names of all your input source files, so that rather than having page123overlay2.jpg, you might find img000172.jpg instead, in your new HTML page called split0000159.xhtml. Good luck finding the correct file for page 123. Or the images it contains, if you happen to want to change one.

This is made all the worse by the fact that all of your carefully labelled CSS entries will have been changed in just the same manner, so that your styling instructions for #page23panel1 is now called data-app-amzn-ke-created-style0126, or some such nonsense. And more than likely all of your neatly organized HTML will be run together in an endless stream of code. Needless to say, this is not helpful in the least.

Unless you're a machine. And a very specific machine at that.

This applies not just to Amazon's proprietary Comic and Kids ebook creator tools, but also in varying degrees to iBooks Author, InDesign's ebook export function, and ebook editors such as Calibre and Sigil. The code they create can only be effectively read by them, thus locking you into using that same platform for all future updates to that file, until such time as their usefulness runs out, they become incompatible with your now-outdated file (or operating system) after an update, another software offers better features, or the company goes out of business.

Then you'll face the same dilemma I did building websites. You'll have to build your ebooks all over. From scratch. Again.

For more in-depth reviews of Amazon's Kindle Comic Creator and the newer Kindle Kids Book Creator (for those still intent on using them), follow those links to my posts on the subject. But don't say I didn't warn you.

Wednesday, September 3, 2014

Kindle Kids' Book Creator - An Analysis

Amazon today released something of a companion to their Kindle Comic Creator application that, like its predecessor, allows for relatively easy graphic layout of Kindle illustrated ebooks, this time geared toward producing children's content rather than comics. Like Kindle Comic Creator (KCC), the new Kindle Kids' Book Creator (or KBC for short) presents a very stripped down interface with a rather limited array of tools. However, this is somewhat deceptive, since users also have access (in both programs) to direct HTML/CSS editing within the application, which makes it useful both for those with some coding skills and those who only want a basic drag-and-drop style interface. For those without, however, the options for styling are somewhat slim, though they should suffice for most cases.

A companion User's Guide is available from the KKBC home page, which details in the usual sketchy technical outlines how to use the program, so I won't go through that here, but suffice it to say that the user guide isn't really necessary, since there's very little that you'll need to learn to use this thing, and most of it is fairly self-explanatory from a cursory glance at the menu bars and buttons; a half dozen random clicks will teach you all you need to know in fifteen minutes.

I ran a quick and dirty test to see how the program fared, and for the most part I was pleasantly surprised, although not without a healthy handful of caveats. Foremost of these right from the get-go is the lack of support for epub as an input source: KBC only accepts PDFs and image files, which means that if you have a nice layout to export from InDesign your only option is to make a PDF and start over from there. You can import pages with text embedded and create text pop-ups to cover them up, or import image-only pages and create both the base text and magnified text from scratch, and then move it around and size it all to fit. If you cut and paste text from another document any styles applied in the original will not transfer, since the underlying CSS will not be copied as well, but any tags you've added to the code still will be in the HTML, so you can manually add the CSS rules to the KBC file easily enough. This, of course, implies the need to delve into the underlying code, which somewhat defeats the point of using a program such as this. But that's the last we'll see of that necessity if you actually want to add some style to your work.

Unfortunately, even though you can add text as a separate layer in KBC, the "childrens" book-type that is automatically added to the output OPF renders the text inactive in the Kindle readers, so that dictionaries, highlights and word search will not work, rendering it essentially the same as text embedded in the image. This is why I discourage the use of a book-type value in fixed layout files as a general rule, but that's a choice each content creator must make for themselves. You can, however, delete the book-type entry from the OPF manually and all the text will then be active. Why Amazon has crippled this ability in children's books and comics is beyond me.

If you create the base text from scratch KBC will automatically add it to the pop-up at 150% magnification (if you add a pop-up, that is, since you don't have to), but you can "unlink" the pop-up from the base text and add any other content to the magnified region that you want. It will still be magnified at 150% of the base text, however, so if you simply want to have the base text change you'll have to learn to do some custom coding in the HTML/CSS editors to make it work (for example, if you want to produce a bi-lingual ebook, or use the pop-up like a question/answer flashcard). Unfortunately, KBC's behind-the-scenes code is a little balky, so unless you already know your way around a KF8 fixed layout file archive pretty well you'll be looking at a lot of gibberish, and even for those of us who do it's pretty messy in there. Also, to create clever tricks like using mag regions to replace one image with another isn't something that can easily by done in KBC. If you're wanting to add that kind of interactivity into your project, KBC is not the tool for you.

Other detracting factors against using KBC for anything but very simple projects are its lack of an Undo button (!), so you better be sure you want to make a change before you do, because there's no way to get it back once it is gone. You can "re-link" a base text box to a pop-up, but this deletes any other custom changes that you've made to the pop-up, replacing them with the default text, and that change cannot be undone, so whatever you had created will be lost. This is a horrendous oversight on the programmers' part, and truly unforgivable in this day and age.

Even though you import images of a given size (that you have no doubt chosen for a reason), KBC will automatically re-size them to a seemingly random dimension of its own choosing, over which you have no control. In my sample test file, for example, all my images were 1118 by 1788 in size. This was chosen as the largest size for which I could achieve the highest quality setting in Photoshop and still come in at just under the "standard resolution" image file size limits the Kindle has in place for older devices (which KBC apparently does not support - only the Fire devices are listed as supported, and available for previewing). At any rate, KBC arbitrarily assigned a landscape layout page size of 1920 wide by 1535 high for my test project, making each half page image only 959 by 1535 rather than 1118 by 1788.

Curiously, however, the auto-generated "scaled-images" folder contained both the original, unchanged images, as well as "thumbnail" versions scaled to 625x1000 pixels and compressed to down around 100 kb in size, rendering them utterly atrocious in quality. Which ones the reader will see is anyone's guess, but since KBC files are apparently not compatible with the eInk devices (even though those devices do support fixed layout KF8 files) my guess is that the larger ones will be delivered to the Fire devices and the smaller ones to Android and iOS apps for phones. But I'm only guessing there, so don't quote me.

You can add single pages in portrait or landscape orientation, or two-page spreads in landscape using either two image stitched together or one image spanned across both pages. This is very nice feature made very easy during the image import stage. However, auto-orientation is unfortunately not an option here; you must choose one or the other. I should also mention that you can add or create additional pages at any point in the process, and insert them anywhere in the page sequence, and even re-sequence pages already added simply by dragging them to a new location. You can also delete pages, but once you do they are gone for good along with any content you included in them, and again this cannot be undone.

You cannot open mobi files that were not created using KBC, even if they're KF8 format children's book-type files. So KBC can import a PDF but not it's own native KF8 format. Go figure.

Above is a snapshot of the formatting toolbar available in KBC, and as you can see the options are somewhat limited. The "Add Pop-Up" button creates a mag region with no underlying text, while the "Add Text option creates both at one go. Just add your text to the base layer text box and it is automagically magnified at 150% in the pop-up box, both of which can be moved around and re-sized to your liking. You can embed custom fonts easily enough using an Add Font menu option, with any added fonts appearing in the drop down menu here. You can also change their size and color very easily here, which is a plus. There is also the slightly unexpected line height and character spacing options to help you make your text fill up the page just right, but be aware that these apply to all text in a single given text box and no others, so if you want all the text throughout the book to look the same you'll once again need to dive into the code and create a custom CSS rule for your base text. It's also not possible, of course, to only change the character spacing for just one line without embedding that line in a separate text box, thereby breaking any pop-up windows into pieces also.

Other than that you have the basic bold, italic, underline, and four standard text alignment options to choose from, and nothing else. There is no way to wrap text around images as in iBooks Author, or even to automatically align text boxes to a standard margin without resorting to advanced CSS editing. Text boxes can be moved and re-sized easily enough by dragging on their corners or edges, but there is no ruler or grid system by which to align them. Also, unless I'm blind I'm finding no color picker to alter the background color in the magnified text boxes, so again you'll need to learn some CSS to use this tool that Amazon is hyping in their promos as being designed for those who don't know any HTML or CSS, and theoretically don't need to learn. In other words, KBC is fairly dismal as a graphic design tool, and if you're used to using Adobe products or iBA for your layout work you'll feel as if you've been drugged and had your arms chopped off while trying to produce something genuinely inventive here.

There is a Console panel available that shows the log file which Kindlegen produces during conversion, but if you're advanced enough to understand that you don't need to be using this kid's toy. Honestly, I was hoping for an adult app at some point from Amazon that can create advanced layouts and complex motion graphics, but all they've given us so far are simple widgets only good for making very basic layouts. And that's probably good enough for beginners, but it's not something professional ebook formatters will ever use.

So to summarize:

  • Easy user interface for a quick and simple learning curve
  • Easily create magnified text pop-ups from base text
  • Text layers isolated from background art (for potential live text upgrades)
  • Editable CSS/HTML code from within the program
  • Two-page spreads with facing pages made super simple
  • PDF / image input for really simple import most users can manage
  • Generates the necessary file structure, including OPF & NCX, which can be edited externally
  • No epub or mobi file input
  • Cannot edit mobi files not originally created with KBC
  • Very limited layout and formatting tools
  • Only basic text interactivity (pop-ups) without advanced HTML/CSS coding skills
  • No support for eInk devices (exports only to "Fire-compatible" KF8 according to the guide)
  • Auto-Orientation not available (locked to landscape or portrait)
  • Auto-resizes images without your input (cannot select your own "original-resolution" value)
  • No Undo!
  • Previews only available for Kindle Fire devices currently
  • Did I say No Undo?

A few notes for the more advanced users:

There are a couple of new "ke-" prefixed entries in the metadata section of the OPF that are interesting:
<meta content="doublepagespread" name="ke-layout-type"/>
<meta content="start-right" name="ke-start-side"/>
<meta content="true" name="ke-hd-images"/>
The first is what allows two pages to be knit together into one page spread or a single image to be spanned across two pages, but I haven't had the chance to dive into the generated CSS to see what's being done just yet to know what all the options are, or why this is necessary when the KF8 code already has the ability to create facing pages from two images. In this case it's a two-page spread consisting of two individual images, so "doublepagespread" is the value. The value is for a two-page spread made from a single image is, not surprisingly, "singlepage".

Second up we have the "start-side" which for the first time allows a Kindle fixed layout file to have a single standing first page on the right side (or presumably left in rl-horizontal writing mode ebooks). Previously this was attempted with the page-id "layout-blank" value, which never worked correctly, and single pages always appeared centered in the viewport rather than to one side or the other.

Finally, we have an "hd-images" entity that I presume is part of the new "standard resolution" versus "high resolution" image distribution for old model versus HD devices. Here the value "true" tells me that only the HD images are going to be transferred to whatever device the file is destined for, regardless of the model. Or perhaps it means that only HD devices are supported for this file, as implied by the absence of any mention of eInk screens in the documentation. But if "false" is also a legitimate value for this entity it begs the question of just what the heck is going on here. Of course, each of these is generated automatically by KBC, so you really aren't intended to alter them outside of the settings found in the program menus, and there isn't one for this third entry. But here they are for what it's worth.

Incidentally, KBC only produces a toc.ncx rather than a nav doc with landmarks, and the OPF contains none of the EPUB3 additions that had previously been implemented in KF8.

Well, that's all I've got for now after a quick first look. I'll ponder on it more and maybe play with it a bit tomorrow. But I doubt I'll ever use it much, except for previewing my HTML/CSS code on the fly. And honestly, that's the best use I can see right now for this thing.

Sunday, May 25, 2014

Kindlegen "-dont_append_source" Option

I discovered this week that there is a command line option for Kindlegen that keeps the source file zip from being added to the compiled mobi archive. It was hidden in the Kindle Publishing Guidelines under the "Building Dictionaries" section (7.5), appearing first in Version 2014.1 back in January, and was not specifically listed in the Revision History, where it simply said "Dictionary Overview (and all subsections of same)." I went through that section, noting all the changes but this one. Go figure.

At any rate, if you add this option to your kindlegen command it will bypass the addition of the source file archive to your output file:
Kindlegen will output this message as the first line in the conversion log:
Info:I9018:option: -donotaddsource: Source files will not be added
I tested a fixed layout file to be sure it worked, and what I found was that a 19.6 Mb source epub which normally converts to a 44.9 Mb file, resulted in a 25.2 Mb mobi when appending this conversion option. Extracting the contents using KindleUnpack showed that the source zip file was indeed absent.

So for those who have been using KindleStrip to remove the bulky source files from your files, you can now just add this option during conversion instead. KindleStrip is no longer necessary.

Bear in mind, however, that with the larger image file size allowance, if you're including higher resolution images for HD devices that are over the standard definition limits (i.e. 127/256/800 kb), these source files may be required by Amazon in order to send them to the end user's HD device.

Saturday, May 17, 2014

Kindle Image File Size Increase Confirmed

After a lengthy wait, I finally received confirmation from Amazon this week regarding the increase in allowed image files sizes for Kindle ebooks listed in the newest version of the Kindle Publishing Guidelines, as discussed in recent posts.

While the image size limit is now stated to be 5 Mbper image, rather than the previous 127 kb for reflowable files and 256 or 800 kb for children's and comic fixed layouts respectively, there has been no official announcement from Amazon on the matter. Moreover, there has been no release of an update to Kindlegen that might incorporate this change, and files converted via Kindlegen or Previewer still compress the images to the previous levels, as revealed by extracting the contents using KindleUnpack.

How, then, is it possible for the higher 5 Mb image limit listed in the Guidelines to be true? Is this just a future upgrade waiting to be made? A simple typographic error? Or is there some unwritten information hidden between the lines?

In an email response to my query on the matter received this week, an Amazon rep said:
From Version 2.9 of Kindlegen, images up to 5MB are retained for High Definition devices. For Standard Definition devices, however, the images will be compressed.
The answer, then, is different images for different devices! In fact, Amazon has apparently been doing this for some time already without telling anyone. Since the converted mobi file contains both the compressed images and an untouched source file zip archive, Amazon is able to send alternate versions of each ebook to different devices, depending on their resolution.

Of course, I wanted to know for certain if, and how, this worked in practice, so I did some tests this morning to determine if it was, in fact, the case. To do this I created a fixed layout, comic book-type test file with 6 pages, each containing just one full page image, ranging in file size from 1.8 to 4.94 megabytes. This created a source file of 22.2 Mb in size, which Previewer converted into a mobi file of 48.6 megs. I also uploaded the source file to KDP, which resulted in a download preview file of an only slightly smaller 45.5 Mb. In addition, I tested variations with the children's book-type, and as a reflowable file, all of which resulted in visible artifacting in the compressed images when extracted. However, when these same files were side-loaded to my HD 8.9" Kindle, every image looked crystal clear, showing that the reading system was accessing not the compressed images, but the high definition source files instead!

Here is a screen-cap comparison of the high quality image file being rendered on the HD 8.9 (left) and the compression employed on the same image for the standard definition file (on the right):

The HD device is clearly accessing the uncompressed image from the source archive file, rather than the highly compressed version found in the mobi8 image folder. Now, bear in mind that this is a 127 kb image that has been compressed down from one nearly 5 mb in size, so an incredible amount of compression has occurred. Moreover, Kindlegen will shrink the actual resolution of the image in reflowable files and fixed layouts with the children's book-type, although not in ones with the comic book-type, so this comparison is between the two extremes and probably not a common occurrence. But you should know that this can happen, and is, in fact, happening to your ebooks already.

One more important note must be mentioned here with regard to including very high resolution images, which is that it severely bogged down the device response, causing delays in page turns, sluggish response to scrolling and zooming, and in general behaving in a most unpleasant way, much like the old days of eInk displays (oh, wait, they still have those, don't they!). What this means in practical terms is that, although you can now include very large images in your Kindle ebooks, you will want to carefully manage the ultimate file size at which each image can be delivered, since you can no longer assume Amazon will do this for you. If you include 5 Mb image files, they will now send those full size images to the end user's HD device, for good or ill.

Which brings me to my final point, and that concerns Amazon's delivery fees. One major concern with image file size increasing has been the prohibitive cost of a .15 cent per megabyte bandwidth charge incurred under the 70% royalty option, since a single 5 Mb image would therefore cost .75 cents to send! But surprisingly, this does not prove to be the case.

While the KDP preview file I downloaded today weighed in a 45.5 Mb, the "book size after conversion" listed on the product pricing page was only 3.36 Mb - the size Previewer's log listed as the "deliverable file size" after conversion. This makes the total download fee just .50 cents for the full size HD ebook, leaving $1.74 profit for a $2.99 list price title. Amazon has essentially decided to subsidize content for the HD devices by only charging for the lower file size. This makes perfect sense, since you cannot rightly charge the higher fee for files sent to lower resolution devices, and charging different rates for different devices is simply impractical (not to mention a bookkeeping nightmare). It's also probably why they haven't bothered to publicize this much. Or at all.

Sunday, February 23, 2014

Image Size Limit Increased in Kindle eBooks

Although Amazon has not yet announced it publicly, there is an updated version of the Kindle Publishing Guidelines available from the link on the KF8 page that includes some very significant changes. Foremost among these is the long-awaited removal of the highly restrictive image file size allowances, and their relation to the booktype value.

To be clear, there is still technically a limit for image file size, but it has been (or will be) increased to 5MB per image, regardless of booktype. This effectively removes any restriction on image size, since virtually any image file can fit within this limit easily, even at high resolution, and adding images of even this size will increase the overall file size dramatically. 
But more on that in a minute.

First, I have highlighted and annotated the changes in my copy of the Guidelines, which you can download here. Or just get a clean version from the link above, since the first change you'll notice is the addition of a Revision History:

This is a seriously welcome addition that makes the task of finding out just what exactly has been altered from one edition to the next far easier. Still, for your benefit, I will share with you here some of the details. KindleGen Messages

This section received some editorial clarification to distinguish between Errors and Warnings in the KindleGen message logs, and their respective results. Nothing technically has changed, but it is made much clearer that Errors result in an abort of the conversion, while KindleGen will attempt to fix issues with Warnings, but this may or may not work, or work to satisfaction (for which Amazon declaims all responsibility).

3.2.1 Cover Image Guideline #1: Marketing Cover Image Is Mandatory

The recommendation for the "marketing" cover image size (the one you upload to KDP) is now 2560 x 1600 pixels, with 350 dpi resolution "to insure image clarity on Kindle HDX devices." This is altered from the last edition from a 2500 pixel recommendation for the longest side, with a minimum of 1000 pixels, but no specs given for the short side. An additional note is given stating that you will now receive a "reminder message" during upload if this image is smaller than the recommended size. The absolute minimum is still 500 pixels on the smallest side, so the warning can be safely (though not wisely) ignored if displayed.

Incidentally, the dpi resolution is essentially irrelevant on digital displays, since it is the total number of pixels that determines what is displayed onscreen. Amazon has previously always recommended 300 ppi, but with the Fire HDX 8.9" packing in 339 ppi they apparently felt obligated to add the higher dpi value, even though it doesn't matter in the least, especially since the marketing image is only used for the book's web page.

Lastly, here is where we are presented with the first indication that image file size has been given some consideration by the powers that be behind the scenes, as it states that "the image file size should be 5MB or smaller," which is a new addition.

3.6.2 Image Guideline #2: KindleGen Performs Automatic Image Conversions

Now we get to the crux of the matter. The entire section relating the various image files size allowances for the various booktypes (i.e. 127, 256, or 800 KB for flowing, childrens, and comics, respectively) has been removed. In its place we get this:

"The maximum size of an individual image file is 5 MB. The maximum size of an epub is 650 MB."
Moreover, while the header to this section still references "Automatic Image Conversions", the portion that formerly detailed the manner in which KindleGen handles "quality factor reduction" (i.e. image compression) has been removed. Apparently this will no longer be the case.

Now, with that said, bear in mind that the current version of KindleGen is still the 2.9 build that was released last September, so we will need to wait until the (presumably impending) release of the newest iteration (2.91? 3.0?) for any of this to take place. In addition, the posted Release Notes, as well as the relevant KDP Help section, still list the older version and lower size limits (although the KDP Help has never even been updated from the original 127 KB image limit for all Kindle ebooks, so I wouldn't put much stock in it as far as accuracy is concerned).

At any rate, we will presumably see an update to the Kindle Publishing Tools quite soon, if these changes are any indication. A few other entries add additional support to this assertion, as well as offering a further update in Amazon's outlook on image size:

4.3.3 Recommendation #3: Optimizing Content for Full Screen

Here (as well as Section 5) the reference to the original Kindle Fire's 1024x600 resolution has been altered to the newer Kindle Fire HD 8.9" display's 1920x1200 pixel depth. This not only increases the recommended image size, but alters its aspect ratio from the prior 17:10 to the newer HD models' 16:10 ratio, now apparently the preferred format (and a step in the right direction, although I'm still a strong advocate of 4:3 on e-readers for the sake of two-page spreads in illustrated works).

Additionally, the previously laughable statement that in order to "support 2X magnification with high quality" in children's books, "image pixel dimensions should be at least 2048x1200" - an utterly unwieldy resolution for a 256 KB file if quality is a concern - has now been increased even further to a recommended 3820x2400! There is no way in this world or any other that an image that size will look even reasonably decent at less than 256 KB.

5.2 Asset Requirements

This is the section that gives the breakdown for "Zoom Factor" values for region magnification, which again have been increased to accommodate the HD displays:

100% - 1920 x 1200
125% - 2400 x 1500
150% - 2880 x 1800
250% - 4800 x 3000 [!!!]
Curiously, this section still states that images must be smaller than 800 KB in size, but I chalk that up to Amazon's standard sloppy editorial practices, since this sort of thing has happened before (and often) throughout the various editions over the years.

Again, these "recommended" image sizes are simply unattainable at the previous (or rather, still current) file size limits. 4800 x 3000 is simply comic in this respect. But at 5 MB I can give my readers glistening crisp and brilliant detail even when zoomed to the highest value. Overall file size, of course, will still be an issue, which brings up a point I sort of glossed over in the earlier section, and that is the mention of a 650 MB file size limit for epubs.

This could possibly mean that the heretofore consistent KDP portal limit of 50 MB for file uploads might be increased (although I doubt it). Since this is not a well documented limit, only a test will determine the truth, and I haven't done one yet. The KDP Help page referenced above still lists the 50 MB limit (which is, as far as I know, the only place it's actually given), but again, it still lists 127 KB as the image size limit too, so don't put too much faith in that.

However, we should remember that even though flowable Kindle files have had the ability to contain audio/video files of up to 600MB for some time now, the portal limit has not changed because of it.

What this really refers to is the maximum input file size for KindleGen itself, as mentioned in Section 6.13 on audio/video file size, where it explicitly states that "the total maximum audio/video file size that can be converted from EPUB via KindleGen is 650 MB." This apparently now applies to fixed layouts in general now as well, and not just audio/video content.

3.6.11 Image Guidelines #11: Use Supported SVG Tags and Elements

A few other sections should be mentioned, of which this one is entirely new. This two page entry provides a list of Supported SVG Elements, along with an example and notes on tag usage. Also included is a link to the SVG specification for reference.

3.12 External Link Guidelines

And speaking of links, this is another new section laying down some laws regarding proper hyperlink behavior. Most are standard "no offensive content" warnings, but it is also made explicit that links to other online retailers are forbidden, and that Amazon "reserves the right to remove links in its sole discretion," a significant stipulation in legal terms.

Dictionary Overview

There are no functional changes to this section, but I mention it because it contains what appear to be the very first indications that Amazon has actually acquired an editor, since all of the several alterations here are purely for the sake of phrasing. Changes, for example, from "quickly search" to "search quickly" or "the entry they want" to "the desired entry" are entirely aesthetic in nature, and bear no substantive difference. 

That said, it is made more clear that dictionary functionality is for "in-book search and lookup" and that they must be marked as such, and with the correct language tag(s) applied, for what that's worth.

11.1 Appendix A: HTML Tags... / 11.2: Appendix B: CSS Selectors...

Although these are each listed in the Revision History, I cannot see any changes made here. All the "No's" are still "No" (i.e. no audio/video support in KF8; no max-width/height attributes, etc.) and there are no additions or deletions that I can discern. If anybody spots them, let me know!

*  *  *

With the release of the HDX displays and resolutions reaching the limits of human perception, the Kindle format is long past due for a major update in this respect, and I can't express how pleased I am to see the changes to the Guidelines. I have been waiting as patiently as possible for technology to reach the point where e-readers can accurately replicate the full size print experience in pristine image quality, since it is clear that digital is the future of the reading experience, or at least a very large part of it. Unfortunately, illustrated content has been sadly left behind in this regard (aside from iBooks on the iPad, which is superior as a graphic format, if not in terms of sales and store support).

With this update Amazon's Kindle platform will once again have a chance to truly shine where visual ebooks are concerned.