Thursday, August 1, 2013

iBooks Tutorial Update: Version 3.0

I've been meaning to do an update to my iBooks Tutorial series for about five months now, ever since Apple released "Revision 1" of the iBookstore Asset Guide 5.1 back in February. But being deeply buried in the Kindle Fixed Layout Tutorial at the time, I've only now found time to address it.

Many of you may have already read the updated Asset Guide, or looked at Apple's new 3.0 sample files, and been confused about the absence of both the NCX and the com.apple.ibooks.display-options, or by some of the new metadata properties found in the OPF. These are just a few of the changes made to the iBooks format that we will discuss here. So with that, let's look at what's new in iBooks!

NOTE: All mentions of the "iBooks" format here refer to the iBooks-compatible .epub file format, and not the proprietary .ibooks format, which can only be created using Apple's iBooks Author app (and which for some inane reason can only be run on a Mac). Much of this discussion applies to the .ibooks format as well, but its implementation is entirely dependent on the iBooks Author tool, and so not relevant here.

EPUB3 Additions

First, the most notable alteration made to the structure of an iBooks fixed layout file was the removal of Apple's proprietary com.apple.ibooks.display-options file, which previously controlled the overall layout and functionality of the fixed layout properties for the publication. This change was made in order to incorporate the portion of the EPUB3 specification that pertains to these attributes, thus making it easier for ebook creators to produce an iBooks-compliant file based on common ePub standards.

These features are now contained within the OPF file, along with all the other relevant package metadata, and employ EPUB3 nomenclature and semantics. Amazon, incidentally, has also incorporated a portion of the EPUB3 spec into their Kindle format. For those of you who have read my Kindle fixed layout tutorial, some of what's to follow will be familiar.

It should be pointed out here that the com.apple.ibooks.display-options file is still required for "Version 2.0" iBooks files, as the new EPUB3 code is fully functional only in "Version 3.0" ebooks, which, of course, also requires iBooks 3.0 or better. Therefore, if you want to produce strict 2.0 ebooks for backwards compatibility, none of this applies. However, going forward you should create iBooks 3 compliant files, for future compatibility, as 2.0 files will eventually be unsupported.

As further evidence of that, Version 5.1 of the Asset Guide removed all references to the EPUB2 file structure, so Apple is not looking back on this one. For info on the EPUB2 file structure you must resort to Version 5.0 or earlier of the Guide (or the relevant portions of my iBooks tutorial here on this blog). Files built using EPUB2 will currently run just fine in iBooks 3, of course, but will become outdated sooner than EPUB3 files as iOS and iBooks continue to be updated.

So with that out of the way, let's look at the specifics of what's been changed...

[By the way, if you're interesting in seeing what has changed in the EPUB spec itself between versions 2 and 3, the change-log can be found here: EPUB 3 Changes from EPUB 2.0.1.]

EPUB3 METADATA

In order to define your publication as an EPUB3-compliant file, the first thing that you need to do is declare the version number in the header of the OPF:
<package xmlns="http://www.idpf.org/2007/opf" version="3.0">
All that's changed here from before is the version number; the namespace declaration remains the same.

And although this is not EPUB3 related, while we're up here messing with the header I should point out a set of new declarations that have been added to the one just mentioned:
prefix="rendition: http://www.idpf.org/vocab/rendition/# ibooks: http://vocabulary.itunes.apple.com/rdf/ibooks/vocabulary-extensions-1.0/"
These define the semantic rules used by some of the new additions, including media overlays and javascript. If these prefix references are not present in the header your ebook may fail validation.

With the com.apple file gone, the entries that define your layout options are now found in the OPF, in the metadata section, along with the title, author, publication date, etc. Incidentally, the "modified date" is now listed for the first time as a required item (along with the title, identifier, and language as the only other three being required rather than optional):
<meta property="dcterms:modified">2012-05-04</meta>
Apple (like Amazon), has incorporated three new EPUB3 properties into the OPF, which are defined as rendition attributes, although Apple has also included a few ibooks-specific entries as well, just as Amazon has included some of their own Kindle-specific entries. First, of course, is the entry that defines the publication as a fixed layout:
<meta property="rendition:layout">pre-paginated</meta>
This replaces the "fixed-layout">true/false option in the display-options file. Options here are "pre-paginated" for fixed layouts or "reflowable" for flowing ebooks. This is obviously required for fixed layouts. Some of what follows also applies to reflowable ebooks, but for the most part I'm only interested in fixed layouts here.

Next, there is a replacement for the "orientation-lock" option in the com.apple file:
<meta property="rendition:orientation">
values for which are "portrait", "landscape", and "auto". Note that the values are not "portrait-only" or "landscape-only", although "none" is a valid alternative to "auto", and is the default value.

The third and final EPUB3 entry is more-or-less (though not exactly) the equivalent of the earlier "open-to-spread" option:
<meta property="rendition:spread">
Here the values specify whether the viewport displays a one- or two-page layout. Allowed values are "none" for a single-page layout in either orientation, "both" for two-page spread in either mode, and "auto" for a "best-fit" option that theoretically displays one page in portrait and two in landscape for standard aspect ratio layouts, and the reverse for wide page layouts. This last, however, does not always work as intended, and in fact, Apple list it as just another two-page spread option, although that is not what it is intended to be in EPUB3. In addition to these, the EPUB3 spec allows the values "landscape" and "portrait" in order to force two page spreads in the given orientation, but iBooks does not support these.

One additional EPUB3 metadata property will be addressed in due course (see VERSIONING below), but these are the three that affect ebook functionality.

IBOOKS METADATA

In addition to the above properties, Apple have included a few proprietary entries to define some of the other iBooks-specific attributes previously found in the deprecated display-options file.

The first of these functions as something of an addendum to the rendition:orientation property, and replaces the "platform" tag in the com.apple file, which allowed for setting separate values on the iPad and the iPhone/iPod Touch. Here the entries are given as:
<meta property="ibooks:ipad-orientation-lock">
<meta property="ibooks:iphone-orientation-lock">
where the value in this case is either "portrait-only" or "landscape-only" (as they originally were in the display-options file). If provided, these values override any global settings entered in the "rendition:orientation" entry (if given), allowing you to define layout preferences for three separate scenarios - which becomes more relevant now that OSX will finally be getting iBooks. If no platform-specific values are entered, of course, the global settings (or defaults) will apply.

Next is the substitute for the "specified-fonts" property, which is now given as:
<meta property="ibooks:specified-fonts">
Valid values for this are true or false. If you include embedded fonts in your publication this must be provided and the value set to true. Otherwise your fonts will be ignored, and the file will fail validation due to it containing "unused" fonts.

Finally, Apple has added an entirely new property with this latest iteration, one that has not been available before:
<meta property="ibooks:binding">
This allows you to turn off the faux book binding if you choose, by entering a value of "false" here. This will also remove the shadowed "spine crease" that runs down the center of each two-page spread, rendering your two-page spread as a single unbroken layout. The value is set to "true" by default.

One other property must be mentioned before moving on, and that is the now-deprecated <option name="interactive"> element found in the com.apple file (though little known and rarely used, it seems). This has been supplanted by the script tag, which is inserted directly into HTML pages where javascript is present. Scripted code, however, is a complex subject which demands its own tutorial, so I'll leave it be for now. Refer to the Asset Guide and the Fixed Layout 3.0 sample file for specifics on implementing Javascript interactivity.

THE MANIFEST

Moving further down the OPF, there are two new additions to the manifest portion of the file, both of which have also been adopted by Amazon. Each of these add a property value to one of the listed items in the manifest that identifies its function in the EPUB3 schema. The first of these defines the cover, and is appended, appropriately, to the line listing the cover image file:
properties="cover-image"
This replaces the <meta name="cover" content="cover"/> entry in the metadata section of previous ebook files. You can leave it in or delete it as you see fit, but it is no longer needed.

The second property defines the new Navigation document, and again is appended to the relevant line in the manifest section of the OPF which lists that file:
properties="nav"
We will look at the Nav doc momentarily, but for now just be aware that you must give this property in the manifest listing for the Nav file (toc.xhtml, or whatever you name it) in order for it to be recognized by the reading system. In addition, since the NCX has been replaced by the Nav doc, you obviously no longer need to add the toc="ncx" attribute to the Spine tag as before.

Finally, one last change has been made to the OPF that must be mentioned before moving on, and that is the complete removal of the Guide section, which has been replaced by the new EPUB3 Navigation document. The Guide is now completely obsolete in iBooks 3, and serves no further function.

The NAV Doc

New in version 3.0 is the addition of the EPUB3 Nav doc, which supplants the outdated NCX. The Nav doc is simply an XHTML file with some unique entries, each of which we'll look at in turn. The nav tag is an HTML5 element, and therefore is not supported in EPUB2. Again, Amazon has added support for portions of the navigation doc to the Kindle format (excluding the page-list attribute), so if you've got my Kindle formatting guide be sure to read that section for more info on this subject.

First, be sure to add the new prefix declarations to the Nav doc header here, in addition to the standard idpf namespace:
xmlns:ibooks="http://vocabulary.itunes.apple.com/rdf/ibooks/vocabulary-extensions-1.0"
epub:prefix="ibooks: http://vocabulary.itunes.apple.com/rdf/ibooks/vocabulary-extensions-1.0">
Note that these both reference the same place, just using different syntax. These are required to define the rules of semantic usage for the entries in this page, and if not included the file will fail validation.

There are three sections of the Nav doc: the toc, landmarks, and page-list, each of which are given as an epub:type within a nav tag.

THE TOC

The first and primary section of the Navigation document is the Table of Contents, which provides the reading system with a list of entries and links for the contents menu. This is the only required element in the Nav doc in all cases (landmarks are required in given situations, which I'll get to shortly, and page-list is not required in any instance). This section replaces, and greatly simplifies, the content of the NavMap in the superseded NCX.

Please note: the Nav toc is not the same as an internal HTML Table of Contents within the book itself, although it can be if you list this file in the Spine (which will make it show up as a rendered page in the linear sequence of the book - otherwise, it will not appear, apart from the contents menu). That said, however, it's generally better to create a separate page for internal TOC's, since the other content in this file will also appear on the rendered page (assuming you include it), which you undoubtedly will not want. Thus, unfortunately, the Nav doc does not provide a consistent means of creating one universal TOC for all purposes. It does, nonetheless, greatly improve the process.

To define the TOC section, add a nav tag with the following values:
<nav epub:type="toc" id="toc">
Note that the epub:type value is case-sensitive, so it must be "toc" and not "TOC" or any other variation, in all instances where it appears. This is true for each of the three values given on this page, which use all lowercase letters.

The id value can be anything you like, so long as it matches what you enter in the OPF manifest for this file's id, otherwise the reading system will not find it. This is the only section of the Nav doc that requires in id value.

All three of the sections give their entries as an ordered list (with or without a header - you can add one if you like, but it's not required):
<h1>Table of Contents</h1>
<ol>
        <li><a href="page.xhtml">Entry Title</a></li>
</ol>
One, and only one, ordered list can be included in each nav section.

List each entry in this manner, providing a link to the file name and a title. The file name can be appended with a fragment (#name) linking to an anchor on a page, but of course in fixed layouts this is not really necessary, since the full page is displayed, unlike flowing files. The text you enter between the angle brackets (> <) will appear as the table of contents entry in the menu, exactly as given, so be sure to use correct capitalization for these, as well as in the header, if given.

Do not add page numbers to these entries, as iBooks will add them automatically, using the values given in the page-list, if provided, or simply by numbering them sequentially if not.

Finally, be sure to close each line item individually, as well as the ordered list at the bottom and the nav section itself (</nav>), since each section requires its own nav tag.

LANDMARKS

The landmarks section of the Nav doc replaces the deleted EPUB2 Guide items in the OPF, providing structural information about the content. However, unlike Amazon's clearly defined functions for the three entries the Kindle recognizes, the purpose of this section in iBooks is more or less opaque. With the sole exception of the "bodymatter" attribute, none of the myriad entry values available have any actual effect on a publication's functionality.

That said, you are required to provide a "landmarks" section with at least the "bodymatter" attribute included if, and only if, you do not provide a custom sample file for your ebook to the iBookstore during title setup. If one is provided by you, the landmarks section is not required. The reason for this is that Apple uses the "bodymatter" tag to locate the start of the ebook's content for the sample it creates automatically if one is not provided by you. Apple recommends you include landmarks for all structural components in your ebook (i.e. preface, dedication, appendices, etc.), and there are good reason for doing so, since they provide additional metadata about your publication that may be useful other along the distribution chain, as well as the eventual end user, since there's no way to know what reading system will ultimately display it.

To create a landmarks section, add a new nav tag with the following attribute:
<nav epub:type="landmarks">
Within this include an ordered list as you did in the toc section, with each line item containing the following attributes:
<li><a href="page.xhtml" epub:type="value">Irrelevant Text</a></li>
Again we have a referenced link to the page location, but in this case we also get a second embedded epub:type attribute for each individual line, the values for which are nearly endless, but one of which is bodymatter, which signifies the start of the main body content. You can find an exhaustive list of possible values on the EPUB 3 Structural Semantics Vocabulary page at the IDPF website, where something like 70 entries are given, detailing the component parts of nearly any given document. Here again the values are case sensitive, with all entries given in lower-case.

Note that you can only include one line item for each epub:type, so for example, you cannot add multiple items with the "chapter" value, even though you may have many chapters. This is more the function of the Table of Contents. The purpose of the landmarks is to distinguish significant sections of a publication, such as its front or back matter, from its main content.

As I mentioned, only the "bodymatter" attribute has any stated functional value in the Asset Guide, although others may be used to pull data at some point in various reading or distribution systems, so it certainly can't hurt to add them. The "bodymatter" link should point to the first page of actual content in the book, after any front matter that's been included.

On that note, there is one other entry that Apple has added which has a unique epub:type all its own:
epub:type="ibooks:reader-start-page"
This sets the "start reading" page to which a book opens on first reading. According to the Asset Guide this applies only to flowing ebooks, but this is where you would add it if you were making a reflowable file, so I thought I'd mention it at this point.

Lastly, with regard to the "title" text provided for these entries, there is no purpose to it whatsoever, as it shows up nowhere, does not contribute to the entry's functionality, and in Apple's examples generally just repeats what is given as the epub:type value. Whereas the Kindle creates active links from the contents of the landmarks section for some of its "Go To" menu entries, iBooks has no such feature, so aside from the sample file function, there is no practical reason to include this section in the Nav doc at all, let alone the pointless titles.

But again, these entries may show up in databases somewhere down the line, such as those used by libraries or retailers, so as always, more information is better than too little.

PAGE-LIST

The final section of the Nav doc provides page mapping functionality. This allows you to asign page numbers, Roman numerals, or letters to any given page, rather than have them simply numbered sequentially in the device menu Table of Contents. iBooks pulls from this list for the page numbers given in the TOC, if supplied.

In addition, these show up at the bottom-center of the screen, just above the thumbnail scrubber, when you tap the display to access the menu bars, and also in the larger thumbnail when you scrub through the images. They do not show up in the actual image grid version of the TOC, but they do appear on any bookmarks the reader adds, so in these cases they are quite helpful.

Incidentally, page maps can be added to reflowable ebooks as well, in order to line up the flowing text with actual page numbers from a print edition, which is enormously important for listing references or for classroom study, or for that matter anywhere two or more readers need to sync up their page locations, such as book clubs.

To create a Page Map, add a new nav section with the page-list attribute:
<nav epub:type="page-list">
And again include within this an ordered list containing links to each page:
<li><a href="page.xhtml">#</a></li>
You can add any alpha-numeric character you like (1,2,3 - i,ii,iii - I,II, III - a,b,c), using capitals or lower-case, or you can leave it blank for an unnumbered page. You can even enter a short word or phrase, such as "Intro" or "fig.1" (although this is best left to a Table of Illustrations). Page numbers are probably more useful, but whatever you use keep it short and relevant.

This is the one feature of the Nav doc that Amazon opted not to incorporate for some reason, possibly because the Kindle has no thumbnail function. As much as Amazon has touted the Kindle's ability to display "real page numbers," one would have thought this would have been included. But unfortunately there's no way to add any page numbers in a Kindle file. It would be nice to have page numbers in the menu TOC at least, as this would make using the "Go To Page" function much more useful (particularly given that the "Go To Location" function is essentially worthless)!

Other Additions

A few additional EPUB3 features have been included in the Version 3.0 format, which I will outline quickly here. These include the ability to add pop-up footnotes (very cool!), change both text direction and page progression, including Tatechuyoko integration (critical for many non-western languages), and adding version numbers in updated editions (with automatic customer notifications of the changes!). These are all significant improvements to the platform, so let's look at them each a little closer.

POP-UP FOOTNOTES

EPUB3 provides an easy means to create active links that trigger a pop-up overlay containing additional content. This is intended to be used for footnotes, but of course can be used for many other reasons, much in the way I used the Kindle magnification function to create many types of interactive content. I have not yet tested this extensively to see how far it can be pushed (as, for example, to what extent the styles can be modified via CSS), but so far as I can tell there is no reason why you could not do much the same with this code as with the Kindle region magnification feature.

To create a basic pop-up you need two elements: a link and a target. First, you create the active link using a standard anchor, but with the addition of an epub:type attribute applied:
<a href="page.xhtml#name" epub:type="noteref">Content to be linked</a>
For footnotes the "content to be linked" is generally a numeric digit of some variety, although it can really be anything, even an image, since the "noteref" simply functions as a trigger to activate the target. You can add a class to style the link as well.

For the target content you use another epub:type attribute contained within an <aside> tag:
<aside id="name" epub:type="footnote">Content for the popup</aside>
This hides any content held within the <aside> tags until the link is activated. Note that the id value must match the fragment in the anchor reference (#name in this case). IBooks applies some default styling to the pop-up, but you can override this to some extent with your own custom CSS classes. Again, I have not yet tested this thoroughly, so you'll have to try it out yourself to see what works and what doesn't.

Also note that any page which includes an epub:type attribute must contain the epub namespace in the header declaration:
xmlns:epub="http://www.idpf.org/2007/ops
This, of course, is in addition to any other declarations required for the content on the page, as given above.

PAGE & TEXT DIRECTION

As I mentioned earlier, you can now define both the text direction and the page progression in order to accommodate vertical and/or right-to-left languages. Page turn direction is, of course, a global setting, since it defines where the ebook starts and ends. This is set by adding an attribute to the Spine tag in the OPF:
<spine page-progression-direction="rtl">
Since the default value is left-to-right, you only need to add this if you wish to alter the page direction, although you can add the value "ltr" if need be (say, for example if you're working with an Asian version of iOS that has a default of "rtl"). In addition, you can enter the value "default" to let iBooks define the page progression, based on the system or reader settings, for example in the event your title has multi-language support (in a technical document or instruction manual, for example).

Text rendering can also be determined by adding a dir attribute to the package header in the OPF in order to define a global text direction:
<package dir="rtl"...
Again, the default is ltr, so you only need to add this to change the default behavior to produce a right-to-left text direction.

Vertical text is also now supported, but this is done at the page level using the CSS3 writing-mode property:
-epub-writing-mode: vertical-rl;
This allows you to create individual pages with alternate direction text, if so desired. Allowed values here are vertical-lrvertical-rl, and horizontal-tb (top-to-bottom). This must be added to either the <body> tag or the <html> container itself for each page, and not to nested elements on the page. Thus, only one value can be applied to each page (or content file in the case of a flowing ebook). You can, however, add a section of alternate-direction text (known as "Tatechuyoko") using a class with the "text-combine" attribute applied:
-webkit-text-combine: horizontal/vertical;
You then append that class to the section of text to be displayed in the alternate direction. In this way you can add both horizontal and vertically aligned text to a single page.

VERSIONING

You can now add a version number to your title in order to distinguish a new edition, with either major or minor changes. Readers who have purchased the book will be notified automatically that an updated file is available, and upon download the new version will replace their earlier one.

The version number is added as a metadata value in the OPF, along with the other new properties mentioned at the top of this post:
<meta property="ibooks:version">####.####.####</meta>
The data string provides for up to three sets of four numeric digits separated by dots, which signify an edition number. The first set signifies a major revision or edition change, as from version 1.x to 2.x. The second set would represent a relatively important alteration and/or addition to the content, while the last set is used for minor changes such as stray typos or non-content modifications to the file structure or layout. Thus, a version 1.1.1 ebook would have seen one minor and one moderate update, but not yet been revised or rewritten to the extent that a new full edition update is required.

Non-numeric characters are not allowed, only standard 0-9 integers, and initial position zeroes are ignored (i.e. 1.01 is the same as 1.1, although 1.10 is higher than 1.9).

The inclusion of the version number property also requires adding the prefix attribute to the package header, just as in the Nav doc mentioned earlier:
prefix="ibooks:http://vocabulary.itunes.apple.com/rdf/ibooks/vocabulary-extensions-1.0/">
Be very careful not to confuse the header version="3.0" attribute, which defines the EPUB format version on which the publication format is based, with the ibooks:version metadata property described here, which signifies the current edition number of this particular title.

CONCLUSION

That pretty well covers it for where we are with the iBooks version of the EPUB3 format, which with this revision has moved much closer to a common open source platform than ever before. The removal of the com.apple file and inclusion of standard properties with far broader support goes far towards making content production a less fragmented process.

It is unlikely that we will ever see a single format that is supported on all (or even the three most popular) devices, since each reading system and hardware device has its own unique requirements and quirks. Moreover, as with any business, platform providers succeed by identifying their product as unique, with features superior to (or at least different from) the others.

The hope is that these platforms will start to differentiate themselves based on the quality of the hardware and the user interface - not by adhering to proprietary file formats, but by how well their device displays a file any other system can also display. The user experience would be greatly improved with the frustration and inconvenience of proprietary file formats removed.

Happily, both Apple and Amazon have made small, but important steps in that direction with their latest updates.

UPDATED 8-2-2013 with some slight editorial changes to make certain passages read more clearly. Added a bit more detail to a few sections as well.