Less than two weeks into 2012 Amazon released the main specs for their new KF8 format, based mainly on ePub, but with a whole new set of a proprietary requirements for fixed layout features such as area magnification, and with severe restrictions on others such as image size and orientation. In addition, they use their own metadata entries to declare fixed layout properties. While KF8 is in essence an ePub in its structure, once converted into mobi via KindleGen it becomes proprietary and can only be read on a Kindle system. ePub files themselves are not recognized by Kindle devices.
Then, just a week later, Apple announced their new .ibooks format, along with the only tool that can be used to make it, iBooks Author. Again, the underlying structure is essentially an ePub, but with a host of new code elements that are entirely unique to Apple, with no relation whatsoever to anything in ePub. So complex and undefined are these new elements that there is no possibility of coding them by hand, so that what was once in essence an ePub is now something altogether different. And of course, it can only be read in Apple's iBooks system (and currently only on the iPad).
This is the direction things are going. As mentioned in my prediction, advances in technology and innovation in individual applications are far outpacing ePub's ability to keep up, to the point that very soon these formats will likely do away with the ePub structure altogether, since there's no inherent reason that they need it if the end goal is to produce a unique proprietary format. With KF8 you can at least begin with an ePub file and then modify it to meet the Kindle Fire requirements. But Apple has removed even that possibility. IBooks format files must be built from scratch, using their software, or not at all (and they must be sold via Apple's iBookstore, and consumed via Apple tablets).
Of course, Apple still accepts ePubs in their standard iBooks model (albeit with some custom modifications, such as the com.apple file), as does Barnes & Noble (though only for reflowable texts: BNKids format is so proprietary they won't even release the spec), but for fixed layout ebooks ePub3 has never really had a chance.
The idea of creating an open standard that many reading systems can handle is a noble, but unrealistic goal. It's just not a practical business model on which to build an empire. Success at the level at which Apple and Amazon are competing is accomplished by creating brand identity and loyalty. This is how Apple has sustained itself for many years, and how the Kindle became the first successful ebook reader. Certainly there are many other factors, among which producing a quality product is foremost. But to create a product of quality one must first create a product that stands apart from the others, and that by its very definition is a proprietary product.
While we will likely never see a major ePub3 reader, the market will be swamped with generic devices and off-brands that will do the job. But none of them will be truly great, relegating ePub3 to the sidelines and the cheap or free titles. Because of this it is unlikely ePub3 will ever reach its full potential.
EPUB3, KF8, and .ibooks files all build on the same standards (e.g. ASCII, Unicode, XHTML, CSS, SVG, PNG, etc.). It's not like they're completely different. One can convert between them, more or less.
ReplyDeleteSomething similar has happened on the web. Over time, the various web browsers added their own special extensions (e.g. a way to do rounded corners in CSS), but over time, many of those ideas have been subsumed into the standards. Over time, if you stuck to using standards only, the set of things you could do got larger.
You can think of the proprietary extensions as being experiments. Some turn out to be useful or popular and become standards, whereas other turn out to be stupid and get abandoned. I think that's healthy.
Just as proprietary extensions didn't kill standards on the web, proprietary extensions won't kill standards for ebooks. Indeed, many of those standards are the same standards.
There's a fundamental difference between the web and ebooks that you're missing here. The Internet by its very nature is designed to be inclusive and accessible everywhere on any platform. That is the purpose of a network. This forces standards to be used.
ReplyDeleteEBooks on the other hand are exclusive, designed and intended to be accessed on a proprietary system (and not just any, but your device specifically). The changes being made to the epub "standard" are not just extensions to it, but radical alterations to it,turning it into something else entirely.
You cannot simply convert from one format to another, at least in the case of KF8 and iBooks, which are the only formats that really matter in terms of sales and where the advances are all happening. Try converting the new .ibooks format into anything else, and if you succeed please let me know.
"the idea of creating an open standard that many reading systems can handle is a noble, but unrealistic goal" - but, we already achieved this via EPUB 2, as the vast majority of Kindle sales are of content submitted to them by publishers as EPUB, and Apple iBooks is as you noted still EBUB-centric, not withstanding a couple handfuls of textbooks authored for the iBooks Author proprietary "fork" of EPUB. In a certain way, EPUB 3 will be easier because everyone is following Apple's example and using WebKit as the engine, that already ensure a base level of capability and compatibility that is better than we had with EPUB 2. And, the players backing Open Web for digital publishing have just banded together, with IDPF acting as catalyst, to do the rest collaboratively. See: readium.org.
ReplyDeleteAnd the logical direction of conversion is from EPUB as the authoring tool output and publisher standard to KF8/iBooks only when necessary. Amazon's own conversion tools have EPUB as the preferred input. And even with somewhat loosened licensing terms publishers doing development for .ibooks format are still incentivized to do it in an open format first. And the leading authoring tool used by publishers is about to get a major upgrade that will put the EPUB platform ahead of iBooks. See: http://idpf.org/epub/pgt/ .
Fundamentally, publishers want to get their content out to all channels, and readers want the choice of where to consume it. Proprietary silos in the middle are just a toll collector. So neither party has incentive to help the middlemen. And devices are about to mushroom into tremendous variety. Last but not least, eBooks esp. in areas like education are by no means going to stay closed "exclusive" objects but need to get connected to other content and social graphs. All this pushes things to open as the ultimate result.
I don't want to understate the risks, and one reason we worked to "herd the cats" to do Readium Project was to counter the very real risks of fragmentation and domination by proprietary silos. Is EPUB 3 a sure thing to be the universal digital publishing standard for the open web? Nope. But DOA? I'm betting you're wrong, and hoping you're there with me in spirit.
Bill, while I agree with you "in spirit", philosophy as always is useless here. Indeed, in the Socratic spirit one might say that open standards equate to communism, while proprietary systems equal capitalism at its fundamental root. But I'll leave that argument for another day.
DeleteYou make several incorrect assertions in your statement. Firstly, Amazon only began accepting EPUB as a source format last year, so your comment that "the vast majority of Kindle sales are of content submitted to them by publishers as EPUB" is flat wrong. The three prior years of Kindle sales were all of content from non-EPUB sources, as much of last years would be too.
Secondly, nowhere does Amazon say that EPUB is the "preferred input" format, either for KindleGen or KDP. In fact, EPUB is always listed last wherever options are given, with html generally listed first. Indeed, in the KDP help topic for Types of Formats it states unequivocally, "For best results, we recommend that you upload in DOC (.doc) or PRC (.prc) format." So to say that EPUB 2 has already achieved some universal goal is hardly accurate.
I agree that ideally authors and publishers want their content on every possible channel. But the fact is that the only channels that matter are the ones that generate sales. And right now that's Amazon. To say that "proprietary silos" are mere "toll collectors" is to gravely misunderstand the revolutionary changes that have occurred. One might just as readily call traditional publishers, promoters and agents "toll collectors" and be far more accurate. In truth, Amazon has done more to remove those true middlemen than anyone.
More importantly, those proprietary silos are also the retail outlets that are generating all the revenue, via their branded devices and enormous web presence. 90% of the device market is dominated by the Kindle and iPad, with a handful of Nook Tablets thrown in. Devices can "mushroom" all they want and it will matter not one whit if no one buys them, and without the support system of an online ecosystem with massive content that won't happen.
There have been six major publishers who produce the bulk of what is read for many decades now, and a lot of smaller presses to fill in the cracks. All of those products are proprietary to those publishers. If you want a Penguin book you have to buy it from Penguin, exclusively (via "toll collectors"), and no one else can produce it. If you want to change from hardcover to paperback, or audio, or large print format, you have to buy it again. How is that any different in principle than buying a specific ebook format?
There are now three major ebook producers who provide the bulk of what is consumed digitally, and a lot of smaller outlets that serve to fill up the corners, as it were. All three of the majors publish their own content, with varying degrees of success. But that is where we're headed. Amazon doesn't need publishers, they need authors. It's the publishers who need them, because they have the customers. That gives them every incentive to produce and maintain an increasingly proprietary product. EPUB is doomed precisely because it is non-proprietary. The only content providers who want EPUB are those who have no successful device and format of their own.
For the record, I want EPUB. As a content creator I would much rather produce a product once than multiple times. But that's just not where we're headed. In creating fixed layout ebooks there is now absolutely no standard at all. To create a fixed layout iBook and the equivalent KF8 format you all but have to start over from scratch. I don't see that changing.
There used to be dozens of software platforms, and shopping for programs was a nightmare. Now there's Windows or Mac. Android apps or iOS apps. Tomorrow it will be Kindle or iBooks.
I'm not a pessimist, I'm a realist.
Scot,
DeleteRE: Amazon sales, I thought context was clear I was talking about the current situation "at the margin" not historical. But since Amazon sales have been on such a rapid rise it may even still be a true statement that the majority of sales for Kindle for all time have been of titles submitted to Amazon as EPUB. Amazon doesn't provide the data, and it's not entirely public when various Big 6 publishers started submitting EPUB to them, so we can only speculate - which is part of the problem. You're also right that Amazon is less EPUB-friendly for smaller programs like KDP. But my main point remains valid:for al the big publishers, which represents most sales of eBooks to readers, their conduit to Amazon right now is already EPUB.
RE: your analogy of book publishers producing different proprietary products. I see the big drive to a universal format as coming from the publishers POV. A Penguin hardcover is sent to all their physical channels, they don't have to make different physical books for different bookstores. It's not practical for them to do this even if there were only 2 or 3 channels to be concerned with, and there are a lot more than that. So it's entirely possible that EPUB could be a wild success as a universal distribution format from publishers to channels, but fail miserably as an interoperable distribution format because it gets ingested into all kinds of different proprietary clouds and converted and/or wrapped in proprietary DRM en route to being file downloads (although that would be a suboptimal outcome IMO).
I think you are BTW sorely underestimating the impact of Nook (and Kobo and Sony Reader and and and) on the total market for devices actively used for reading. It's not just a Kindle + iPad game, Apple is nowhere close to being the #2 eBook sales channel for general trade publishers. And, in the long run there's going to be a zillion different tablets - iPad won't be 70%+ of the market forever.
Lastly, fixed-layout can be done with standard EPUB 3 (which supports CSS abs pos as well as SVG) and IDPF members are (with Apple participation) in the process of "dotting the i's" left out of the EPUB 3 spec re: fixed-layout metadata. So again you "don't see that changing" . But, it is already changing, rapidly.
Really the big picture that we disagree on, perhaps, is what the ultimate platform for digital publishing will be. I see it necessarily convergent with the Web, with Kindle and iBooks being at best a short-term diversion (a la AOL and Compuserve, which also seemed uber dominant in their day). EPUB's role is thus simply to grease the skids for this convergence, by being the publication packaging of Web content. This ain't rocket science, so I think I'm the realist here - an optimistic realist, for sure, I'll give you that.
Yeah, I couldn't disagree with you more about web convergence. Looking at the past four decades since ebooks first arrived, there has been ample opportunity to read ebooks in browsers of one variety or another, and none of them have taken hold (I've been reading ebooks since Project Gutenberg had fewer than a thousand titles). Platforms and formats have come and gone but none have ever met with any success until the Kindle. One would be wise to look at why this might be.
ReplyDeleteThe convergence that finally met success was not "web + cool css templates", but eInk + great device + instant access + massive selection + huge pre-installed customer base + proprietary format. Remove any one of those and the numbers crumble. Sony has bailed out of the ebook business altogether for this very reason. They make great devices, but have none of the other critical factors.
As for underestimating the impact of various devices on the market, see my posts on the recent Aptara survey - http://authoradventures.blogspot.com/2011/12/ebook-publishing-infographic-analysis.html. According to the figures provided by the major trades, Sony had 0% of trade publishers' digital sales revenue. The Nook, with 1% of trade and 2% of educational, has met with only moderate success at best, given BN's base infrastructure; they're making more on device sales than content. Kobo, with 3% of trade ebooks, is growing primarily because they serve an international market with less direct competition from other devices. As for Apple, they came in at 4%, ahead of BN, Kobo and Sony, so "nowhere close to being #2" is again a bit inaccurate. As for tablets, a couple dozen have come and gone already and it hasn't fazed iPad sales a bit.
The more important figure, however, is that Amazon meanwhile dominates the market with 56% of trade ebook sales. You can believe that Kindle and iBooks are "short term diversions" all you want, but that won't change the facts and figures. And businesses deal in facts and figures, not speculation and hopeful thinking. What's working currently, for the first time ever, is a proprietary format on a dedicated device with a broad content ecosystem.
I wish you the best of luck with the Readium Project and the EPUB 3 templates and updates, but without the aforementioned support structure it's frankly irrelevant. Besides, no one wants to deal with all that code behind the scenes, or pay someone else to do it, which is where iBooks Author made such a huge advance. That's they way I see things going in the future. The major trades with decline into virtual oblivion, with new digital publishing empires ascendant in the form of Amazon and Apple, as well as B&N if their brick and mortars don't drag them down first. There's room for others, though, so maybe Google will hook up with Sony to produce an EPUB superstore that can rival the current competition. But that's just wishful thinking.
Well, I think the Aptara data is highly suspect as a basis for generalizing about total sales - there's no indication of what % of the total eBook sales it covers, it's lumping micro/self-publishers in with big folks, and most of the big pubs didn't give them data. I stand by my statement that Apple is a long way from being #2.
ReplyDeleteThat aside, what's working currently is major publishers submitting EPUB files to all channels including Apple and Amazon. The transition of this to HTML5-based EPUB 3 is, IMO, far from wishful thinking. Given that KF8 and .ibooks are based on HTML5 and frankly imitative of EPUB, it's the expected outcome. If - a big if - we don't go all doomer.
As far as reading via browser stacks... are you even aware that iBooks is built on WebKit? It's basically Safari, under the chrome. And, KF8 is HTMl5-based too.
So we're supposed to take your vague guesses and personally beliefs over a published study? Numbers are always a bit suspect, and to be taken as a broad generalization in most cases, but surveys such as this tend to be fairly accurate, with a relatively narrow margin of error. With over 1350 publishers participating, it provides some very useful data that represents a reasonably large segment of the industry, as well as some informative analysis. But yes, there are further breakdowns I would like to see as well.
DeleteMost lists place Barnes & Noble second, with iBooks, Kobo et. al scrabbling for the bottom single digits. I'm frankly surprised the iBookstore hasn't taken off as well as it might, but that just goes to show that most iPad users aren't buying their devices for reading, whereas with Kindles and Nooks they are.
As for the browser issue, who cares? If an ebook can only be produced for and read in a specific app or platform it's irrelevant if the underlying program code is based on WebKit or whatever. You can't open your iBooks in Safari. The fact is that iBooks requires one set of code, and KF8 another, each with specific implementation, and while they share many similarities via their EPUB foundation, they're diverging, both from EPUB and from each other. And they're not coming back.
Maybe the common ground is where EPUB will find its niche, or maybe it will rise up as a dominant contender. But that's all to be seen. I have to deal with what's in front of me today if I want to get my books in front of readers tomorrow.
If you look at what iBooks, KF8, and EPUB have inside, you'll find that they all have the same guts, more or less. They're built out of XHTML, SVG, CSS, and other standards --- the same standards which are also used on the web.
ReplyDeleteI wouldn't be at all surprised if KF9, or whatever Amazon calls its next e-book format, is even more like EPUB3. As it stands, KF8 is exactly the same as EPUB3, only with some bits removed. Amazon could have called KF8 "EPUP3-lite" and they wouldn't have been far off.
I couldn't disagree with you more. KF9 (or whatever) will very likely be even more divergent and proprietary than KF8, because it's in Amazon's best interest to keep their ebooks locked into their ecosystem. An open standard will only make their product the same as every other product, and that's not how successful businesses are run.
DeleteYou've not only missed the point of my post, but you're completely wrong about what the iBooks and KF8 formats have inside. Yes, they're based on "ePub-lite", but it's what they ADDED that makes the difference, particularly where it comes to their fixed layout formats. Little by little, both Apple and Amazon are moving away from ePub. Try converting either .ibooks or .azw files to .epub and see how far you get.