Which EPUB Preflight Step Catches Which Failure?
An author preparing an ebook needs more than a reassuring green result. The practical question is which check will stop a particular mistake, where to send the repair, and which decisions still require someone to inspect the book. We tested that question with nineteen small, original EPUB archives: one unchanged control and eighteen deliberately selected changes. EPUBCheck 5.4.0 and DAISY Ace 1.4.6 completed every archive.
The tools provided different information. Five changed archives produced diagnostics only in Ace, one only in EPUBCheck, and four in both. Eight produced no reported diagnostic in either tool. These are counts of chosen experimental cases, not failure rates for books, an estimate of tool sensitivity, or evidence that eight changes would be harmless to readers. The useful result is a repair map: missing files, absent attributes, contradictory links, weak descriptions and missing semantics do not all reach the same checkpoint.
What we built and froze
The canonical work is an original two-chapter miniature, The Lantern at the Quay. Mara arrives before sunrise with a blue lantern; Ivo arrives later with an empty crate; they leave through the western gate. The package contains an original map, a descriptive caption, a three-row, two-column supply table including its header row, a short MathML equation, a contents document and a local stylesheet. No commercial book, unpublished manuscript or third-party test publication supplies the text or images.
Before running the changed archives, we froze the case list, hypotheses, admission rule and machine-observable endpoints. Every archive uses the same canonical publication identifier, urn:uuid:08c94180-0adb-5f8b-b3ae-d1a455a9bbca. Filenames distinguish the experimental versions, so the identifier is not an additional difference between archives. The downloadable fixture manifest records archive hashes and changed members, so another analyst can verify which packaged file differs from the control.
Most changes alter one attribute, element, reference or resource. Two are semantic substitutions with necessary accompanying edits: removing table headers also removes attributes that would be invalid on ordinary cells; replacing MathML by an image also removes the manifest declaration that the chapter contains MathML. Those are disclosed operations, not claims that every comparison differs by one byte. A caption-only label in the plan is an alias for the missing-alt case, because the original caption remains in both. It is excluded from the nineteen-archive denominator.
The control had to produce zero reported findings in both tools before derivative comparisons could be interpreted. Two control-only setup attempts were retained: one removed a redundant navigation attribute rejected by EPUBCheck; another supplied the OPF package language Ace requested. Neither changed the selected derivative cases after seeing their outcomes.
A missing file belongs at the package check
Deleting map.png while preserving its manifest entry and image reference produced an EPUBCheck RSC-001 error. Ace completed a report for that same archive without a violation. That result is especially useful for a production checklist: a completed accessibility report did not replace a resource-integrity check, even on a very small publication.
The appropriate first repair is concrete. Restore the file, correct the reference, or remove the unused manifest and content entries as appropriate; then rerun both tools on the new export. A designer should not begin by rewriting the image description when the immediate defect is that the referenced resource is absent. Likewise, a production system should not treat a successfully written Ace report as proof that every required asset exists.
This experiment does not establish that Ace always overlooks missing assets. It establishes what these pinned versions reported for this exact deleted PNG fixture. Different content, engine revisions and failure combinations can produce other results. Keeping the archive and diagnostics available makes that limited statement useful and testable without converting it into a universal tool claim.
Language syntax and language truth need separate attention
Removing both language attributes from the first chapter produced an Ace html-has-lang finding and no EPUBCheck diagnostic. Changing those attributes to the malformed value en_US produced findings in both: EPUBCheck RSC-005 and Ace html-lang-valid. These cases route an author toward the document declaration rather than the prose itself.
The third language case was different. We changed the first chapter declaration from English to the syntactically valid French tag fr while keeping the English prose unchanged. Both tools reported no diagnostic. A tag can satisfy a format check without accurately describing the words it labels. This is a particularly relevant export check for authors maintaining several language editions from related templates.
The practical review is to open representative files from each final language edition and compare the declarations with the actual text, including any passages in another language. The experiment did not measure pronunciation, screen-reader behavior or comprehension. It shows that checking whether a declaration exists and is well formed answered a different question from checking whether its language was truthful in this controlled example.
Contents checks catch contradictions, but cannot choose the intended scene
Changing the first contents link to a nonexistent fragment produced EPUBCheck RSC-012 and Ace epub-toc-order. Reversing the two chapter entries in the spine while leaving the contents unchanged produced EPUBCheck NAV-011, a warning, and Ace epub-toc-order. These are useful consistency findings. They identify a relationship that deserves investigation before distribution.
The warning-only spine case also explains why a workflow must retain severity. EPUBCheck's command returned zero for that case, while its report contained NAV-011. Counting only nonzero exits would miss a disclosed warning. A publisher can decide how to review warnings, but it needs to read or store them before making that decision. A warning is neither an error nor an empty report.
We also changed the Arrival contents entry to the existing Departure destination while preserving the visible Arrival label. That case produced no diagnostic in either tool. The destination existed, but it was the wrong destination for the editor's stated intention. This is a direct reason to click the final contents entries and verify their landing points, rather than inspecting only whether a link resolves.
The reversed-spine result should not be inflated into a claim that the tools understand narrative chronology. This test left the contents and spine inconsistent, and both tools noticed that inconsistency. We did not test a fully consistent rewrite of both structures into the wrong story sequence. A human still needs to know whether arrival should precede departure; a structural comparison cannot establish the intended plot on its own.
A caption did not replace a missing image alternative
The control's map has a meaningful alt value and an adjacent caption. Removing only alt, with that caption left in place, produced Ace image-alt and no EPUBCheck diagnostic. The paired example is useful because an author can see a descriptive sentence below an illustration and assume that the image itself has already been handled. In this fixture, the visible caption did not make Ace accept the absent attribute.
Two other edits were silent. Replacing the meaningful alternative with map.png produced no finding; changing it to an empty value also produced no finding. The image and its informational role did not change. These outcomes separate three review questions: whether an alternative is provided, whether its words convey the image's purpose, and whether the decision to mark an image decorative is appropriate.
An empty alternative can be an intentional choice for a decorative image, so the result should not be described as proof that empty alt is always wrong. Here the original map was deliberately informative, and the mutation removed its description without changing that role. The author or editor must make that content judgment. Neither a filename nor an empty string should become an automatic substitute for reviewing what an image contributes.
Ace also included the image alternative and caption in its report's image inventory. That inventory can support a deliberate review of the relationship between an illustration, its caption and its alternative. Our measured endpoint remains the reported diagnostic, not whether a particular assistive reading setup would present the image satisfactorily. That would require a different test with a declared reader and device protocol.
Heading presence differs from heading appearance
Changing the first chapter's h2 subheading to h4 produced Ace heading-order and no EPUBCheck diagnostic. An author investigating that result should check the document outline and the export template that assigned heading levels. Correcting the template may prevent the same skipped level from recurring in later exports.
Replacing that same h2 with a styled paragraph, while preserving its words, produced no diagnostic in either tool. The paragraph was styled as a subheading through its class and styling, but it no longer contributed that heading to the semantic outline. We did not perform a visual-equivalence or reading-system test; the relevant controlled difference was the element's role in the document.
The author-facing action is to compare the intended section list with the exported outline. A book can have an orderly set of remaining headings and still omit a section that should have been a heading. The experiment supports checking both skipped levels and missing intended entries. It does not estimate how frequently word processors or ebook converters introduce either problem.
Tables reveal why small fixtures need precise limits
Changing one data cell's headers reference to absent produced EPUBCheck RSC-005 and Ace td-headers-attr. This comparison supplied a specific broken relationship that both tools could identify. The repair should address the table's actual association between data and heading cells, followed by a fresh export and check.
Removing all four header cells' heading semantics, converting them to ordinary data cells and removing the corresponding header-related attributes, produced no diagnostic in either tool. The fixture remained a small three-row, two-column table. Its original row and column labels still existed as text, but their header roles had been removed.
This result is deliberately bounded. It does not show that Ace cannot detect header problems in other tables, nor that every table needs the same markup. Layout tables, larger data tables and more complex relationships require their own interpretation. An author with tables should inspect the intended meaning of each row and column rather than infer semantic completeness from this small table's empty diagnostic result.
The combination is the useful observation: a dangling header reference was detected, while the complete absence of those declared relationships in this particular small table was silent. For a checklist, reviewing only reported broken references leaves a different question unanswered—whether the relationships that ought to exist were created in the first place.
Equations require correspondence review
The control contains the equation two plus three equals five in semantic MathML with a matching spoken label. Changing only that label to say six instead of five produced no diagnostic in either tool. Both the equation and the label remained present, but their meanings no longer matched.
A separate declared representation substitution replaced the MathML with an original image carrying the weak alternative equation.png. Removing the now-unused MathML manifest property was part of that substitution. Again, neither tool reported a diagnostic. A nonempty alternative did not establish that the mathematical information had been preserved for a reader using the alternative.
These two cases tell authors of mathematical or technical material to compare the expression, its label and any image alternative directly. They do not demonstrate a particular reader's success or failure with MathML, and they do not rank image equations against MathML across reading systems. The measured result is narrower: these altered representations and descriptions did not trigger the pinned tools' diagnostics in the chosen fixtures.
Low contrast and an empty link reached Ace
Changing the body text color from #111111 to #eeeeee on a white background produced Ace color-contrast findings, while EPUBCheck reported none. Removing the visible words from the final return link produced Ace link-name and no EPUBCheck diagnostic. These are additional examples of problems that reached the accessibility-oriented check after the publication-format check remained quiet.
The low-contrast export generated several assertion instances because the changed stylesheet affected multiple rendered elements. Those instances are not independent books or independent experimental interventions. The outcome matrix counts that archive once. Reporting the number of assertion instances as a number of distinct production failures would exaggerate the evidence and obscure the single underlying color change.
Neither case establishes how every reading app applies user themes, font settings or contrast adjustments. It records the engine's response to the packaged default styling and link content. Authors should preserve those distinctions when deciding whether to repair the source stylesheet, the link label or both before producing another distribution file.
A successful command can still contain findings
Every default Ace invocation in the experiment returned exit code zero, including the nine changed archives with violations. We then reran the exact missing-alt archive with the documented --exiterror2 option. The strict invocation returned two and retained the image-alt finding. The file had not been repaired between the runs; only the command's failure-signalling mode changed.
This is an actionable workflow distinction, not a newly discovered undocumented defect. Ace's official command-line documentation explains that the default exit codes report execution success rather than publication violations, and it documents the strict option. Our paired run demonstrates the consequence in a reproducible ebook case. An automated handoff that reads only the default return code can complete successfully while forwarding a book with reported violations.
A useful release process stores the reports, checks that processing completed, distinguishes violations from execution failures, and uses a deliberately chosen stop rule. EPUBCheck warnings need explicit review as well. Authors who use a graphical app should inspect its report; authors who automate a build should verify what the command actually signals. Neither workflow should turn a zero return code into an unqualified accessibility claim.
A practical preflight order and the boundary of this benchmark
The measured results support a short sequence. First run EPUBCheck and resolve missing resources, malformed declarations, broken fragments and other reported format problems. Review its warnings. Then run Ace, inspect its reported violations and confirm that any automated stopping rule uses the intended failure mode. After repairs, regenerate the final archive and rerun both, because an export can reintroduce a defect.
Then review content correspondence deliberately: declarations against language, contents labels against destinations, the story against its intended reading order, intended sections against the outline, informative images against their alternatives, table meaning against header relationships, and equations against descriptions. Those actions are derived recommendations from the observed gaps, not measured human test outcomes. Screen-reader checks, reading-system testing and formal conformance evaluation remain outside this experiment.
The tools were pinned to the current official releases verified on 31 October 2026. The dependency lock and engine-version record identify the actual runtime. All fixture assets were embedded locally; an application transport guard blocked ordinary external HTTP and WebSocket requests and disabled background-networking features. No attempted external requests were logged during the benchmark. This was a guarded local test, not an independently verified operating-system airgap.
DAISY's Ace tests, W3C's EPUBCheck tests and the published accessibility guidance are important prior work. We did not copy their publications or claim to invent the distinction between automated and human evaluation. The original contribution is the openly reproducible paired archive set, exact diagnostic matrix and repair-routing examples for a small author workflow. The deliberately chosen cases do not certify a book, establish WCAG conformance, guarantee distributor acceptance, or estimate the accessibility of the ebook market.
Sources and reproducibility
Official releases: EPUBCheck 5.4.0 and Ace 1.4.6. Operational behavior: Ace command line and configuration. Evaluation limits: DAISY's Ace guidance and EPUB Accessibility 1.1. Prior test work: Ace tests and EPUBCheck tests.
Download the frozen plan and corrections, original fixture bundle, member-hash manifest, outcome table, detailed diagnostics, engine versions, dependency lock and reproduction scripts with this study. The result table can be recalculated from the saved reports without downloading tools. Re-executing the benchmark requires the declared official tools and their retained licence notices. Credit the original fixtures and analysis to Alisson Bento and the checkers to their respective maintainers.
Which check reported a diagnostic?
Which check reported a diagnostic? 18 deliberately chosen changes; one original control omitted. EPUBCheck: 5; Ace: 9; Neither tool: 8.
Full chart and source
Scroll horizontally to inspect the chart. ↔
Chart and values
Scroll horizontally to see all columns.
| Original controlled case | EPUBCheck messages | Ace rule diagnostics |
|---|---|---|
| Unchanged control | — | — |
| Missing document language | — | html-has-lang |
| Valid tag, wrong language | — | — |
| Malformed language tag | RSC-005 | html-lang-valid |
| Missing TOC fragment | RSC-012 | epub-toc-order |
| Valid but wrong TOC target | — | — |
| Reversed spine, unchanged TOC | NAV-011 (WARNING) | epub-toc-order |
| Missing image alt; caption kept | — | image-alt |
| Image alt = filename | — | — |
| Informative image alt empty | — | — |
| Skipped heading level | — | heading-order |
| Heading replaced by paragraph | — | — |
| Table header roles removed | — | — |
| Invalid table headers reference | RSC-005 | td-headers-attr |
| Low default contrast | — | color-contrast |
| Empty link label | — | link-name |
| Missing PNG resource | RSC-001 | — |
| MathML replaced by weak image | — | — |
| Wrong mathematical label | — | — |
A dash means no reported diagnostic. NAV-011 is warning-only. Counts concern chosen archives, not published-book rates.
Different checkpoints, different findings
Ace only: 5; Both tools: 4; EPUBCheck only: 1; Neither tool: 8. 18 deliberately chosen changes; one original control omitted
Full chart and source
Scroll horizontally to inspect the chart. ↔
Chart and values
Scroll horizontally to see all columns.
| Reported by | Changed archives |
|---|---|
| Ace only | 5 |
| Both tools | 4 |
| EPUBCheck only | 1 |
| Neither tool | 8 |
A dash means no reported diagnostic. NAV-011 is warning-only. Counts concern chosen archives, not published-book rates.
Data & downloads
Scroll horizontally to see all columns.
| Original controlled case | EPUBCheck messages | Ace rule diagnostics |
|---|---|---|
| Unchanged control | — | — |
| Missing document language | — | html-has-lang |
| Valid tag, wrong language | — | — |
| Malformed language tag | RSC-005 | html-lang-valid |
| Missing TOC fragment | RSC-012 | epub-toc-order |
| Valid but wrong TOC target | — | — |
| Reversed spine, unchanged TOC | NAV-011 (WARNING) | epub-toc-order |
| Missing image alt; caption kept | — | image-alt |
| Image alt = filename | — | — |
| Informative image alt empty | — | — |
| Skipped heading level | — | heading-order |
| Heading replaced by paragraph | — | — |
| Table header roles removed | — | — |
| Invalid table headers reference | RSC-005 | td-headers-attr |
| Low default contrast | — | color-contrast |
| Empty link label | — | link-name |
| Missing PNG resource | RSC-001 | — |
| MathML replaced by weak image | — | — |
| Wrong mathematical label | — | — |
A dash means no reported diagnostic. NAV-011 is warning-only. Counts concern chosen archives, not published-book rates.
Scroll horizontally to see all columns.
| Reported by | Changed archives |
|---|---|
| Ace only | 5 |
| Both tools | 4 |
| EPUBCheck only | 1 |
| Neither tool | 8 |
A dash means no reported diagnostic. NAV-011 is warning-only. Counts concern chosen archives, not published-book rates.
Evidence & method
How we calculated it
- Before running the changed archives, we froze the case list, hypotheses, admission rule and machine-observable endpoints. Every archive uses the same canonical publication identifier, urn:uuid:08c94180-0adb-5f8b-b3ae-d1a455a9bbca. Filenames distinguish the experimental versions, so the identifier is not an additional difference between archives. The downloadable fixture manifest records archive hashes and changed members, so another analyst can verify which packaged file differs from the control.
- Most changes alter one attribute, element, reference or resource. Two are semantic substitutions with necessary accompanying edits: removing table headers also removes attributes that would be invalid on ordinary cells; replacing MathML by an image also removes the manifest declaration that the chapter contains MathML. Those are disclosed operations, not claims that every comparison differs by one byte. A caption-only label in the plan is an alias for the missing-alt case, because the original caption remains in both. It is excluded from the nineteen-archive denominator.
What the data cannot tell us
- The tools provided different information. Five changed archives produced diagnostics only in Ace, one only in EPUBCheck, and four in both. Eight produced no reported diagnostic in either tool. These are counts of chosen experimental cases, not failure rates for books, an estimate of tool sensitivity, or evidence that eight changes would be harmless to readers. The useful result is a repair map: missing files, absent attributes, contradictory links, weak descriptions and missing semantics do not all reach the same checkpoint.
- DAISY's Ace tests, W3C's EPUBCheck tests and the published accessibility guidance are important prior work. We did not copy their publications or claim to invent the distinction between automated and human evaluation. The original contribution is the openly reproducible paired archive set, exact diagnostic matrix and repair-routing examples for a small author workflow. The deliberately chosen cases do not certify a book, establish WCAG conformance, guarantee distributor acceptance, or estimate the accessibility of the ebook market.
- The results apply only to the pinned EPUBCheck 5.4.0 and Ace 1.4.6 releases, to one original two-chapter fixture and to one execution of each archive. They do not generalise to other tools, versions, content or reading systems, and no human, screen-reader or distributor testing was performed.
Original sources
- EPUBCheck 5.4.0 release
Official pinned EPUBCheck release and scope.
Accessed October 31, 2026
- DAISY Ace 1.4.6 release
Official pinned Ace release; runtime separately recorded.
Accessed October 31, 2026
- Ace command-line documentation
Documented ordinary and strict exit behavior, demonstrated in a paired run.
Accessed October 31, 2026
- Ace configuration documentation
Default return-2-on-validation-error is false.
Accessed October 31, 2026
- DAISY Ace evaluation guidance
Automated evaluation limits and separate checks.
Accessed October 31, 2026
- EPUB Accessibility 1.1
Accessibility conformance context; no certification claimed.
Accessed October 31, 2026
- Prior work: Ace test suite
Existing Ace test suite credited; no test publication copied.
Accessed October 31, 2026
- Prior work: EPUBCheck test suite
Existing EPUBCheck test suite credited; original workflow cases are separate.
Accessed October 31, 2026
Cite & reuse this study
Alisson Bento. Which EPUB Preflight Step Catches Which Failure? Original paired-fixture workflow benchmark, 2026. https://alissonbento.com/research/epub-preflight-paired-workflow-benchmark-2026/
Reuse the original fixtures and analysis with credit to Alisson Bento; credit DAISY and W3C for their respective tools and prior tests. Retain versions, chosen-case scope and non-certification limits. Preserve software notices.
Corrections or editorial questions