Incorporating XML in Documentation

[Jason Willebeek-LeMair <jlemair -at- cisco -dot- com>]

Names have been changed to protect the ignorant.

This describes the XML project of a single documentation
team and does not represent any corporate projects.

I deny everything and accept responsibility for nothing.

Background
----------

This describes an XML solution for a single product. It is not
an ideal solution by any means, but it does demonstrate a low-cost 
alternative to expensive document management systems for small
groups or projects.  The system described here WILL NOT scale
to an enterprise-wide solution.  

It is also a good method for a proof-of-concept or learning 
implementation without shelling out a lot of $$$.

XML Implementation
------------------

Phase I: Defining the Markup and Selecting the Tools

We chose to create our own DTD rather than use an existing one.  

We chose DTD rather than schema because 1) schema had
not yet been finalized, 2) there were several competing 
schema specs (TREX, RELAX, W3C XML Schema, MS Schema), and
3) most importantly, there was not (and still is not)
widespread support for schema.

We went with home brew rather than an existing DTD (such as
DocBook) because our information set was already pretty
well structured and because the existing DTDs did not quite 
meet our needs (they either offered too much or too little).
Besides, it was fun.

<side.note>Six months after we went live, IBM came out with
their DITA architecture, which is remarkably similar to ours
at a high level. So, we like to think that we are as smart 
as IBM.</side.note>

Some tips to remember when creating your own markup:

* Think structure, not format.  Formatting is handled by 
another process.  This has been the most difficult part 
of our system for our writers to grasp.  They keep wanting
to see how things will look, which is impossible (after 
all, our hardcopy style is much different than our online
style). 

* Think semantics.  What role does the markup play?  Is it
a menu, a command, a button, a step, an item in a list, etc.

* Think about tag names.  You may be tempted to abbreviate 
tag names, and it may be good to do so.  On the other hand,
if you are anticipating getting people up to speed quickly, 
you may not want to.  For example, which is more explanatory:
tl or tasklist?  Also think of the authoring environment; 
most XML authoring tools allow you to pick tags like you 
paragraph styles in Frame or word, so having longer names 
does not burden the writer.  On the other hand, if you are 
authoring in NotePad or vi, you may want to abbreviate the 
names to save wear and tear on the writer's fingers and 
keyboards.

Our XML information architecture has three layers. 
The top layer has four DTDs: Book, HTML Help, WinHelp, 
and Workbook (we'll get to workbook later).  They each 
correspond to the type of deliverable.  They also organize
our topics slightly differently.  For example, the 
book DTD has chapters and only allows nesting of topics 
within the chapters to go 4 deep.  The HTML Help DTD 
contains help project information and allows infinite 
nesting of our topics (as does the help).

These four layers rest on top of the real meat of the 
architecture -- the topics.  We defined a topic as a heading
plus the text after that heading.  This is basically our 
smallest reusable chunk of information, aside from some 
boilerplate text and warnings. We have several types of 
topics: Procedurebox, Conceptbox, Taskbox, Checklist, and
Fieldbox. Each topic has a unique content model.  For example,
a procedurebox contains a Title, an Intro, and a Procedure. The
procedure contains one or more steps, which in turn can have
actions, notes, warnings, cautions, results, etc.

Underlying the topics is our common markup.  Typical stuff, 
like paragraphs, lists, inline markup, etc.

With this architecture, we can quickly create new top-level 
forms (java help, man pages, etc) for new types of 
deliverables or insert a new information type (such as a 
glossary) into the architecture without too much fuss (i.e.
new work and re-work).

Tools were another matter. Open source XML processors are 
pretty abundant.  This was important, because this started 
out as a side project with no budget.  We used Saxon
(http://users.iclway.co.uk/mhkay/saxon/instant.html) and 
XALAN (http://xml.apache.org/xalan-c/index.html) at the 
beginning, because they were free, good, and 
compliant.  We have added MSXML 
(http://msdn.microsoft.com/xml/default.asp) to our bag of 
tools also, since their latest release is compliant to the 
specs, fast, and has a command-line utility (important for 
us--I will get to that in the output section).  Since our budget 
was US$0.00, NotePad was our editor. You really do not need more 
that that to create a DTD, XML, or XSLT file, but I eventually
upgraded to a US$25.00 copy of TextPad (http://www.textpad.com/).  

Phase II: Testing

Naturally, we tested the markup and the process of converting
our HTML source to XML.  Found a lot of bugs and overlooked 
issues. Always test before you convert.

Total Project Cost So Far: US$25.00 (Don't worry, it gets more 
expensive later)

Phase III: Conversion of Existing Source

Since all of our source was in pure HTML, we used HTMLTidy 
(http://www.w3.org/People/Raggett/tidy/) to convert the
existing information source to XML. Then, where possible, 
we used an XSLT transform to convert it to an approximation
of our markup.  

This is an important point -- we could not transform directly 
to our markup because our XML is semantically richer than our
HTML (despite the liberal use of Class attributes).  For 
example, items marked as <b></b> in the HTML may be converted
to <menu></menu> or <field></field>.  These items required manual 
cleanup.  

I suppose we could have been clever and written a script that 
examined the contents of the tag and applied the appropriate 
XML tag, but we were not clever at that time.

For the cleanup work, we assigned all of the writers a set of 
files to work on and gave them a month. We used a commercial 
authoring tool (XMetaL was our tool of choice, but there are
other commercial editors available that are arguably just as 
good or even better--before selecting an authoring tool, ask 
for demo copies and give them all a try).  XMetaL gave us on-
the-fly tag choosing and validation, so that the authors could
know when their files were valid. (http://www.xmetal.com)

Total Project Cost So Far:  In the 4-digit range (buying 11 copies
of the authoring tool).

Phase IV: Authoring

Authoring in a flat-file, rather than documentation database-
driven environment, posed some unique challenges due to the 
nature of XML.

For example, we do not have a doc database that can 
automatically chunk a chapter into its component information 
types. We can configure our Authoring tool to do this, but 
have not had time. So, instead, we create the chunks and then
collect them in the higher-level document.

The problem with this approach is that the chunks, or XML 
fragments, cannot have a document type declaration in them 
(see production 78 of the XML Specification--
http://www.w3.org/TR/2000/REC-xml-20001006). So, you cannot
edit them directly in the authoring tool and have the nifty
dynamic tag chooser (which gets its information from the DTD).

To work around this problem, we created templates for all of 
the information types and an artificial top-level document 
type called a WorkBook.  When the author needs to create new 
information types, they use the templates to create the new 
files.  Then, they add the new files to the WorkBook (as 
referenced, external entities) which contains a document type 
declaration (and provides the authoring tool the DTD from 
which to extract the tag-choosing information). They can then
open the fragments from within the workbook file and edit away.

It sounds more complicated than it is in practice.

An additional benefit to the WorkBook is that the writers now
have a logical grouping of XML fragments.  They can also swap
WorkBook files for editing or information sharing.

Total Project Cost So Far:  Quadruple Digits (for tools) + Time

Phase V: Output

This is perhaps the most asked-about part of XML authoring.
How do I get a book/help system/web page/whatever out of it
once I have authored it.

All of our output is generated using XSLT.  We have XSLT files 
for: HTML and HTML Help generation, RTF for What's This help 
generation, and MIF for FrameMaker (hardcopy) generation.

The Help (HTML and What's This) are kicked off automatically from 
a build script (which is why command line support was important when
choosing the tools).  Hardcopy generation is currently manual.

As an example, our main HTML Help system is a single XML file with 
all of the help topics included as references to the external 
entities.  The build script first parses that file and creates a help
project file.  Then it parses it and generates the TOC.  Then, 
it parses the file a third time and generates all of the HTML files
for the help system.  Finally, the build script kicks off HTML Help 
Workshop and generates the CHM file, nicely plopping it in the 
appropriate build directory.

Similarly, the build script picks up the WinHelp XML file (based on 
the WinHelp DTD that we defined) and creates the What's This help project
file and RTF file, generates the help, and plops it in the appropriate 
build directory.

For hardcopy, we manually kick off the build because we do not rev those
as often.

Now, you may be thinking that this is nothing that FrameMaker, conditional
text, and WWP can't handle.  And you would be right if you think of it 
as a book to help process.  But our HTML Help contains more information 
than our books -- our books are just targeted subsets of the Help
information.
Some of that information may be reused across the books (yes, books, it is 
a rather large product) or reused across the HTML Help and What's This help.
Being in XML makes it easy to reuse that information. 

We can also reorganize our books in a matter of minutes simply by 
rearranging, adding, or deleting entities.  We can quickly create a new 
book about a specific subject by collecting and arranging those 
entities in an XML document, thus targeting a particular user or feature
of our application. And finally, the ultimate goal is to allow the user to 
create their own books, either by on-the-fly construction based on the 
features they ordered or by selecting topics from a web form.

There are countless other benefits, harped on ad nauseum on the web.

Total Project Cost So Far:  Quadruple Digits (for tools) + Time

So, you can create an XML solution without an expensive document management
database or workflow utility.  But there are tradeoffs, such as 
scalability. However, for testing the feasibility of such a system, it is 
a good compromise between functionality and cost.

XSL-FO -- A Look Towards the Future
-----------------------------------

Although in the planning stages, we are thinking about using 
FO technology to bypass FrameMaker and go directly to 
hardcopy (PDF or PS).

Our actual implementation will depend upon what we find 
during our trials.  We will probably use FOP or XEP during
our development.  FOP (http://xml.apache.org/fop/index.html) 
is free and XEP (http://www.renderx.com/FO2PDF.html) offers 
a free trial.

Recommended Reading
-------------------

These books are listed in order from basic to advanced.

The XML Handbook, Third Edition
Charles Goldfarb and Paul Prescod

For those of you who are new to XML, this book provides an excellent
introduction to the vocabulary, concepts, and markup.  Also includes
information about XSLT and real-life scenarios where XML is used in 
B2B, publication, and applications.

Structuring XML Documents
David Megginson

Provides information about DTD syntax and includes 5 industry DTDs to 
learn from. A must for those who are going to write their own DTD 
or modify an existing one to meet their needs.

XML: The Annotated Specification
Bob DuCharme

An invaluable resource once you get past the basics.

XSLT: Programmer's Reference
Michael Kay

Once you master the basics of XSLT, you will need something heavier.
This is one of the best XSLT references that I have found.  It is 
full of examples.


--------------------------------------------------------------
Standard Weasel Disclaimer
--------------------------------------------------------------
All opinions and plans are strictly my own and in no
way represent the opinions or plans of my company.

AKA: I may be full of s**t.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

*** Deva(tm) Tools for Dreamweaver and Deva(tm) Search ***
Build Contents, Indexes, and Search for Web Sites and Help Systems 
Available 4/30/01 at http://www.devahelp.com or info -at- devahelp -dot- com

Sponsored by DigiPub Solutions Corp, producers of PDF 2001 Conference East, 
June 4-6, Baltimore, MD. Now covering Acrobat 5. Early registration deadline 
April 27. http://www.pdfconference.com. 

---
You are currently subscribed to techwr-l as: archive -at- raycomm -dot- com
To unsubscribe send a blank email to leave-techwr-l-obscured -at- lists -dot- raycomm -dot- com
Send administrative questions to ejray -at- raycomm -dot- com -dot-  Visit 
http://www.raycomm.com/techwhirl/ for more resources and info.