Embedded Indexing Survey Results (LONG, 242 lines)

["Nancy C. Mulvany" <nmulvany -at- WELL -dot- SF -dot- CA -dot- US>]

Background
   In December 1993 I posted a Request for Comments (RFC) about
   embedded indexing software on three discussion groups on the
   Internet. Specifically, I asked if people were satisfied with
   the embedded indexing tools that they used. I asked people to
   indicate what tools they used and to indicate any features they
   would like to see added.

   The RFC was posted at the following locations:
       Indexing Conference on The WELL
       INDEX-L -at- BINGVMB -dot- BITNET
       TECHWR-L -at- VM1 -dot- UCC -dot- OKSTATE -dot- EDU
       FRAMERS -at- UUNET -dot- UU -dot- NET

   The RFC was re-posted by others on other lists. People were
   asked to respond during Dec. 9-31, 1993. During this period I
   received 85 comments about embedded indexing tools.

Disclaimer
   Absolutely no effort was made to conduct a "scientific"
   survey. Discussions about whether or not these results reflect
   a fair sample of the population that uses embedded indexing
   software are pointless. The fact is that during a typically
   busy time of the year (the Christmas holiday season) I
   received approximately 185 Kb of comments from people who are
   users of this type of software. The results overwhelmingly
   confirm my own experiences with users of this software.

Results
   Question: Are you satisfied with embedded indexing software
   that you have used?

     5% are satisfied (n=4)
     95% are not satisfied (n=81)

   Satisfied respondents are those who said they were satisfied
   and/or had no suggestions for improvements in the product(s).
   Not satisfied respondents are those who said they were not
   satisfied and/or had suggestions for improvements in the
   products.

Products Represented
   The number following the product name indicates the number of
   comments received about the product.

   Satisfied with the Embedded Indexing Module (n=4; 5%)
     FrameMaker (1)
     IBM BookMaster (2)
     WordPerfect (1)

   Not Satisfied with the Embedded Indexing Module (n=81; 95%)

     FrameMaker (27)*
     IBM BookMaster (1)
     Interleaf (8)
     LaTeX (1)
     Microsoft Word (15)
     PageMaker (3)
     Ventura Publisher (4)
     WordPerfect (9)
     WordStar (2)
     XyWrite (1)
     Product Not Specified (10)

   * It should be noted that the relatively large number of
   responses about FrameMaker are very likely due to the fact
   that the RFC was posted to a list devoted to FrameMaker
   discussion which of course biased the number of responses
   about FrameMaker.

Respondents' Suggestions for Improvements
   Many of the respondents included thoughtful comments about
   features they would like to see added or improvements they
   would like to see made in embedded indexing software. It is
   significant that so many respondents took time to compose
   these comments. This indicates to me a serious concern on the
   part of users of this type of software.

   While some of the comments are specifically related to a
   particular product, many other comments are universal and can
   applied to any tool used for indexing electronic material. I
   will present the general comments that I consider to be of
   interest universally. Of course, not all comments apply to all
   products. I have listed suggestions that for the most part
   were suggested by more than one respondent.

o  Index entry methods
   Many find the methods used to create index entries to be
   tedious and time consuming. Some prefer a "fill-in the fields"
   methods as seen in Ventura's dialog box or Interleaf's
   "Levelx" sheet. Others want a method for creating index
   entries without having to remove their hands from the
   keyboard.

o  Index display
   Many are frustrated by the not being able to see index entries
   that have been created previously. Some products allow the
   writer to see the last few entries created, many would like to
   be able to see all entries already created. One user comments,
   "PageMaker's indexing at least lets you see other entries in
   the same alphabetical range you have made, but forces you to
   break your index editing task into two modes: editing already
   existing entries, and then going back and adding tags for new
   entries. It is really slow."

o  Index editing
   Many, many comments were devoted to the index editing task.
   Since many programs generate the index as a static document,
   editing changes must be made in the text files by locating
   individual tags/index tokens, changing them, and regenerating
   the index.

   Suggestions for improvement include:
   The generated index should be dynamic document that is linked
   to the index tags. Changes made in the index file should be
   updated in the text file tags. As noted above, this feature is
   available in PageMaker; it is also available in Describe, a
   product not mentioned in this survey. It should be possible to
   spell-check the embedded index entries and globally change
   entries and their tags.

o  Citing continuous discussion of a topic
   Many complain about the methods used for citing the range of
   pages on which a topic is discussed. In some programs the
   method is so tedious that writers do not attempt to provide
   page ranges (e.g, files, copying, 5-9). Essentially people
   want an "easy" way to indicate that a discussion begins right
   here and ends there.

o  Increase the size of the index window/panel/dialog box

o  Sorting: Better algorithms that allow index entries to be
   sorted correctly; faster; ignore leading function words in
   subentries

o  Cumulative Indexes: should be easy to merge multiple
   indexes

0  Reference Locators: Ability to properly handle modular page
   numbers (e.g., 3-2) and lettered appendix page numbers (e.g.,
   A-3); reminder of incomplete range references (missing
   startrange or endrange string)

o  Index Text Formatting: Ability to retain text formatting in
   the document in the index entry. Ability to set pre-defined
   styles, such as all main headings will be capitalized.

o  Index Document Formatting: automatically take care of bad
   breaks with "continued" phrases; widow/orphan control; more
   control over the final index format

o  Attach a Thesaurus for Vocabulary Control

o  Pick entries from a list of previous entries

o  Quickly be able to invert entries (i.e., make the subentry a
   main heading and the main heading a subentry)

o  Easily repeat any heading level instead of re-typing

o  Interactively display all entries with specified string of
   text

o  Cross-references: Easy insertion of cross-references; specify
   their position with the entire entry and for subentries;
   cumulate multiple cross-references so that they do not appear
   separately

o  Automatic Text Processing Tools: Generate a concordance that
   the writer can edit to create the index. It is interesting to
   note that most writers requesting this feature insisted on
   human intervention, noting that a concordance is not an index.
   One summed up the matter by writing, "The biggest problem with
   a scheme like this one, though, is the overwhelming potential
   to implement it poorly."

o  Display embedded entries in the document in a clear way

Conclusion
   This report is already getting too long for general
   circulation via Internet discussion groups, so I will be
   brief. First, I want to thank everyone who contributed
   comments. Second, I want to say that you have verified the
   work I have doing in this area since 1988.

   The features and functionality you request are all "doable".
   Many of these features have been incorporated into standalone
   indexing software for the past 11 years. My company publishes
   standalone indexing software (Macrex) for IBM PCs. This
   software is primarily used by indexers indexing from printed
   pages of text; it does not embed tags in text files. Users of
   this type of software work with their index in alphabetic
   order. They are able to reduce keyboarding significantly
   through the use of dedicated commands for this purpose.
   Because you can see exactly where an entry will go, it is
   possible to significantly reduce inconsistencies.
   Cross-references are verified instantly so that you will know
   if you create a cross-reference for an entry that does not
   exist. It is possible to group together entries that contain
   specified strings of text. This way the indexer can see all
   entries for a particular phrase; both main headings and
   subentries will be displayed.

   It is my opinion that the design of embedded indexing tools
   has been "wrong-headed" from the start. The context has been
   wrong. The context has been the document being indexed rather
   than the index being written. With many of the these programs
   you are indexing in the dark. You cannot easily see the
   emerging index structure. Imagine your word processor causing
   each sentence to disappear as soon as it is ended. As you work
   you are unable to see the sentence in context with other
   sentences or paragraphs. An overhaul of the user interface
   design of embedded indexing software is called for.

   Problems with incorrect alphabetization, reference locator
   handling, formatting of bad breaks with indexes are due to
   sloppiness. There's no excuse for these problems. We solved
   them with the CP/M version of our software during the days
   when computers came with 64Kb of memory!

   Designers of embedded indexing software would do well to
   consult the standard references in indexing (Chicago Manual of
   Style, 14th edition; Indexing Books, Nancy Mulvany, Univ. of
   Chicago Press, 1994; British Standard 3700:1988). There are
   rules and guidelines for presentation of indexes that have
   been widely ignored by developers. It would also be helpful if
   designers spoke with professional indexers and find out how we
   work. Before developers can provide efficient tools they must
   understand the process of creating authored indexes.

A more in-depth discussion of embedded indexing software will be
presented at the 26th Annual Meeting of the American Society of
Indexers in San Diego (May 13-14, 1994). The paper will appear in
the ASI Proceedings for that meeting (available in May 1994).
Contact ASI at: P.O. Box 386, Port Aransas, TX  78373; or via
email at asi -at- well -dot- sf -dot- ca -dot- us

Nancy Mulvany
Bayside Indexing Service
(510) 524-4195
nmulvany -at- well -dot- sf -dot- ca -dot- us