14 October 2007
Human-computer hybrids, in indexing
I recently completed an (as-of-yet unreviewed) article for Information - Wissenschaft & Praxis (IWP), the premier German journal on information science. The topic was the intersection of computer-based indexing and human indexing, and how these two approaches to indexing are unequal but in many ways compatible.
The biggest challenge in writing the article comes from the simple fact that I'm a human indexer, and that I believe that automatic indexing fails every important quality test. On the other hand, since it's unlikely we're going to have people typing away to index the World Wide Web (see my entry "A needle in a haystack with 100,000,000 blades"), it seems we're going to need something faster than human fingers and brains to get the job done.
I'm not going to repeat the article's ideas here, except to say that I tried to give an even-handed view of automatic indexing -- even as I tend to rip it to shreds in this blog when I can. The distinction is that people constantly overestimate when it's necessary. Automatic indexing is overused and misused.
Still, I thought my loyal readers might knowing that even on this, there are two valid opinions.
The biggest challenge in writing the article comes from the simple fact that I'm a human indexer, and that I believe that automatic indexing fails every important quality test. On the other hand, since it's unlikely we're going to have people typing away to index the World Wide Web (see my entry "A needle in a haystack with 100,000,000 blades"), it seems we're going to need something faster than human fingers and brains to get the job done.
I'm not going to repeat the article's ideas here, except to say that I tried to give an even-handed view of automatic indexing -- even as I tend to rip it to shreds in this blog when I can. The distinction is that people constantly overestimate when it's necessary. Automatic indexing is overused and misused.
Still, I thought my loyal readers might knowing that even on this, there are two valid opinions.
Labels: business of indexing, future of indexing, indexing tools, keywording
17 March 2007
Notes on automatic indexing
"Automated indexing software" is, according to the common definition, software that analyzes text and produces an index without human involvement. I'm a firm believer that the technology doesn't exist, and that a human being is required to write an index. Thus I don't use the software, and I also don't recommend it.
There are those who advocate it, arguing that it's "not as bad as an indexer would have you think." These people are often coming from the standpoint that automatic software is faster and cheaper, and they're right. Thus the issue surrounds quality.
I believe that good automatic indexes will exist once there's good artificial intelligence, something that presently doesn't exist. In very limited circumstances, however, it does; a machine can easily cull capitalized words from a textbook to create an approximation of an index of names -- although, again, the machine isn't going to differentiate between names like "David Kelley" and places like "San Francisco," since they are both of the same format and used the same way. It also won't know that "Bill Clinton" is also "William Jefferson Clinton." And certainly it can't tell when the name is being mentioned in an unuseful and trivial way, as are the names in this paragraph! So imagine the problems trying to get a machine to parse full sentences of ideas and recognizing the core ideas, the important terms, and the relationships between related concepts throughout the entire text.
FYI, those who advocate automatic software, however, would argue that the machine gets "close enough" so that a human being can edit the resulting product. However, expert evaluators unanimously agree that the software fails; those who disagree are likely those who are sufficiently ignorant of indexing in the first place such that they are unable to determine the quality differences.
Oh, I should mention that there are software programs that human indexers use to simplify and speed up the mechanics of the index process. For example, it would be silly to disallow a computer to alphabetize the entries, reformat the index, and manipulate page numbers. There are a few software packages that do this exclusively, which are considered top of the line; other applications that have indexing capabilities, such as Microsoft Word and Adobe FrameMaker, have some of these capabilities, with notable limitations.
For information on the various software available, see http://www.asindexing.org/site/software.shtml. If you have feedback, especially differing opinions, I'd love to hear them. Write me at seth@maislin.com.
(This article was originally published in 2002 and 2004 -- and it's still 100% accurate.)
There are those who advocate it, arguing that it's "not as bad as an indexer would have you think." These people are often coming from the standpoint that automatic software is faster and cheaper, and they're right. Thus the issue surrounds quality.
I believe that good automatic indexes will exist once there's good artificial intelligence, something that presently doesn't exist. In very limited circumstances, however, it does; a machine can easily cull capitalized words from a textbook to create an approximation of an index of names -- although, again, the machine isn't going to differentiate between names like "David Kelley" and places like "San Francisco," since they are both of the same format and used the same way. It also won't know that "Bill Clinton" is also "William Jefferson Clinton." And certainly it can't tell when the name is being mentioned in an unuseful and trivial way, as are the names in this paragraph! So imagine the problems trying to get a machine to parse full sentences of ideas and recognizing the core ideas, the important terms, and the relationships between related concepts throughout the entire text.
FYI, those who advocate automatic software, however, would argue that the machine gets "close enough" so that a human being can edit the resulting product. However, expert evaluators unanimously agree that the software fails; those who disagree are likely those who are sufficiently ignorant of indexing in the first place such that they are unable to determine the quality differences.
Oh, I should mention that there are software programs that human indexers use to simplify and speed up the mechanics of the index process. For example, it would be silly to disallow a computer to alphabetize the entries, reformat the index, and manipulate page numbers. There are a few software packages that do this exclusively, which are considered top of the line; other applications that have indexing capabilities, such as Microsoft Word and Adobe FrameMaker, have some of these capabilities, with notable limitations.
For information on the various software available, see http://www.asindexing.org/site/software.shtml. If you have feedback, especially differing opinions, I'd love to hear them. Write me at seth@maislin.com.
(This article was originally published in 2002 and 2004 -- and it's still 100% accurate.)
Labels: indexing process, indexing tools
18 December 2006
"Can I Delete All My ___ Entries in MS Word?"
(This is the newest entry in my MS Word Indexing FAQ.)
Every now and then, there's nothing you want to do more than globally delete a bunch of entries. The problem is how this is supposed to happen. For example, suppose you have a common main entry for "publicity," when you decide that you're better off with a cross reference like "publicity. See marketing." In addition to creating this cross reference, you need to remove all of your original publicity entries. Although you can search for marker text, you can't search for whole markers. In other words, you can search for the word "publicity" when it's used within index markers (look for hidden text), but you can't search for a whole marker like {XE "publicity"} or {XE "publicity:methods for"}. For this reason you can search globally and delete.
The easiest approach to deleting all publicity entries is the manual approach: generate your index, then delete everything that starts with the word publicity. Unfortunately, manual edits will be undone as soon as you generate the index again; you'll have to remember that you want to make these manual changes every time you create a new version of the index. To help you remember to make these manual changes, I recommend changing the format and/or language for the word publicity to make sure it jumps out at you. Search for XE "publicity, the unique text for all publicity entries, and replace it with boldface, all caps, and a shocking color like red. I also recommend that you change the word publicity with something that will sort at the very beginning of your index, such as aaa DELETE ME. Now, when you generate your index, you'll see some red, boldface, all-caps reminder at the top of your index file. Hopefully this will be enough for you to remember deleting your entries.
Another approach, and by far the one I prefer, is to replace the marker syntax with something that Word can't interpret. Instead of using the letters XE in your marker, use something like DELETE_ME. In other words, globally change XE "publicity with DELETE_ME "publicity. Since markers are hidden text, your DELETE_ME markers will remain hidden from publications; further, they'll fail to become index entries since Word won't interpret them as XE markers. The biggest advantage to this method is that it works globally, and you only have to make these changes once. Another advantage is that you aren't actually deleting the entry, just rewriting it; if for any reason you need to reconstruct entries, you can always change DELETE_ME to XE. (This is a kludgy way of creating conditional text, but it might be just what you need.) The disadvantage is that you're not actually deleting anything, potentially cluttering your documentation.
As a side note, whenever you remove an entry from your index, remember that you have to delete any cross references that target those now-removed entries. For example, if you replace your publicity entries with "publicity. See marketing," you'll need to rewrite or delete entries like "public relations. See also publicity."
Every now and then, there's nothing you want to do more than globally delete a bunch of entries. The problem is how this is supposed to happen. For example, suppose you have a common main entry for "publicity," when you decide that you're better off with a cross reference like "publicity. See marketing." In addition to creating this cross reference, you need to remove all of your original publicity entries. Although you can search for marker text, you can't search for whole markers. In other words, you can search for the word "publicity" when it's used within index markers (look for hidden text), but you can't search for a whole marker like {XE "publicity"} or {XE "publicity:methods for"}. For this reason you can search globally and delete.
The easiest approach to deleting all publicity entries is the manual approach: generate your index, then delete everything that starts with the word publicity. Unfortunately, manual edits will be undone as soon as you generate the index again; you'll have to remember that you want to make these manual changes every time you create a new version of the index. To help you remember to make these manual changes, I recommend changing the format and/or language for the word publicity to make sure it jumps out at you. Search for XE "publicity, the unique text for all publicity entries, and replace it with boldface, all caps, and a shocking color like red. I also recommend that you change the word publicity with something that will sort at the very beginning of your index, such as aaa DELETE ME. Now, when you generate your index, you'll see some red, boldface, all-caps reminder at the top of your index file. Hopefully this will be enough for you to remember deleting your entries.
Another approach, and by far the one I prefer, is to replace the marker syntax with something that Word can't interpret. Instead of using the letters XE in your marker, use something like DELETE_ME. In other words, globally change XE "publicity with DELETE_ME "publicity. Since markers are hidden text, your DELETE_ME markers will remain hidden from publications; further, they'll fail to become index entries since Word won't interpret them as XE markers. The biggest advantage to this method is that it works globally, and you only have to make these changes once. Another advantage is that you aren't actually deleting the entry, just rewriting it; if for any reason you need to reconstruct entries, you can always change DELETE_ME to XE. (This is a kludgy way of creating conditional text, but it might be just what you need.) The disadvantage is that you're not actually deleting anything, potentially cluttering your documentation.
As a side note, whenever you remove an entry from your index, remember that you have to delete any cross references that target those now-removed entries. For example, if you replace your publicity entries with "publicity. See marketing," you'll need to rewrite or delete entries like "public relations. See also publicity."
Labels: indexing tools
05 September 2006
Troubleshooting Microsoft Word indexes
I've been archiving various questions people have asked me about Microsoft Word indexes. It's amazing how many things can go wrong, and then can't be easily solved. In response, I've started building a page for troubleshooting Word indexes. You can access that page here:
http://taxonomist.tripod.com/indexing/wordproblems.html
If you're having a problem with Word, first know that you're in good company, and then write me about it.
http://taxonomist.tripod.com/indexing/wordproblems.html
If you're having a problem with Word, first know that you're in good company, and then write me about it.
Labels: indexing tools, Microsoft Word indexing
