Seth Maislin's Indexing Blog

Human-computer hybrids, in indexing

2007-10-14T20:58:00.000-05:00

I recently completed an (as-of-yet unreviewed) article for Information - Wissenschaft & Praxis (IWP), the premier German journal on information science. The topic was the intersection of computer-based indexing and human indexing, and how these two approaches to indexing are unequal but in many ways compatible.

The biggest challenge in writing the article comes from the simple fact that I'm a human indexer, and that I believe that automatic indexing fails every important quality test. On the other hand, since it's unlikely we're going to have people typing away to index the World Wide Web (see my entry "A needle in a haystack with 100,000,000 blades"), it seems we're going to need something faster than human fingers and brains to get the job done.

I'm not going to repeat the article's ideas here, except to say that I tried to give an even-handed view of automatic indexing -- even as I tend to rip it to shreds in this blog when I can. The distinction is that people constantly overestimate when it's necessary. Automatic indexing is overused and misused.

Still, I thought my loyal readers might knowing that even on this, there are two valid opinions.

External deadline forces

2007-09-11T11:33:00.000-05:00

"I need this book in hand by Friday because..."

Outside of the natural production process there are, definitely, many different kinds of external circumstances that impact the timing and schedules of indexing. Many are sector- or medium-specific, but of course there are indexers who work among several of these and thus feel the impact all year. Here are some examples that I know:

Textbooks that are used in American public schools tend to appear in time for the Texas and California state adoption processes. If a book isn't published on time to be reviewed by the school officials in these states, it's unlikely that the book will be used in public schools at all.
College textbooks need to be on the shelves in time for traditional semester beginnings, in September and January.
Books that are budgeted for one year are pushed to get finished during that budget (fiscal) year, to avoid (a) losing the opportunity to spend money already allocated for the publishing process, and (b) spending money needed in the next year. This impacts the indexers around U.S. Thanksgiving.
Software books targeted toward the general public need to be first to market to catch the wave of early sales; these schedules are irregular but can be predicted by looking at the various technologies that are coming out. For example, we're still near the beginning of the Windows Vista wave, since the new operating system was only recently released.
General-readership books based on cultural events (news items, holidays, anniversaries) are similar to software books, in that being first to market matters equally.
Politics is a special kind of cultural event in that it's ongoing. Books on politics tend to appear in advance of events that are potentially influential in the political world. For example, books about presidential candidates tend to appear in parallel with their campaigns: early books to define the brand, later books to strengthen the message, and post-election books to analyze the results and consequencies. Other than elections, books related to policy making, international relations, and larger political issues (like national security and environmental conservation). Corporate politics can fall into this category as well, though these publications may double as marketing and promotion documents.
Professional conferences occur in clusters (lots in the summer, for example), and so publications that are relevant to conference events tend to get published (and re-published) in clusters.
New printing and publishing technologies, which the layperson doesn't hear about, can drive new publications in a way similar to first-to-market publishing. For example, when CD envelopes were first made available in books, there was a market-driven desire to include CDs with more books. Most printing technologies are small variations on what exists today, but when a new possibility exists, it's a trend that some publishers chase right away. For example, if the quality of color rendering took a small leap forward, books where color is particularly critical (art, medical imaging, etc.) would appear more frequently for a while.

If you have more to suggest, let me know.

Books I really, really, really want to index

2007-07-24T23:48:00.001-05:00

Perhaps because most of my indexing work is on books that are rather typical for nonfiction reference books -- technical titles like Measurement, Analysis, and Control Using JMP and resource guides like Dx/Rx: Colorectal Cancer -- I jump for joy when I get something so off the beaten path that I renew my love for this job. For example, I recently completed the index for First Position, a collection of biographies of ballet dancers; more recently, I indexed Sensual Knits and Sensual Crochet, both beautifully photographed books of designs and patterns.

But now, working in the wee hours of the night, I find myself fantasizing about the books that I really, really, really want to index, books that are just asking to be written so that I, Seth Maislin, can be assigned their indexes. So here's my wish list:

Chihuahuas for Dummies
Don't laugh. You probably have no idea just how far-reaching the Dummies series has become since its long-ago inception as a series for computer use. There's Fantasy Football for Dummies, a book about imaginary sports playing; Stretching for Dummies, a book about limbering up, perhaps in advance of reading Sex for Dummies; Guitar for Dummies, Bass Guitar for Dummies, and the upcoming Rock Guitar for Dummies, which I have to believe compete with each other somehow; and Jewish Cooking for Dummies, a book that, dare I say it, would make me feel guilty to own. Nevertheless, let me make myself clear here. Chihuahuas for Dummies is a real book. I want to index the next edition, you see, because I'm dying to see what changes.

10.9 Seconds: The Joey Chestnut Story
(see http://origin.mercurynews.com/valley/ci_6297731 to get the joke) Yes, this book is my own invention, but the fun part about indexing sports books is that they are so completely self-reverential. (Yes, reverential, not referential.) Written by sports geeks for sports geeks, the authors' language captures the awe-hubris-humor combination achieved by fans and record-breakers when it comes to the sport that is most of their life. It doesn't matter what the sport is, either, so I'm all for those esoteric things like Ultimate Frisbee (I was offered such a book once) and so on. I recently indexed the comprehensive Chasing the Hunter's Dream, a directory of hunting opportunities around the world. This book included both descriptions of "dream hunts" -- think lion hunts in Africa -- and an entire section in the back dedicated to recipes, including a few meals for squirrels -- I mean, of squirrels. And I mentioned First Position in my intro, where at times I felt like I was reading an artist's diary.

How to Work My Body: A Manual
There are a number of sex books out there -- including one for Dummies -- and most of them have indexes. I just finished indexing Him and Her, short and photograph-filled manuals of the sexes, along with instructions to make them work. And I do mean "work": the book about men attempts to explain why they tend not to do chores around the house. (Oh come on, you didn't think I'd use an erotic example of "work", did you? :-) These books, produced by the same group of people who made Sensual Crochet, were a joy of sex to index, especially once I realized that most of the anatomy-filled books that I index are about abnormal anatomy: prostate disorders (100 Questions and Answers About Prostate Diseases), gunshot wounds (Criminal Investigation, 2nd edition), and the like. And unlike the traditionally polite sex-instruction book, Him and Her are more about the art than the words -- something that, for eunuchs at least, would make the indexing go much faster.

The user manual to anything only cool people own
I had the honor of indexing the user manual to the Class E series Mercedes-Benz automobile. This full-color production was totally awesome; I spent a lot of time trying to convince myself that reading the manual long before the car's official release was as envy-worthy as owning the car itself. (For many months my friends and family joked that I should paid in cars instead of dollars.) I've indexed the manuals to software applications before, but I have more memories from editing the user guide to a long-since-extinct universal remote control ... and I'm talking back when these things were large control panels. So what other cutting-edge production is taking place? I missed indexing the iPhone manual, but maybe someday I'll get to index the field guide for a nasty-looking military weapon.

Instructions to the 1040 Form
Indexing gets so little press, but that doesn't stop me from wanting to index something that's so popular or high-profile that I can't feel proud. I'll never be a household name, but if I had landed that one magical indexing project with the U.S. Internal Revenue Service, my work might have reached every household. They really were looking for someone, at least for a little while. Even the newest Harry Potter book isn't as popular. Which reminds me: is someone out there indexing Rowland's books? If not, there ought to be. The Unauthorized Index of Harry Potter would be a big seller ... despite use of the word index in the title. Move over, back-of-the-book indexing. We're on the cover now.

Okay, I'm starting to drool.

Printing Word documents with XE fields visible

2007-07-09T16:28:00.000-05:00

(This is taken from my Word Indexing FAQ.)

Can I Print My Documents with the XE Fields Visible?

Yes, you can. Microsoft Word can make all hidden-text codes visible, whether they're for indexing or not. Go into Page Setup (available from the File menu) and look for something that says "print display codes" or "print hidden text" or something like that. Until you uncheck that box in the future, all of your codes will show up in your printouts.

Be aware that printing with your indexing fields visible will affect the pagination. Don't write an index using your hard copy this way.

On a related note, remember that you can track changes when you work. Every time you insert, edit, or delete an XE field, you'll get a note in the margins. These marginal callouts can speed up your ability to find your XEs, although it might also clutter up your work. Use Tools > Track Changes to turn that feature on. Additionally, these changes can be made visible when printing as well, using a similar process as described above. Keeping XEs invisible but marginal notes visible allows you to see the index pointers without messing up the pagination. Be warned, however, that if changes are already being tracked, don't turn that feature off! You could lose that information for good. Instead, use the View menu to make those changes visible.

Technology that ignores indexing

2007-06-27T09:33:00.000-05:00

It's true, I'm including a link to this video because, frankly, it made me laugh. Witness a satire of Microsoft Surface. Yes, it's funny, but I found myself thinking about how technologies are so often designed to create needs, not to meet existing needs. I mean, it's fascinating to imagine a table that has a computer screen as a surface, but what about building height adjusters into the legs so they don't wobble? And as I continued to think about this, I discovered I have two reasons to put this video in my blog.

First, there's the opening sequence in which someone is looking at digital photos and videos scattered across the tabletop. With his fingers, the Surface user can move them around, open then, and even video the videos. In other words, the engineers of this expensive table have managed to reproduce the worst part of photographs: the pile of undifferentiated images. If someone came to you and dumped a box of photographs on your table, would you be happy? Now, what if all those photographs were digital? This is technology that completely ignores indexing. Compared to tools like Picasa, which puts metadata to work, this product does its best to create an interface option where metadata is ignored. And if you're one of those "old-timers" who longs for the physical-contact nostalgia of long-ago days of printed photographs, such that you might think shuffling through a pile of photographs would be fun, think again. Remember, these are digital photographs. They have no width and no weight.

Second, there's the reality that in real life, we use table tops as horizontal storage surfaces. Whether you're a neat freak who has only a magazine or a coaster rack on top, or you're more like me and live with your tables essentially camouflaged by life's detritus, either way you've essentially buried your workspace. In other words, this tool seems to forget the environment in which we look things up. The voiceover in the ad jokes about the convenience of a handheld machine in comparison to this table, but I'd like to suggest that this table would make more sense as a vertical hang-on-the-wall flat-screen TV. Take a lesson from the many-years-old television industry: there is no market for a horizontal television.

Indexing has at least one fan

2007-06-20T10:11:00.000-05:00

It seems my blog got noticed just the other day, which is pretty neat. It seems I have at least one fan ... other than myself, of course.

Why does the field of indexing have so few fans outside of the profession itself? I heard many stories from and about thankful authors who swear by the quality of the indexes written by professionals. My favorite is something like this: "Until I looked at that index, I didn't even know I wrote all that!" I'm talking about something more general.

No, we're not firefighters, bursting through burning walls to save people we've never met, but I like to think that we make the world a better place anyway. We're the traffic cops of information, tour guides for books, instructors and librarians, a taut rope in the rough seas of data storms....

If you know anything about indexing -- not indexes, but indexing -- then you know it's not a boring profession. Think about the public perception of lawyers, and how we don't consider that profession boring, and yet the reality of law is lots of books, lots of reading, lots of research. Those "exciting moments" brought to you on the television, along with the anxiety and intrigue of any moral or ethical battles regarding the implementation of law, represent only a small piece of the whole system. There is a lot of boredom in lawyering. No, the part of the law that brings so many students into the law schools (other than the potential for income, perhaps) is the idea that law governs our every-day lives, the sociological analogy to science.

Indexing is the process of analyzing and re-representing information, the lifeblood of everything we do. It's the Matrix; we hold the Da Vinci Code. We are responsible for getting data from one place to another in an efficient format, so that we can actually talk. Whenever you get frustrated by a failure to communicate, remember that an indexer can change that.

Maybe the reason indexing seems so boring is that the word is so inexorably tied to that alphabetized, indented thing you see in the back of books. Despite the applications indexing has for the Web, in search, and with taxonomy, people associate what we do with good old-fashioned paper. And gosh, they've been around, like, forever, so of course they're as boring as dirt -- note that dirt is not boring to some people -- and a whole lot less inspiring of nostalgia. Such a shame.

Maybe we need a movie, the way Top Gun got people signing up for the U.S. Air Force. Here's one. It's called Cross. Jack Hannah, Agency "prep consultant" who can find out anything about anyone, is double-crossed when a routine inside investigation of an agent turns out to have the exact same life Jack has. Part No Way Out and part Blow Up, Cross follows Jack on a dangerous journey into government archives to answer what should have been a simple question: Who is the real Jack Hannah?

Exemplary indexes

2007-06-09T13:40:00.000-05:00

Historically, the H. W. Wilson Award has been given to indexers of scholarly books, often because of the very complications and challenges you're talking about. What makes Do Mi Stauber, who won this year's award, so great at her work is that she has a knack at doing this without slowing down very much. I like to think I have the same knack when it comes to technical and reference books.

Scholarly indexing is WAY hard. I recently accepted a book I didn't realize was scholarly, tried to index it myself, and realized almost immediately that I was in way over my head. (Note that I'm talking about that irrational fear an indexer experiences at the start of every project, but rather something quite objective: an inability to understand the sentences and paragraphs well enough to parse them into indexable ideas.) I subcontracted the index to another indexer, someone who specializes (or at least doesn't mind) scholarly works, and the result was great.

By the way, you need to see the award-winning book's index to really understand what I'm talking about.

Scholarly works are exceptionally difficult, even if you know the basic subject matter, because of how they are written. Many scholarly publishers underpay their indexers, too, because scholarly books rarely have large audiences: they're library-books-to-be, really, put there for students and faculty. Given that a book won't sell well, publishers are often reluctant to put more money into the production process. However, for the kind of book that Do Mi indexed -- and even the one I gave to someone else -- the indexer had better be making closer to $6/page (U.S.). In comparison, I think $4/p is reasonable for the average technical book, like a book on mathematics or computer programming. See, a technical book requires expertise in or a strongly sympathetic understanding about the subject, whereas scholarly books require a tremendous amount of time spent synthesizing what's in there. Think poetry and "Shakespeare," not of prose and "John Grisham." :-)

But the H. W. Wilson Award can be given to indexers of other kinds of books, including technical. What makes the award possible is an exemplary show of knowledge and cunning, something that many technical books don't allow for. You also need the kind of working environment in which a publisher won't chop your index down to size, use a lousy design, or force you to complete the job too quickly to produce an exemplary product -- the kinds of things that are more likely to happen in technical fields than scholarly, in fact. But even a coffee table book can win the award, if the index shows that extra something special. :-)

Given the kinds of things I index -- and the circumstances in which I index them -- I often think the only way I would win the Wilson Award is if I wrote the book myself, specifically for the purpose of making an awesome index. For example, maybe I would write a book that would require me to use symbols as entries. :-) Then again, I'm still trying to write a mystery index, too. I wonder if that would win a Wilson...

Throughput in indexing

2007-05-29T20:10:00.000-05:00

I gave a presentation last year (at the Toronto conference of the American Society of Indexers) about money. I synthesized some statistics to come up with something I hadn't seen expressed before.

According to the 2004 survey of ASI members (all numbers in U.S. dollars):

median per-page rate: $3.26 to $3.50
median hourly rate: $30 to $40
median per-entry rate: $0.70 to $0.79

Note that MEDIAN rates do not necessarily match an indexer's lifestyle, workload, or typical projects. For example, some indexers work exclusively on the kinds of projects that earn more (or less) than the median. In other words, these are NOT target numbers; rather, they are reflective of the variety of everything that indexers do.

Synthesizing these numbers:

typical indexing speed: 10 pages per hour or 45 entries per hour
typical index density: 4.5 entries per page

From the survey:

average annual income: $33,325
part-timers (<32>
full-timers (40+ h/wk) in survey: 12% median
income for full-timers: $45k-$49k [from 2000 survey]

Synthesis:

To make $45k/year at $4/page = 11,125 indexable pages per year
= thirty-seven 300-page books per year
At 10 pages/hour, you must index for 185 six-hour days/year
At 20 pages/hour, you must index for 100 six-hour days/year

If you want to make more money, focus on throughput: projects that are easier for you, more effective indexing tools, improved marketing, stronger client relationships, etc. In fact, the reason that advanced indexers tend to make more money is that they have been given the opportunity to build these skills: speed, marketing, relationships. For example, indexing a single book for a single author might be short-term lucrative, but building relationships with the author's institution is more lucrative in the long term. Also, experience clearly counts toward speed, too, while short-cutting quality can seriously damage relationships.

If you think of your career in terms of throughput, you might think about your day-to-day tasks differently. For example, there have been debates among indexers regarding the sharing of book mistakes caught (like misspellings); when thinking about throughput, sending such mistakes (a) slows you down, but (b) improves repeat business. On the other hand, when you've got a client who provides you with only one book a year, it's all loss, and no trade-off, in terms of income.

Finally, when I gave this presentation I made it clear that income isn't the only reason we're doing what we do. After all, there are more lucrative professions out there in the world. If you're earning a ton of money but destroying your health, sacrificing your happiness, hurting your family, or failing yourself in some other important way, then please reconsider your priorities.

Indexers indexing infinitely ... like monkeys

2007-03-25T18:38:00.000-05:00

Three ideas have merged.

First, there's the idea I published last December as "A needle in a haystack with 100,000,000 blades," where I argued how the Web, or an approximation thereof, could be indexed by humans for a reasonable amount of money.

Second, there's The New York Times article "Artificial Intelligence, With Help From the Humans," in which we learn that the Amazon Mechanical Turk service subcontracts human workers to perform tasks that are especially challenging for computers to accomplish, such as matching images to textual descriptions. For some jobs, Turkworkers might make one penny per transaction.

And finally, there's the infinite money theorem, which states that a monkey hitting keys at a typewriter for an infinite amount of time will "almost surely" type the complete works or Shakespeare, or something similar. I first heard this ideas as a "million monkeys and million years," but I bet the math's a bit different. After all, "infinite" is much bigger than a million million.

Putting these ideas together seems to provide a rather obvious solution: third-world indexers. After all, if it costs only a nickel to get someone to write a few keywords for something, we can get a lot of indexing done very cheaply; I say "third world" because no indexer I've ever known is willing to work for a penny per word.

The indexing industry is facing the very real possibility that our workload will be taken from us and delivered to those in economies that allow lower prices. But what if we went a step further and, instead of looking for less expensive indexers with good qualifications, we decided to look for dirt cheap indexers with no qualification other than time to waste? What if, I ask, we asked monkeys to pound away at their keyboards?

I find the idea amusing but too close to the truth. After all, the intelligence behind Google is the social intelligence, the uneven and culturally biased workings of millions of Internet users plugging away at their disparate tasks. What Mechanical Turk has going for it, then, is the human decision making at the back end. Whereas most search engines look for better and greater stores of metadata with which to judge content, one man in a back room can make smarter decisions upon command. No, the real problem is that today's human intelligence is worth only pennies per word. Computers do their best, and humans sweep up afterwards. Our natural intelligence isn't worth a whole lot, I guess.

That's how we know computers are smart. Computers own us monkeys.

The passive-aggressive bullies of the information world

2007-03-24T10:29:00.000-05:00

An indexer, while building an index of historical documents for a small township on Cape Cod, Massachusetts, came across an old diary written during the American Civil War. She scanned the pages, filled with small and semi-illegibly handwritten words, and realized that nothing important had been written.

The diary went unindexed.

This anecdote, shared by indexer Marilyn Rowland at the March 24 (2007) meeting of the New England Chapter of the American Society of Indexers, struck me as surprisingly uncomfortable. Certainly I agree that when something seems unimportant to the indexer, it should not be indexed; in fact, I've claimed many times within this blog that one of the biggest failings of computer-generated lists and search engine algorithms is that they cannot identify the true value (or correctness) of content, even when using social algorithms.

Still, not indexing someone's diary? This sounds passive-aggressive. So does this instruction: "Don't index the names of everyone in that photograph. Mention these two important people, and don't bother with the rest."

Just as scientists are often accused of sacrificing ethics and social responsibility in favor of "pure scientific exploration" (the temptation to clone human beings is a fun example), so might indexers be accused of excessive marginalization or trivialization of content. It may be human nature to filter out everything we don't need to survive or enjoy ourselves in our lives, but it is an indexer's nature to impose these filters upon future users. In other words, indexers are responsible -- on a daily basis -- for rewriting history.

Everything we create in our lives -- email messages to diaries, family snapshots to oil paintings, back-of-the-napkin notations to dissertations -- is subjected not just to the entropy of time but also the red pen of the indexers. We may speak about the value of individuals, but in reality it's just a big game of Survivor, where the indexers are the ones to vote our creativity out of existence.

There is no good way to remove indexers from the equation, of course. If nothing were indexed, and no content were ever deemed to be more valuable (worth finding) than something else, content would be lost in the same way a paper cup with a lipstick stain inevitably disappears into a landfill. But who would have believed that indexers are the ones in control, that indexers are the Langoliers, who like the big kids in school get to decide who gets picked first for the schoolyard team, and who doesn't get picked at all. We are, let's face it, the bullies of the information world.

Don't mess with me. I'll erase you.

Notes on automatic indexing

2007-03-17T22:11:00.000-05:00

"Automated indexing software" is, according to the common definition, software that analyzes text and produces an index without human involvement. I'm a firm believer that the technology doesn't exist, and that a human being is required to write an index. Thus I don't use the software, and I also don't recommend it.

There are those who advocate it, arguing that it's "not as bad as an indexer would have you think." These people are often coming from the standpoint that automatic software is faster and cheaper, and they're right. Thus the issue surrounds quality.

I believe that good automatic indexes will exist once there's good artificial intelligence, something that presently doesn't exist. In very limited circumstances, however, it does; a machine can easily cull capitalized words from a textbook to create an approximation of an index of names -- although, again, the machine isn't going to differentiate between names like "David Kelley" and places like "San Francisco," since they are both of the same format and used the same way. It also won't know that "Bill Clinton" is also "William Jefferson Clinton." And certainly it can't tell when the name is being mentioned in an unuseful and trivial way, as are the names in this paragraph! So imagine the problems trying to get a machine to parse full sentences of ideas and recognizing the core ideas, the important terms, and the relationships between related concepts throughout the entire text.

FYI, those who advocate automatic software, however, would argue that the machine gets "close enough" so that a human being can edit the resulting product. However, expert evaluators unanimously agree that the software fails; those who disagree are likely those who are sufficiently ignorant of indexing in the first place such that they are unable to determine the quality differences.

Oh, I should mention that there are software programs that human indexers use to simplify and speed up the mechanics of the index process. For example, it would be silly to disallow a computer to alphabetize the entries, reformat the index, and manipulate page numbers. There are a few software packages that do this exclusively, which are considered top of the line; other applications that have indexing capabilities, such as Microsoft Word and Adobe FrameMaker, have some of these capabilities, with notable limitations.

For information on the various software available, see http://www.asindexing.org/site/software.shtml. If you have feedback, especially differing opinions, I'd love to hear them. Write me at seth@maislin.com.

(This article was originally published in 2002 and 2004 -- and it's still 100% accurate.)

Interpretation, not computation

2007-03-05T00:12:00.000-05:00

After explaining the limitations of Microsoft Word's auto-indexing feature to one of the many people who write me asking for indexing advice, I got an interesting response. Clearly frustrated by the nonexistence of computer tools to do something as simple as generate a name index, he wrote:

> I'm amazed at the poor development of the science of indexing for printed matter such as books.

I wrote back, "You misunderstand!"

The science of indexing is quite broad, given that it has a history in long-ago library science. What seems undeveloped in this case are the tools, but that's a misunderstanding of what indexing is. Indexing is an editorial field, not an automatic one. You might say it's a lot like writing, in that the writer must decide what their readers want to read, and then the writer must communicate those ideas in an organized and approachable way. Indexing is the same: analysis of text to discover what readers might find interesting, and then multiply labeling and organizing those ideas so people can find them.

Computers will never be able to write indexes because they can't (a) interpret importance of a concept, (b) understand concepts over simple words, and (c) connect ideas in contextually relevant ways. As much as I admire the Google.com search engine for what it can do, once again I will demonstrate what it can't do. Google finds 10,000,000 things when we really only want 3 (or 10 or 20). It finds what we type, but it doesn't find synonyms. And there's no guarantee that Google is searching everything that's out there, though it appears to come close; in book indexing, however, there's a human to make sure every page was considered.

How often has Microsoft Word attempted to auto-correct you in a completely inaccurate way? Spell-check? Auto-format? Auto-complete? Half-intelligent humans don't make the kinds of mistakes that these tools do.

Here's what I wish he had written:

> I'm amazed that people who know full well that computers could never write newspaper articles still believe computers can write indexes.

Another problem, of course, is that indexes aren't respected in the industry. The reason Microsoft Word even has an automatic indexing feature is because the people who wrote that software have no idea of the damage such a tool provides. That Word's {XE} functionality is so miserable is even further proof. There's a nasty cycle: people use inferior tools, quality indexing grows less likely, and inferior tools become the standard.

Indexing is an editorial process, just like writing and editing. Indexing requires interpretation, not computation.

Computers will not and should not be used as indexers. If my job ever dies because computer programmers have found a way to make me obsolete, at least I know I'll be in the enlightening company of human writers and artists.

Indexes are the speed limit on the information highway

2007-02-07T22:13:00.000-05:00

The growing demand for indexes that point to online, changing, and custom content is forcing a huge gap into the indexing industry, and that gap is physical.

With traditional publishing, if I wanted a reader to find content on page 114, I would simply have the number 114 show up in my index: "credit card fraud, 114." To make this work, however, that page number must be immutable across the lifetime of my index. Should the content be republished in a different format, layout, or language, or with significant edits within the first 100-plus pages, my index could be rendered inaccurate. In other words, if I type "114" in my index, that content had better be on page 114.

The appeal of fluid content, however, is slowly making traditional information delivery obsolete. Not only are books republished for lots of "traditional reasons" (e.g., updated editions, new languages, different book and print sizes), but technology is enabling books to be published without a single physical page. With the possible exception of the Adobe PDF format (which purposefully preserves the overall book-like format in an electronic file), page breaks are optional and subjective. A Web page or HTML document can have a scroll bar, such that there are no pages; an e-book intended for a handheld reader is paged according to the size of the reader; a news or magazine article of any length can be broken in two or three simply to increase ad sales; and some electronic documents can be edited by the readers such that anything goes.

Ah, how I miss the days when 114 meant 114.

Indexing content that changes is going to be hard, but the fundamental challenge isn't about keeping up with what was newly published today, or even in the last twenty minutes. It's about content ownership. When content is moving around all the time, indexers don't have a good way to tracking where that content is going.

As an analogy, consider a classroom filled with thirty students, with one student at each desk. If you have a photograph of where every student is sitting, you could leave the room and generate a spreadsheet that lists each person's name and seat location. But what happens when the students are playing musical chairs? Every photograph you take is outdated almost immediately; even staying in the room wouldn't be good enough, because your typing speed will never match the speed of twenty kids jumping around. In fact, the only way you could manage a spreadsheet that shows where each student is sitting at all moments is if that spreadsheet operated in real time, by reference. In other words, if all thirty students carried GPS locator chips in their pockets, you could track the chips -- and thus the students -- by satellite. Your map could be as dynamic as what it is you're mapping.

Embedded indexing, or indexing by reference, is a rudimentary and imperfect example of this process. With embedded indexing, I can have some kind of information inserted into the content -- like the GPS chip in the student's pocket -- and then I can generate an index based on where that information is at any one time. This blog entry, for example, has keywords attached to it; the website where my blog is published can, at any time, generate a list of all entries with that keyword. This kind of dynamic indexing is not uncommon these days; website content is served according to a number of immediate rules, and the result can be as simple as a website that publishes "Hello Seth Maislin" on my page but no one else's, or as complicated as an online stock trading program that keeps track of millions of private transactions.

I say this is rudimentary, however, because it's still a snapshot. Perhaps it's convenient to have that snapshot taking at the moment I arrive at a website, but if I leave my browser at a website and walk away for 20 minutes, the picture doesn't have to change. The "Hello Seth Maislin" greeting made sense when I was sitting at the computer, but if I walk away and my wife sits down, it's now wrong. The snapshot is old. Google search results can change from one minute to the next. Even stock trading programs sport copious warnings that despite the best efforts of the website, the price you think you're getting may not be the *actual* price when you complete a transaction; the delay between your clicking the mouse and the machines at the other end doing something is a legitimate and unavailable delay. Some website attempt to minimize this by taking a snapshot every fraction of a second, as if you were watching what was happening "live." In reality, there's still a delay, and there's still no way to truly synchronize everyone's machine.

My point is that indexes to changing documentation must live apart from the documentation. If they really lived completely together, the content and the index would be essentially the same thing, just as the GPS chip and the student are really one merged object. But because indexes are interpretations of content, there is always going to be a gap. The generation of the index be removed from the content that is being indexed, in order for that interpretation to take place.

The only way for indexing to survive, I think, is for content to slow down. And because I believe indexing -- interpretation -- is critical for learning, the only logical conclusion is that content will slow down.

The need for an index is the logical limit of just how fast data can travel.

TAA Conference in Buffalo, June 22-23

2007-02-03T13:12:00.000-05:00

I have been scheduled to present twice at the 2007 conference for the Text and Academic Authors Association (TAA). I'm excited about these presentations -- actually, one of them is a roundtable -- because this will be perhaps the first time when my audience is predominantly authors. Although I have taught indexing to numerous technical writers over the years, the nature of their writing is significantly different to that of other authors. TAA members tend to write journal articles and textbooks; they are writing because they want to share information (whether driven by a simple desire to share knowledge or by more complicated goals like industry prestige, peer respect, or job security), whereas technical writers are obligated to write documentation as part of a larger project.

In some ways, having this opportunity to reach out to the authoring community represents a longer reach than usual, in that most indexers ply their trade among the publishers themselves, who manage the book production but don't do any of the writing. Although any business benefits I receive from these talks won't be as lucrative as the others -- convincing one author to hire me for the job isn't as valuable as convincing one publisher to hire me for several jobs -- the advocacy benefits are likely bigger but unknown. I often think that the indexing process is hidden from authors, despite their desire to see quality indexes appended to their work.

If the indexing industry is going to grow, it won't be because the indexers have advocated for themselves. No, indexers will be prominent only when others -- like writers -- advocate for them and their products.

Foreword to Heather Hedden's upcoming book

2007-01-20T11:15:00.000-05:00

I was asked to write the foreword to Heather Hedden's upcoming Indexing Specialties: Web Sites, to be published in 2007 by ITI. Given the importance of this book in the indexing industry, I am reprinting that foreword here. For more information on the book itself (not yet available), visit either ASI's publications page or a list of ITI's indexing publications.

- - - - - -
Foreword

Indexing is not a popular profession by any stretch of the imagination. Not only is it almost completely unknown in lay circles, but let's be honest: writing indexes sounds about as exciting as cleaning the house, but a hundred times harder. Also, if you were born in any year before 1990, the idea of Web indexing sounds like cleaning a house in outer space. I mean, there are no houses in outer space.

The Internet and the Web -- this monstrously huge and growing system of sharing data -- desperately need more information sorcerers like Heather Hedden. Not only does Heather have the talent to recognize when knowledge is missing, but she also has the ability to make that knowledge visible. She starts by learning for herself, and then she loves to share.

Heather and I first crossed paths in my classroom, where I taught a course called "Writing Indexes for Books and Websites." My course was written to explore the questions and theories of indexing, and so couldn't be limited to just books. Heather’s interest went much further, and since then she has explored writing web indexes as a singular discipline. For me, Heather has been a student, an apprentice, and a role model. She's someone I count on to get things done. She has vaulted across the lines from library science to book indexing to web indexing, each time with surprising success, and has since become a renowned and respected expert in the web indexing community.

Indexing Specialties: Web Sites is a book filled with honest, get-it-done advice. Heather is not afraid to talk about the code and the tools, because she has faith in her readers. In her hands, the complicated stuff looks straightforward. Besides, when the technical lessons are over, Heather shows readers how to think about web indexing as well: as a process and as a business. Until now, if book indexers wanted to graduate to the Internet frontier, they had no unified place of reference, no single source of everything they'd want to know. In fact, some of the tools Heather includes in this book were almost completely unknown to indexers until now.

I am excited and pleased to see Heather compiling this knowledge in a book. She has put into print an indexer's Rosetta Stone, which will lead book indexers toward other information management topics like taxonomies, information architecture, and search tools. It's not about complicated coding practices and computer programs, but about the guidelines to getting that A-to-Z index published on the Internet, and doing it right.

She begins by exploring the boundaries of web site indexing, clarifying what kinds of sites need indexing, how they should look, and how they should work. Then she immediately provides the HTML building blocks to making your indexes appear on the Web, the surprisingly simple code you'd need to create index pages, index entries, indentations, hyperlinks, and cross-reference links. If you've never programmed on the Web before and are afraid it's over your head, you’ll be kicking yourself once you see how easy Heather makes it.

Once you're armed with the grammar, you next need the tools to actually write. Heather gives you the detail about the tools (CINDEX, HTML Indexer, HTML/Prep, Macrex, SKY Index Professional, and XRefHT) to create or generate indexes that are ready for web publication. She takes more time exploring the specialized tools of XRefHT and HTML Indexer, two stand-alone web indexing applications, and shows how you can use their features with agility.

The last third of the book is dedicated to the "mindspace" of web indexing. There's more to indexing than just the tools, and so Heather writes carefully about how indexers should approach the job. She addresses the challenges of working out of order, adding anchors, indexing periodicals, and knowing which pages and at what level of detail you should index. She deals in detail with cross-references, language, subentry structure, and format. Finally, Heather dives into the nitty-gritty of the web indexing marketplace, including how to market yourself as a web site indexer.

Web Sites is going to satisfy you immediately and in the long term. On behalf of the American Society of Indexers -- and myself, personally -- I am honored to welcome Heather as an esteemed author in our community.

Seth Maislin
President of the American Society of Indexers (2006-2007)

Constructing a mysterious index

2007-01-09T21:26:00.000-05:00

[NOTE: Edited on 6-9-07 to fix missing indentations for index entries. -SM]

For years I have puzzled over the possibility of writing an index, to an imaginary book, in which a mystery is revealed and potentially solved. The book itself would not have to be a mystery, but there would have to be some kind of secret.

For example, suppose you had these entries:

La Traviata, clandestine meeting at, 145
Marters, Francine
meeting at La Traviata, 145

From these entries you would learn that Francine Marters met someone at La Traviata surreptitiously. An additional entry

Rapiere, Evan
   accidental discovery of La Traviata matches, 166

implies not only that Evan was not the person at the restaurant, but also that Evan might have been the reason the secret was necessary. A final entry,

Pfiser, Victor
   confronted by Evan Rapiere, 231

allows for the possibility that Victor was the other person with Francine (on page 145), such that Evan's discovery of the matches led to this confrontation.

What would make an index like this potentially interesting as a puzzle would be (a) the randomization of information, caused by the alphabetization of entries; (b) the summary-style labels in the index, which must naturally leave out much of the story; (c) the creativity of the labels, which can emphasize or omit interesting facts without destroying the quality of the index itself; and (d) the ability to tell many overlapping and long stories across just a few pages.

On the other hand, what makes an index puzzle challenging -- and the reason I've had no success so far -- is that the index must articulate all the facts; an index can have no secrets if it's going to work as a puzzle. For example, if this were a murder mystery, wouldn't the murder have to be indexed? If the index is going to be a good one (and that's a requirement for me, because otherwise it would seem too contrived), you'd have to have entries like these:

Rapiere, Evan
   murder of, 235
Marters, Francine
   guilty confession of, 469

Is there any way that a legitimate index could obfuscate information sufficiently enough to leave some mystery? In a way, an index must be too "honest" to allow for secrets.

The other problem, opposite to the honesty problem described above, is that if an index isn't specific enough, it's impossible to put the facts together in the first place. For example, if I changed the matches entry above to this:

Rapiere, Evan
   accidental discovery of match book, 166

there's no way to connect this to La Traviata without help. Similarly, if I change the murder entry to simply this:

Rapiere, Evan, 235

or

Rapiere, Evan
   surprised by an intruder, 235

then there is nothing in the index to clarify that Evan actually died.

I'm looking for ideas on how to get around these challenges. How much integrity can the index maintain without either giving too much away or leaving too many holes in the story?

My waitress, my audience

2007-01-03T22:46:00.000-05:00

For some reason I felt like hugging my waitress this morning.

There’s a diner in my hometown that I visit up to three times weekly, while my daughter is at school. They know me here. I have my special table, my eminent domain, where I plug in my computer. Kathleen, my regular waitress, brings me coffee even as I sit down and tries to guess what I want for breakfast, with some accuracy. She knows my name, too, which I feel is the best part. So in this new year, after I happily kissed 2006 good-bye, I was inspired to welcome my diner lifestyle in true living style. “Kathleen, can I give you a hug?” I asked, and she said yes.

Then we get to talking. Her age, my age. Her career path, my career path. The kinds of things people always talk about at the start of a new year. To sum up her half of the conversation in just one sentence, I’d write this: She came to the realization that at the age of 54, if she had taken her father’s advice and gotten a job at the phone company when she was 20, she would be retired instead of working at the diner. (I have to add that she’s never had a vacation in her entire life, except for two weeks when this diner was closed temporarily, and that her job pays no benefits.)

As a self-employed indexer working voluntarily where she works, what can I tell her?

First, I tell her that my profession is dominated by women over the age of 45. Many of these women raised their children and wanted to do something different with their lives. Many felt oppressed by their imaginings of the traditional workplace, or were afraid of having insufficient skill to reenter the job market. Others no longer had the financial support of a spouse and simply needed to find work to stay comfortable, or even solvent. I see these women at every indexing meeting and in my classrooms. Almost all had never heard of indexing before, and they’re giving it a try. After all, being self-employed and reading books sounds a lot better than being buried in a cubicle.

Then I tell her that once you’re truly self-employed, making enough money to pay monthly bills and quarterly taxes, in your life you will never experience unemployment again. You can’t be fired, you can’t be laid off, and you can’t be transferred to another division, location, building, team, or employer—not without first choosing it for yourself. As long as you have your core skills, you’re not much different from a child who can entertain himself with sticks, sand, and even an empty parking lot: the world is your workplace.

Finally, I tell her that in my profession, everything I have every learned (and continue to learn) has a direct application to my job performance. My grandfather says “no knowledge is ever wasted,” and in my case he’s literally correct. Every conversation I have gives me better background on people and information, and every experience provides me with a potential story to share with my students.

It’s not hard for me to be positive about what I do, because I love what I do. There are lot of perks, too, from being my own boss to sitting at my favorite table in the diner, where I am now. Kathleen, on the other hand, is pretty grumpy about her job. She gripes about a lack of benefits, a dearth of Social Security earnings, a regular 5:30am wake-up call. I have no doubt that I have it better than she does, at least in these ways.

If there’s a professional lesson for me here, it’s that Kathleen is my audience. She is the person who reads the books I index, the person who might sit in my classroom. I have to remember that if I do my job correctly, I am building a bridge from me, the college-educated small business owner who works off his laptop, to people like her. The gap in our lives is the challenge I face every time I sit down to invent a keyword, and index entry, or a label for a hyperlink.

In the world of indexing, these life differences are surmountable.

Eighteen million people can't be wrong

2006-12-28T10:49:00.000-05:00

No matter how much you and I might like Google, the fact is that Google has some very serious problems with it comes to finding content. More specifically, if you're looking for the "right" answer, or if you're attempting to do any serious research, Google is likely to fail you miserably.

The flaw lies in Google's strength: social algorithms. Social algorithms are processes in which decisions are made by watching and following the majority of people in a community. If blogger.com tends to be the place people go to create blogs, then a social algorithm will see blogger.com as "better." When a search engine is managed by a social algorithm, a website might appear first in search results not because of the quality of site, but rather because a larger number of people treated the site as if were of higher quality. In other words, social algorithms equate "majority" with "best," something that often looks right but actually is patently untrue.

When you perform a search at Google.com, your results are sorted based on majority behavior and little else. For simple questions about anything -- as well as complex questions about cultural issues, for which "lots of people" is critical -- frequently the majority opinion is rather close to what you want -- which is why Google is so successful. But the gap between "close to what you want" and "accurate" is an invisible one, and that makes it insidious and dangerous.

For example, search for "Seth Maislin." The first hit is my website. The second hit is this blog. The third hit is an interview I did for O'Reilly & Associates in July 1999. An investigation of why these are the top three sites is rather interesting. First of all, these are the only results in which my name actually appears in the title; the fourth link and beyond have my name in the document, but not the title. Second, my website appears at the top not because it's the definitive website about "Seth Maislin," but because Google knows of 24 people linking to it. In comparison, the only person who ever created a link to this blog is me -- a number far less than 24! The same goes for the O'Reilly interview, except that the single linker isn't even a valid site any more: it's broken. The popularity of my home page (in comparison to this blog, for example) is why it's a better hit for my name. But if you folks out there started to actually link to this blog, that would change.

You should look into the search results for the word "Jew." A website known as JewWatch.com, an offensive and inflammatory collection of antisemitic content, had appeared as the number-one result at Google.com for this one-word query. This happened because a large number of supporters of this site tended to build links to it; then, those were were outraged or amused also linked to it within their protestations. In the end, the social algorithms at Google recognized how popular (i.e., "linked to") this site was, and in response rated it very highly -- in fact, rated it first -- compared to all other websites with the word "Jew" in the title. Eventually, those who were enraged by this content fought back by asking as many people as possible to link somewhere else -- specifically, the Wikipedia definition of Jew -- just as I have here. Over time, more people linked to Wikipedia than to JewWatch, and so the latter dropped into second place at Google. This process of building networks of links in order to influence Google's social algorithm is called "Google bombing." In other words, when the people who hated the site acted together in a large group, Google's social algorithms responded.

(By the way, you'll notice that I do not create a link to the offensive site. I see no reason to contribute to its success.)

Do you see the problem? The success of Google bombing is analogous to the squeaky wheel metaphor, that the loudest complainer gets the best service. Social algorithms reward the most popular, regardless of whether they deserve it. JewWatch made it to the top because it was popular first; Wikipedia's definition moved to the top because those offended banded together to demonstrate even more loudly. And in the end, there's no reason for me to think either of these links is best.

Whether popularity is a good thing or a bad thing is often subjective. In language, some people lament the existence of the word ain't, while others applaud its existence as an inevitable sign of change; either way, the word is showing up in our dictionaries because more and more people are using it. But I'm not talking about language; I'm talking about truth.

Do you think vitamin C is good at preventing colds? Well, it isn't; there have been no studies demonstrate its effectiveness, but there have been studies that show it makes no real difference. (It's believed that vitamin C will shorten the length of a cold, but studies are still inconclusive.) But after a doctor popularized the idea of vitamin megadosing, our entire culture suddenly believes taking the vitamin will keep you extra healthy. Untrue.

Do you know why "ham and eggs" is considered a typical American breakfast? Because an advertising executive in the pork industry used Freudian psychology to convince people to eat ham for breakfast. He did it by asking American doctors if they thought hearty breakfasts were a good thing (which they did); the ad-man then asked if ham were a hearty food. Voila: ham, sausage, and bacon are American breakfast staples, and the continental breakfast vanished from our culture.

In both of these examples, majority belief trumps the truth. And look at the arguments about global warming! I won't repeat the arguments laid out by Al Gore in An Inconvenient Truth, but his argument is that as long as enough people insist that global warming isn't true, its dangers will remain unheeded. In fact, I'm not even going to argue here whether global warming is a real thing or not; it doesn't matter what I believe. What matters is that the debate over global warming isn't a fight over the facts. Instead, it's a shouting match, in which the majority wins. Right now, so many influential people have argued that it doesn't exist (or isn't such a big deal) that very little has been done in this country in response to its possible existence. But as more and more people start to believe it's at least possible, it's becoming a reality. Doesn't that just drive you nuts? Why are the facts behind global warming driven by democracy? Can't something be true even if no one believes in it?

One last look at this "majority rules" concept, only this time let's avoid politics and focus on simple word spelling. If you search for the word millennium, correctly spelled with two Ls and two Ns, you'll get about 54 million hits at Google (English-language pages only). If you search for the word millenium, misspelled with two Ls and only one N, you'll get 18 million hits. Twenty-five percent of all websites have this misspelling in them! For content that's published, that by its very nature is biased toward having only correct spellings, this error rate is monstrous! But does Google let you know that millenium is misspelled? Does it ask you if you "meant to type millennium?" No! After all, Google considers the misspelled word correct.

I mean, eighteen million people can't be wrong, right?

I met a famous indexer the other day

2006-12-24T15:33:00.000-05:00

In my March 20 post ("Frustrated by a lack of meaning"), I made reference to a Microsoft clip art mess that was quite public. The story is that the keyword "monkey bars" caused certain images to appear when someone searched for the word "monkey," and that these results were misinterpreted in a strongly negative way.

Well, I met the indexer who actually wrote those keywords -- someone I've known for a long time -- and I have to say, there's something really cool about realizing that one of your good colleagues was behind that story. I also find it reassuring that the indexer is someone who really knows what she's doing, because it emphasizes just how far apart good indexing is from good search: smart people, dumb tools.

For more on this subject, I recommend reading The Inmates Are Running the Asylum. The book is about computer programming in general, but the sentiment is dead on.

A needle in a haystack with 100,000,000 blades

2006-12-24T13:58:00.000-05:00

The Internet has more than 100 million websites, according to the November Netcraft survey. If you were standing on top of the growth curve, by now your stomach would have nothing left to vomit up.

I did some math, and I've figured out a way to make sure that all of these websites are indexed. Here's what I discovered.

Between October and November 2006, approximately 3.5 million sites were created. Assuming that my team would be responsible for inventing a set of keywords for the whole site -- and not for individual pages or parts of pages -- we would have to build 3.5 million keyword sets.
Let's further assume that on average, every website would have four keywords or key phrases. For example, this blog would get the keywords "Seth Maislin," "indexing," "blog," and perhaps my company name, "Focus Information Services." Ideally we'd have the time to invent many more, since it's our goal to help the website perform well at the various search engines, but this team simply can't give everyone special attention. So I'm making the executive decision to limit ourselves to creating 4 terms each for 3.5 million sites.
Assuming that we can invent and type one keyword every two seconds -- a conservative estimate, given that my company name takes me a minimum of two seconds to type -- we'll need 28 million seconds to get the job done.
Now remember, we're just taking about the new sites created in October 2006. Consequently, we have only a month to get the job done before we have to start indexing the November 2006 sites. For this reason, I'm going to build a team of several people, with each one putting in eight hours per day, twenty days each month. That's 576,000 seconds per person per month.
Dividing 28,000,000 seconds per month by 576,000 seconds per person per month gives me 48.1 people, which I'll round to a nice 50 people. That means I need a team of just 50 people to get the job done.

So there you go: a team of 50 people can index the Internet. That doesn't sound nearly as bad as I thought. Of course, everyone will have to type rather quickly, and we'll need a system in place to prevent us from accidentally indexing any one website more than once, but that shouldn't be too bad. And yes, I'm assuming that all of these websites are in English, but most of them are; I'll bring a few translators to work on the few remaining.

At U.S.$50,000 per year per indexer, which is quite modest for a highly intense round-the-clock job like this, plus $100,000 for me as manager, I could probably put together a bid of about $350,000/year to get the job done. Given how many billions of dollars are spent or exchanged over the Internet today, that seems quite reasonable, too. Heck, I should triple the whole thing, since we'd have to re-index the old sites every once in a while. Maybe I should double it again, too, so we'd be allowed to use eight keywords instead of four.

So let's see, that brings the total bill to to $2.1 million. Gosh, that isn't bad at all, is it? I mean, we all agree that indexing the Internet is at least a two-million-dollar-per-year business, right?

Except it's not. Indexing the Internet is a zero-dollar-per-year business. No one is doing it. Just about no one seems to care about quality keywords. In fact, there are only two industries that exist around keyword creation. One of them is misnamed "search optimization," which is about spamming the heck out of the Web. Optimize, I think not: this is the opposite of the intelligent product my team would be build. The other business is the search business itself, companies springing up around those fancy algorithms that Google, Yahoo, Lycos, Ask Jeeves, and the rest use. The thing is, those algorithms are just word-matching machines. These engines are looking for keywords, but none of them is actually writing any. So you see, no one with indexing training is writing any keywords. The inexpensive market for human indexers is being completely overlooked.

Guess it's not worth the two million.

"Can I Delete All My ___ Entries in MS Word?"

2006-12-18T15:31:00.000-05:00

(This is the newest entry in my MS Word Indexing FAQ.)

Every now and then, there's nothing you want to do more than globally delete a bunch of entries. The problem is how this is supposed to happen. For example, suppose you have a common main entry for "publicity," when you decide that you're better off with a cross reference like "publicity. See marketing." In addition to creating this cross reference, you need to remove all of your original publicity entries. Although you can search for marker text, you can't search for whole markers. In other words, you can search for the word "publicity" when it's used within index markers (look for hidden text), but you can't search for a whole marker like {XE "publicity"} or {XE "publicity:methods for"}. For this reason you can search globally and delete.

The easiest approach to deleting all publicity entries is the manual approach: generate your index, then delete everything that starts with the word publicity. Unfortunately, manual edits will be undone as soon as you generate the index again; you'll have to remember that you want to make these manual changes every time you create a new version of the index. To help you remember to make these manual changes, I recommend changing the format and/or language for the word publicity to make sure it jumps out at you. Search for XE "publicity, the unique text for all publicity entries, and replace it with boldface, all caps, and a shocking color like red. I also recommend that you change the word publicity with something that will sort at the very beginning of your index, such as aaa DELETE ME. Now, when you generate your index, you'll see some red, boldface, all-caps reminder at the top of your index file. Hopefully this will be enough for you to remember deleting your entries.

Another approach, and by far the one I prefer, is to replace the marker syntax with something that Word can't interpret. Instead of using the letters XE in your marker, use something like DELETE_ME. In other words, globally change XE "publicity with DELETE_ME "publicity. Since markers are hidden text, your DELETE_ME markers will remain hidden from publications; further, they'll fail to become index entries since Word won't interpret them as XE markers. The biggest advantage to this method is that it works globally, and you only have to make these changes once. Another advantage is that you aren't actually deleting the entry, just rewriting it; if for any reason you need to reconstruct entries, you can always change DELETE_ME to XE. (This is a kludgy way of creating conditional text, but it might be just what you need.) The disadvantage is that you're not actually deleting anything, potentially cluttering your documentation.

As a side note, whenever you remove an entry from your index, remember that you have to delete any cross references that target those now-removed entries. For example, if you replace your publicity entries with "publicity. See marketing," you'll need to rewrite or delete entries like "public relations. See also publicity."

ASI President's Letter (December 2006)

2006-12-16T21:27:00.000-05:00

Below are the first few paragraphs of my letter as ASI president, published in Key Words. The full letter is available in the December 2006 issues of the bulletin, available to ASI members at the ASI website, as well as by request.

ASI: Prospective and Retrospective, a Presidential Perspective

At the end this year, I’ll write a letter to “Seth of 2008.”

For a number of years I’ve been sending snapshots of my life to future “selves,” capturing a year’s events, achievements, and desires onto a couple pages. Even though I’m writing to myself, however, I’m trying to communicate with versions of me that don’t yet exist. Who will I be in 2008? Why will I want to know about today’s “me”? What about the Seth of 2014? So, after warming up my pen with details about family, house, job, art, and health, I inevitably get to the tough stuff: ambitions, anxieties, hopes, and disappointments. There’s an irony to the whole thing, knowing I’ll be reading the letter with perfect hindsight. It’s an incentive to improve every year.

ASI’s strategic plan is just such a letter. With its many strategies and priorities, we’re informing our future society about some critical information. Our members have shared with us a vision in which indexing will be recognized and respected more; to reach this vision we’ll have to look critically at who we are, now and soon. With the hindsight we’ll have in 2008 (and 2010 and 2014), I don’t want us to feel nostalgic when we look back. I want us to feel successful. I want us to be glad that we live in better times.

The conflict between the needs of the immediate and our goals for the future is real. To function as a society, we need people in charge of what’s happening right now, as well as people in charge of what’s happening in the future. Week to week, ASI manages a long stream of important details: chapter name changes, SIG formations, PR construction, training course materials, administrative shifts, the Philadelphia conference, membership drives, and so on. The board gets a few dozen reports from committees, fourteen chapters, SIGs, and task forces. This is the “ASI of 2006,” focused on bylaws and meetings and content development.

Seth, a different enabler

2006-12-16T21:17:00.000-05:00

Most coincidentally, taxonomist Seth Earley wrote about the enabling process in his own blog, "Not Otherwise Categorized...." Of course, his entry is less comic, more professional, and thus more meaningful than mine (12 Dec 2006), so it would be a shame for me not to reference it.

"Just tell me the answer" (blog entry, 7 Nov 2006)

There are more enabling Seths out there than you might have noticed at first.

Seth, the enabler

2006-12-12T23:56:00.000-05:00

I rediscovered the two roles that a consultant can play in business. He can step forward, propose and perhaps implement a solution all by his lonesome, and then walk away; or he can sit quietly to what everyone has to say, push and prod in strategic ways, and get everyone else to do the work for him.

Here's an analogy. Suppose a friend who needs a resume approaches you for help. "Would you write me resume?" she asks. On approach is to say "yes," ask a couple of questions, and then crank out a complete resume. Handing it to her you say, "Go ahead and make some edits, if you want." There are some wonderful advantages to this process: you get to work on your own, on your own terms, and for a very short period of time. On the other hand, what you're really supposed to do is sit down with your friend and say, "Well, what have you got so far?" Then you ask all sorts of clever questions like, "What do you think you do best?" and "What kind of job do you think you want?" She answers these questions, and as you nod wisely, you tell her to write all that stuff down.

The greatest part about being an enabler is that you never have to make a decision at all. You're a Freudian psychologist asking all sorts of provocative questions, getting paid by the hour to watch someone else do all the work. The better they do, the better you look.

I've discovered that being an enabler is the smartest, most lucrative, and most effective way to be a consultant -- but the fact that I never have to make a decision is very interesting. "What do you think? How would you do this? Do you think this would work? Before tomorrow, see if Joe agrees." I'm amazed at the power these kinds of questions have.

Ask yourself how much enabling you do in your job. I'm starting to realize that helping people do things on their own is more rewarding than doing it myself. Frankly I'm unnerved by this. This wasn't at all what I learned in engineering school.

But everything I've read says that this is now the right way to do this. Decisions made by people who don't actually use the system are less likely to succeed. Evidence-based practice is about moving forward not on what you think, but on what you know, such as from testing. So yes, it's about asking the right questions, and not about what you know. In fact, psychologists who do know the answer have to play dumb if they're to succeed.

If you had told me years ago that the subject matter experts are far less valuable than the subject matter dunces, I'd have said you were full of... what's the word? (I trust your opinion.)

Indexing moving content

2006-12-12T23:41:00.000-05:00

Has it been three months? Almost!

Fact is, the world has a way of throwing curve balls on a regular basis. For me, those curve balls include a family-wide influenza epidemic, teething babies, travel plans, and the like. Trying to keep a grip on life is like trying to catch fish with your hands.

Tonight I give a presentation about trying to index moving targets. I was surprised to discover that of all the presentations I've ever given, this was absolutely the hardest to write. In fact, I just finished a few minutes ago. I've taught three-day classes, with eight hours of material on each day, but this 45-minute presentation really stymied me. There are two reasons for this.

First, trying to index moving content is, no matter what, a mess. The simplest example of a problem is creating an index entry like "software development, 111-121," and then finding out that pages 111 and 121 have moved respectively to pages 113 and 123. With standalone indexing (where you type in the page numbers), the only real way to fix this is manually: go back and rewrite all your page numbers. It's a MESS. So here I am, hoping to provide some tips to indexers and technical writers, something to help them avoid these kinds of corrections -- only to realize that there's no good answer. (A bad answer is to not index at all. :-)

The second problem is that even if I did have a list of useful tools, they don't make for interesting presentation materials. The first draft of my presentation would have resembled a public reading of the weather report for ever American city, in alphabetical order: if you're lucky, you're interested in Albuquerque and Atlanta and can walk out early.

The fact is, our growing reliable on live and custom information is wreaking havoc on the indexing world. It's becoming harder and harder to collate information in relevant chunks. Search will never do it; even if there were human beings out there developing controlled vocabularies, full-text search still retrieves a tremendous amount of flotsam. But creating keywords for something that won't live an hour seems kind of pointless, too. We're all just pounding sand.

I'm looking forward to what the participants have to say. Must we accept the false imprisonment of uncatalogued real-time information flow, or will writers finally catch on that indexers have an important role on the creation side as well?

Unfindable (a virtue)

2006-09-08T12:06:00.000-05:00

Indexers work to make things findable, but there's another side to this coin. Indexers also work to make things unfindable.

An important and often overlooked consequence of the culling process that indexers hone when deciding what should be indexed or labeled, and how, is that every decision an indexer does not make makes something that more unfindable. In fact, just as there are infinite number of misspellings for any one word, there an infinite number of indexing choice that an indexer can choose. But unlike misspelled words, which by definition are "mistakes," every indexing choice and every keyword is a good keyword in the right context. The information space is too big for mistakes; for each conscious and unconscious inaction, the best we can hope for is "highly unlikely." If we don't do it, perhaps no one will need it.

There's a sign in a general store in Lake George, New York: "If you don't see it, you don't need it." This is the inadvertent motto of all indexers. We can only pray that everything concept we leave unindexed, every word we don't choose, and every relationship we don't articulate is unneeded. Then again, there are an infinity of choices we never even see, aren't there?

Unfindability is a pandemic, a glorious desert that stretches beyond our senses and imaginings. In today's word of RFID technology, in which every object and person (and object-person combination) can potentially be mapped and tracked over an arbitrary length of time, the vast wasteland of unfindability starts to rank up there with a good vacation.

Let's turn the indexing process around. As indexers, what do we want to make lost?

Systems of belief that destructively conflict with our own. Even if you believe in an open exchange of ideas, perhaps the close-minded ideas and people could disappear to make your world a better place. Some indexers face this challenge regularly, being asked to index materials that they disagree with on political or religious grounds, or to use indexing methods that conflict with their professional ethics.

Falsehoods. Wisdom is knowing what you don't know, but what if you believe something that is completely untrue? Certainly there are layers of truth, and ignorance isn't inherently bad (e.g., it drives scientific research and discovery), but what of the urban legends and purposeful deceit that scatters our information space? If they can't be labeled as more than 90% false, shouldn't they just disappear?

Hurtful knowledge from which there is no clear benefit. Constructive truths hurt but spur growth; destructive truths are better left unsaid. We've all had the experience in which someone commented on our selves, perhaps even politely and with good intention; I know a woman who was told she could never excel at tennis without surgery, because her body shape interfered with a good backhand. I'm sure the instructor was trying to helpful, but she's never enjoyed tennis since. If these comments can't be left unsaid, unfound is the next best option.

Private information, and the deep past. Is my street address on the World Wide Web? What about my childhood photos? Medical and financial records? Candid post-mortem comments about my behavior in past relationships? Or should I accept that my life is an open and Google-accessible book? With millions of blog pages being written by today's schoolchildren, what will happen when someday they run for office and the world (re)discovers their underage exploits? There is no expiration date on personal content, something we sometimes regret.

And finally,

Stuff no one needs. How do indexers judge what nobody needs now, or in the future, under any circumstances? And yet we do it every day at work. We tell ourselves that we can take out our biases, ignoring our beliefs and lives and emotions and ethics and needs, just as so many others become detached emotionally when performing their jobs. But unfindability is inevitable; sooner or later (and usually sooner), our choices will have a direct effect on the abilities of others to find something they want, from historical knowledge to insider secrets, from biographical summaries to photographed nudity. I say that it is our responsibility to make these decisions, to apply our beliefs and biases as well as our knowledge, to unmap the information space we want left behind.

As indexers, we shape the worlds that no one sees. Now let's do it on purpose.

Troubleshooting Microsoft Word indexes

2006-09-05T22:00:00.000-05:00

I've been archiving various questions people have asked me about Microsoft Word indexes. It's amazing how many things can go wrong, and then can't be easily solved. In response, I've started building a page for troubleshooting Word indexes. You can access that page here:

http://taxonomist.tripod.com/indexing/wordproblems.html

If you're having a problem with Word, first know that you're in good company, and then write me about it.

The salesman's cross reference

2006-09-02T12:34:00.000-05:00

My wife and I burst out laughing.

We have a short book called Baby Animals. (Apparently there are a lot of books called Baby Animals.) It's filled with page-sized photographs of very cute creatures, like these pictures of chicks or bunnies or whales. In fact, there's a cute baby lion on the cover of our book. I'm telling you, these baby animals are CUTE!

After reading cool facts about these babies -- dogs and cats are born blind, chicks eat small stones because they have no teeth, whales drink 100 gallons of milk daily -- we get the inside back cover, where the publisher writes the following:

If you liked learning about Baby Animals, you will also enjoy

Fighting Fires
Giant Machines
Killer Whales
Planets Around the Sun
Wild Bears

What am I missing?

Information is owned by the few

2006-08-27T20:38:00.000-05:00

Consider the history of manufactured items. At one time in United States history, the manufacturer in your neighborhood was the primary source of whatever it made. If Maytag had a plant in your city, you bought Maytag. There was almost no question of buying a competitor product in a distant city, with reasons ranging from the practical (delivery requirements) to the social (your family was employed there) to the unconscious (you heard about this company every day in the news). Manufacturers were king: they made, you bought (assuming you could afford), and no questions were asked.

Then came the middlemen, resellers like Sears, who discovered that if you brought a number of competing products into the same show room, customers came to that show room to make an educated decision. No longer convinced to buy from one manufacturer, you could shop among several models. This was how resellers made their money: providing you a service you'd pay extra money for. Products and manufacturers that failed to compete well in side-by-side arrangements were abolished in the face of consumer choice.

And finally came the Internet. The World Wide Web provided you with not only all the same information the resellers had, but much more: professional and amateur reviews, community-level and industry-specific emails filled with recommendations and warnings, and manufacturers' contact information in case you had questions. Now you could shop intelligently around the world. Much of the resale industry was demolished, now that their services paled in comparison to what consumers could do themselves. Look at the fate of independent bookstores, who all-but-vanished in a wired world where consumers read reviews and compare prices among Amazon.com, BN.com, and Borders.com, only to buy the book from an Internet-based reseller with massively discounted prices. Travel agents, too, disappeared in the face of Expedia.com and Travelocity.com.

It is thus believed, therefore, that the Internet has empowered the individual.

Not true. I'm sorry to say that it's all an illusion.

First of all, the online resellers are no better than the brick-and-mortar resellers. After browsing the options available at an online travel agency, it's often cheaper to then go to the airline site itself to buy your tickets. For example, if I want to fly from Boston to San Francisco, I'll plug my dates into a search engine at Expedia (and later, Travelocity), find the cheapest option at the best times, and then buy my ticket at Delta.com, Southwest.com, or another airline, or perhaps call a human travel agent after all at AAA and start again. As long as I have access to the source of that service or product -- the manufacturer, the service provider, etc. -- the reseller is a source of information without sale.

Second, the online resellers are limited in scope. Thanks to partnerships and other marketing choices, not all of my options are provided. For example, both Expedia and Travelocity tend to overlook small, unaffiliated airlines. Additionally, at one time (and perhaps still today) Expedia charged extra money if I wanted to buy a ticket for USAir flights, without telling me. The bottom line is that going through an online reseller is not necessary more comprehensive or cheaper than my other options.

But biggest of all, however, is that for me to perform ANY search these days, I'm going to have to use a search engine, like Google.

Without even getting into problems with spam, search engines are responsible for providing me with the information I'll need to do anything on the Web, if I don't already know precisely how and with whom to do it myself. Google is the next Sears. If I wanted to find some good choices for a boy's name, Google will provide me with so many choices that I'll inevitably stop after the first twenty (and more likely, stop after three). Google is filtering my search, valuing some choices above others just as my supermarket creates end-of-aisle displays to sell me things. The only difference is that I know the supermarket makes money from the sale. With search engines, you have no way of guaranteeing you're not clicking on a link the search engine company prefers.

Consider the unscrupulous used car salesman. Let's step through the process.

I approach the salesman asking a simple question: "I want a reliable automobile for a good price."

The salesman immediately points out a few models. The first one he shows me is way too expensive. The second one is terrible. In comparison, the third one he shows me seems wonderful at first glance, but then I ask more questions.

The salesman doesn't give me precisely the information I want. Some of his answers sound ridiculous. He's reluctant to show me any more cars. But when I keep pushing, he finally gives in and shows me a fourth car, without much enthusiasm.

Finally, I ask for specific kinds of cars, things I've heard rumors about. "What about a Toyota Sienna? Is there a good Ford minivan?" The salesman is completely unhelpful. Clearly this was a terrible place to come shopping. Maybe I'll visit some dealers, or talk to my neighbor.

Let's compare this to a Google search for boys' names. I choose Google here because it's currently a very popular search engine that, people seem to believe, does an honest job in helping people search both online and offline content.

I start with a simple request: "I want to find a good boy's name." My query is "boys names."

Google gives me some immediate results. Some of them are immediately terrible and can be skipped over, but it doesn't take long to find something promising. I visit the website and, although looks like what I want might be there, I have a hard time using it. I decide to give up and return to Google and its search result list.

I try a second website, but I've lost confidence. Maybe it's not Google's fault in any obvious way, but none of these websites is helping me in the way I want to be helped.

I decide to try some new queries. Maybe "boy names"? Do I need an apostrophe? Or perhaps, because I'm interested in a boy's name that isn't too ethnically different from the names I know in the United States, I should try a search like "American boy names." Unfortunately, my search choices are even worse. I give up. The Web is a terrible place to search for boy names. I'll try the bookstore.

You see? No practical difference.

You might think this exercise was a bit silly, but I'm not wrong. The people, companies, or machines that control what you want are the same entities that control the process. The car salesman controls which cars you buy; even if you trust him, the process is his, not yours. He's just nice about it. The same is true with Google. Sure, we all tend to trust Google -- and what's not to trust or like -- but we do not own the information-seeking process. Google owns it. Here's why:

Not only doesn't Google find everything, it doesn't tell you there are things missing. The Google database isn't as up-to-date as the Web. Your search words don't match every relevant result in every relevant language. Sure, it looks as if there are 2,600,000 hits for you search, but that doesn't mean it found everything. What's more, you can't even see all 2,600,000 hits if you wanted to! Google shuts you out after only a few hundred.

Google doesn't explain what it's doing, or why. The search algorithm is never explained; it's a patent secret. We know what kinds of ingredients go into the mix, but we don't know the precise details. And although sponsored links appear separate from search results -- something not all search engines do -- we have no certainty that there are some other sponsorships happening in there.

If Google is biased, we have no way of knowing. I guarantee Google is biased, because its algorithm is based on how people use the Web. Google News collects stories more often from the AP Wire than the Boston Globe, and more often from the Globe than the Arlington Tab. That's well-intentioned bias. There are less favorable biases, too, like social biases. Because there are fewer computer users who are poor or homeless, the websites of interest to these people never show up at the tops of list. Because the Google default language in the United States is English, U.S.-based news articles are far favored over newspapers in other countries, even when the news takes place in those countries. And because most people have heard of large companies like Amazon.com, smaller companies like independent booksellers are pushed into obscurity. There are also language-based biases. It's easier to find websites related to money because this word is both singular and plural, whereas finance has a plural form. It's easier to search for words like mistress and misogyny, which exist, than for the nonexistent gender-opposite versions. And it's nearly impossible to find a company that sells windows because your search results will be overwhelmed by companies that sell [Microsoft] Windows.

But we don't have a choice. There is too much information in the world. We must go through an information repackager if we're not going to do the work ourselves. (Librarians do the work themselves; the results are of excellent quality, of limited quantity, and of almost negligible relevance for our day-to-day needs of airline tickets and boys' names. Libraries have some excellent information with which we can arm ourselves -- like using Consumer Reports to choose a quality used car -- but in general we still have to take the final steps on our own.)

Regardless of their motives, search engines OWN the information access. Maybe that's good enough. Maybe you're comfortable performing your searches in ignorance of the engine's inner workings, generally satisfied with the results most of the time. But please, that doesn't make it a good thing. What if Google started charging you for some of your searches? What if Google integrated its sponsored links into the search engine (as other engines did or do)?

Here's a real-life, immediate example. Search for Pluto. There has been a ton of recent press regarding Pluto's demotion as a planet in our solar system. Where is all that news in the search results page? There's just a tiny news area that most people won't see because it looks different, and then there's a bunch of sponsored links. This is a branding decision; Google thinks "news" and "sites" are very different things and doesn't even combine their results.

Don't kid yourself. The power of the Internet has moved, but not to you.

We're lost without an information education

2006-08-02T15:44:00.000-05:00

A few years ago, one chapter of the American Society of Indexers created a bumper sticker: "If you don't teach your children about indexing, who will?" Now that my daughter is old enough to repeat words if I ask her to -- "Say cucumber." "Kuku." -- I tried one of my job words on her.

"Say index," I said.
"Icks," she replied.

Given how my colleague Rachel taught her three-year-old to recite how bad a book is if it doesn't have an index, it seems I have some work to do. My daughter shouldn't respond to indexing with icks.

In the United States, children learn about indexes when they are old enough to visit the school library and get instruction on how to use its resources. And while many of the printed card catalogs of my youth have been replaced with computer systems, students are still taught how to use the indexes in the backs of some books. After that, their indexing education is complete. They probably never talk about indexing with the librarian again.

Though brief, even this index education is extremely important. Instinctively, children unfamiliar with indexes will look up information just as adults use a dictionary to look up spellings. For example, if you think deceive is spelled decieve, you'll go to the dictionary to look up decieve. Not finding it, you'll look for a neighboring word that looks somewhat similar, and discover the correct "deceive." In other words, you'll enter the dictionary looking for one word, but be satisfied with another. This is how children use indexes, too. They'll look up "Civil War," not find it, and be satisfied with "civil engineering." Then, of course, they'll fail.

(It is worth noting that adults demonstrate this behavior with indexes, too. I might attempt to look up "potatoes" in a cookbook, yet be satisfied with a result of "potatoes and yams.")

Meanwhile, adults don't instinctively understand the metaphor of things inside things inside things. The well-known marushka dolls, in which a large bowling-pin-shaped doll holds a smaller doll that holds another doll, and so on, is endlessly fascinating for children. As adults, we're fascinated by the plots to suspense novels. Each step along our way -- an uncovered doll, a turned page -- is built upon the past in a linear way. We follow events, from first to last, in linear sequence, and we succeed.

Hierarchical organization, in contrast, has no obvious place in human existence. To survive, it's enough to separate things into only two groups at a time: dangerous vs. safe, edible vs. inedible, alive vs. dead, something we like vs. something we don't like, family vs. nonfamily. As intelligent creatures we might create a few more categories at a time -- family, co-workers, non-work friends, acquaintances, strangers -- but rarely do we construct them into layers like "people I know > people I like > people I like to work with." Layering is completely unnecessary in our daily lives. Perhaps it is for this reason that human beings cannot instinctively organize things in a hierarchical way -- in the same way we can't tell the (very big) spatial difference between one million miles and one billion miles. To do these things, we need training.

You know, we don't do math naturally, either. Our instincts tell us the difference between one item, two items, a few items, many items, and very many items, but that's it. We also understand more and fewer. But we don't have an instinct that tells us how to add or multiply, let alone solve calculus problems. (If you don't believe me, then I dare you to cut a pizza or a cake into five equal slices without making a mistake.)

Today, we have math classes. Before math was taught as its own course, certain elements of math were taught within specific subjects. Shipbuilders and shoemakers learned enough math to do their jobs, and that was it. The idea of teaching math independent of application must have seemed very strange. What good is shipbuilders' math to shoemakers? But eventually, the math-proficient individuals in each field spoke to one another and discovered exactly what they had in common: a need to add numbers together, a need to calculate weight, and a need for geometry. Now math is an integral part of standardized testing, which means students aren't allowed to graduate from school without proving themselves in basic math skills, separate from their application.

So why aren't we teaching information the way we teach math? Information classification exists in every field of human exploration, from literature (divisions of author style or message) to sales (styles of negotiation), and from biology (life classifications) to auto mechanics (systems of function). If a student is going to learn anything about anything, he should learn a little something about how information itself fits together.

The impact a basic, application-independent information education can have is astounding. As an example, consider driving directions. In general, we give directions to people in a linear order, something that makes sense given how we travel. Here is how you can get to the post office near my home: "(1) Take route 95 until exit 26. (2) Take route 2 East until exit 59. (3) Take route 60 into Arlington Centre. (4) Turn left onto Massachusetts Avenue. (5) After three blocks, turn right onto Court Street. (6) The post office is on your left at the end of the street." As I said before, you don't need information hierarchy to survive; following these linear directions is quite easy. But suppose you make a wrong turn, or miss your exit? To find your way back to the path I provided, you need to know something about the geographic layers that make up these regions: "greater Boston > north Boston suburbs > town of Arlington > Arlington Centre area > Court Street." You need a hierarchical knowledge of the area! Put another way, what many of us refer to as "a great sense of direction" is actually "a deep understanding of relevant geographical hierarchies." That's why someone who knows their way around New York City will get lost in the woods: they learned how NYC streets fit together (NYC > Manhattan > Upper East Side > etc.) but learned nothing about forests. Get my point? Sense of direction is taught and learned.

It's time for us to start teaching information construction in schools. We're lost without it.

Meta-indexing

2006-07-10T19:16:00.000-05:00

Many indexers in the business are frustrated that search engines are getting so much publicity and credit, when in fact indexes are, well, more effective!

Usability testing has shown that people prefer indexes over search when it comes to accuracy and comprehensiveness, and yet the same tests show that people prefer search as a technology. In other words, people prefer search engines even though they unanimously agree that indexes are more accurate. This is not that different from the person who insists on lifting the heavy box himself, even though someone stronger and better able has volunteered. "No, no, I'll do it myself," says the searcher.

And then he injures himself. Silly, silly person.

What is at stake, apparently, is something more psychological or emotional. Search engines may offer users a sense of power and control, or a sense of speed, that indexes don't. Further, indexes seem so much more complicated when you glance at them -- words, words everywhere -- and in comparison search is so simple: an empty box. Just type a word and bingo! If you were to stop and look at this behavior you'd realize that there's something subsconscious going on; rationality is losing to some deeper sense of emotionality and self. Search simply feels right in a way that using an index does not, at least not instinctively.

Some indexers take this news with a strong sense of pessimism, seeing this "shift toward the emotional" as paralleling our current lifestyle of sensationalist news and entertainment. They believe that indexes will become extinct in most practical circumstances, because search engines are psychologically preferred -- not to mention faster, cheaper, online, and scalable.

These pessimists aren't wrong.

However, I contend that the pessimists are also looking at the situation completely upside-down. Ask yourself what makes a search engine effective or likeable at all -- that is, what does Google have that seems to draw a majority of Web users not only to the Google.com website but also to license Google technology at their own sites -- and you'll realize that there's indexing on the back end. People don't call it "indexing," necessarily, but the intellectual, rational processes that comprise indexing are still taking place.

The difference, however, is that a search company like Google doesn't really look at the individual words and their instances. Instead, the designers of Google search (and other tools) are looking at how people respond to these words. They are looking at behavioral patterns, and using those patterns to do the indexing for them.

My brother and I used to play a game at ballparks. One of us, when it was his turn, would attempt to turn as many heads as possible without speaking. My brother would turn his head and look over his shoulder casually, then allow his eyes to lock on something imaginary but far behind all the people sitting behind us. He'd tap my on the shoulder and get me to look; I'd play along. He'd point. I'd point, and he'd correct me. Then he'd stand up. And so on. After a while, some of those people who can see us directly in front of them would be curious to know what we're looking at, and they'd turn their heads to see. This would inspire other people to turn their heads, and so on. If we'd done our job well -- it was a game of timing as well as body language -- we could get hundreds of people to look behind them, at nothing.

This kind of behavior explains the popularity of some really stupid websites. Get enough people to visit your website, and Google will acknowledge that there's something about this website worth looking at. Then more people will look at it. Internet-based fads occur weekly, from paparazzi photos to cool advertisements.

If a human were indexing this, the indexer might think, "This isn't so important that it needs to be found a million times." That human is right. But the meta-human looks at what all the humans are already doing and thinks, "There is a cultural need for this content."

For a back-of-the-book indexer to break into the world of mass search, he'll have to give up the words and instead figure out the rules -- linguistic as well as social -- behind how these words are being used. Those rules, which govern how we find things (and not what we find), don't describe the indexing we know at all.

If indexing as we indexers know it is going to survive, we'll have to find that nifty middle ground between the words and the people. It should be easy, given that we already do this, but so far we haven't managed to break into this field at all. Hopefully we'll evolve.

In the next generation, we'll index the indexes.

(Brian Pinkerton developed the first full-text retrieval search engine back in 1994. "Picture this," he explained. "A customer walks into a huge travel outfitters store, with every type of item, for vacations anywhere in the world, looks at the guy who works there, and blurts out, 'Travel.' Now where's that sales clerk supposed to begin?")

The detailed analysis of indexing mistakes

2006-07-04T11:04:00.000-05:00

In linguistics, the analysis of error is one means of learning how we cognitively process language. For example, when someone accidentally misspeaks "unplugged the phone" as "unphugged the plone," we discover that both the speaker is a visual learner (because he switched the P blends in the phrase, despite their different sounds) and that the speaker processes language in its component sounds. On the contrary, a speaker who says "unphoned my plug" processes language in morphemes (e.g., root words), and a speaker who says "unplugged my feet" is an aural learner (because phone and feet start with the same f sound). There seems to be an infinity of spoken-language errors possible, including absences, duplications, inclusions, misalignments, substitutions, and transpositions of letters, sounds, morphemes, words, and phrases.

When I evaluate an index, my job is to look for mistakes. As a now-experienced indexer who himself has made mistakes, I know that I can learn much about how an indexer thinks (or doesn't think) by analyzing her errors and accidents. And as with speech, there are innumerable kinds of mistakes available for the unwary indexer: absences, duplications, inclusions, misalignments, misrepresentations, and missortings of page numbers, letters, words, structures, and ideas.

Consider the incorrect page number, such as when content on page 42 is indexed as if it were on page 44. This kind of error tells us that the indexer did not attend properly to detail, perhaps because the working environment (deadlines, tools, etc.) was less than ideal. When a page range appears simplified to a single number, such as when 42-45 appears simply as 42, I am more likely to consider the indexer lazy instead of scatterbrained, though again it is also possible to blame the working environment (including client demands).

Entries that appear in an index but have no value to readers (e.g., the inclusion of passing mentions and other trivia) demonstrate the indexer's ignorance of the audience, or of the indexing process itself. Entries that fail to appear in an index but should (e.g., the under-indexing of a concept) demonstrate either the indexer's ignorance of the audience, the indexer's ignorance of the subject content, or a sloppy or otherwise rushed working process.

Awkward categorizations, such as entries that are mistakenly combined or that doesn't relate well to their subentries, are a clear sign that the indexer misunderstands the content or is too new to indexing to understand how structure is supposed to work. For example, an indexer who creates

American
....Idol (television program), 56
....Red Cross (organization), 341

doesn't think of indexing as a practice of making ideas accessible, but rather as a concordance of words without meaning. Under no circumstances should American Idol or American Red Cross have been broken into halves, let alone combined. Since categorization can be subtle, however, evaluators can learn something interesting about indexers by looking closely at their choices:

writing
....as artistic skill, 84
....fiction vs. nonfiction, 62

In this example, the first subentry defines writing as a trade; it's clear the indexer is comfortable with the idea of a writer. The second subentry defines writing as a process, with a start and finish, such that the process (or journey) of writing could be different when you're writing fiction instead of nonfiction. Analysis of this entry tells us that the indexer doesn't recognize or appreciate the difference between writing (trade) and writing (process). Is the indexer revealing her inner disdain for writers, does she believe that all writers are the same no matter what they produce, or does she simply know nothing about the writing life?

One of the big challenges for indexers is to provide the language that readers will need to find the content they're looking for. When an indexer either offers language that no one will look up or omits the terms that readers prefer, she is demonstrating an ignorance of the audience or of the content, or hinting that the overall indexing process or environment is inadequate. Further, when the indexer fails to provide access from an already existing category entry (for example, if the index has an entry for "writing, fiction vs. nonfiction" but fails to provide the cross reference "See also author" when there are author entries), she tells us clearly that she is unfamiliar with the material. No other combination of errors speaks of subject ignorance as clearly; by failing to connect existing concepts, the indexer shows us gaps in her knowledge of the information map.

There are several kinds of text errors. Misspellings and other typographical errors are a sign of carelessness or insufficient tools. Accidental missortings are a sign of ignorance, poor tools, accelerated schedules, or a failure of communication among publication staff. Ambiguous terms that aren't clarified are caused by indexers who are too limited in their thinking or their assumptions about the audience, indexers who don't know the material, and authors who failed to communicate the ideas clearly enough for the indexer to understand. Finally, odd grammatical choices usually signal a poor production process, such as when two indexes are combined automatically with insufficient editing effort, or a brand new indexer with no formal training.

Before concluding, I would be amiss to ignore errors of formatting. A failure to use consistent styles signals a deficit in tools or attention, whereas awkward or unreadable decisions regarding indentations, margins, and column widths are a big sign that the index designer (who is not necessarily the indexer) has no clear idea whatsoever how indexes work. Missing continued lines communicate the same thing. (On the other hand, exceptional use of formatting, such as the isolated use of italics within a textual label, is a clear sign that the indexer really does understand both the audience and how they approach the index.)

Ignorance, sloppiness, indifference, and confusion: these are shortcomings even a professionally trained, experienced indexer might have, but thankfully they often manifest as isolated exceptions in her practice of creating quality work. But when a single kind of mistake appears multiple times throughout an index -- numerous misspellings, huge inconsistencies of language, globally insufficient access, awkward structures -- we need to be concerned. When we see these, we have an obligation to analyze the indexer. By properly arming ourselves with this knowledge, we can determine for ourselves if the indexer was the wrong choice for a particular project, struggled with the challenges of inferior tools, or simply had a bad day.

Meanwhile, if indexes written by different indexers are plagued by the same exact problem, it's unmistakably clear that the problem is in the systemically faulty publication process: ridiculous deadlines, uncooperative authors, uncaring editors, poor style guides, and so on. In other words, you shouldn't evaluate indexes in isolation. Instead, look at the work of other indexers for the same publisher, as well as the work of other publishers by the same indexer.

Okay, but what if the index is essentially perfect, with no errors at all? Can we still learn something? Yes, we can. The absence of all error tells us something very important about the indexer: She's being underpaid.

With respect to our loved ones

2006-06-26T08:31:00.000-05:00

There are many who believe that books -- and by this I refer to the traditional object of bound paper and print -- are sacred objects. Anne Fadiman wrote in Ex Libris that books are often marketed as if they were toasters, and yet remembered as if they were friends. Certainly a block of paper, ink, and glue would not hold such an esteemed place in our hearts if there were nothing transcendent about it. The way in which we archive old books on our shelves because of the memories they inspired in us, whether as a favorite book from early childhood or an intellectual realization from our older years, is not unlike the way a museum places found bones under glass. Unlike the skeletal remains of an ancient animal, however, our old books are neither unique nor unused nor truly old.

The loss of such a book -- from spilled juice, from a disrespectful borrower, from a forgetful moment on a bus -- can be devastating. Almost all titles can be repurchased, in some cases with benefits like a new introduction by the author, an improved detail of scholarly footnotes, or a respectful commentary written with the benefit of time. Rarely, though, is it the corporeal book itself that brings us such pleasure. Had the book been empty, like those ubiquitous writing books and diaries sold at the checkout displays of almost every bookstore, its loss would have gone mostly unnoticed, valued at approximately the retail cost printed on its back cover. No, it is the content that brings us pleasure, with its memories of having been explored.

Content is what makes our books cherished items. Porcelain figures, music boxes, ticket stubs, and toy animals have memories but no inherent content, whereas books can be reopened, reread, and rediscovered. Even when the words don't change -- and they rarely do -- the experience of seeing them with changed eyes and minds is different each time.

Listen, friends, for this is the romantic side to indexing.

As readers -- gentle, voracious, impulsive, or any other adjective that best defines the nature of your reading relationships -- we have an obligation to provide access to these memories, past and future. Even if we do not write, we must endow the writings of others with every tool at our disposal. We cannot guarantee that any single book won't get lost in the attic or destroyed by fire, but we can, as indexers, guarantee that every important sentence within is flagged with accuracy and passion. We can, in the end, turn the writings of others into useful thoughts, moments of learning, and renewable tools for discovery and self-discovery.

Bias in indexing

2006-05-18T21:55:00.000-05:00

The greatest advantage that indexing processes have over automated (computer-only) processes is the human component. Of course, as someone who has worked with humans before, you probably recognize there can be imperfections.

I was reading Struck by Lightning: The Curious World of Probabilities earlier this week, in which the author writes of biases in scientific studies. I realized that these same biases occur with indexes and indexers as well, and I wondered if I can list them all.

(The biggest bias in indexing isn't one of the index at all, but rather the limitations on what the authors write. For example, if a book on art history didn't include information about Vincent van Gogh, I would expect van Gogh to be missing from the index; this absence might be caused by an author bias. However, I am going to focus on biases that affect indexing decisions themselves.)

Inclusion bias. Indexers may demonstrate a bias by including more entries related to subjects that appear more interesting or important to that indexer. For example, I live in Boston, and so I might consider Boston-related topics to be less trivial (more important) than the average indexer; consequently, documentation that includes information about Boston is more likely to appear in my index. I imagine inclusion bias is a common phenomenon in documentation that includes information about contentious social issues -- immigration, tobacco legislation, energy policy -- because the drive to communicate one's ideas on these issues is stronger. I also believe that inclusion bias is not entirely subconscious, and that indexers may purposefully choose to declare their ideas with asymmetric inclusion. It should be noted, however, that biased inclusion would not necessarily provide insight into the indexer's opinion on the subject; creating an entry like "death penalty morality" does not clearly demonstrate whether the indexer actually disagrees with capital punishment.

Noninclusion bias. Similar to inclusion bias, indexers might feel that certain mentions in the text are not worth including in the index because of their personal interests or beliefs. Unlike inclusion bias, however, I suspect noninclusion bias does not appear in regards to contentious issues; conflict is going to be indexed as long as the indexer recognized the conflict has value. Instead, an indexer is likely to exclude things that "seem obvious"; rarely are these tidbits of information controversial. For example, an indexer who is very familiar with computers is likely to exclude "obvious computer things," subjectively speaking; you probably won't find "keyboard, definition of" in such a book.

Familiarity (unfamiliarity) bias. When an indexer is particularly interested in or knowledgeable about a subject, the indexer is likely to create more entry points for the same content than another indexer might. For example, an indexer who is familiar with "Rollerblading" might realize that Rollerblade is a brand name, and that the actual items are called inline skates. This indexer is more likely to include "inline skates" as an entry. Unfamiliarity bias would be opposite, in that multiple entry points are not provided because the indexer doesn't think of them, or perhaps doesn’t know they exist.

Positive value bias. An indexer who has reason to make certain content more accessible for readers to find (and read) is likely to create more entry points for that idea. At the extreme, the indexer will overload access by using multiple categorical and overlapping subtopics, where those subcategories are at a higher granularity than the information itself. For the generic topic of "immigration," for example, an indexer might include categorical entries like "Hispanic immigrants," "European immigrants," and "Asian immigrants," as well as overlapping topics like "Asian immigrants," "Chinese immigrants," and "Taiwanese immigrants," with all of them pointing to "immigration" in general.

There are three types of positive value bias. Personal positive value bias is demonstrated when the indexer himself believes that the information is of greater-than-average value. Environment-based positive value bias is demonstrated when the index is swayed by environment forces, such as social pressures, political pressures, pre-existing media bias, and so on. Finally, other-based positive value bias is demonstrated when the index bows to pressures imposed by the author, client, manager, or sales market (i.e., the person paying the indexer for the job). Although it can be argued that this last type of bias is not the indexer's bias, strictly speaking the indexer can choose to fight any bias forced upon him. For example, a client who instructs the indexer to "index all the names in this book" might interpret this instruction as some kind of market bias, and thus refuse to follow this guideline. In reality, however, most indexers do accept the pressures placed upon them by the work environment, and thus in my opinion take on the responsibility and ethical consequences of this choice.

Negative value bias. It's possible for an indexer to provide fewer entry points for content that he feels is not of great importance to readers -- the direct opposite of positive value bias -- but the reasons for limiting access to that content are probably not related to indexer's perceived value of that content for readers. Instead, indexers are likely to limit access to content when there is a significant amount of similar content in the book, and as such including access to those ideas would either bulk up the index unnecessarily or waste a lot of the indexer's time. For example, if an indexer were faced with a 40-page table of computer terms, it's unlikely that each term would be heavily indexed, even if such indexing were possible and even helpful to readers.

For this reason, I believe that there are three kinds of negative value bias: time-based negative value bias, in which the indexer skimps on providing access in an effort to save time; financially motivated negative value bias, in which the indexer skimps on providing access in an effort to earn or save money; and logistical negative value bias, in which the indexer skimps on providing access in response to logistical issues like software limitations, file size requirements, page count requirements, controlled vocabulary limitations, and the like.

Topic combination (lumper's) bias. This bias is exhibited by indexers who are likely to combine otherwise dissimilar ideas because they find this "lumping together" of ideas to be aesthetically pleasing or especially useful. This kind of bias is visible in the ratio between locators (page numbers) and subentries, in that entries are more likely to have multiple locators than multiple subentries, on average. For example, an entry like "death penalty, 35, 65, 95" shows that the indexer believes the content on these three pages is similar enough that subentries are not required or useful. Topics that start with the same words might also be combined in a more general topic (such as combining "school lunches" and "school cafeterias" into a combined "school meals.") It is worth noting that some kinds of audiences or documentation subjects may tend toward topic combination bias; for this reason, it may be difficult to recognize lumper's bias.

Topic separation (splitter's) bias. This bias is exhibited by indexers who are likely to separate otherwise similar ideas because they find this "splitting apart" of ideas to be aesthetically pleasing or especially useful. As with lumper's bias, splitter's bias is represented by the ratio of locators to subentries throughout an index, in that splitters are likely to create more subentries than would other indexers, on average. It is worth noting that some kinds of audiences or documentation subjects may tend toward topic separation bias; for this reason, it may be difficult to recognize splitter's bias.

These are all the biases I've found or experienced. If you think there's another kind of bias that indexers exhibit, let me know.

The remaining question is this: Is it wrong for an indexer to have bias? That is, should indexers study their own tendencies and work to avoid them? I don't think it's that simple. The artistry that an indexer can demonstrate is fueled by these biases -- experiences, opinions, backgrounds, interpretations -- and perhaps should even be encouraged. An indexer's strengths come from his understanding of not just the material, but also his perceptions of the audience, the publication environment, and the audience's environments. Further, indexers who know and love certain subjects are going to be drawn to them, just as many readers are; these biases aren't handicaps so much as commonalities shared between indexers and readers. Biases will hurt indexers working on unfamiliar materials in unfamiliar media, but under those conditions the biases are the least of our worries; when the indexer is working without proper knowledge, the higher possibility of bad judgment or error is a much greater concern.

If anything, indexers should be aware of their biases because they can serve as strengths -- especially in comparison to what computers attempt to do.

Demands for quantity are misplaced

2006-05-14T18:17:00.000-05:00

There is a continuing trend in search engines: more, more, more.

Press releases from Google, like “Google Checks Out Library Books” [December 14, 2004] and “Google Tunes Into TV” [January 25, 2005], hit the media waves in a grand style. For the first time, entire libraries of books, from Harvard and Stanford Universities to the Universities of Michigan and Oxford, and soon the New York City Public Library, will be available from Google’s website. Within limits of copyright, the words of entire books can be searched. Information philosophers are all over this story, essentially declaring that Google will become the public library of the next generation, excited about how the very nature of libraries might change, and scratching their heads over how the book publishing industry is going to survive yet another hit in the market.

In the second, Google (as well as Yahoo!) applauds themselves for once again providing access to a greater diversity of the world’s information, because television’s closed captioning content has been indexed into a Google Video database. Viewers of public broadcasting and basketball are early adopters, and why not? Finally, all those oh-so-deprived sports consumers can satisfy themselves on more than just the videos, statistics databases, press releases, articles, blogs, commentaries, and (don’t forget) live games themselves. Because now they can search among the announcers’ words.

Now when I search for 76ers, instead of getting 1.61 million hits, I’ll get 1.62 million. Phooey.

Our instinctive reaction is to be impressed. I’m thinking of all those Ph.D. theses gathering dust in the Physics-Optics-Astronomy Library at my alma mater, 150-page books without indexes. I’m thinking about Red Sox fans who, for the first time in a very long time, are interested in the World Series.

But I’m also thinking about the catalog search system at my public library, which won’t improve with Google’s additions. For a book already in the catalog, adding its content doesn’t help at all. Instead, we’d be cluttering up the database with a few trillion new words.

So our instincts are wrong.

Why Quantity Hurts

Every few years, search engine companies are finding new ways to promote themselves by bragging about how much they can find. In the early 1990s, Northern Light was independently rated top among competitors because they searched the largest percentage of the World Wide Web: sixteen percent. Time and Newsweek contributors warned, “We’re not finding everything.”

In the late 1990s, articles about the “invisible web” appeared in popular magazines and newspapers, explaining how search engines cataloged only text, image, and sound files, thus skipping over the good content stored as spreadsheets, databases, and fonts. Again came the cry, “We’re not finding everything!”

And now, in just two months, Google and Yahoo have added more to the huge pile of information: library books and closed captioning data. No longer are our searches limited to the billions of files already on the Web. Yay!

It’s all about quantity. Nobody seems to care about quality any more.

Libraries have been struggling to redefine themselves ever since the Internet (and more, the World Wide Web) reached people’s homes. Book publishers also have suffered. The failure isn’t that of these institutions and industries, however, but of the public. The public seems unaware of the natural filtering process inherent in human behavior. Publishers choose which titles to publish, and libraries choose which titles to add to their catalogs. You might not agree with their reasoning or results, but you do have to admit that there are human beings at the helm.

(By the way, Google's library program was put on hold because of criticism. This isn't because it's a bad idea or anything, but rather because the traditional publishing industries got scared they'd lose money. It's an I-was-here-first money-by-copyright battle.)

For many, this filtering-by-design is a major disappointment, which explains the astounding popularity of the World Wide Web. The Web allows everyone to speak up: to post pictures of their pets and babies, their ideas about government, their Harry Potter fan fiction. But in a room where everyone is shouting, nothing gets heard. Northern Light and Yahoo earned their money offering a way through the noise. Other companies, calling themselves search optimization experts, profit by offering their clients the means to be noticed by these search engines, the hypertext equivalent of megaphones.

The filtering process is missing.

Okay, yes, adding content to search databases is a good thing. I might make fun of sport fanaticism, but the desire to retrieve information of choice is a valuable privilege of the individual. I might not care about the Red Sox, but I respect that there are others who do. I also feel extremely happy for the researchers who now have access to volumes of scientific research. In many ways, adding content to a database is like translating content into new languages. Really, these are not bad things.

Even so, we are only adding to the number of people shouting in a room. Google and Yahoo are definitely improving the scope of what we can find, but they are not improving our ability to find. I might proudly accumulate more and more in my attic, while simultaneously making it harder and harder to retrieve anything.

Here’s a real example. My wife and I had been expecting our first child (15 months ago). We were struggling in our decision of her middle name. When we searched for “baby names” on the Web, with quotation marks around the phrase, we found 2.5 million sites at Google, 0.9 million at Yahoo, 0.7 million at MSN. Now, imagine that all the contents of library books and scientific articles and sports broadcasters are added to the Web. Although there may exist a few anthropology articles that would have helped us choose a name, I sincerely believe that over 99% of this new content would have proven unhelpful. I also believe that some of this unhelpful content includes the unusual phrase “baby names.” For example, consider this sentence, which appeared on the Web in November 2004: “Julia Roberts now joins the list of celebrities who have jumped on the Hollywood bandwagon, which gives license to choosing odd baby names.”

After my wife and I decide on a candidate name, we search for that name online, looking for its meaning. The query “Ryan meaning” (without quotes) for the boy’s name Ryan gets 1.2 million hits at Google. (By the way, Google suppresses near-duplicated content and never displays beyond the first 1000 results, so the “true” result set is quite inaccessible.) Because Ryan is a common name, it likely appears numerous times within bibliographies. The word meaning is also extremely common among scholarly articles. If Google indeed adds university libraries to it’s already large database, 1.2 million will become a very small number. As the scientists benefit from a library search, my wife and I will find it that much harder to learn about a particular name using Google.

It is common knowledge among library scientists and search engine experts that you cannot improve the accuracy of a search at the same time you improve its comprehensiveness. Either you get perfect relevance but miss something useful, or you get everything you want along with content that you don’t. As search engines trend toward larger and larger databases, results pages grow more cluttered.

Please, Sir, May I Have Some Less?

Google’s popularity as a search engine has nothing to do with the numbers of results. When I ask people why they like Google (or whatever search engine they prefer), they answer, “Because what I want is usually within the first few results.” I’ve also gotten the answer, “Because it thinks the way that I do.” Most people don’t want millions of results. They want three. Three quality results.

I can’t remember the last time I heard about a search engine improving its algorithms. Perhaps they do this all the time in secret, inventing features behind the scenes. I do know that if a search engine started regularly serving up nonsense, it would go out of business.

So why are these efforts at improving quality so unpronounced? Did you know that while Google pays attention to quotation marks, Lycos doesn’t? That you can type a zip code into Google to get a map? Many of Google’s best features are published in books like Google Hacks and even Google Maps Hacks (O’Reilly & Associates, 2004 and 2006 respectively), where few people are going to look for them. Either nobody cares, or nobody knows the difference.

But when Google adds sports commentary to its search engine, watch out! The story appears in all the major newspapers.

At times like these, I get rather discouraged. I feel as though I am trying to hold back the ocean. It takes me more than a week to index a single, average book. The information world is growing at such an insane pace, my job seems absurd.

At times like these, I have to remind myself of two perspective. First, context. When I write the index for a book with 350 pages, it doesn’t matter that the “book of Google” has over 8 billion. Someone decided that these 350 pages needed to be written, and it’s my job to make them accessible. My work improves this book. No, I haven’t changed the world, but I have made a difference within the context of this one book, in a segment of this one industry, to a small set of readers. For me, indexing is like the civic duty of voting: few win by one vote, and yet every vote counts. It’s also contagious, because voting begets voting. And indexing does beget indexing, because 5% of the people I talk to about my job want to know more, offer me work, or express a desire to become an indexer themselves.

The second perspective is one of application. I don’t have to index books. If I wanted to make a difference at the source, there are many other applications of skills.

The key to these perspectives is a willingness to become activists. We are environmentalists in an information world. Just as scientists show concern with over a one-degree rise in ocean temperature, so should we show concern with a one-percent increase in information dissemination. Bulk up our search engines? This is not an environmentally friendly choice.

I want a search engine—it doesn’t even have to be Google—to announce that they’ve found a way to help me filter out the pages I couldn’t possibly want. The important word here is announce. The modus operandi of these companies is to bring more shouting people into the room, and then publicize this with pride. No wonder the libraries and publishers are in trouble: they’re not being praised for what they do. Neither are the indexers. In the public media, quality gets a whole lot less attention than quantity.

If the search engine is being improved, they’re not telling anyone. Apparently it’s a secret. I don’t want to hear that someone has added billions of pages to the database, unless I hear also about a system that filters billions of pages away.

I think it’s wonderful that more esoteric content is being added to the database. I applaud search engine companies who continue to improve their algorithms. What drives me crazy is that everyone is talking about the first, but not the second. The publicity is lopsided. Why won’t anyone talk about quality any more?

When we asked search engine companies for more, that’s what we got. And we lost precision. Maybe it’s time for us to ask for less.

Library of Congress Control Number sj 96004876 ("Babies")

2006-05-12T08:34:00.000-05:00

If any of my faithful readers were growing concerned about my e-silence, I'm pleased to say that my reason for muteness is wonderful: Oren Harold Maislin, born April 24, 2006. Perhaps coincidentally, this is the 207th anniversary of the establishment of the United States Library of Congress by President John Adams. For a full story, visit here.

Whatever happened to "indices"?

2006-04-04T20:44:00.000-05:00

My uncle (among others) asked me, "Is the word indices no good anymore? It seems that indexes has won."

I didn't know those words were contesting, but yes. If a U.S. winner were to be declared today, I'd have to go with indexes.

Although strictly speaking the correct term is indices, I think in common speech a distinction has been made for an item that is rarely plural. For example, when referring to an appendix in the back of a book, often there is more than one: appendices. However, when referring to the (vermiform) appendix in the human body, rarely do you talk about more than one at a time, and thus "appendixes." (And then there's the acronym, APPENDIX, which simply doesn't count.)

With book indexes, there is rarely more than one -- although you can certainly talk about the indexes across the books, as I do. But in database programming and similar constructions, often each line in the database (each record) has its own index. And so you can have thousands of indices. When you're working with indices (as opposed to indexes), you're working with large quantities of small bits of information.

To me, this logic is what's also behind such oddities as the words "persons" and "peoples." These terms, though related, are attempts at showing quantity in environments where quantity is much less likely. Said another way, these words are attempting to emphasize the value of the singular, even while referring to more than one. Thus "persons" is used in legal contexts where the individual is important, "people" is referring to a group of beings in which individuality is not important, and "peoples" is referring to a collection of groups of beings in which the nature of each group remains important.

How's THAT for an answer?

Of course, the only real test is if there are other words that seem to follow the same pattern. So far, I can think of only index and appendix. Others?

Standards in the indexing community

2006-04-02T21:28:00.000-05:00

The American Society of Indexers has been deluged with opinions regarding its venture into credentialing individual indexers. Leaving aside all legitimate concerns about the implementation of credentials, there are still many strongly voiced opinions about whether credentialing is a good idea in the first place.

One thing about indexing certification and index standards building that I keep getting stuck on is the question of who they're really for. There are indexers who believe credentialing will impact all indexers in a negative way, or just ASI members. There are many who suspect new indexers will benefit, and that everyone else will get hurt.

But I don't believe the truest benefits to standards building are about the individual indexer. There will be effects, and certainly those effects will be different for different people, but the whole reason standards are created are to improve the industry as a whole. At least, that's how I understand it.

There is a standard for indexing already out there. It's the ISO 999 standard. Does it benefit you? If you were indexing when it was updated in 1996, did you feel any repercussions in your business? Gosh, it sounds as if I'm joking.

Recently I learned that Massachusetts now requires carbon monoxide detectors on every floor. That means I need to install two more in the house. If I don't install them, I won't have a problem until I attempt to sell the house. And truthfully, I'm kind of annoyed that safety regulations are being pushed onto me -- the vehicle seat belt law, the bicycle helmet law, and now this. I'm not saying that seat belts and helmets are stupid things, but I don't like feeling forced into wearing them.

ASI is attempting to create a standard -- certainly one that is harder to define than "wearing a seat belt or not," I'll freely admit -- and so yes, it is a bit disconcerting to be on the receiving end. I'm doing just fine without my carbon monoxide detectors, and I'll do just fine without professional credentialing.

But ask yourself if you believe in the ideal here. Do you believe that standards *should* exist? You don't have to.

I do.

There are many industries that survive very well without some kind of license, certificate, or degree, and some might think that indexing is one of them. But no matter how it affects me personally, I really believe indexers should be a part of a larger entity, something more important than a simple networking community. Do we -- and I mean ALL of us who write indexes -- have anything in common? Do we have anything to fight for as a group (other than higher rates)? Nah. As an industry, we really don't stand for anything without a standard. We're just a bunch of word inventors -- tinkerers -- working alone in our attics.

Most days, I'm fine standing for nothing. I like my income, I like most of my clients and projects, and I absolutely love being able to work at home and raise a daughter who stops to smell the flowers. Some people like Sudoku and crossword puzzles; I like indexing and teaching.

Other days, I feel like someone bailing out a rowboat with a hole in it, with indexes as buckets. I am frustrated by a complete lack of growth in the industry, by repeating myself to every new production editor I meet, by fighting for the right to use page ranges or decent indexing software, and again and again by having to justify my very reasonable rates.

The deeper meaning that can come from credentialing also comes from other things: professional development, education, and research -- including all those great index usability research ideas. Credentialing isn't the only pursuit of this association, nor would it truly succeed in isolation. But I need to be a part of an association that advocates not just for its individual members, but for the meaning of the industry itself.

Credentialing can be part of the solution.

The granularity of an online "page number"

2006-03-24T09:23:00.000-05:00

When writing a hyperlinked index (where hyperlinks are used instead of page numbers), to what should those links point?

Some people think they should point to the section title in which the information is provided; other people like to point right to the specific word used in the index. The "answer," obviously, is that hyperlinked index entries should take the readers to where the information is, right? The problem -- the reason there's this question of "where do I point my entries" in the first place is that readers of hypertext might find themselves bounced somewhere they don't understand. How many times have you followed a link, only to find yourself fiddling around with the scroll bar to figure out where you ended up? Following a hyperlink is like being blindfolded and transported to an unknown destination.

What you may not know is that book indexes aren't much different. :-)

Think about how book indexes actually work, and you realize that direct readers to the page on which the information starts. An entry like "buoyancy, 164" tells the reader to look somewhere on page 164; an entry like "global harmony, 164-167" tells the reader to start looking somewhere on page 164. The granularity of an index is defined as the smallest unit of area that can pointed to. For printed indexes, this area is the page number. Rarely will you find locators that use fractional or qualified page numbers like 164-1/2 or 164top. (There are such things as qualified locators, like 164f, which might point to the footnote on page 164, but even in the books in which they're used they comprise only a small number of all locators used.)

If you follow the standards of the industry, then, the granularity of a printed index is one physical page. For this reason, books that have lots of words on a page -- big pages, narrow margins, tiny print -- are less friendly to book indexers. It's like telling someone that there's a needle in that 164th haystack over there. Maybe we should count our blessings that someone bothered to number the haystacks, but ideally this is where the book designer starts earning her salary. Book pages don't have to look like haystacks -- more accurately, wordstacks -- if the book has legible headings and subheadings. Books can be written with quickly visible landmarks within the pages, like italics and boldface, larger and smaller font sizes, headings and callouts, footnotes, and so on. Going back to the blindfolded analogy, there's no reason we have to drop our readers into deserts of information, when we can drop them in a place surrounded by location clues and navigational signs, like at a train station.

On the Web, however, there is no such thing as a printed page. Web pages can be any length, from tiny pop-up windows with only a sentence fragment of information within, to long scrolls of endless paragraphs and images. Additionally, you don't have to direct the reader to just the page any more, but rather you can deposit him anywhere within the page. The granularity of a Web page is a word! You can send someone into the middle of a paragraph.

When you have tiny little windows of information, using that window as a destination is a no-brainer: the reader arrives at a single sentence of information, which is what he needs. It doesn't matter if you point him to the beginning, middle, or end of that sentence, because it's all they get to read. Pointing someone to an isolated window of information -- what Web authors call "chunks" -- is as easy as looking into a food pantry that contains only a single can. But when you have longer pages, and you have the ability to point someone to any spot within those longer pages, you have a decision to make. And it's a decision that didn't exist in the printed world, with its larger granularity.

The solution is to connect the text of the index entry with the text of the documentation. Not the meaning, but the actual words. If the index entries are written to almost identically match those of the documentation, then the reader won't mind as much because it won't look like a desert. They'll have exactly the landmark they need right in front of them. The entry "cancer, prevention of," for example, could point directly to this line without a problem:

... cessation of smoking. In fact, many physicians are well aware that one way to prevent cancer is to quit ...

That's because the words of your index entry, which are cancer and prevention, appear almost verbatim in that line of text. And if this information were part of a section titled "Using Peer Pressure to Help Patients Quit Smoking," then you really wouldn't want to point to the heading for context. That's because it's unclear to the reader that you're actually directing him to information about cancer or prevention. You're making them work at it.

And then there's the other situation. Using the same sentence and heading as above, where should the indexer point readers who look up the entry "smoking, how to quit"? Clearly they should go right to the heading. If they went to the line that talked about physicians, they wouldn't know where they are.

Our original question here was this: When writing a hyperlinked index (where hyperlinks are used instead of page numbers), to what should those links point? Clearly the only way to answer this question comprehensively is to suggest that the language of hyperlink indexes has two contexts: the index entry itself and the destination location. These two contexts need to work together. And as we saw, the same is true with the printed book: having arrived at page 164, how quickly can you find the idea you were looking for?

Looking this closely at hyperlinked indexes only emphasizes something we need for all indexing: use index entries that match the documentation text. If you have to write a slightly longer entry, that's okay. Instead of "cigarettes," use "cigarette smoking, quitting." Instead of "social networks," use "social networks and peer pressure." The people who work with search engines and Internet marketing are familiar with the term trigger words, which refers to visible language that matches the mental language of the searcher. If you're thinking of the words "white elephant," then a result of "pale pachyderm" doesn't work because it doesn't trigger your sense of recognition.

So the next time there's a white elephant in haystack 164, be sure to tell someone as explicitly as possible.

Frustrated by a lack of meaning

2006-03-20T21:49:00.000-05:00

Not for the first time, search engines have been wrongly criticized for the politics of their results.
As reported by The New York Times today ("Amazon Says Technology, Not Ideology, Skewed Results," March 20, 2006), an abortion-rights organization discovered and reported the appearance of biased results in its search engine. Apparently books with anti-abortion leanings appeared as more relevant on Amazon's search results pages. I am not taking sides on this highly charged issue; I am taking offense at the ignorance demonstrated by people who don't seem to understand how search works. (And I'm not singling out this issue either, as you'll see from my later examples.)

See, there isn't a search engine on the planet that can cull actual meaning from its databases. They can only look at the words themselves. Even search engines that analyze the behavior of their users still look at words and numbers, without interpretation.

Let me explain what really happened with Amazon, and why Amazon is not automatically in the wrong. Someone went to the search engine and typed in the word abortion. Now imagine that you're the search engine, and you have two results to give back. Result one is a book whose title is simply Abortion. The second is a book whose title is Understanding Abortion. Tell me: which result is more relevant? Answer: you have no clue.

When faced with this impossible question, the search engines at Amazon and elsewhere attempt to apply certain generalizations that might work in other situations, but simply don't work here. For example, there might exist a rule that puts Abortion ahead of Understanding Abortion because the title of the first book matches the query exactly, whereas the second title is only "half right." Or perhaps one of the books is 500 pages long, but the other is 200 pages long, and Amazon favors longer books. Maybe Amazon is interested in selling you the more expensive book, the book more recently published, or the book that gets a higher rating from all the people visiting the website. In the end, however, all of this analysis fails -- completely and utterly fails -- to answer a very simple question: which of these books is against abortion? Heck, even I don't know, and I invented them!

With search, meaning is irrelevant. Search engines can look only quantitatively at the letters of the words, and at innumerable statistics (e.g., number of Web views) that have at best a tangential relationship with meaning.

Before we look at another example, let me also talk about another thing that Amazon did at one time. If you searched for the word abortion, in addition to your results you received what should be interpreted as a helpful search hint: "Did you mean adoption?" This might sound political, but the logic of this lies in the similar spelling of the words adoption and abortion. Given that there are many more books about adoption than abortion at Amazon, the search engine guessed that someone typing the word abortion might have misspelled something; the computer offered what it considered a reasonable alternative. Had that suggested word been something different -- "Did you mean apportion?" -- no one would have cared.

By the way, I will admit that it is always possible that a company, like Amazon, could consciously manipulate its search results to accomplish some kind of selfish ends. Lycos puts sponsored links at the top; Yahoo promotes its internal products over those of others; Amazon presents the products of its more lucrative partners over all others. It is not far-fetched to imagine a company exercising editorial control for political or religious purposes, especially in today's age. The problem is that some issues are perceived as so volatile that no one is willing to consider coincidence of language as just that, a coincidence. Language is powerful stuff; spelling women as womyn to avoid the "men" letter subset is a powerful choice, whether you agree or not.

Here's another story, from the late 1980s. A search for the word monkey within a database of clip art provided by Microsoft produced a seemingly offensive result: a picture of African-American children. There was an uproar, and although Microsoft denied that it had done anything intentionally racist, it quickly removed the image from the database. The real problem, however, is that the children in the image were playing on monkey bars. Interestingly, if you stop to think about it, the only racism in this example is caused by the person who performed the search! That's the person who actually connected the word monkey with the children (and not the playground equipment); no one at Microsoft did. In this example, the giant void where meaning should have been was automatically filled in by the searcher, by association and as a reflex.

Here's another story, from last year. An article (I can't remember where) expressed how a bad critical review of a specific performer appeared more relevant in a Google search that the good reviews -- of the performer's own website. This would be equivalent to searching for me ("Seth Maislin") and getting a top result of "Seth Maislin Has Bad Teeth" instead of this blog, my website, or one of my interview at O'Reilly & Associates. In this case, Google isn't passing judgment, but it certainly feels like it! Instead, it's looking at how popular that Bad Teeth article might be, or its host (for example, it might be a Wall Street Journal or People Magazine article, periodicals that have readerships thousands of times larger than anything I've ever done), and using that popularity to push the article to the top. It's assuming -- wrongly, in this case -- that people looking for me are less interested in my website than in what Teen People or the WSJ has to say.

Search just doesn't care. If you're looking for meaning, don't ask a search engine.

Bikini emergencies

2006-03-15T21:15:00.000-05:00

Index entries can be divided into two categories: the things people look up, and the things they don't. The first category is straightforward, but the second category is inspiring.

I think it's fair to say that index entries that no one looks up are, well, better left out of the index in the first place. If you're reading a book about the cardiovascular system, there's little point in including the index entry "jack-o-lanterns."

It's at this point that I've love to say, "Enough said," but I'd be wrong. That's because having entries that no one looks up isn't really a problem, as long as you don't have too many of them. Ignoring the costs of the physical space they require, or the indexers' and editors' resources in making them actually appear in that physical space, unused index entries can exist without anyone really caring, like pennies in a penny jar. As long as the jar isn't full, no one cares.

Why was the entry there in the first place, though? There are some legitimate reasons, of course, with the most legitimate being a reflection of an author's non sequitur. It's not too far-fetched to imagine a cardiovascular surgeon authoring a textbook about the cardiovascular system, taking a moment to wax poetic in a footnote about how the surgically cutting into the chambers of the human heart always remind him pumpkin carving on Halloween. If he takes even a half-sentence to explain why heart surgery has something in common with ritualistic gourd mutilation, the indexer will notice this and create that silly entry for "jack-o-lanterns." No one will look it up, but that's not the indexer's fault, is it? She's just doing her thorough-as-usual job.

Then there are the indexers who include things without realizing who the audience is. They include ideas that are too esoteric, off-topic, general, or inappropriate for people to look up. For example, in a book about pet care, they might include an entry for "schnausers. See dogs." Alternatively, they index the ideas people will look up, but under labels that no one would look up, like using the term "octothorpe" as a name for the # symbol.

And of course there are always the honest mistakes: spelling or typographical errors, document file anomalies, outdated indexes for new materials, and so on.

What most indexers seem to forget, however, is that just as these odd index entries might be created by accident -- author silliness, indexer ignorance, production oversight -- those very same index entries might be discovered by accident, too: reader serendipity. One of the great advantages of printed indexes over search results (or search-accessed indexes) is the serendipity that results from browsing. Just as we're likely to discover interesting words in the dictionary while trying to look up something else -- look up "Jefferson, Thomas" and find yourself reading about jeffing and jeffus -- so we might discover interesting things those indexes.

If you found "jack-o-lanterns" in a cardiology textbook, wouldn't you follow the entry? I know I would. And that's why there's one last reason for including these unlikely-to-be-used entries in indexes: sheer joy.

Did someone say, "bikini emergency"?

Fixing the books that leak

2006-03-14T21:45:00.000-05:00

The more that I think about the presentation I am giving in April to the ASI New England Chapter, the more I find myself contemplating the environment in which indexers work. Why do so many indexers have trouble bidding for work?

Because clients don't grasp what this industry is about.

I'll make the analogy between a book without an index and a leaky faucet. The client recognizes that his sink faucet is malfunctioning. He is unable or unwilling to fix the faucet himself. He investigates the services of a professional faucet fixer.

Taking the three most common bidding environments for indexers, we get these three models for an owner of a leaky faucet.

Model 1: The Drowning Homeowners at Midnight. Having waited too long, or perhaps because another faucet-fixer failed to complete the job, these clients need help NOW. They're willing to pay any amount of money, but their standards are very low. "Please," they say, "just stop the leak." If they happen to know how to reach a specific faucet-fixer at midnight, that faucet-fixer is going to make a lot of money. But if there's no one they can call at midnight, they're going to call everyone, leave a lot of messages, and wade knee-deep in hundreds of early-morning responses. It's a bidding war, and the least expensive person wins.

For indexers to make this model work, they need to have their names right there on the clients' desks, and they need to be prepared to sacrifice quality for the sake of an important deadline. However, this model is terrible for the industry, because it leads to bidding wars where indexers underbid each other for the privilege of doing a lousy job.

Model 2: The Paranoid Homeowners. Here are clients who have worked with so many bad contractors over the years that they simply don't trust anyone. They're looking for someone who will fix the faucet according to ridiculously robust specifications, and who will allow them to stand over their shoulders, watch the fixer's every move, and offer suggestions and instructions all along the way. It's obvious this homeowner would do the work himself if he could -- which means all of those specifications, suggestions, and instructions are coming from a place of very hostile ignorance. With this model, indexers will find themselves burned almost every time.

This model doesn't help the individual indexers or the indexing industry, yet the model exists only because indexers as a whole failed to uphold any kind of standards, or to educate their clients about indexing. It's not uncommon for a more experienced indexer to clean up the mess left behind by someone without appropriate experience or sufficient resources to get the job done right the first time. When faced with this model, the indexer must be prepared to uphold his own principles, explain his choices, demonstrate precedent, and remain friendly throughout. I suppose it's like trying to talk about love to someone who just got divorced.

Model 3: The Proud Fixer-Uppers. Then there are the folks who would be happy to fix the faucet themselves but don't have the patience, the know-how, or the resources. They want help, but on some level they resent needing it. These are the people who, when presented with a faucet-fixing cost of $80, ask, "Will you do it for $40?" If you say no, they'll say, "What if you just put the pieces in a line and I'll wrench them together myself?" Say hello to the silly goons who would rather let their faucets leak than admit they need help worth paying for.

This model exists because the industry is underrespected. If someone had an overflowing toilet, do you really think they'd try to haggle with the plumber? But with indexing, this happens all the time. "Just index the headings" and "Do the best you can in only three days" are all too common in an industry where people aren't aware of the advantages of a good index. The individual indexers perpetuate this problem by accepting these underpriced, undervalued jobs and creating mediocre products.

Yes, there is a fourth model, and that's the ideal situation: someone who understands that indexing is a specialized trade that requires a professional with the appropriate education, background, and resources to get the job done. If this model were the most common, though, would indexers still be afraid of bidding? Nope.

What would it take to convert the clients of the three models above into the clients of a more reasonable, respectful model? I've heard a lot of suggestions -- education, indexing standards, indexer credentialing -- but the problem runs a lot deeper than most people think. There are two fundamental issues here: knowing what a good index is, and knowing that an indexing profession exists.

With a faucet, all we need is water on the floor to know it's broken, but you can't "replace a washer" to make a bad index into a good index. And while most home dwellers have heard of plumbers, few of the world's literate population have heard of indexers! Trying to educate the world about indexing is like trying to explain the ideal gas law to toddlers: they may breathe the air, but that doesn't mean they know how it works. (And don't ask their parents, because they don't know either!)

The law that mandated the wearing of seat belts in motor vehicles raised awareness of their value. Would they work for indexers?

College content management

2006-03-01T12:09:00.000-05:00

If you thought online education was a growing industry, you underestimated. It's about to explode. That's because Congress is now allowing federal financial aid to students of colleges that teach more than half of their courses off-campus, including over the World Wide Web. (See "Online Colleges Receive a Boost From Congress," The New York Times, March 1 2006.)

Let me rephrase that. Colleges are going to explode. Like scambling an egg.

There are two types of courses that are likely to appear online at exponential speeds. First, you have the low-end requirement courses, like Calculus I and English 101, which are courses everyone has to take to earn a bachelor's degree. These courses are regularly populated with large numbers of students, and yet they are introductory-level, tried-and-true courses that are relatively simple to teach and grade. (In fact, graduate students usually teach these courses.) Instead of wasting valuable classroom space and valuable instructor time, these courses will appear online. For large universities, there's an even greater advantage: hundreds of students can be taught at once without that "giant lecture hall" atmosphere, and without having to create numerous sections (class segments) for advising lessons or grading purposes.

The other type of course that is going to end up online are the courses that professors are just climbing all over themselves to teach, either because it's a cutting-idea, the product of a personal info-lust or -peeve, or simply because it's so esoteric that their departments have never granted them the opportunity to teach something that almost no one will attend. These kinds of courses will prosper online because students from all over the world can attend. That sociology elective called "What License Plates Tell Us About Our Culture" won't get just one student any more, but tens of students, and that makes professors and universities happy. I'm an online instructor of a rather esoteric subject -- indexing -- and so being able to offer my indexing course over the Internet has enabled me to reach dozens of indexing professionals and enthusiasts from around the globe every year. Had I continued to teach only in person, the course might have been cancelled for lack of interest.

So suppose every college in the United States retools that Calculus I class into an online course. It's not easy building an online course, but talk about unnecessary redundancy! No, if department heads are smart, they'll team with the department heads from other colleges and share a course. For example, I can imagine a single English 101 course offered to every college student in Massachusetts. I'm not saying this is a perfect idea, but then again, neither is having 14538 of them.

This is a content management problem (which is why I'm blogging about this). Content management is about many things, including (a) avoiding redundancy in communication; (b) avoiding the communication of inaccurate, outdated, or contradictory things; and (c) providing the correct information for each audience subset, whether it's a single person (like "Welcome, Seth!") or a large group (like English speakers); and (d) communicating everything that needs to be said. Content management is a huge issue in the distribution of information, one that got even more obvious with database use and the Internet.

The magic question here is this: How many different Calculus I courses do we need? From a production standpoint, our goal is to have only one. In reality, however, there are many reason a student might prefer one version of this course over another: the instructor (charisma, ability to teach, ability to communicate, educational background, current interests, track record in past courses, reputation in the industry, etc.); the course materials (ability to relate to examples, quantity of independent and groups exercises, immediate relevance of exercises to students [e.g., local interest], ethical choices, strictness of prerequisite management, etc.); the course delivery (tools requirements, number of lectures, number of students, grading methods, student-to-teacher and student-to-student interactions, what percentage of the course is different from that in previous semesters); and the course environment (reputation of university, opportunity for real-time meetings [chat, voice, face-to-face], textbook requirements); and so on. As you can see, there are a lot of reasons one course might be "better" than another, for a particular student.

And so you have a battle: competition for students vs. need to teach the basic materials. Colleges and universities are already doing this at a macro level -- compare MIT to Bunker Hill Community College, as in the film Good Will Hunting -- but the competition at the lower level is going to be very interesting.

Further, if course development follows the path of computing, parts of courses are going to be delivered as if by subscription. Imagine some guy in his attic churning out math problems for fun. (Believe me, it's real.) Professors can subscribe to this guy in the same way you find those Sudoku puzzles. In computer parlance this is called distributed computing: where a bunch of computers are working together, but each is working on a separate problem. (It's sort of like wearing a wristwatch to tell the time, carrying an iPod to listen to music, and attaching a bottle opener to your keychain, instead of investing in a WatchPod Opener.)

Distributed education means the end of campus life as we know it! Professors moderate courses written by dozens of international specialists, students take courses moderated in other countries, and grades are applied to the diploma of your choice. Campuses become less about learning and more about community events, just as public libraries and shopping malls have been forced to evolve.

Congress has made the right choice, but is this country truly ready for college management?

Custom organization schemes

2006-02-25T13:20:00.000-05:00

In previous entries I've talked about exact and ambiguous schemes. Now I want to talk about "everything else that remains," the custom organization scheme.

The three most common exact schemes -- alphabetical, chronological, and spatial -- aren't the only exact schemes in the world. There are also numeral (1, 12, 166, 2, 22, 266, 3, ...) and numerical/counting (1, 2, 3, 12, 22, 32, 166, 266, ...) schemes, and there are schemes where the sort style is based on some other intrinsic characteristic or standards, such as the periodic table of elements (numerical order by proton count) and the colors of the rainbow (numerical order by wavelength values). Although it's challenging to think up exact schemes that aren't based on numbers, characters, time, or space, it is incredibly easy to think up subjective (ambiguous) ways of organizing information. Below are several suggestions. Although some of these possibilities seem quite similar to numerical order, number order won't ever change with time, although these can.

by frequency of use
by importance
by logical complexity
by sensory intensity (e.g., brightness)
by mood
by personal interest
by profitability
by likelihood to inspire controversy
by necessity to avoid a lawsuit
by the order in which you thought it up

Further, the subjectivity for each of these goes further, because you can choose your audience for these sorting schemes:

by importance to the author
by importance to the common user
by importance to the experts

... and so on.

It's easy to get confused between complicated exact ordering schemes -- like the periodical table, which is in order of proton count -- and custom ambiguous schemes. In both cases, the sorting scheme may not be obvious, at which point it's easy to assume it's obtusely subjective or completely arbitrary. For example, the following are three ways to organize the English alphabet:

a b c d e f ...
q w e r t y ...
e t a o i n ...

The first is alphabetical order. The second is spatial order (letters on the standard keyword, affectionately known as the QWERTY keyboard). The third is in order of use in the American English language (the letter E is the most commonly used letter in the lexicon). I would argue that for all practical purposes, these are all exact schemes. There is no subjectivity here. To sort the alphabet in a subjective way, I'd suggest by ease of pronunciation. :-)

My point is that in the end, most custom schemes are exact and not ambiguous. Instead, they are translations of intrinsic orders into something exact (again, like the periodic table being in numeric order by proton count or atomic weight). Alternately, they are exact or ambiguous schemes that are time-dependent, such that the scheme itself varies over time or application.

In summary, you should consider using a custom scheme when (a) you have an internal order that you want to obey, or (b) you have an order that changes over time. In both cases, however, it's important that your users understand that your schemes are not necessarily obvious. The periodical table requires training to understand and use; meanwhile, because the search results at Google.com tend to vary over time, some people are disturbed by getting different results for the same search they used yesterday. If you can't use an intuitively obvious scheme (like alphabetizing) or a subconsciously obvious scheme (like task or topic order), it's probably a good idea to find a way to impose a numeric scheme on top of your results. The periodic table has a key, and search engines often have a "relevance percentage."

Toronto conference

2006-02-23T15:33:00.000-05:00

The international indexing conference is scheduled for 15-17 June 2006, in Toronto, Ontario, Canada. It's my job, as president-elect of the American Society of Indexers, to plan this event. Boy, planning a conference can be a challenging thing sometimes.

Finally, though, we have a schedule in place. The ASI website will have that information soon, but in the meantime you can join the asiconference mailing list for current and archived information.

Even bad entries can be good, in context

2006-02-20T16:42:00.000-05:00

In technical documentation, often I've seen main entries like "creating user accounts, 325" and "managing settings," where the first word is a gerund that has questionable value. When I'm asked why this isn't a valuable entry, I explain that the ideas of "creating" and "managing" are rather vague. I also explain that if someone wants to know how to create something, he is more likely to look up the something. For example, if you were interested in how to bake a cake, would you look up "baking" or just "cakes"? I believe you'd look up "cakes" first.

Strictly speaking, however, there is nothing wrong with entries like "creating accounts" and "baking cakes," even though accounts and cakes are the more specific and more likely targets of most users. Entries under these generic or vague terms (creating, managing, etc.) become valuable, however, when these terms have greater meaning within their context. In religious texts, the concept of creation (as in "world creation," often written with a capital C) is worth indexing. In business textbook, the concepts of management is worth indexing. And in an instructional cookbook that explains how ovens are used in the most general sense, an entry for "baking" might be appropriate.

The guideline to avoid these general terms, declaring the resulting entries as "bad," ignores context. The argument that readers are more likely to look up the objects of these actions -- "messages, writing" instead of "writing messages" -- doesn't negate the value of having these gerunds as available access points. Actions are still concepts and deserve to be indexed; in technical documentation, task-oriented language has even greater value than average. Consider these entries:

filtering email messages, 000
spell-checking your document, 000
upgrading applications, 000

The reason these work so well is because the ideas of filtering, spell-checking, and upgrading are distinct ideas of importance to technology users. In fact, the above term document is itself a bit vague (whereas file is not), and beginners know applications better as programs. So you can see just how valuable using those gerunds can be. Dismiss them at your peril.

By the way, if you need a practical clue regarding their use, take a look at the full set of documentation and ask yourself if you can use that gerund much more often than you already are. For example, you might have a section where "creating accounts" is obvious, but does that same book talk about the creation of other things, like passwords, files, security filters, network connections, and so on? Just because the word "creation" isn't used (passwords are invented, filters are applied, networks are initiated, etc.) doesn't mean it's wrong. If after some serious thought you realize that you'd have dozens or more subentries under that gerund, consider getting rid of it as not specific enough. Other candidates for nonspecific gerunds in technical documentation include installing, configuring, customizing, opening/closing, hiding/showing, deleting, starting/stopping/quitting, editing/modifying/altering, and accessing.

You have sixty days until Tax Day (U.S.)

2006-02-15T17:50:00.000-05:00

I have two months to finish preparing my taxes. I know a lot of people out there who much prefer working with tax accountants and advisors, but I much prefer doing the work myself. All year long I throw anything related to my financial condition in a big red crate. Every year, usually in mid-February, I upend that crate and start putting everything into piles. After three days of labor, those piles of papers have been sorted, organized, reviewed, mined for valuable data, calculated, recalculated, tabulated, recorded, and sent in one big envelope each to the Internal Revenue Service and the Commonwealth of Massachusetts.

Three days, yeah. You probably think I'm crazy.

Here's why I go through that, without an accountant. First of all, the three days that I spend sorting and tracking has to be done--by someone at some time--before Tax Day. I could do a little bit of that work every day using software like Microsoft Money or a spreadsheet, or even using a pencil and a ledger. Alternatively I could create folders for myself and sort the paperwork as it arrives, exchanging that one big red crate for two dozen hanging file folders. And if I wanted, I could instead bring that big crate to the nice accountants and say, "Add this for me," and not worry about it all. But because it has to be done, and because I'm much more qualified to know when a certain phone call is a business expense, when a certain receipt is a medical expense, or when a piece of paper fell into the crate by mistake, the person who should be doing most of that adding is, of course, me. And since it really is just addition, why should I pay an accountant when I am fully capable of using a calculator on my own?

Second, the one arena where the ability of accountants completely overshadows my own is in their understanding of tax law, and how those laws apply to my earnings, debts, and expenses. But I don't like not knowing about the laws that affect me. With the same curiosity I feel about who in the world might have my social security number, I want to know how much of money is going to fund my government. Going through the distress of all that mathematics pays off, because now I understand my how Social Security payments are calculated, where the tax tables were derived, and how much of my money was actually spent on medical expenses, mortgage payments, and investment fees.

And there's a corollary to all this, and it's a consequence of waiting until mid-February to look at this paperwork: Doing my taxes is when I figure out exactly how much I make! Sure I watch my checkbook balance go up and down, but I don't waste my time looking at those numbers on a daily, weekly, or even monthly basis. Once a year is good enough for me. February is therefore a big day of numerical realization. For example, this year I discovered that 40% of my gross indexing income came from a single client. Other discoveries include how much less interest I'm paying on my mortgage from last year; how much money I invested on the house (which is important because I work at home and can deduct some of it as an expense); how much money I spent on postage and shipping; how often I used my car for business; and all the other little things like phone bills, photocopying, client and colleague lunches, medical expenses, bank interest, retirement investment, and so on. If ever I needed a reality check, this is it, and it's a lesson on a global scale. It's one thing to see how much money I spent on business travel, but quite another to see that number next to how much money I spent on business advertising.

Not only do I learn about the tax-related information, however, but I really learn about everything related to money. Once a year I thoroughly read my credit card statements, line by line. Under the pretense of looking for those $4.95/mo payments for website hosting at tripod.com, I have the delicious opportunity to reminisce about each year's events, like the birth of my daughter, that vacation with my wife, the day I bought TiVo, the huge party catered by Blue Ribbon Barbecue, etc.

And finally, if ever there's a reason to say "Yes, I do my own taxes," it's to impress everyone! It takes them a moment to cough and gasp with incredulity, and to tell me that I'm insane not to have an accountant because they and everyone they know has an accountant, but then the light dawns. "Wow," they think, "this guy must really ____."

a) be smart?
b) have a lot of patience?
c) enjoy building character?

Fill in your own blanks. That's what I do.

Organizing ambiguously

2006-02-11T15:54:00.000-05:00

In a continuation of my post from 8 Feb, I want to say a few words about ambiguous or subjective sorting. First of all, ambiguous sorting is cool. Unlike the exact methods I wrote about, ambiguous sorting schemes actually pay attention to the meaning of what's being sorted.

Topical schemes sort categories based on what they're about, like the organization of a textbook (simple to complicated). Task-oriented schemes are organized in "doing" order, such as the steps one takes to make a sandwich. Audience-oriented schemes separate items according to who wants them, like the split between members and nonmembers, or the MPAA categories for movie age appropriateness (e.g., PG-13).

Consider how you might organize the steps involved of visiting a website. In task order they would appear as (1) turn on computer, (2) open the browser, (3) type in the website address, (4) read the page, (5) close the browser, and (6) turn off the computer. In topic order, the "turn on" and "turn off" items would be combined, since people pair these together; same with "open the browser" and "close the browser." In alphabetical order, on the other hand, the browser is closed before it's opened, and the computer is turned off before it's turned on. Using alphabetical order is pretty stupid here would be pretty stupid, eh? (Did you ever wonder notice that the options under the File menu aren't alphabetized?)

The problem with ambiguous systems, of course, is that people don't necessary categorize things in the same way. In fact, this is why people can't find things in other people's kitchens! (See my post from 7 Feb.) We don't put measuring spoons, soup spoons, and serving ladles in the same drawer, even though they're all spoons. Instead, we interpret categories according to how we perceive these connections, subjectively.

Think about how restaurant menus are organized. First they are organized in task order: appetizers in the front, entrees in the middle, and desserts near the back. Then they might be organized by ingredient (e.g., all the pasta dishes appear together), although it's unclear in what order these ingredients are listed. And within these categories, what's the order? Maybe it's profitability; maybe it's to show off the chef's skills or the breadth of available selections; maybe it's to put the most popular or intruiguing items at the top.

By the way, in my opinion audience-oriented categorization is the most powerful and useful of all sorting techniques, and yet it's woefully underutilized. Organizing items by popularity is simply a variation on this idea.

As soon as you start really looking at how people use information, you will run away from exact schemes almost immediately, as much as possible. (Long flat lists, like the entries in an index, still demand some sort of umbrella sorting. Short lists, like the options in a computer menu, are fair game. Have you ever noticed that the choices under the File menu are generally in task order?)

Nothing in life is exact, so why do we force it?

International indexing conference in Toronto, Ontario (15-17 June 2006)

2006-02-09T19:41:00.000-05:00

If you're just tuning in, you may not realize that the international indexing conference, co-sponsored by the American Society of Indexers and the Indexing and Abstracting Society of Canada is coming up fast! Conference information is available at both websites.

I'm the American in charge of the conference (and president-elect of ASI), and my Canadian counterpart is Ruth Pincoe. If you have any questions about the conference, please write conference@asindexing.org and I'll be happy to provide an answer. A preliminary schedule of events is about to be released on the asiconference mailing list. (To subscribe, send a blank mail to asiconference-subscribe@yahoogroups.com.)

Organizing as exactly as possible

2006-02-08T18:36:00.000-05:00

It should come as no surprise that information can be organized in many, many ways. According to Rosenfeld & Morville, there are two major types sorting schemes: exact (objective) and ambiguous (subjective). Personally, I like to imagine a third category, custom, that overlaps both and therefore deserves special treatment.

Exact sorting schemes are things like alphabetically ordered, numerically ordered, chronologically order, and spatially ordered. These schemes require that you follow an accepted and hopefully well-known sequence, such as from A to Z, from 1 to 9, or from top to bottom. Exact schemes do allow you to reverse order -- my blog, for example, sorts each entry in reverse chronological order, with the newest entries at the top -- but they don't allow you to start mixing things up. In an alphabetically ordered list of words, words starting with C will always appear between the B-words and the D-words.

There are three major problems with exact schemes:

They assume that your audience knows the scheme. In most situations this isn't hard, but it's a big assumption. There are many native English speakers who still have to sing "The Alphabet Song" when trying to remember whether J comes before K. For spatially ordered data, the audience must know the associated spatial map; in the United States, there aren't many people who can tell you exactly where each state is. (People in New England, where I am, tend to have problems with the block-shaped states in the middle of the country; people outside of New England can't remember which tiny state is New Hampshire, and which tiny state is Vermont.)
Your scheme is limited to its own items. Alphabetical order doesn't govern where numbers go. Some people sort numbers as if they were spelled out, putting "16 Candles" under S for sixteen. Other people will put all the numbers before the A-words, or after the Z-words. Punctuation also doesn't sort easily; which comes first, it's or its? And then there's the question of where spaces go (look up word-by-word sorting and letter-by-letter sorting) or if capitalization makes a difference (windows vs. Windows). I've given you all alphabetical examples, but the same is true in all other exact schemes. On a timeline, for example, how do handle repeating items? On a calendar of days, how do you record something that lasts two weeks, or breaks for lunch hour? Do you record time ranges (like hour-long appointments) as different from single moments (like 5:35pm)? In spatial ordering, how do you identify things that constantly move, or places that can be identified in multiple ways?
Exact schemes have no meaning. It's true that A-words appear before B-words in the alphabet, but is that because B isn't a lesser letter? In a week that runs from Sunday to Saturday, why must an important Thursday event be buried in the middle? Do the units that make something sort a certain way -- the letters in a word's spelling -- have any meaning whatsoever? No, they don't. In fact, if you're going to sort something exactly, you have to completely ignore what it means and chop it into little meaningless or random components. If you have a friend whose name starts with Z, your friend knows what it's like to be treated as inferior for no valuable reason whatsoever.

Of course, these three reasons aren't enough to toss exact schemes into the rubbish bin completely. Exact schemes are easy. You can get a computer to sort things almost instantly, and most audiences have no trouble using them despite their shortcomings. However, my last point -- that they're meaningless -- is why there are so many better options.

I'll talk about ambiguous and custom schemes in my next posting.

Organizing the kitchen

2006-02-07T23:16:00.000-05:00

I've always been fascinated by how challenging it can be to make analogies between indexing and the "real world," when in fact we organize and retrieve things all the time. So I'm always looking at the kitchen as my model of information organization.

First of all, why is it so hard to find things in other people's kitchens? Doesn't everybody keep the trash can under the sink? Isn't cutlery always in a waist-high drawer near the sink? Don't people keep their drinking glasses and coffee mugs in the same cupboard? Apparently not.

We organize our kitchens for ourselves. If we are living alone, we only need to put things where we want them to be. If we are living with others, we do our best to compromise with our home-mates and protect our children. The things we rarely use go way up high; the things we don't want our kids to get are up high or behind a lockable door. Everything else goes where it fits, where we can reach, and next to the areas where we're most likely to use them. So for some people, coffee mugs and water glasses are stored together because they fit neatly beside each other (unlike glasses and bowls). For other people, the coffee mugs are stored closer to the coffee maker, in the same cabinet as the sugar bowl and the coffee filters. Both of these choices involve organizing by function -- the function of drinking, the function of enjoying coffee -- but the results are personal. The kitchen is ours.

Indexing involves turning your kitchen into a place that other people can use just as easily. This means you have to organize your kitchen in such a way that people don't have to ask you where the spoons are, but instead could just walk in and find exactly the spoon or other object they need. Your personal guidance should become unnecessary, because the kitchen is intuitively and universally organized. No one will ever open the wrong drawer or door or canister again.

Yeah, right.

Basically, you have four choices. The first choice is to label everything. Every drawer, every cabinet, every appliance, and every countertop object should have a little piece of paper attached to it. The cultery drawer might be labeled CUTLERY. The refrigerator might be labeled COLD FOOD. But this is not as easy as it sounds. What, other than cutlery, is in your cutlery drawer? A can opener? Twist ties? Napkin rings? Meanwhile, your refrigerator may contain cold food, but what kinds of food are kept cold? Are your apples in there, or are they in a bowl? Do you use fresh milk, or do you buy your milk in those boxes? You see, labeling is only as good as your labels. Don't you dare create a label for SPOONS, because you have teaspoons, dessert spoons, wooden spoons, slotted spoons, sugar spoons, serving spoons, antique decorative spoons, plastic spoons, and sporks in your kitchen.

Clearly the problem is that your kitchen isn't perfectly organized. Why aren't all your spoons in one place? So pull everything out and lay it down on a freshly washed floor, and reorganize it. Put all of your spoons in one place. Everything you might call a plate or a platter goes together. Everything you eat goes against one wall, and everything you don't eat goes against the other walls. And finally, your labels make sense. Of course, you've sacrificed your kitchen for the sake of everyone else, but wasn't that the point? No! This is the problem with the Dewey Decimal System in some public libraries: nobody knows how to find anything except the librarians! But I'll tell you, if you want to learn about a topic, you might just discover that everything on that topic is the same exact place.

Almost guarantee. Go the library with an interest in World War II, and you'll find yourself in the history section to read about history, the romance section to read historical fiction, the fiction area to find some spy thrillers, the newspaper archive to read old news articles, the magazine section to read current articles, the science area to read about radar, the aeronautics section to read about the airplanes, the humor section to read those funny WWII joke books, and so on. The same is true with our kitchen, where the same knife can be used to cut food, spread jam, open envelopes, and even unclog the drain. Your kitchen objects, like words in the English language, are used in many different ways; categorizing them becomes rather subjective. So when that guest comes in looking for fruit, will he find it in the refrigerator, in a bowl, in a box or can, or in the compost bin? Yes, yes, yes, and yes. Wow, I guess we need a FRUIT category.

The third approach, then, is to put everything everywhere! Put a teaspoon in every drawer, on every horizontal surface, in and next to every appliance, in each cabinet, and on every shelf. Now, when someone goes looking for a spoon, it doesn't matter where he thinks the spoon is, because he's right! There's a spoon on top of the microwave, in the Crisper drawer of the refrigerator, and in the sink. Of course, not only are spoons everywhere, but so are everything else: can openers, slices of bread, blenders! One of everything, everywhere! (Of course, to be truly practical, you'd need more than one at every location, since sometimes people need more than one spoon at a time. :-) By the way, this is how people use search engines, like Google. We create a web page, and then we attach as many keywords as possible. We want to make sure that everyone will find our stuff, no matter where they're looking. In fact, some people want their content discovered even when people aren't looking -- stumbling over spoons everywhere.

The final approach is some combination of all of these things: decent labels, better organization, and as much redundancy as the cabinets can stand. It won't be perfect for everyone all the time, but very few people are going to have to open more than one or two drawers until they find what they want, even if what they want is a tiny whisk or an egg timer. Everything is categorized, labeled, and multiply placed.

That's indexing.

Introducing...

2006-02-07T23:05:00.000-05:00

I like to think the world needs yet another forum to talk about indexing, but perhaps it's just me. These days, my absolute favorite part of being an indexer, an information architect, a trainer and educator, and an "information guru" (not my words!) is simply talking about the possibilities that come with indexing.

Let's explore together.