036. Fancy Wizard and Red Squiggles

Office94 (still the working name for what would become Office 95) was primarily about working with Windows 95 and shipping on the same day. That led to the constraint of having a relatively small team, less than 20 software engineers across the product compared to almost 60 on just Word 6.0. We knew that simply working with the new operating system would not be enough to get people to shell out a couple of hundred dollars (at the time, about 1 in 10 PC owners were also legal owners of our apps). The apps were also constrained by not changing the file format, which limited fancy features. There was a surgical approach to choosing features that we hoped would garner winning reviews of the suite and each app. Winning reviews remained the highest priority for the team. As part of writing this book, I am making an effort to include stories about features that everyone uses but often exist without a sense of where they came from or why. Back to 035. Windows 95, August or Bust [Chapter VI]Using a PC mystified most people, even in the workplace. Features designed to assist or help customers were almost always viewed positively even necessary as part of product reviews in magazines. Answer Wizard was our first attempt at using natural language processing and early artificial intelligence techniques to provide assistance in using ever more complex products. The early days of PCs were days filled with in-person courses to learn the products, and these had given way to 600-page books that filled the shelves at Barnes & Noble and Tower Books, regurgitating every feature, menu, and dialog box in alphabetical order. We had written thousands of pages, provided reference materials, created online “computer-based training,” and more, but still the first days of the PC era were marked with too much complexity and too high a hurdle to begin to use. Most people buying a new computer were also enrolling in courses that met for a couple of hours every week for a month (courses were the upsell much like today’s extended warranty).One of the core problems was jargon. There are dozens of phrases in the user interface that look like words a normal person would know but used in a way only a techie might grok (grok is a common techie expression for “to understand” that comes from Stranger in a Strange Land.) For example, PowerPoint, a tool few were comfortable with or had any understanding of, defied any logical English word (or any native language). What is a “slide master” or a “meeting minder”? What’s a “snap to grid”? The worst were features that were not so obscure but used English words in ways that most people could not understand, such as Word’s “mail merge” or Excel’s “lists.” We used to joke that we could probably put the version of Office with German language menu commands in front of English speakers and they probably wouldn’t even notice the difference. Answer Wizard was designed as a bridge between humans and computer jargon. If someone typed, “How do I send letters to a list?” Answer Wizard would find the Mail Merge feature (without literally going through all the menu items trying things at random). Sometimes commonly used symbols in Word such as the pilcrow, ¶, were totally unknown to regular people who would type questions into Answer Wizard such as, “How do I get rid of the elephant character?”Answer Wizard was a collaboration with Microsoft Research and proved to be the foundation of future work in Office96 being developed in parallel. Answer Wizard was the underlying technology of the natural language interface to what would become the Office Assistant, or Clippy. The earliest research group at Microsoft was the natural language research group where they were working on big hard problems of translation and understanding. That technology was more than a decade away and ultimately delivered by Google. Instead, we collaborated with the MSR’s first group of AI researchers using Bayesian mathematics to probabilistically select from among a set of choices. Basically, we tried to add an element of probabilistic guessing to the solution rather than relying on a traditional full text search or index. The guessing was based on a small database of words not already in the index or help database that we could map to the various articles in the help system. We called these metanyms because they were not precisely synonyms but somewhat close in meaning. We brought together those who wrote documentation, called User Education, or UserEd, and for the first time they were working on much more technology baked into the product. We renamed the team and the effort and called it User Assistance, or UA. It would be a few more years until we stopped referring to customers and humans as users, as we would often remark that only one other industry called customers users.One of the seemingly minor changes we made was that hitting the universal help key, F1, would always bring up Answer Wizard. We designed the new experience with a flashy animated icon, the first use of animation in Office, and even broke our own style guide with the font in the interface. Answer Wizard arrived as a feature just as people started typing queries into web search engines looking for help. Surprisingly, we quickly learned how much easier it was becoming to find answers to usage questions on the internet than it was within our own assistance features. Nevertheless, Answer Wizard proved to be one of our first suite-wide features and reinforced the commitment we had to making Office, not just the apps, the easiest to use product.While the marketing story of Office94 was told through the lens of the suite and consistency and the few significant changes to the product we made in order to emphasize the suite, the constraints of not changing the file format and a small team led to some of the most innovative and memorable app features, under the guise of IntelliSense. Some of these seemed so small and trivial, yet they had to be invented at some point in history, and when they were they were often the work of a small set of people with a clever idea and the runway to get things done.IntelliSense had become the branding moniker describing the intelligent features in Office 4. The canonical IntelliSense example was the newly introduced AutoCorrect, first released in Word 6.0 but brought to all the products in Office94. The feature was such a big part of Word 6.0 that the print advertising campaign for Word 6.0 often featured a large “teh” changed to “the.” As with so many things that are incredibly helpful, it wasn’t much work. The genesis of the feature was a remarkable story as it set a tone for developing data-driven features for years to come while also learning a great deal about global scale and sensitivities to users of vastly different backgrounds and cultures.DHach joined the Word team straight out of Harvard’s Math department as a program manager on the “basic use” team of Word, which was the part of the team tasked with making the product easier to use for core functionality (versus focused on long documents or on fancy magazine layout features). Word already had many fascinating unused features. One of those features was the “glossary,” which was a way to type a short phrase, hit a keystroke (the F3 key, thus explaining why no one used this feature), which would then replace the short phrase with the longer text. Dean’s insight was that in English the spacebar could replace the awkward F3, and then he realized that he could prepopulate the list of glossary entries with a library of common typos and misspellings. This was the origin of correcting “teh” to “the” and hundreds of other words. Other insights included replacing the accidental caps lock key (“dEAR sIR” turns into “Dear Sir” with caps lock turned off).One of the more aggressive uses of AutoCorrect was turning off the “sticky” shift key, which caused typos at the start of sentences such as “THis is the start.” Because so many acronyms were used in business and often with plurals (such as PCs), this feature was held back from Word 6 until an elegant solution could be devised for Office94.As the first do what I mean feature, AutoCorrect was revolutionary. The key lesson in building automatic features was their value . . . when they worked. And when they did not, the frustration level soared. This tension over doing more for people while also not introducing errors and mistakes, or breaking muscle memory, was a theme in the evolution of IntelliSense and also proved to be a wedge issue with other products. For example, Excel resisted the idea of AutoCorrect for common formula typos because of the potential to insert the wrong formula or wrong reference to a named cell. This was a real concern but at times seemed somewhat stubborn from the OPU perspective. It was a classic example of “Excel users are different,” to which the OPU refrain was “Why, because they don’t make typos?” These would be worked out, but navigating these cross-group opinions was always time-consuming.Given the success of AutoCorrect, it became a star feature of Office, embraced by all the applications with some app-specific constraints. Importantly, this feature was one of the first features shared across all the apps—the same typos got fixed the same way no matter where they were typed.AutoCorrect could not catch everything, though—a lesson from Word 6.0 was that being too aggressive and making mistakes was far worse than leaving extra work for the user. As a result, AutoCorrect didn’t replace traditional spellcheck, but the idea of helping automatically when possible informed us on ways to introduce a dramatic improvement to spelling correctly. In developing IntelliSense features, we learned that our corrections needed to be right 100 percent of the time—being wrong even a little bit felt to customers like we were wrong all the time. This is a lesson the industry continues to learn with today’s autocorrect on phones.Spellcheck was the original feature that distinguished word processors from typewriters. At first, most companies that sold word processors sold companion spell checkers often costing as much as the word processor itself. These largely competed among products by the size of spelling dictionaries and the ability to add custom dictionaries, such as legal or medical terms. Using these (and Microsoft’s) spell checker was a modal experience, that is the user would invoke the spell checker and begin to identify and correct spelling errors one at a time unable to do anything else. This was time-consuming and frustrating—the process was stopped when a correction was needed and then restarted, and every falsely flagged word required the user to click on an “Ignore” button. A spelling error resulting in a substantial change often reset this process requiring a full scan of the document again.Office94 took two major steps in spelling, AutoCorrect and background spelling, which became iconic Office features.Originally, spellcheck was a feature of Word. There was never any thought given to using it in Excel, where there weren’t a lot of words—another case of Excel users are different. Excel users had been evolving in their desire to use Excel all the time. On Wall Street, Excel was such a hammer that it was being used to nail nearly everything. One of the biggest new scenarios was using Excel to create pitch books for financial products. By removing grid lines and making clever use of fonts, borders, table widths, and the sophisticated macro programming that was a hallmark of Excel, one could create a pitch book without ever leaving the comfort of a spreadsheet. In Excel 3.0 there was even a presentation template that made Excel look like PowerPoint complete with animated transitions between slides (which were really ranges of cells). It made sense, therefore, that adding spellcheck would finally become a useful feature.Word, Excel, and PowerPoint adding robust spellcheck proved to be a nice addition and emphasized the suite. Word had created a breakthrough idea, which went on to be a universal feature anywhere people typed, background spellcheck or the ubiquitous little red squiggles under (mostly) misspelled words.The origin of the feature was a product of multiple small ideas and thoughts that piled up over time, starting with an incredibly novel research approach the Word team had done using Word 6.0. Reed Koch (ReedK), a longtime program manager in DAD, was one of the early proponents of studying how people used the product in the real world. This was not as easy as it sounds before the internet and cloud computing. To study the product “in the wild,” the Word team created a special variant of the product, called the instrumented version, or IV, which was the same product in the marketplace except it recorded what Word commands were used (menus, toolbar buttons, keyboard shortcuts, dialog box choices, and more) and in what order and frequency. Data was gathered from a small set of selected informed volunteers who knew that their actions (but none of the content) was being collected once we installed this special version on their computer. After a few weeks we returned to the customer site and collected the data, using a stack of floppy disks, and replaced the IV version with the regular product. This use of real-world data was a pioneering effort and formed the foundation of how the applications would collect and use data on the internet in just a few short years.Program managers pored over the data and analyzed it (using Excel of course as well as Access because the datasets were so large), trying to understand patterns and places where customers were getting stuck, using too many steps, or failing to use a feature that would have made things easier. The wealth of insight gathered from this approach could not be underestimated and building IV versions became a significant part of customer research that led to many of the internet and cloud innovations in future releases.Aside from learning things, like the fact that Print was the most common command (more so than Save!), or that as many people used each of the menu command, keyboard shortcut, and toolbar button for cut/copy/paste, or that features for assembling long documents (table of contents, index) were not frequently used—all of which most might think obvious—some important things were annoying users. One of those was the message that would pop up: “The spelling check is complete” with an OK button. It was a pointless message that did nothing but interrupt workflow, thousands of times.In addition, the IV helped to inform the team that with great consistency, if during a spellcheck a suggestion for a misspelling was chosen as a correction then it was one of the first suggestions listed or it was ignored. This insight would form the foundation of one of the biggest advances in IntelliSense, spellcheck while typing. Getting to that feature required connecting a few dots.Invisible to users, Word did work behind the scenes to keep the document up to date for printing while typing or reading. For example, if a document had page numbers and added text in the middle of a document caused flow to the next page, then in the background, without slowing Word down, page numbers were adjusted to repaginate the document. In a world of operating systems with limited memory and CPU, this was a nifty engineering trick (essentially Word was its own mini operating system). Today we take for granted the ability of computers to do work in the background, but prior to Chicago this wasn’t supported and took incredible trickery to pull off.This was called the idle loop, because it was where the program looped or did nothing while a person was taking those tiny little breaks when typing or thinking. Could this background processing capability also check the spelling of documents without taxing the system and slowing everything else down? Writing code to take advantage of this idle processing power was somewhat of an art and the developers often referred to it as a devil’s playground of sort—one wrong addition or ordering of background tasks and the whole thing would grind to a halt and be very difficult to debug.PCs were getting more powerful, so much more powerful than when the original spellcheckers were programs purchased separately from word processors and run after typing simply because there was not enough memory on the computer. The first word processor I used was called WordStar and it came with a separate program called SpellStar to check spelling. It was cumbersome. Integrated spellchecking was a big improvement, but it was still modal—a separate step and manual operation. PCs were 100 times faster since SpellStar but word processing documents had not grown at the same pace. What could the team do with the power that was otherwise sitting there idle?Running the spellcheck in the background while typing was simply taking the idea of background processing to the next level. It was so important that background processing was a key part of the patent application. The implementation frustrated our partners on the Chicago team and at Intel who were evangelizing the idea of using Win32 threads as the way to add background processing. It also happened to be a feature of these modern operating systems and chips, but the overhead and complexity of rearchitecting for threads (for background spelling, printing, pagination, etc.) was far too high for such a critical capability, especially when processors were already becoming far faster than required for editing.The lesson from the IV was that showing the closest matching words from the dictionary using a convenient right-click menu would have a high likelihood of being correct. The red squiggles were simply reflective of a proofreader’s style of mark (also one of the early uses of color in the interface). Just in case, Word left the existing interface in the product for good measure. It also made use of the right click menu, which had been introduced across the products in Office 4 and had become a defining feature for power users.The feature did need a way to communicate to users that “something was happening,” and so a small animation was placed in the status bar of Word at the bottom of the screen. Originally the team wanted to use a tiny buzzing bee, as in “spelling bee.” But as was quickly pointed out by the diverse team that made up Word, such an iconic representation did not translate to other languages and cultures. The result was a little notebook with a squiggling pencil. Whimsy was difficult at global scale.The feature was not without critics. There was an unintended side effect of opening a document which was as soon as the background processing kicked in the misspellings were identified. Maybe it was a company name, or a city, or a product name but these all looked like misspellings particularly in a world of sharing email attachments when the custom dictionary was different on the other end. Word introduced a number of subtle tricks to help with this, such as not marking words until an edit and providing for an easy ignore button that would unmark the incorrectly flagged text.Still some people were so frustrated they wrote letters (actual letters) to BillG to complain and those would invariably end up in my inbox. Often these were just irate at proper names being flagged as misspellings, such as from members of the Phenis family who often exchanged letters and did not like to see their name underlined, and they especially did not like the suggestion. We quickly removed the offending suggestion in the next update. The letters back and forth continued for some time as the product update took a while to reach all customers.Sometimes things got a bit over the top. A public radio broadcaster hosting a music show from Oregon was rather irate each week as the playlist was assembled. Every time they cued up their favorite Queen of Soul the red squiggles offended them. Again not just the squiggles but the alternate suggestion for Aretha Franklin was really what got them going. As you can sense from these two examples, the words referring to anatomy had to be reviewed even though Word was used by medical doctors and scientists all the time. The host threatened to “unleash the indignation” of their viewers if it was not fixed. The broadcaster was so annoyed they started sending the letters to their congressional representative claiming that Microsoft’s monopoly power was to blame. That got me in a long thread with what I assumed at the time was a zealous intern. Once again we removed the words and added names to the dictionary. I only had to endure one last broadcast where the host read the letters of campaign’s success on air.While such letters were obviously not representative, they proved extremely important to Microsoft’s culture. Most every product hallway had a wall of letters from customers, usually at the extremes of loving the product or incredibly awful experiences. Other than the instrumented versions in Office and the samples of telephone support calls, anecdotes were the primary real-world inputs. While there was a special product support phone number that executives could use to escalate an incident, the company did not have a systematic approach to the onslaught of problems that came from millions of customers, whether a problem was Microsoft’s creation or not. Nevertheless, I enjoyed these letters and the dialog that came from replying, including the dozens of times I refunded the price of a product for whatever reason brought great frustration to a customer.Background spelling with red squiggles ultimately became a showcase feature for the product and made the lofty phrase IntelliSense make sense to reviewers and customers. Groups all around the company asked for the IntelliSense code so they could add it to their products, not even realizing it was marketing. IntelliSense spawned endless jokes for wanting red squiggles under things in real life, as in “This idea should have red squiggles.” In brainstorming sessions at a whiteboard, someone would invariably put red squiggles under a misspelling or lame idea. Scott McNealy, the founder and CEO of Sun Microsystems, famously joked that Microsoft had a whole team of people deciding on red for squiggles and an option to change it—his attempt at making a joke (misguided and incorrect) about bloated features people don’t use. The feature became commonplace in every word processor and every browser and more.On to 037. Capone and Email Without Typos This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit hardcoresoftware.learningbyshipping.com