Did Google always dominate the web search market? In the second of three posts on the history of search engines, I look at the pioneers of the early search market, including the first web crawler, WWW Wanderer. Did you know that Disney used to be one of the biggest players in the business? Or that Altavista was more technically advanced, in many ways, in 1998 than Google is now? Keep reading!
Pioneering web search engines
Actually, the point where modern search engines start to appear is after the development and popularization of the MOSAIC browser in 1993. In 1994, Internet Magazine was launched, along with a review of the top 100 websites considered to be the ” more extensive “. ready to appear in a magazine. A 28.8 Kbps modem was priced at $ 399 and put the Internet within reach of the masses (albeit slowly).
At this point and for the next 4-5 years, it was almost possible to produce print and web-based directories of the best sites and for this to be useful information for consumers. However, the rapid growth in the number of www sites (from 130 in 1993 to more than 600,000 in 1996) began to make this effort seem as futile as producing printed yellow pages of every business, media and library in the world.
While WAIS was not a lasting success, it did highlight the value of being able to search and click the full text of documents on various hosts on the Internet. Internet magazines and nascent web directories further highlighted the challenge of being able to keep up with an Internet that was growing faster than any human being’s ability to catalog it.
In June 1993, Matthew Gray of MIT developed the PERL-based web crawler, WWW Wanderer. This was initially conceived simply as a tool for measuring the growth of the World Wide Web using “collection sites.” Later, however, Gray (who now works for Google) used the crawled results to build an index called “Wandex” and added a search interface. In this way, Gray developed the world’s first web search engine and the world’s first self-contained web crawler (an essential feature of all modern search engines).
While Wanderer was the first to send a robot to crawl websites, it did not index the full text of the documents (as WAIS did). The first search engine to combine these two essential ingredients was WebCrawler, developed in 1994 by Brian Pinkerton at the University of Washington. WebCrawler was the search engine in which many of us pioneers first toured the web and will be fondly remembered for its (at the time) attractive graphical interface and the incredible speed with which it returned results. 1994 also saw the launch of Infoseek and Lycos.
However, the growth scale of the web was beginning to put indexing out of the reach of the average college IT department. The next big step required a capital investment. Enter, on the stage to the right, the (then huge) Digital Equipment Corporation (DEC) and its ultra-fast Alpha 8400 TurboLaser processor. DEC was an early adopter of web technologies and the first Fortune 500 company to establish a website. Its search engine, AltaVista, launched in 1995.
Founded in 1957, DEC during the 1970s and 1980s led the minicomputer market. In fact, most of the machines that the first ARPANET hosts ran on were DEC-PDP-10 and PDP-11. However, in the early 1990s, DEC was a company in trouble. In 1977, its then CEO, Ken Olsen, said that “there is no reason for an individual to have a computer at home.” While a bit of context was taken out at the time, this quote was in part symptomatic of DEC’s slow response to the rise of personal computing and the client-server revolution of the 1980s.
When Altavista was being developed, the company was besieged on all sides by HP, Compaq, Dell, SUN, and IBM and was losing money like it was going out of style. Louis Monier and his research team at DEC were “discovered” internally as the latest public relations hit; the entire web captured and searchable on a single computer. What better way to show the company as innovative and demonstrate the blazing speed and 64-bit storage of your new baby?
During 1995, Monier unleashed a thousand web crawlers on the young web (at the time an unprecedented achievement). By December (site launch) Altavista had indexed more than 16 million documents comprising several billion words. In essence, Altavista was the first commercially strong web-based search engine system. AltaVista enjoyed nearly 300,000 visits on its first day alone and, in nine months, served 19 million requests a day.
Altavista was, in fact, technically well ahead of its time. The search engine pioneered many technologies that Google and others later took years to catch up. The site carried natural search queries, Boolean operators, machine translation services (babelfish), and image, video and audio search. It was also insanely fast (at least initially) and (unlike other engines) was well suited to indexing legacy internet resources (and in particular the still popular UseNet newsgroups at the time).
After Altavista, Magellan, and Excite (all released in 1995), a host of other search engine companies made their debuts, including Inktomi & Ask Jeeves (1996) and Northern Light & Snap (1997). Google was launched in 1998.
Of these early engines, each enjoyed its own enthusiastic following and a slice of the then nascent search market. Each also had their own relative strengths and weaknesses. Northern Light, for example, organized its search results into specific folders tagged by topic (something that arguably still needs to be improved today) and, as a result, acquired a small, but enthusiastic, following. Snap pioneered ranked search results, in part, because of what people clicked on (something Yahoo! and Google are just toying with right now!)
In January 1999 (at the beginning of the dot-com boom), the largest sites (in terms of market share) were Yahoo!, Excite, Altavista, and Disney, with 88% of all search engine referrals. Market share was not closely related to the number of indexed pages (where Northern Light, Altavista, and a then relatively unknown Google led the pack):
Search engine referral participation (December 99)
Yahoo! – 55.81%
Excite Properties (Excite, Magellan and WebCrawler) – 11.81%
Altavista – 11.18%
Disney Search Properties (Infoseek & Go Network) – 8.91%
Lycos – 5.05%
Go to (now Overture) – 2.76%
Snap / NBCi – 1.58%
MSN – 1.25%
Northern light