In the pantheon of internet history, WebCrawler is a name that looms large. Launched in 1994, it was one of the very first search engines, allowing users to scour the World Wide Web by keyword at a time when surfing the net meant browsing manually curated directories. WebCrawler‘s impact was swift and significant – it quickly became the web‘s most popular search engine, processing millions of searches per day at its peak. For many early internet users, WebCrawler was the primary tool for finding and discovering content on the then-nascent web.
But that was nearly 30 years ago – several lifetimes in the fast-moving world of technology. Since those pioneering early days, the web has evolved dramatically, and the search engine landscape has transformed many times over. Giants like Google and Bing now dominate the market, with sophisticated algorithms and a dizzying array of features. Amidst all this change and competition, what ever happened to WebCrawler? Does this former titan of web search still exist in 2023?
The short answer is yes – WebCrawler does still exist, and you can visit it right now at webcrawler.com. But the WebCrawler of today is a very different beast than the groundbreaking search engine of the 1990s. To understand WebCrawler‘s journey and its place in the modern search landscape, let‘s dive into its fascinating history.
The Early Days: WebCrawler‘s Rise to Prominence
WebCrawler was created in 1994 by Brian Pinkerton, a computer science student at the University of Washington. At the time, the web was still in its infancy, with only a few thousand websites in existence. The most common way to find content was through curated web directories like Yahoo!, which organized websites into categories. Search engines did exist, but they were primitive, only indexing website titles and URLs.
Pinkerton saw an opportunity to create a more powerful search tool. His idea was to create a "crawler" program that would automatically browse the web, following links from page to page and indexing the full text of each page it found. This would allow users to search not just page titles, but the actual content of web pages, greatly expanding the amount of information that was searchable.
WebCrawler launched on April 20, 1994, running on a single computer under Pinkerton‘s desk. Despite these humble origins, it quickly gained traction. In November 1994, WebCrawler served its 1 millionth query. By the end of 1995, it was serving over 1 million queries per day, making it the web‘s most popular search engine.
Several factors contributed to WebCrawler‘s rapid rise:
Full-text search: WebCrawler‘s ability to search the full text of web pages was a game-changer. Suddenly, users could find pages based on any word or phrase, not just the limited metadata provided by directories. This made web search far more useful and powerful.
Speed: Despite running on a single machine, WebCrawler was fast. Pinkerton optimized his crawler to efficiently index pages and his search algorithm to quickly return relevant results. At a time when web pages could take minutes to load, WebCrawler‘s speed was a significant advantage.
Coverage: WebCrawler wasn‘t the only search engine, but it indexed more of the web than its competitors. By late 1995, WebCrawler had indexed over 4 million web pages – a significant chunk of the entire web at the time. More comprehensive coverage meant more relevant results for users.
Partners: WebCrawler partnered with high-traffic web portals like Excite and Netscape, powering their search functionality. This gave WebCrawler massive exposure and helped drive its growth.
By the mid-1990s, WebCrawler had established itself as the leading search engine, a position it would hold for several years. It was processing millions of searches per day and continuing to innovate, launching features like image search and category-specific search engines for topics like news and travel.
But even as WebCrawler dominated web search, the seeds of upheaval were being sown. A wave of new search engines emerged in the late 1990s, each bringing new technologies and approaches. Some, like Excite and Lycos, followed a similar crawler-based model to WebCrawler. But others, like Yahoo! and Altavista, pursued a different path, using human curation and more complex query languages. And then there was Google, founded in 1998 with a radically new approach to search based on link analysis and a minimalist user interface.
The Competitive Era: WebCrawler‘s Decline
As the web continued its explosive growth in the late 1990s and early 2000s, the search engine market became intensely competitive. WebCrawler, once the undisputed leader, found itself struggling to keep pace with a constant stream of innovation and a proliferation of well-funded rivals.
Several factors contributed to WebCrawler‘s decline during this period:
Acquisition and neglect: In 1995, WebCrawler was acquired by America Online (AOL), one of the largest internet service providers at the time. AOL saw search as a strategic area and wanted to own a leading search engine. However, under AOL‘s ownership, WebCrawler languished. It received few updates and lost ground to competitors in terms of index size and search quality.
Rise of the portals: In the late 1990s, web portals like Yahoo!, Excite, and Lycos became the dominant starting points for internet users. These sites offered not just search, but also news, email, weather, and other services. By leveraging their popularity, portals could drive massive traffic to their own search engines, siphoning market share from standalone search providers like WebCrawler.
Google‘s ascent: Google‘s launch in 1998 marked a sea change in web search. Google‘s PageRank algorithm, which analyzed the linking structure of the web to identify the most important pages, proved to be a breakthrough in search relevance. Combined with a fast, minimalist interface, Google quickly gained a reputation for delivering the best search results. As word spread, Google began to eat into the market share of established players like WebCrawler.
Lack of innovation: As competition intensified, innovation became critical for search engines to maintain their edge. Unfortunately, WebCrawler fell behind on this front. While Google and others were constantly tweaking their algorithms and adding new features, WebCrawler remained relatively stagnant, with few major updates or improvements.
By the early 2000s, WebCrawler had fallen far from its former position of dominance. Its market share had dwindled to the single digits, and it was no longer seen as a leading search destination. In 2001, AOL sold WebCrawler to InfoSpace, a company that aggregated search results from multiple engines. Under InfoSpace, WebCrawler became essentially a meta-search engine, with its results sourced from other providers.
Over the next decade and a half, WebCrawler continued to operate, but in an increasingly diminished capacity. It changed hands again in 2016, acquired by web advertising company System1. While System1 gave WebCrawler a visual refresh in 2018, the fundamental product remains little changed – a basic search interface powered by results from Google and Bing‘s syndicated search partners program.
WebCrawler Today: Niche Player or Faded Glory?
So that brings us to today. WebCrawler still exists as an operational search engine, but it‘s a shadow of its former self. According to traffic analytics firm SimilarWeb, WebCrawler currently receives around 240,000 unique visitors per month. That‘s not nothing – but it‘s a far cry from the millions of daily searches WebCrawler processed at its peak, and it represents a minuscule share of the billions of searches conducted each month on market leaders like Google and Bing.
Visiting WebCrawler today is a bit like stepping into a time capsule. The site‘s design is clean and functional but dated, with a simple search box, a handful of filtering options, and little else. There‘s no auto-suggest in the search box, no knowledge panels or rich snippets in the results. Ads are pervasive, with multiple sponsored results and display units on every page. It feels very much like a relic of an earlier, simpler era of web search.
Under the hood, WebCrawler is no longer really crawling the web itself. Instead, its results are provided through syndication deals with Google and Bing. Essentially, when you do a search on WebCrawler, you‘re seeing a subset of results from Google and Bing‘s indexes, filtered and reordered by WebCrawler‘s own algorithms. For the most part, the results are serviceable – you can generally find what you‘re looking for. But the experience feels like a stripped-down, ad-heavy version of searching on Google or Bing directly.
So who uses WebCrawler these days, and why? It‘s a bit of a mystery. The site doesn‘t seem to serve any particular niche or use case that isn‘t better met by the major search engines. Some users may be attracted to WebCrawler‘s simple, no-frills interface. The site makes a point of emphasizing privacy, stating that it doesn‘t track or profile users. But the same is true of DuckDuckGo, which offers a more sophisticated privacy-focused search experience.
It‘s possible that some of WebCrawler‘s traffic comes from users who simply haven‘t changed their habits. If you learned to use the web in the mid-1990s, WebCrawler may have been your first search engine. For a subset of those users, WebCrawler could be the comfortable, familiar option, even if it‘s no longer the best tool for the job.
More likely though, a significant chunk of WebCrawler‘s current usage is driven by its ownership. System1 is an advertising technology company that specializes in placing ads on search engines and other digital properties. Owning WebCrawler provides System1 with a captive platform for serving search ads. Every search conducted on WebCrawler is an opportunity for System1 to display paid results and generate revenue. In this context, actual user satisfaction and search quality may be secondary concerns to simply driving traffic and ad impressions.
Looking ahead, it‘s hard to envision a future where WebCrawler mounts any kind of resurgence. The search market today is thoroughly dominated by Google, with Bing a distant second. There‘s little room or appetite for new entrants, and the cost and complexity of building and maintaining a competitive web index is immense. WebCrawler‘s best bet for continued relevance may be to focus on its niche as a privacy-centric, ad-supported alternative for searchers who prefer a simple experience. But without significant investment and innovation, it‘s hard to see WebCrawler gaining significant traction.
The Legacy of WebCrawler
Despite its diminished present-day circumstances, it would be a mistake to dismiss WebCrawler as merely a footnote in internet history. Its impact and influence in the early days of the web were profound and far-reaching.
As one of the first full-text search engines, WebCrawler played a pivotal role in transforming the web from a disorganized collection of hard-to-find pages into a navigable and searchable resource. It introduced millions of early web users to the power and possibility of keyword search, setting expectations for how the web could be used and explored. In many ways, WebCrawler helped establish the very idea of web search as we know it today.
WebCrawler‘s legacy can also be seen in the many search engines and web companies that followed in its footsteps. Many of the key figures involved in WebCrawler‘s early development and success went on to play influential roles in the internet industry. Founder Brian Pinkerton became Chief Architect at Excite and later AOL. Other early WebCrawler employees went on to high-profile roles at companies like Microsoft, Amazon, and yes, Google.
Perhaps most significantly, WebCrawler‘s meteoric rise and eventual fall serve as a case study in the dynamic, fast-moving nature of the internet business. In the span of just a few short years, WebCrawler went from a dorm room project to the web‘s most popular destination to a struggling also-ran. Its story is a reminder of how quickly the digital landscape can change, and how even the most successful and innovative companies can be disrupted by new technologies and competitors.
In the end, WebCrawler‘s legacy is one of pioneering innovation, rapid ascent, and eventual eclipse. It‘s a story that has played out again and again in the history of the internet, with once-dominant players like AOL, Yahoo!, and MySpace giving way to upstarts like Google, Facebook, and Twitter. It‘s a testament to the web‘s ceaseless cycle of creative destruction, and a reminder that no position of dominance is ever truly secure.
So while WebCrawler may no longer be a major player in the world of web search, its place in internet history is secure. As one of the first and most influential search engines, it helped chart the course for how we find and discover information online. And while the web of today is a vastly different place than it was in 1994, the fundamental need that WebCrawler first served – the desire to make sense of the web‘s vast troves of information – remains as vital as ever.
