As someone who‘s worked in data extraction for over a decade, I‘ve seen firsthand the business value that responsible web scraping provides. But I‘ve also witnessed questionable practices that risk giving this rapidly growing industry a bad reputation.
That‘s why I‘m excited to see groups like the Ethical Web Data Collection Initiative (EWDCI) bringing companies together to define standards and best practices. Their efforts to self-regulate and prioritize transparency can help secure the future of ethical web scraping.
Let me explain more about the EWDCI‘s goals, why standards are needed, and how following guidelines benefits scrapers and sites alike.
The Rapid Growth of Data Aggregation
Web scraping tools and services have exploded in popularity the past several years alongside the wider data economy. The industry already generates tens of billions in economic value annually.
Market research predicts global web scraping revenues will grow at 14% CAGR to reach over $13 billion by 2028 as more companies integrate scraped data. That‘s comparable to the search and social media marketing industries.
This massive growth comes from scraping‘s ability to extract insights across the open web. Use cases span:
- E-commerce sites aggregating product info for price monitoring
- Businesses tracking brand mentions, reviews, and other social listening
- News monitoring services compiling articles and commentary
- Search engines scraping pages to index content
- Investors accessing alternative data for analysis
However, as web scraping expands in scale and scope, we‘ve seen more examples of overly aggressive or deceptive practices that violate site terms and hurt server performance.
Let‘s discuss why standards can help address these issues…
Questionable Practices Demonstrate the Need for Guidelines
While most web scrapers operate ethically, lack of clear industry standards enables irresponsible approaches by some actors. Concerning examples I‘ve seen include:
- Scraping at exponentially high frequencies that overload servers
- Failing to respect sites‘ expressed data usage policies
- Using scrapers disguised as normal users to evade restrictions
- Reselling scraped data without permission
- Extracting complete databases of info versus sampling
- Scraping personal/non-public pages and content
As Andre, CTO of analytics firm Novae, told me: "A ‘move fast and break things‘ mindset by some scrapers has caused backlash. We need cooperation to prevent overregulation."
Establishing best practices provides a way to differentiate responsible scraping from predatory practices. This will improve trust in data aggregation overall.
Key Areas for Web Scraping Standards
So what should an ethical web scraper code of conduct actually include? Here are some key areas I believe industry guidelines should cover:
Respecting robots.txt restrictions – The robots exclusion protocol enables sites to communicate policies on scraping. Ethical aggregators should comply rather than looking for workaround strategies.
Rate limiting requests – Scrapers should implement reasonable delays between requests and limit overall volume to avoid overloading servers. Specific thresholds could be defined.
Securing scraped data – Companies collecting data have a duty to safeguard it via encryption, access controls, limited retention periods, etc.
Honoring opt-outs – If a site explicitly revokes permission to scrape, ethical aggregators must cease data collection immediately.
Transparency – Scrapers should identify themselves to platforms when possible rather than disguising their activities.
Limited private data collection – Guidelines should restrict scraping of personal and non-public pages/content without permission.
Permitted uses – Standards around only collecting data for appropriate purposes like research, journalism, and statistical analysis based on public interest.
Attribution – Requiring scrapers cite sources and respect copyrights/trademarks when republishing third-party content.
Compliance processes – Ways to monitor adherence and remedies for violations to hold aggregators accountable.
By following principles like these, the web scraping industry can demonstrate commitment to fair and community-focused data collection. But guidelines require participation…
Building Consensus Through an Open Process
That‘s where groups like the EWDCI come in. Their goal is bringing together companies from across the web scraping ecosystem to collaboratively develop standards.
Rather than manufacturers or regulators imposing top-down rules, an open process enables the industry to shape guidelines itself. ColdJust, an EWDCI member, says this grassroots approach is key:
"We want all voices heard so the standards created meet the needs of various data aggregators and site owners."
The EWDCI seeks to facilitate outreach and convene working groups of diverse stakeholders. Scrapers, platforms, publishers, regulators, and public advocacy groups all have valuable perspectives.
Through forums, consultations, and multistakeholder meetings, common ground can emerge on guidelines that balance access to public data with site owner rights.
Benefits of Joining the Effort
So why should web scrapers get involved? I believe EWDCI membership has significant advantages for ethical companies looking to differentiate themselves.
By cooperating in standards development and making a public commitment to follow best practices, aggregators can:
- Showcase trust and transparency – Following EWDCI guidelines provides 3rd party validation of responsible practices for current/prospective clients.
- Improve site partnerships – Demonstrating you honor platforms‘ scraping policies builds goodwill and facilitates collaboration.
- Avoid regulatory risk – Helping define reasonable standards is preferable to potential future government restrictions on data collection.
- Level the playing field – Guidelines prevent unethical competitors from undercutting responsible companies on cost.
In sum, the EWDCI presents a great opportunity to engage in shaping the future of web scraping for the better.
What Sites and Platforms Can Do
For their part, publishers and platforms benefit by clearly communicating their data usage expectations. This includes:
- Implementing machine-readable robots.txt policies
- Providing APIs or partner programs for approved aggregators
- Specifying rate limits, opt-outs, and other restrictions
- Outlining consequences for violations like blocking scrapers
Transparency and dialogue on acceptable practices allows ethical aggregators to comply while deterring bad actors.
Securing the Future of Web Scraping
As data aggregation continues rapid growth, establishing standards provides a way to build public understanding and prevent misuse. The EWDCI‘s mission to promote cooperation and accountability comes at an ideal time to steer the industry toward maturity.
I encourage fellow data professionals to get involved in this effort. Together, we can ensure web scraping provides economic and social value responsibly for years to come. Reach out if you have any other thoughts!
