Hey there!
LinkedIn is like a treasure trove of 850+ million professionals ready for the taking. Whether you‘re in sales, recruiting, research, or just job hunting, there are mountains of data that can help you succeed.
But sifting through each profile manually is mind-numbingly slow. So how do we extract that data from LinkedIn efficiently?
Web scraping.
Scraping tools let you automate LinkedIn searches to extract exactly what you need, when you need it. As a web scraping expert who‘s extracted millions of LinkedIn profiles, I‘ll show you how to tap into this goldmine.
By the end of this guide, you‘ll be able to:
- Scrape thousands of LinkedIn personal and company profiles with just a few clicks
- Uncover the right prospects and candidates hidden in LinkedIn‘s 850M users
- Monitor LinkedIn for new job postings, hires, and other real-time alerts
- Legally and ethically extract the data that matters to you (don‘t worry, I‘ve got you covered)
So if you‘re ready to take your LinkedIn game to the next level, let‘s get started!
LinkedIn by the Numbers: A Data Goldmine
Let‘s scope out the enormous scale of information unlocked by LinkedIn scraping:
850 million members – LinkedIn‘s user base is a massive pool of prospects and talent.
40 million companies – With over 40 million company pages, LinkedIn is a prime B2B data source.
21 million open jobs – Lots of fresh opportunities for job seekers and recruiters.
570,000+ skills – Detailed skill data helps match candidates to roles.
Users from 200+ countries – LinkedIn is global, with users everywhere.
Those are just the high-level stats. But here‘s where it gets interesting…
Each LinkedIn profile also contains dozens of data fields like:
- Job titles and company names
- Emails, phone numbers, and social media links
- Backgrounds, skills, and certifications
- Education history and degrees
- Job descriptions and responsibilities
For companies, you can extract fields like:
- Company size and industry
- Technologies and products
- Leadership names and changes
- Hiring trends and new job postings
- Newsfeed posts and employee growth
This high-fidelity data powers all sorts of business use cases, which we‘ll dig into next.
LinkedIn Web Scraping Use Cases
Let‘s explore some of the top ways companies use LinkedIn scraping:
Sales Prospecting
LinkedIn is every sales rep‘s dream. You can identify relevant prospects and quickly access their contact details and backgrounds.
With web scraping, sales teams can:
Get lead lists – Scrape company names and titles for outbound prospecting
Enrich CRM data – Add social profiles and employment histories on leads
Track new triggers – Monitor companies for leadership changes, new hires, etc.
This transforms random cold outreach into targeted, personalized selling.
Pro Tip: Sort prospects in your CRM by LinkedIn connection strength to prioritize your warm leads.
Recruitment and Headhunting
LinkedIn has become the #1 recruiting channel. Over 75% of job seekers use it in their search.
Recruiters scrape LinkedIn profiles to:
Source candidates – Discover qualified prospects even if they aren‘t actively looking.
Research prospects – Review work history, skills, and education on applicants.
Identify turnover – Monitor competitor pages to see if employees depart.
Build talent pools – Curate databases of potentials candidates in key roles.
This opens up passive talent pools and surfaces candidates that Indeed or Monster may miss.
Pro Tip: Try scraping profiles of employees who recently earned hot skills like Kubernetes or AI expertise.
Market and Competitive Research
Company showcase pages provide a trove of intel for researching target markets and monitoring competitors.
Researchers can scrape LinkedIn for:
Leadership changes – Reveal new executives, departures, and org structures.
Hiring trends – Identify growth areas by job openings and new hires.
Newsfeed posts – Get product launches, announcemnts, and culture posts.
Employee growth – Track headcount over time as a growth metric.
Continuous LinkedIn monitoring builds a vivid image of external threats and opportunities.
Pro Tip: Set up an automated scraper to check competitor pages weekly and alert you to anything useful.
Job Hunting
Finally, don‘t forget job seekers! Scraping tools help candidates by:
Finding openings – Get alerts when target employers post new jobs.
Researching companies – Vet cultures and management styles before applying.
Sourcing insider contacts – Identify employees to network with during the process.
This can make a huge difference when trying to land the perfect role.
Pro Tip: Look for recent alumni of your school now working at the company to find friendly connections.
Of course, these are just a sample of possible use cases. The key is identifying what data would be most valuable to you and then extracting it at scale.
Scraping LinkedIn Legally and Ethically
Now before we dig into the how-to, let‘s cover the rules of the road. Is web scraping LinkedIn even allowed?
Here are a few key principles to scrape by ethically and legally:
Only use public profiles – Avoid logging into private accounts or evading authentication.
Don‘t spam – Add delays and limits in your scraper to avoid flooding LinkedIn.
Rotate proxies – Mix up your IPs to prevent overscraping a single one.
Obey robots.txt – Block any off-limits pages defined by LinkedIn.
Attribute data properly – Don‘t portray scraped data as officially from LinkedIn.
Mind Terms of Service – Stay updated on LinkedIn‘s usage terms.
If you follow these guidelines, you can scrape most public LinkedIn data without issue. Businesses do it every day within reason.
However, scraping sensitive fields like contact info brings some risk. It‘s wise to consult legal counsel before scraping and selling private data in bulk.
But for low-volume internal personal use – lead enrichment, research, recruitment, etc. – you should be fine. Just mind your manners!
Step-by-Step: Scraping LinkedIn Company Pages
Alright, time for the fun part – actually scraping LinkedIn!
Let‘s walk through how to extract company profiles using Apify, my tool of choice:
1. Get Apify LinkedIn Company URL Scraper
First, grab Apify‘s LinkedIn Company Finder from their site or store.
Apify has tons of handy scrapers anyone can use for free – no coding required. We‘ll focus on their LinkedIn tools here.
2. Add Your Target Companies
Next, add your list of companies you want to get LinkedIn URLs for. Put each one on a new line.
You can add individual names or upload whole lists in CSV/Excel formats.
3. Set Concurrency (Optional)
This field determines how many browser instances run in parallel. The higher, the faster it scrapes.
I‘d recommend starting with 15-25 concurrent browsers. You can scale up as you get more experienced.
Too many browsers risks getting blocked, so build up gradually.
4. Click "Run" to Start Scraping
Once configured, just click "Run" to kick off the scraper. It will automatically search LinkedIn and extract matching company profile URLs.
The status will change to "Running" until it finishes.
5. Export LinkedIn Company Data
Finally, click "Storage" to see your scraped LinkedIn company URLs. You can export them as a CSV, JSON, Excel, etc.
And we‘re done! It‘s super quick and easy to extract LinkedIn company pages with Apify – no coding required.
Now let‘s look at getting LinkedIn personal profile URLs.
Step-by-Step: Scraping LinkedIn Personal Profiles
The process for LinkedIn personal profiles is nearly identical. We‘ll use Apify‘s LinkedIn People Finder tool.
1. Get Apify LinkedIn People Finder
Grab the LinkedIn People Finder scraper from Apify Store.
2. Add Names to Scrape
Next, add the names of the LinkedIn profiles you want to lookup. One name per line again.
You can enter individuals or upload long lists as usual.
3. Select Language (Optional)
If you want to restrict results to a certain language, you can set that here.
For example, choose "French" to only get French profiles.
4. Click "Run" to Start
Hit "Run" and the scraper will start searching LinkedIn for those names and return any profile URLs it finds.
5. Export Results
Once finished, go to "Storage" to download your scraped LinkedIn profile URLs in CSV, Excel etc.
And that‘s it! With Apify‘s tools, anyone can scrape both LinkedIn personal and company pages without any technical expertise.
Now let‘s dive into some pro tips and advanced tactics to level up your LinkedIn scraping.
Level Up Your LinkedIn Scraping: Tips from a Pro
Beyond basic URLs, there‘s so much more data you can extract from LinkedIn to get truly actionable insights.
Here are some advanced ways to get even more value out of LinkedIn scraping:
Scrape Emails and Phone Numbers
Contact info is invaluable for sales and recruiting, but LinkedIn hides emails and phone numbers.
Luckily, scrapers can still extract them from LinkedIn in a few different ways:
Parse emails from LinkedIn profile texts, which some users add publicly
Generate email addresses with names and companies
Buy targeted contact lists legally collected from across the web
Email append services like Clearbit or ZoomInfo are great sources here as well.
Monitor Company Changes Over Time
Instead of one-off scrapes, run scripts on a schedule to watch target companies. Have them track:
- New job openings and hiring trends
- Leadership additions and departures
- Major newsfeed announcements
Then push alerts to your sales or recruiting team when changes are found.
This automates lead gen and competitive intelligence.
Scrape International Locations
LinkedIn supports over 20 languages and every country. Make sure to expand your scrapes globally.
Run separate scrapers focused on regions like DACH, LATAM, APAC, etc. And translate any search terms into local languages.
This taps into 700M+ international profiles most competitors ignore.
Target Specific Skills and Titles
The best prospects aren‘t just any old profiles – you need the right roles and skill sets.
When scraping, add filters for:
Job titles like Founder, Developer, CEO
Seniority like Director, VP, Manager
Skills like "Python", "Full Stack Development"
Certifications like PMP, CPA, Six Sigma
Then you‘ll get prospects truly relevant to open positions or customer needs.
Extract Full Profile Data
URLs are just the start. To really power automation, extract full profile data like:
- Job titles, companies, and tenures
- Emails and phone numbers
- Skills, education, volunteering
- Links to social media profiles
This rich data feeds everything from candidate scoring to lead segmentation and nurturing.
Using Proxies for Large-Scale Scraping
When you‘re ready to take LinkedIn scraping to the next level, you‘ll need proxies.
Proxies act as middlemen for your requests, routing each through a unique IP address. This prevents IP blocks from hammering LinkedIn from the same IP.
Here are a few top proxy services I recommend:
BrightData – 40M+ residential IPs worldwide starting at $500/mo
SmartProxy – 1M+ mixed IPs from $200/mo
GeoSurf – Residential IPs with country targeting
Soax – Budget residential proxies from $50/mo
With enough proxies, you can scale up to extract millions of LinkedIn profiles without getting shut down.
I‘d suggest starting with a pool of at least 5,000 IPs – more are better. Rotate them at each request.
LinkedIn Scraping Results and Benchmarks
So what can you expect from your shiny new LinkedIn scrapers? Here are some benchmarks to set your expectations:
Profiles per day: 20,000+ per day scraping at scale
Companies per day: 50,000+ company pages per day
URLs per IP: I recommend rotating proxies after ~500 URLs each as a safe measure. LinkedIn blocks individual IPs scraping too much.
Scraping speed: Web scrapers can crawl pages in < 1 second, but I‘d suggest setting a 2-5 second delay between requests to be safe. You want to avoid flooding LinkedIn.
Accounts: Use one account per ~500 URL scrapes. Regularly rotate to new test accounts to maintain access.
With the right configuration optimized to balance speed and scale, you can extract millions of profiles over time. But start small and work your way up.
Handling LinkedIn Blocks and CAPTCHAs
When scraping heavily, LinkedIn may throw some obstacles your way:
HTTP status 429 – This means you are rate limited. Slow down your scraper or increase proxy rotation.
HTTP status 503 – Service unavailable error. Usually temporary. Wait a bit and retry.
CAPTCHAs – The dreaded "are you a robot?" tests. Use tools like DeathByCaptcha to solve them automatically.
Profile not found – If you get lots of these, LinkedIn may be blocking your scraper IP. Rotate proxies.
The key is having enough proxies to rotate constantly. I recommend 40,000+ for heavy 24/7 scraping. Budget under $1,000 monthly for residential proxies.
Advanced LinkedIn Scraping Tools
Apify makes it easy to start LinkedIn scraping without coding, but for advanced users, some other powerful tools include:
Octoparse – Visual scraper with built-in LinkedIn scrapers
parseHub – GUI web scraper with LinkedIn templates
ScraperAPI – API for JavaScript scraping scripts
Python – Libraries like Selenium and BeautifulSoup for custom scrapers
ProxyCrawl – Residential rotating proxies designed for scraping
These take more technical skill but provide added flexibility for custom LinkedIn scrapers beyond basic URLs.
Scraping LinkedIn Data: Final Thoughts
And there you have it – everything you need to start scraping LinkedIn profiles like a pro.
The key points to remember are:
LinkedIn is a goldmine for sales, recruiting, research, and job hunting if you can extract the data.
Focus on ethically scraping fields valuable to your goals from public profiles.
Use tools like Apify to scrape LinkedIn at scale fast without coding skills.
Rotate proxies constantly to avoid blocks when running large scrapers.
Expand globally and target niche roles and skills to get quality prospects.
Monitor company changes over time instead of just one-off scrapes.
Go beyond basic URLs to get phone, emails, and full profile data.
Scraping unlocks LinkedIn‘s full potential. But use these powers for good! With great data comes great responsibility.
If you have any other questions, feel free to reach out! I love helping people harness the power of LinkedIn data.
Happy extracting!
- Oleg
