VIS Paper Tracker

A Python tool that automatically tracks the online availability of IEEE VIS conference papers on arXiv (Semantic Scholar optional). The tool scrapes paper information directly from the VIS 2026 website, maintains an incremental index showing when papers become available online with timestamps and abstracts, and can download the PDFs it finds.

🚀 Quick Start Guide

New user? Follow these 4 simple steps:

1. Install dependencies: uv venv && uv pip install -r requirements.txt
2. Get papers: python3 vis_paper_tracker.py --scrape
3. Start tracking: python3 vis_paper_tracker.py --add-papers vis2026_papers.json
4. Search & see results: python3 vis_paper_tracker.py --update --report

Daily updates: Just run python3 vis_paper_tracker.py --update --report

Current status (VIS 2026, last run 2026-09-01): 172 of 287 papers found online (59.9%), 161 PDFs downloaded.

Features

• Automatic Web Scraping: Fetches papers directly from IEEE VIS 2026 website with proper encoding handling
• Multi-Source Search: Searches arXiv by default, with Semantic Scholar available via --sources
• Paper Classification: Detects Full Papers and Short Papers (posters are excluded)
• Session Tracking: Captures conference session information
• Progress Logging: Optional detailed logging with statistics
• PDF Downloads: Optional fetching of arXiv PDFs into a single folder
• Data Export: JSON index and CSV files for analysis
• Analysis Tools: Built-in author analysis with statistics and visualizations
• Incremental Updates: Tracks discovery dates and maintains search history
• Robust Progress Saving: Saves progress every 10 papers to prevent data loss
• Data Cleaning: Built-in author name formatting and encoding issue fixes

Quick Start

Prerequisites

• Python 3.7 or higher
• Internet connection

Installation

1. Download or clone this repository
2. Create the environment and install dependencies with uv:

uv venv
uv pip install -r requirements.txt
source .venv/bin/activate   # or prefix commands with: uv run

First Time Setup (Complete Workflow)

Follow these steps in order:

# Step 1: Scrape papers from IEEE VIS 2026 website
python3 vis_paper_tracker.py --scrape

# Step 2: Add papers to tracking system (use the file created in step 1)
python3 vis_paper_tracker.py --add-papers vis2026_papers.json

# Step 3: Search for papers online (paced at one arXiv request per 5 seconds)
python3 vis_paper_tracker.py --update

# Step 4: See results
python3 vis_paper_tracker.py --report

Daily Updates (After Setup)

Once you've done the initial setup, just run:

# Check for newly available papers
python3 vis_paper_tracker.py --update --report

Understanding the Data Workflow

The tool uses a two-stage data system:

Stage 1: Seed Data (one-time setup)

• vis2026_papers.json - Static list of papers from VIS 2026 website
• Created by: --scrape command
• Contains: titles, authors, sessions, paper types
• Used once to populate the tracking system

Stage 2: Live Tracking (ongoing updates)

• paper_tracking_data_2026/paper_index.json - Live database with search results
• paper_tracking_data_2026/paper_availability.csv - Spreadsheet export for analysis
• Updated by: --update command
• Contains: search status, URLs, abstracts, discovery dates

Optional Files

• Log files - Detailed activity logs (if you use --log-file)

Data Flow

--scrape → vis2026_papers.json → --add-papers → paper_tracking_data_2026/ → --update → results
   ↑              ↑                    ↑               ↑                        ↑         ↑
 One-time      Seed data         Populate       Live tracking              Search     Reports

Common Commands

First Time Only

# Get papers from VIS website and start tracking
python3 vis_paper_tracker.py --scrape
python3 vis_paper_tracker.py --add-papers vis2026_papers.json

Regular Use

# Search for papers and see results
python3 vis_paper_tracker.py --update --report

# With detailed logging
python3 vis_paper_tracker.py --update --report --log-file daily_check.log

Choosing Search Sources

# Default: arXiv only - fast, and the source of nearly all hits
python3 vis_paper_tracker.py --update

# Add Semantic Scholar, queried only for papers arXiv did not have
python3 vis_paper_tracker.py --update --sources arxiv,semantic_scholar

Semantic Scholar is opt-in because its unauthenticated API is shared across all callers and rate-limits heavily. In the VIS 2026 run it contributed 11 of 172 finds (6%) while increasing the runtime from minutes to hours, and its hits are landing pages with no downloadable PDF. Worth enabling for an occasional thorough sweep, not for routine updates.

Downloading PDFs

# Fetch the PDFs of papers that were found on arXiv into paper_pdfs_2026/
python3 vis_paper_tracker.py --download-pdfs

# Choose a different destination
python3 vis_paper_tracker.py --download-pdfs --pdf-dir ~/Papers/vis2026

Files are named Paper_Title_arXivID.pdf, and papers already on disk are skipped, so it is safe to re-run after each --update. Papers found only on Semantic Scholar are counted as "without an arXiv link" and not downloaded: those links are publisher landing pages with no predictable (or freely accessible) PDF.

Maintenance

# Fix formatting issues in author names
python3 vis_paper_tracker.py --clean-authors

# Re-scrape if new papers are added to VIS website
python3 vis_paper_tracker.py --scrape
# Note: This overwrites vis2026_papers.json with fresh data

Tracking a Different Year

# The tracker defaults to VIS 2026; any year with a papers list page works
python3 vis_paper_tracker.py --year 2027 --scrape
python3 vis_paper_tracker.py --year 2027 --add-papers vis2027_papers.json --update --report

# Each year keeps its own database: paper_tracking_data_<year>/
# (VIS 2025 data stays in the original paper_tracking_data/ directory)
python3 vis_paper_tracker.py --year 2025 --report
python3 analyze_authors.py --year 2025 --report-only

Advanced Options

# Use custom data directory
python3 vis_paper_tracker.py --data-dir my_tracking_data --update

# Enable debug logging
python3 vis_paper_tracker.py --update --debug

# Just generate a report (no searching)
python3 vis_paper_tracker.py --report

Data Format

Input Papers (JSON)

{
  "title": "Paper Title",
  "authors": "Author1, Author2, Author3",
  "session": "Session Name",
  "award": "Award Type",
  "paper_type": "full",
  "venue": "TVCG",
  "location": "Boston"
}

Paper Types

• full: Main conference papers (including TVCG and CG&A papers presented at VIS)
• short: Shorter research contributions
• unknown: When type cannot be determined

The venue field records the journal tag shown next to a title (TVCG, CG&A, empty for conference-only papers), and location records the presentation site listed by the conference (e.g. Boston, Tianjin, Paris, Pre-recorded talk).

Note: Posters are automatically excluded from tracking as they are typically not published as citable papers.

Output Files

• vis2026_papers.json: Scraped paper list from VIS 2026 website
• paper_pdfs_2026/: Downloaded PDFs (only with --download-pdfs; git-ignored)
• paper_tracking_data_2026/paper_index.json: Persistent tracking database
• paper_tracking_data_2026/paper_availability.csv: Export for data analysis
• Log files (optional): Detailed status reports and statistics

Project Structure

The paper lists and tracking databases are version controlled - they are small, and keeping them makes each year's results a historical record even after the conference website changes. Downloaded PDFs and analysis outputs are git-ignored: re-fetchable with --download-pdfs and analyze_authors.py respectively.

After setup, your project directory will look like this:

vis-paper-tracker/
├── vis_paper_tracker.py       # Main script
├── analyze_authors.py          # Author analysis and visualization script
├── vis2026_papers.json         # Seed data (created by --scrape)
├── paper_pdfs_2026/            # Downloaded PDFs (optional, git-ignored)
├── paper_tracking_data_2026/   # Live tracking database for VIS 2026
├── paper_tracking_data/        # Live tracking database for VIS 2025 (legacy name)
│   ├── paper_index.json        # Detailed search results
│   └── paper_availability.csv  # Spreadsheet export
├── README.md                   # This documentation
├── requirements.txt           # Python dependencies
├── LICENSE                    # MIT license
└── .gitignore                 # Git ignore rules

How It Works

1. Web Scraping: Fetches papers from IEEE VIS 2026 website with session names and paper types (excludes posters)
2. Author Cleaning: Removes double commas and normalizes author lists
3. Multi-Source Search: Searches arXiv first, falling back to Semantic Scholar only when enabled with --sources
4. Fuzzy Matching: Uses 70% threshold word overlap for title matching
5. Discovery Tracking: Records first discovery dates and maintains search history
6. Progress Saving: Automatically saves progress every 10 papers to prevent data loss
7. Abstract Storage: Keeps the longest abstract found across sources
8. Progress Logging: Optional detailed logging with paper type and session statistics

Data Analysis

The tracker includes analysis scripts to explore patterns in the collected data:

Author Analysis

Analyze authorship patterns with the included analyze_authors.py script:

# Basic author analysis
python3 analyze_authors.py

# Show distribution statistics
python3 analyze_authors.py --stats-only

# Generate histogram (requires matplotlib)
uv pip install matplotlib numpy
python3 analyze_authors.py --histogram

# Top 10 authors with interactive plot
python3 analyze_authors.py --top-n 10 --histogram --show-plot

Sample Output:

TOP AUTHORS BY PAPER COUNT
- Kwan-Liu Ma: 8 papers (7 full, 1 short) - 24 collaborators
- Huamin Qu: 8 papers (7 full, 1 short) - 35 collaborators
- Cindy Xiong Bearfield: 8 papers (5 full, 3 short) - 31 collaborators

PAPER COUNT DISTRIBUTION
- 1 paper: 886 authors (82.1%)
- 2 papers: 131 authors (12.1%)
- 3+ papers: 62 authors (5.8%)

Outputs:

• Console report with top authors and collaboration statistics
• author_analysis.csv - Detailed spreadsheet with all authors
• author_histogram.png - Visual distribution chart

Analysis Options

# Analysis script options
python3 analyze_authors.py --help

# Key parameters:
--top-n 20              # Number of top authors to show
--histogram             # Generate visual histogram
--stats-only           # Show statistics without full report
--csv filename.csv      # Custom CSV output filename
--hist-file plot.png    # Custom histogram filename
--show-plot            # Display plot interactively

Custom Analysis

The tracking data is stored in standard JSON/CSV formats, making it easy to:

• Import into R, Python pandas, or Excel for custom analysis
• Create visualizations with your preferred tools
• Analyze collaboration networks, temporal patterns, or subject areas
• Compare productivity across institutions or research groups

Troubleshooting

Common Issues

Error: "No such file or directory"

# Make sure you use the correct filename
ls *.json  # See what files exist
python3 vis_paper_tracker.py --add-papers vis2026_papers.json  # Use existing file

Error: "ModuleNotFoundError"

# Install missing dependencies into the project environment
uv pip install -r requirements.txt

Slow performance during updates

• This is normal! The tool waits 5 seconds between arXiv requests to respect their rate limits
• A full update of 287 papers takes ~25-30 minutes with the default arXiv-only search
• Adding --sources arxiv,semantic_scholar can stretch that to several hours
• Progress is saved every 10 papers, so interruptions won't lose much work
• You'll see progress counters like "Checking (45/287): ..." showing current status

Lots of "429" errors in the log

• arXiv is rate limiting; the tool backs off and retries automatically, and papers whose searches could not be completed are marked error rather than not_found
• Re-running --update retries both error and not_found papers, so nothing is lost
• Sustained 429s mean an IP-level cooldown: wait ~15 minutes before retrying

No papers found

• Check your internet connection
• Try running with debug logging: --debug
• Some papers may not be available on arXiv or Semantic Scholar yet

Double commas in author names

# Clean up existing data
python3 vis_paper_tracker.py --clean-authors

Garbled characters in paper titles (like âThey Arenât Built For Meâ)

• This is from encoding issues in older scraped data
• Re-scrape to get clean data:

python3 vis_paper_tracker.py --scrape
rm -rf paper_tracking_data_2026/
python3 vis_paper_tracker.py --add-papers vis2026_papers.json

Papers show "Added" but report shows 0 papers

• This was a bug that has been fixed
• If you encounter this, update to the latest version of the code

Getting Help

If you encounter issues:

1. Run with debug logging: python3 vis_paper_tracker.py --debug --update
2. Check the log file for detailed error messages
3. Verify your Python version: python3 --version (needs 3.7+)

API Information

• arXiv API: No authentication required, 15-second timeout
• Semantic Scholar: No API key needed, returns top 5 results (opt-in; heavily rate limited)
• IEEE VIS 2026: Direct HTML scraping, no authentication needed
• Rate Limiting: Per-host spacing (5 s for arXiv, 3 s for Semantic Scholar) with exponential backoff and up to 4 retries on HTTP 429/5xx, honouring Retry-After

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Authors

Claudio Silva & Claude

Acknowledgments

• IEEE VIS 2026 conference organizers
• arXiv and Semantic Scholar for their open APIs