Distill

Distill is a research-paper distillation project with two main parts:

1. agent_flow\ — a Node.js pipeline that turns a PDF into structured Markdown context and then into a single embeddable HTML visual.
2. distill\ — a Next.js app that lets users upload a paper, run the pipeline, and view the generated summary and visual in the UI.

Repository layout

Path	Purpose
agent_flow\	CLI pipeline for PDF -> summary Markdown -> HTML visual
distill\	Next.js application and API route that invokes agent_flow
design-handoff\	Design/support artifacts currently kept in the repo

What the project does

Given a research paper PDF, the project aims to:

• extract and understand the paper
• create a readable Markdown summary focused on the hardest important ideas
• generate a single self-contained HTML visual that can be embedded in an iframe
• surface the result inside the web app

How it works

1. Paper processing pipeline

The pipeline lives in agent_flow\.

Main commands:

npm --prefix .\agent_flow install
npm --prefix .\agent_flow run summarize -- --pdf "C:\path\to\paper.pdf"
npm --prefix .\agent_flow run visualize -- --context-file ".\out\paper.summary.md"
npm --prefix .\agent_flow run pipeline -- --pdf "C:\path\to\paper.pdf"

Current behavior:

• default summarize mode is extract
• PDF text is extracted locally
• long papers are chunked
• chunk-note model calls are run in parallel
• a final synthesis produces Markdown
• a second step generates an HTML visual

Outputs are written under agent_flow\out\ unless overridden.

2. Web app

The app lives in distill\.

It includes:

• a paper upload flow
• a server route at distill\app\api\distill\route.ts
• client components that render live summary content and the generated HTML visual

The API route saves the uploaded PDF under distill\public\generated\..., runs the pipeline, then returns:

• distilled markdown
• a generated visual URL
• basic paper metadata

Local setup

Prerequisites

1. Node.js 20+
2. npm
3. A locally installed AI CLI used by agent_flow
4. Authentication/configuration for that CLI before running the pipeline

Install dependencies

npm --prefix .\agent_flow install
npm --prefix .\distill install

Run the web app

npm --prefix .\distill run dev

Run the pipeline directly

npm --prefix .\agent_flow run pipeline -- --pdf "C:\papers\paper.pdf" --summary-out ".\out\paper.summary.md" --visual-out ".\out\paper.visual.html"

Important implementation notes

• Relative paths passed through npm --prefix .\agent_flow ... resolve from inside agent_flow\.
• The summary stage is generally reliable with extract, but large papers can still take time because they require multiple model calls.
• The visual stage is the slowest and most fragile part right now; large summary inputs can still time out depending on model responsiveness.
• The generated HTML visual is intended to be served as a static asset and embedded with an iframe in Next.js.
• Debug logging is enabled throughout the pipeline to make slow steps and failures easier to diagnose.

Current limitations

• Direct PDF-native reading has been less reliable than extracted-text summarization.
• Large papers can still be slow even after chunk reduction and parallel chunk-note requests.
• Visual generation may need prompt/input reduction or additional optimization for long summaries.
• The repository currently contains both product code and some handoff/support artifacts.

Suggested workflow

1. Start the Next.js app.
2. Upload a paper in the UI.
3. Let the API route run the agent_flow pipeline.
4. Review the generated summary and visual.
5. If needed, run the pipeline directly from agent_flow\ to debug or tune behavior.

Related docs

• agent_flow\README.md — pipeline-specific usage and behavior
• distill\app\api\distill\route.ts — server entry point for running the pipeline from the app