Skip to main content

How Focal Harvest's Pipeline Works

Focal Harvest runs a six-stage research pipeline entirely inside your terminal. Each stage hands off clean output to the next, from a raw query to a structured Markdown report pushed to your configured channels. This page explains what happens at each stage and which part of the codebase owns it.

Contents​

Pipeline Overview​

Owner: scraper.py

Focal Harvest resolves your query against one of two search backends. If a Tavily API key is configured, it uses the Tavily Search API, which is optimized for programmatic retrieval and returns cleaner, more structured results. If no Tavily key is present, it falls back to a lightweight DuckDuckGo crawler that requires no credentials.

The output of this stage is a ranked list of target URLs to scrape in Stage 2.

Stage 2: Concurrent Fetch​

Owner: utils.py

All target URLs are fetched in parallel threads. The HTTP wrapper implements configurable retry limits, exponential backoff with random jitter (to avoid thundering-herd effects on target servers), and fail-fast timeouts on connections that stall. Before making any live request, the wrapper checks the local cache — if a URL was scraped within the cache_expiration_hours window, it loads from disk and skips the network entirely.

If curl_cffi is installed, the wrapper automatically routes requests through it to impersonate Chrome's TLS/JA3 fingerprints, bypassing Cloudflare and Akamai bot-detection gates. See Configuration → for setup details.

Stage 3: HTML Sanitization​

Owner: scraper.py

Raw HTML is noisy. Stage 3 strips navigation menus, cookie overlays, ad containers, tracking scripts, and footer widgets — leaving only the core prose and structured content. Focal Harvest uses a hybrid parser for this:

  • readability-lxml extracts clean, layout-stripped article content for standard editorial pages.
  • Full-soup structural cleaning takes over for directory-index pages (like Hacker News or GitHub), where readability-lxml would strip content that's actually meaningful.

Platform-specific plugins override both for major sites that require custom handling. See Platform Plugins → for the full list.

Stage 4: Adaptive Replenishment​

Owner: scraper.py

Some pages will block, return empty content, or fail after retries. Rather than passing sparse data to the synthesis stage, Focal Harvest detects these gaps and automatically fetches replacement URLs from the search results to fill them. This keeps the context density high enough to produce a useful synthesis output.

Stage 5: Synthesis​

Owner: analyzer.py

Clean text from all scraped pages is passed to the synthesis engine. Focal Harvest tries your configured AI providers in order — Gemini 1.5 Flash first, then OpenAI GPT-4o-mini, then Claude 3.5 Sonnet — and falls back to the next if a key is invalid or rate-limited. If no API keys are configured at all, it uses the built-in offline PositionRank statistical keyword-density ranker to compile a report without any network calls.

The synthesis output is a structured Markdown report with a summary, key findings, and source attributions.

Stage 6: Dispatch​

Owner: notifier.py

The completed report is saved to reports/ as both Markdown and raw JSON. If Discord or Telegram webhooks are configured, notifier.py pushes a formatted summary embed to your channels. The terminal displays the full report inline and logs cache status for each URL scraped (CACHED vs LIVE).

Codebase Layout​

├── main.py               # Interactive TUI controller and main loop
├── scraper.py            # Search, concurrent crawler, HTML parser, plugins
├── analyzer.py           # Multi-LLM wrappers and offline PositionRank logic
├── notifier.py           # Markdown exporter, Discord/Telegram webhook dispatch
├── utils.py              # HTTP wrapper — retries, cookies, header sanitization
├── config_manager.py     # Reads and writes config.json
├── install.py            # Platform-independent dependency setup
├── std_plugins/          # Built-in platform-specific parser plugins
└── tests/                # Isolated unit test suite

Each file owns exactly one stage of the pipeline. main.py is the orchestrator — it reads user input from the TUI and calls the other modules in sequence.

Now that you understand the pipeline, Platform Plugins → explains how the 17+ built-in modules extend Stage 3 for specific sites.