Skip to main content

Platform Plugins — What Gets Scraped and How

Focal Harvest ships with 17+ built-in parser modules in std_plugins/. Each plugin applies platform-specific cleaning rules on top of the standard HTML sanitizer, extracting structured content that a generic parser would miss or mangle. This page lists every built-in plugin, what it targets, and what it extracts.

Contents​

Developer Platforms​

Hacker News​

Extracts full thread comment hierarchies and front-page listing content. Captures comment scores, usernames, and nested reply trees — not just the top-level post.

Reddit​

Maps URLs to Reddit's .json API endpoint for clean programmatic access. Falls back to Wayback Machine archived snapshots for threads behind login gates or bot blocks. Reconstructs comment trees from the JSON response.

Stack Overflow​

Formats code blocks for readability, identifies the accepted answer, and filters responses by vote score. Strips sidebar ads, related-question panels, and footer navigation.

GitHub​

Handles repository file views, directory listings, issue threads, pull request diffs, and discussion pages. Strips the repository sidebar, file tree navigation, and contributor panels.

Academic Databases​

arXiv​

Extracts paper abstracts, author lists, submission dates, DOI metadata, and citation tables. Strips LaTeX rendering artifacts and navigational elements.

PubMed​

Extracts structured medical abstracts, MeSH terms, author affiliations, and journal metadata. Formats citation data as clean Markdown.

Google Scholar​

Extracts result listings with author names, publication years, citation counts, and linked DOIs. Handles Scholar's anti-bot measures via User-Agent rotation.

Blogs and Communities​

Medium​

Extracts article body prose and strips the paywall overlay, membership CTAs, and clap/comment widgets. Works on publicly accessible articles; paywalled content falls back to Wayback Machine.

Substack​

Strips subscription popups and email capture forms. Extracts article body, author byline, and publication date.

Wikipedia​

Extracts article body prose, infobox data, and section headings. Strips edit links, citation superscripts, navigation boxes, and reference lists.

Dev.to​

Strips sidebar panels, tag navigation, and comment widgets. Extracts article prose, code blocks, and author metadata.

Quora​

Redirects to Wayback Machine archived snapshots to bypass the login gate. Extracts question text and top answers by upvote count.

Finance and Markets​

Yahoo Finance​

Extracts live ticker pricing data, company summaries, and financial metric tables. Outputs data tables as Markdown tables. Uses compliant User-Agents for financial data requests.

SEC EDGAR​

Extracts company filing metadata, report summaries, and structured financial tables from 10-K and 10-Q filings. Handles EDGAR's plain-text and HTML filing formats.

Documentation and E-Commerce​

ReadTheDocs​

Strips the navigation sidebar, version selector, and TOC panels. Extracts the main documentation prose and code examples cleanly.

Amazon​

Extracts product titles, prices, ratings, review counts, and feature bullet lists. Strips recommendation carousels and sponsored product slots.

Product Hunt​

Extracts product descriptions, maker comments, upvote counts, and the top launch comments. Useful for tracking new tool launches in a given category.

How Plugins Are Selected​

Plugins are matched automatically by URL pattern. When scraper.py receives a URL, it checks it against each plugin's registered domain patterns in priority order. If a match is found, the plugin's cleaning function runs instead of the standard readability-lxml parser. If no plugin matches, the hybrid parser handles the page.

You can extend the system by adding modules to std_plugins/. Each plugin exports a can_handle(url) function and a parse(html, url) function — the interface is intentionally minimal.

Now that you know what each plugin targets, Configuration → covers how to set API keys, tune the request pipeline, and enable optional platform features.