Skip to main content

Using Focal Harvest for OSINT Collection

Focal Harvest is purpose-built for two distinct workflows. One is research automation for developers and analysts. The other — covered on this page — is open-source intelligence (OSINT) collection: systematically monitoring named targets, topics, or entities across public platforms, with automated alerts when new information surfaces.

Contents​

What Makes It Useful for OSINT​

Most OSINT workflows involve repeating the same collection steps manually across multiple platforms on a schedule. Focal Harvest automates that loop: you define the query, configure the target platforms, set a monitoring interval, and let the daemon run. When something new appears, you get an alert.

The key capabilities that serve OSINT specifically are the daemon scheduler, the Wayback Machine fallback for protected pages, the platform-specific plugins that extract structured data from community hubs, and the local-first architecture that keeps all collected data on your filesystem — no third-party cloud storage involved.

Continuous Topic Monitoring​

Start a monitoring daemon from the TUI by selecting Option 2. You'll be prompted for a search query and a sweep interval in minutes:

python main.py
# Select Option 2: Start Monitoring Daemon
# Query: "CVE-2024-1234"
# Interval: 60 (minutes)

The daemon runs the full six-stage pipeline on your configured interval. Each sweep saves a timestamped report to reports/ and pushes a summary to your configured Discord or Telegram channel if new content is detected.

For headless server deployments, you can also drive the pipeline directly:

from scraper import search_duckduckgo, scrape_urls_concurrently
from analyzer import synthesize_topics
import time

query = "CVE-2024-1234 exploit"
interval_seconds = 3600

while True:
    results = search_duckduckgo(query, max_results=10)
    urls = [r["url"] for r in results]
    scraped = scrape_urls_concurrently(urls)
    report = synthesize_topics(scraped, query=query, spec_topic="new disclosures and proof-of-concept code")
    print(report)
    time.sleep(interval_seconds)

Platform Coverage for OSINT​

The built-in plugins cover the platforms most relevant to OSINT collection workflows:

PlatformWhat's Extracted
Hacker NewsFull comment threads, scores, nested replies
RedditPost text, comment trees, vote counts via .json endpoint
Stack OverflowAccepted answers, code blocks, vote-filtered responses
GitHubRepository discussions, issue threads, PR comments
arXivPaper abstracts, authors, submission dates, DOI metadata
PubMedMedical abstracts, MeSH terms, author affiliations
Google ScholarResult listings with citation counts and linked DOIs
SEC EDGARCompany filings, financial tables, 10-K/10-Q metadata

These plugins extract structured content — not raw HTML — so the synthesis stage receives clean, high-density input directly relevant to your query.

Bypassing Bot Protections​

OSINT targets are often behind bot-detection systems. Focal Harvest handles this at several layers:

Wayback Machine fallback: For platforms that block automated access (Quora, Reddit when rate-limited, some Stack Overflow pages), Focal Harvest automatically redirects the request to the Internet Archive's public snapshot. This preserves access to content without triggering the target platform's defenses.

User-Agent rotation: The HTTP wrapper randomizes browser User-Agents and sends standard Google Search referrer headers and Chrome Client Hints on each request, mimicking natural browser navigation patterns.

TLS impersonation: If curl_cffi is installed, requests use Chrome's JA3 TLS fingerprint instead of Python's default, bypassing Akamai and Cloudflare 403 Forbidden gates. See Configuration → Enable Cloudflare Bypass.

Local cache: After the first successful fetch of a URL, subsequent sweeps load from disk — eliminating repeat exposure to bot-detection systems for the same targets.

Alert Dispatch​

Configure Discord or Telegram webhooks to receive formatted summary embeds whenever a sweep completes. Set them up via the TUI (Option 3) or environment variables:

export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
export TELEGRAM_BOT_TOKEN="your-bot-token"
export TELEGRAM_CHAT_ID="your-chat-id"

Both channels can be active simultaneously. Each alert includes the query, sweep timestamp, source count, and a condensed summary of findings. The full report is always saved locally to reports/ regardless of webhook status.

Responsible Collection​

OSINT collection using Focal Harvest must stay within legal and ethical boundaries. Key constraints:

  • Public data only — do not use active personal session cookies to access authenticated content. Use dedicated research accounts exclusively.
  • Polite rate limiting — the default retry backoff and cache settings are designed to be respectful of target server load. Do not override these to run aggressive parallel sweeps.
  • Local data only — all collected data stays on your filesystem. The maintainers do not receive, store, or transmit any scraped content.
  • Regional privacy law — if your collection involves Personally Identifiable Information, ensure compliance with GDPR, CCPA, or applicable regional privacy frameworks.

See Legal & Responsible Use → for the full disclaimer and case law context.