How to Configure Focal Harvest
Focal Harvest works with zero configuration out of the box — offline mode activates automatically when no API keys are present. This page covers how to add AI provider keys, set up notification webhooks, tune the request pipeline, and enable optional features like Cloudflare bypass.
Contents
- Set API Keys
- Environment Variables
- Pipeline Parameters
- Cache Settings
- Enable Cloudflare Bypass
- Cookie-Based Authenticated Scraping
Set API Keys
Option A — In-app (recommended for first-time setup):
Launch the TUI and select Option 3 (Configure API Keys & Settings). Paste your credentials when prompted. These are saved to a local config.json file in your project directory.
Option B — Environment variables:
Export keys in your terminal session, or write them to a .env file in the project root:
export GEMINI_API_KEY="your-gemini-key"
export OPENAI_API_KEY="your-openai-key"
export ANTHROPIC_API_KEY="your-anthropic-key"
export TAVILY_API_KEY="your-tavily-key"
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
export TELEGRAM_BOT_TOKEN="your-bot-token"
export TELEGRAM_CHAT_ID="your-chat-id"
All keys are optional. Focal Harvest uses whatever is available and falls back gracefully when a key is missing or rate-limited.
Environment Variables
| Variable | Required | Description |
|---|---|---|
GEMINI_API_KEY | No | Google Gemini 1.5 Flash — first in the failover chain |
OPENAI_API_KEY | No | GPT-4o-mini — second in the failover chain |
ANTHROPIC_API_KEY | No | Claude 3.5 Sonnet — third in the failover chain |
TAVILY_API_KEY | No | Tavily Search API — preferred over DuckDuckGo when present |
DISCORD_WEBHOOK_URL | No | Sends report embeds to a Discord channel on sweep completion |
TELEGRAM_BOT_TOKEN | No | Telegram bot credentials for alert dispatch |
TELEGRAM_CHAT_ID | No | Telegram chat or channel ID to receive alerts |
Pipeline Parameters
Once config.json is generated, you can fine-tune the request pipeline directly. Open the file in any editor:
{
"max_retries": 3,
"backoff_factor": 1.0,
"retry_on_status_codes": [429, 500, 502, 503, 504],
"cache_enabled": true,
"cache_expiration_hours": 24
}
| Parameter | Default | Description |
|---|---|---|
max_retries | 3 | Maximum retry attempts per failed request |
backoff_factor | 1.0 | Base multiplier for exponential backoff sleep times |
retry_on_status_codes | [429, 500, 502, 503, 504] | HTTP status codes that trigger a retry |
Increasing backoff_factor makes retries more conservative — useful if you're hitting rate limits on protected sites.
Cache Settings
Focal Harvest caches scraped pages to reports/cache/ using MD5 hashes of each URL. Subsequent requests for the same URL within the expiration window load instantly from disk.
| Parameter | Default | Description |
|---|---|---|
cache_enabled | true | Enable or disable local file caching |
cache_expiration_hours | 24 | Hours before a cached page is considered stale and re-fetched |
Set cache_expiration_hours to 0 to always fetch fresh content. Set cache_enabled to false to disable caching entirely — useful for monitoring tasks where you need live data on every run.
Enable Cloudflare Bypass
For scraping heavily protected e-commerce sites, documentation portals, or any target that returns 403 Forbidden, install the optional C-compiled TLS impersonation library:
pip install curl_cffi
Once installed, Focal Harvest automatically detects it and routes requests through curl_cffi to mimic Chrome's JA3 TLS signatures and connection layout. No additional configuration is needed.
curl_cffi is optional and only activates when present. The core pipeline works without it — install it only if you encounter consistent 403 blocks on specific targets.
Cookie-Based Authenticated Scraping
Focal Harvest supports scraping pages that require session authentication. Place a Netscape-format cookies.txt file in the config/ directory:
config/
└── cookies.txt
The pipeline ingests cookies automatically and sends them with matching domain requests. You can also configure domain-to-cookie maps directly in config.json for more granular control.
Only use dedicated research or burner accounts for authenticated scraping. Using personal session cookies risks account suspension if a platform detects automated access. See Legal & Responsible Use → for full guidance.
With configuration in place, see Benchmarks → for performance expectations across different hardware and operation types.