Proxy Harvester & Validator: Reference Manual & CLI Guide

1. Architectural Architecture & Async Pipeline

Public proxy scraping frequently involves dealing with dead IP addresses, slow handshakes, and forged headers. Synchronous scrapers take hours to validate a few thousand IP addresses because standard socket timeouts block execution.

Proxy Harvester & Validator utilizes Python's asyncio event loop and aiohttp client sessions with custom DNS resolver caches to test 500+ proxy connections simultaneously in parallel tasks, validating 10,000 candidate IPs in under 90 seconds.

2. Installation & Environment Setup

# Install via pip
pip install proxy-harvester-validator

# Or clone from source repository
git clone https://github.com/Modracx/proxy-harvester-validator.git
cd proxy-harvester-validator
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

3. CLI Usage & Command Examples

3.1. Basic Harvest and Fast Validation

# Harvest proxies and validate with default 200 concurrent workers
python -m proxy_harvester --harvest --validate --output live_proxies.txt

3.2. Strict Filtering for HTTPS Elite Proxies

# Filter for HTTPS proxies only with latency under 800ms and Elite anonymity
python -m proxy_harvester \
  --protocols https \
  --anonymity elite \
  --max-latency 800 \
  --concurrency 400 \
  --format json \
  --output elite_proxies.json

4. Anonymity Classification Matrix

Anonymity Level	HTTP Headers Sent to Target	Target Detects Proxy?	Target Sees Real IP?
Transparent	Sends `X-Forwarded-For` and `Via` headers with your real IP	Yes	Yes
Anonymous	Sends `Via` header, but conceals your real IP address	Yes	No
Elite (High-Anon)	Sends zero proxy headers; connection appears as a standard residential/server IP	No	No

5. Output Formats Reference

Results can be streamed directly into multiple formats via the --format parameter:

txt: Raw ip:port strings (one per line) ready for curl or scraper pools.
json: Detailed metadata including country ISO, latency (ms), protocol, anonymity level, and last verified timestamp.
csv: Comma-separated table for Excel or spreadsheet import.

6. CLI Arguments Reference Table

Argument Flag	Type	Default	Description
`--harvest`	Flag	`False`	Fetch new proxies from built-in public source feeds
`--input`	File Path	`None`	Validate an existing proxy list file instead of harvesting
`--protocols`	String	`http,https,socks4,socks5`	Comma-separated protocols to validate
`--concurrency`	Integer	`200`	Number of simultaneous async validation tasks
`--timeout`	Integer	`5`	Socket connection timeout in seconds

Proxy Harvester & Validator: Reference Manual