Outbound downloader workers¶
For master configuration, node approval, archive semantics and migration, see Multi-worker downloader: master setup.
Deployment runner¶
downloaderworker is an outbound-only worker for the v2 fleet API at
/api/downloader/workers/v2. Run it on a machine whose lawful/institutional egress
can access the desired papers. It invokes the shared downloader.FetchPDF ladder;
remote proxy/fleet delegation and agent execution are not enabled inside workers.
The existing downloaderproxy image and root proxy configuration are unchanged.
Container deployment¶
docker build -f Dockerfile.downloaderworker -t qatlas-downloaderworker .
docker volume create qatlas-worker-data
# worker.env contains DL_WORKER_MASTER_URL=https://qatlas.example.org,
# DL_WORKER_ENROLLMENT_TOKEN=<enrollment credential>, DL_WORKER_NAME=campus-01,
# and optionally DL_UNPAYWALL_EMAIL / DL_S2_API_KEY. Protect it with chmod 600.
docker run -d --name qatlas-downloaderworker --restart unless-stopped \
--stop-timeout 30 --shm-size 256m \
--env-file worker.env \
-v qatlas-worker-data:/var/lib/qatlas-downloader \
qatlas-downloaderworker
Do not publish any ports (-p/-P are unnecessary). The runner has no HTTP
listener. Its sole local browser supervisor binds CDP to 127.0.0.1:9222 inside
the container; CDP is not an administration or public API. The entrypoint only
execs the Go runner and never starts a second browser. It prefers headless-shell
and omits full Chrome's --headless=new flag for that binary. Unexpected browser
exit triggers a delayed restart. SIGTERM cancels download contexts, stops and
reaps the owned Chrome process group, and leaves durable results for recovery.
An externally managed browser is not terminated by this worker.
The image runs as UID/GID 65532. Named volumes acquire the image directory's ownership; a bind mount must already be writable by that UID. Mount one unique persistent data volume per worker. Do not share/copy a worker identity across concurrently running containers. A volume lock rejects duplicate runners. The data volume includes a plaintext independent worker secret (files mode 0600, new directories mode 0700); restrict access and protect backups. Reusing the volume with a different master URL is rejected to avoid credential disclosure. Do not store the data directory on an ephemeral container filesystem.
After first registration, the worker remains pending until an administrator approves it on the master. Its random ID and random 256-bit secret are committed to the volume before registration; retries reuse that identity, including recovery after a lost registration response. The master hashes the worker secret. Subsequent requests use only that worker's bearer credential, not the enrollment credential. Once enrollment succeeds, the enrollment env can be removed on the next restart without changing the volume. Never place credentials in URL query parameters. All redirects are rejected (including same-host redirects), preventing bearer forwarding; configure the final master origin directly.
Configuration¶
Flags override their corresponding environment defaults. Credential variables
are env-only so --help cannot print secret defaults.
Environment variable |
Flag |
Default / meaning |
|---|---|---|
|
|
Required HTTPS master origin; optional reverse-proxy path prefix |
|
env-only |
Required for first enrollment, unnecessary once registered |
|
|
Hostname; descriptive operator label |
|
|
|
|
|
2, validated range 1–32; enforced by admission and active tasks |
|
|
|
|
|
|
|
|
Optional external CDP; disables local browser spawning; use a trusted private endpoint |
|
|
Auto-detect headless-shell, google-chrome, chrome, chromium; image sets |
|
|
Optional Unpaywall/OpenAlex contact email |
|
env-only |
Optional Semantic Scholar API key |
— |
|
False; explicit opt-in only for trusted local development |
— |
|
|
— |
|
Read local health status and exit; never opens a listener |
Production HTTPS is mandatory. --allow-http sends credentials in plaintext and
must not be used over an untrusted network. Standard TLS certificate validation
is never disabled. Control requests have a 20-second overall deadline; uploads
have a 3-minute deadline. Non-2xx, malformed, oversized and redirect responses are
failures, never acknowledgements. Failed control/upload operations retry on a
10-second polling cadence, with uploads independent of heartbeats. The master
client may use standard process HTTP(S) proxy environment settings and may target
a private HTTPS master; the repository's root proxy configuration is unchanged.
Ordinary PDF/landing-page and arXiv HTTP fetches use a separate public-network client: only HTTP(S) URLs without userinfo are accepted, each DNS answer is checked for loopback/private/link-local/multicast/reserved destinations, and the connection dials the validated numeric address (no second DNS lookup). The same checks apply after redirects. Mixed public/private DNS answers are rejected. This fetch client bypasses environment HTTP proxies, because a proxy could resolve an otherwise-unvalidated private destination. This can intentionally reject campus-only private-address PDF hosts; there is no unsafe bypass flag.
This is not a complete browser/network SSRF sandbox. Chromium navigation, JavaScript subresources and metadata-resolver clients are separate networking paths. Deploy the worker/browser in an isolated network namespace with egress firewall rules denying host services, RFC1918/internal networks, link-local/cloud metadata services and other sensitive destinations, except explicitly required master/CDP endpoints. Do not use host networking or mount host service sockets. A public-looking URL is not sufficient protection against browser redirects, DNS rebinding or malicious subresources. External CDP needs equivalent isolation.
Durable results, receipts, TTL and disk pressure¶
A catalog records running, ready, failed, or expired attempts with durable
atomic file replacement and directory fsync. Downloaded PDFs are hashed locally,
written and synced before their ready catalog transition. A restart verifies
ready-file size and digest, resumes delivery, and reports prior running attempts
as interrupted instead of silently dropping or redownloading them. A corrupt or
missing ready PDF fails startup, allowing operator investigation rather than
silently treating a lost result as successful.
Delivery first queries the attempt receipt and then sends the raw PDF with stable
attempt ID, SHA-256, size and bounded provenance metadata. Lost responses cause
idempotent receipt checks/retries. An HTTP 200/202 or staged/received receipt
is not an archived acknowledgement. Before TTL, deletion requires a done
receipt matching task ID, attempt ID, digest and size: done means the master
has durably archived and registered the PDF. Receipt lookup is nondestructive.
Transient upload failures, uncertain responses and stale attempts retain the
local PDF until an explicit archived receipt or retention expiry.
The default 24-hour TTL starts when the result becomes ready. At expiry the
runner first durably marks expired, then removes the PDF and retries a timeout
failure report to the master so the task can be rescheduled. Active upload files
are protected from expiry. If online, receipt/delivery is attempted before that
cycle's expiry cleanup; there is no guarantee an unavailable master archives a
result before TTL. Increase TTL if offline periods can exceed a day.
Admission reserves 100 MiB per active download, accounting for ready PDFs, existing reservations, and actual filesystem free space (plus metadata headroom). Quota/disk pressure stops new claims; unexpired ready results are never evicted to make space. Each returned PDF is bounded by the smaller of 100 MiB and the master's assignment limit. Catalog/identity/browser-profile filesystem overhead is outside the PDF quota; provision extra disk space. Keep the volume on a local filesystem supporting file locking, atomic rename and fsync.
Health and operations¶
docker inspect health runs downloaderworker --healthcheck against the local
health.json. Healthy means approved, browser reachable, a recent successful
master heartbeat, and usable capacity or currently running reserved work. Pending,
draining/revoked, browser failure, communication failure, stale (>90s) health,
and fully blocked spool are unhealthy. Thus newly enrolled containers normally
show unhealthy until approved; this is not a reason to recreate their volumes.
Container restart policy follows process exit, not health status.
Worker logs use fixed error summaries and HTTP status codes rather than tokens,
master response bodies or credential-bearing URLs. Successful provenance strips
source URL userinfo/query/fragment. Failed downloads persist and report the last
12 strategy steps (including duration) through the existing v2 Trace fields;
URLs are omitted from failure traces. Errors contain only allowlisted failure
kinds, available HTTP status codes and allowlisted verification/denial page
titles from the last captured HTML response (not necessarily the live DOM).
Arbitrary titles, page bodies and raw error strings are never exported. Unknown
errors remain generic; not_found means no strategy produced a PDF, not HTTP 404.
worker download failed logs correlate task/attempt IDs; worker download strategy logs contain the bounded sanitized trace. Reports retain these traces
across spool restarts and delivery retries. A master with diagnostic recovery
support also includes them in failed download pipeline traces and retains the
previous worker failure when a requeued task later expires. A final queue timeout
therefore does not necessarily mean a worker ran until its execution deadline.
Healthy only verifies worker readiness, not publisher subscription entitlement.
No protocol/schema migration is required. Update both worker and master binaries for the full diagnostics; a new worker can still report to an older v2 master, but the old master may overwrite the summary on expiry. Preserve each worker's existing data volume and identity when replacing its container. This change does not alter browser navigation, retry budgets, concurrency or publisher access. No inbound endpoint or live CDP port needs firewall exposure. Restrict outbound master access and browser/CDP administration appropriately.
Local verification (no live downloads)¶
go test ./internal/downloadworker ./cmd/downloaderworker
go build ./cmd/downloaderworker
Unit tests use temporary directories, fake fetchers, and httptest; they do not
deploy containers or download real papers. The opt-in TestPostgresOutbound*
suite runs real runner-to-fleet HTTP flows against an in-process httptest master
and a uniquely named temporary schema in a disposable PostgreSQL database:
TEST_DOWNLOADFLEET_DATABASE_URL='postgres://test_user:password@localhost/test_database?sslmode=disable' \
go test -run TestPostgresOutbound ./internal/downloadworker
The test user needs permission to create/drop schemas. Never point this at a production database. Tests remove only their own schema on completion and never modify the public schema or Goose migration state. Coverage includes pending approval, complete PDF archival and spool cleanup, deliberately lost upload acknowledgements, staged archive failure/recovery, nondestructive receipts, two-worker shared capacity, and failure rescheduling to a second worker. Container dependencies, real Chrome supervision, and institutional egress still require separate operator-run staging checks with approved infrastructure.