# QuantumAtlas ↔ RustFS integration > **新不可变内容边界**：fresh PDF 与完整解析成员写独立 `s3.bucket_content` > （默认 qatlas-content，必须区别于旧桶/corpus），本地后端使用同一 content/ 逻辑命名空间。 > 成员 create-only、持久SHA复核、manifest最后发布，PG成功后才切current；PG不可用不延期报成功。 > 同SHA可复用source ID；arXiv vN语义、source字节身份、S3VersionId物理版本、parse revision > 不同。冻结导入别名不同字节409（overwrite也不替换），解析重传新revision。 > 旧三桶/layout/versioning/prune步骤保留为历史与旧资源运维，**不得据此覆盖冻结原件**； > 新功能不bulk回填，不删除旧桶，不复制旧MD/JSON/images，只在内容/PDF访问懒冻结旧PDF。 > 无nightly/boot/入库自动解析，只有显式内容访问或管理员明确RunNow。 > 当前API/存储/错误与上传规则见[论文内容契约](paper-content.md)和[Upload API](upload-api.md)。 > How the Go server (`cmd/qatlasd`) wires to RustFS (S3-compatible > object store) for paper assets. Covers env vars, IAM policy spec, > bucket layout, version lifecycle, the `qatlasd storage prune` > operator command, and known RustFS-vs-MinIO quirks. > > Application-level upload semantics (sha256 dedup, 409 conflict > behaviour, `?expected_sha256=` guard) live in > [upload-api.md](upload-api.md). Wider storage architecture (why > we have separate Raw / Metadata / Graph layers) lives in > [storage-architecture.md](../concepts/storage-architecture.md). ## Backend selection `internal/objstore` exposes a single `Store` interface with two implementations: - `LocalStore` — directory under `cfg.RawDir` (XDG default `~/.local/share/qatlasd/raw/`). Dev / first-boot / CI. No version concept, no presigned URLs. - `S3Store` — RustFS / MinIO / Amazon S3, via `minio-go/v7`. Production. Selection is **all-or-nothing**: setting any of the four `QATLAS_S3_*` env vars without setting all four is a startup error. With all four set, the server logs ``` raw store: S3 backend http://:9000/qatlas-raw ``` on every boot. Without them it logs ``` raw store: local backend /home//.local/share/qatlasd/raw ``` The split is in `cmd/qatlasd/main.go::initRawStore` and the all-or-nothing rule is enforced by `internal/config/config.go::(*Config).ValidateForServe` (called from `cmd/qatlasd/main.go::validateServeCfgAfterFlags` inside the `serve` RunE). `config.Load` itself only emits a `slog.Warn` on a half-set quartet so non-serve subcommands like `qatlasd --help` / `qatlasd pat list` keep working on a partially-configured `.env`. ## 支持的对象存储后端 qatlasd 通过 [`minio-go/v7`](https://github.com/minio/minio-go) 走 **AWS SigV4** 协议，所以**任何 S3-compatible 后端都能用**。验证过 / 强推荐： | 后端 | 部署形态 | 备注 | |---|---|---| | **RustFS** | 自托管（推荐） | 默认选型；NAS / VPS / k8s 都行 | | **MinIO** | 自托管 | RustFS 的前身，行为接近；CE / community 版免费 | | **AWS S3** | 公有云 | 真 S3；按调用次数计费，dev 慎用 | | **Cloudflare R2** | 公有云 | 零 egress 费；endpoint = `https://.r2.cloudflarestorage.com` | | **Backblaze B2 (S3-compat)** | 公有云 | 低价 cold storage 友好 | | **Wasabi** | 公有云 | 同上 | | **DigitalOcean Spaces** | 公有云 | 同上 | | **Ceph RadosGW** | 私有云 | SigV4 全兼容 | | **SeaweedFS (S3 gateway)** | 自托管 | 跨地域 erasure coding 场景 | **部分支持**（需要 path-style URL + 验证 SigV4 实现完整性）： | 后端 | 注意事项 | |---|---| | **阿里云 OSS（S3 兼容模式）** | endpoint 用 `https://oss-.aliyuncs.com`；某些 bucket policy 字段不支持 | | **腾讯云 COS（S3 兼容模式）** | 同上，endpoint `https://cos..myqcloud.com` | | **华为云 OBS（S3 兼容）** | 同上 | **不支持**（不是 SigV4 协议，需要换 SDK）： - Azure Blob Storage（native API） - Google Cloud Storage（native API） - WebDAV / NFS / SMB / FTP（不是 S3） - Synology Drive、Nextcloud（不是 S3） > 想用 Azure / GCS 必须前面挂个 [s3proxy](https://github.com/gaul/s3proxy) 翻译协议；社区有用例但 QuantumAtlas 未官方验证。 ## 分离部署：对象存储放另一台机 RustFS / S3 backend 跟 qatlasd 不需要同机。常见拓扑： - **NAS 模式**：家用 NAS（Synology / QNAP / 自组）跑 RustFS docker；qatlasd 在 VPS / 工作站；走家用宽带反代或 EasyTier mesh - **公有云对象存储**：qatlasd 自托管，对象走 R2 / AWS S3 / Wasabi；省自管存储的运维 - **跨地域多边缘**：qatlasd 在多个 edge（VPS / 不同机房），共享同一 RustFS（active-active 写入；audit 留痕由 RustFS notify 兜底） ### 硬件 / 容量建议（对象存储侧） 对象存储的瓶颈在**磁盘容量**和**网络带宽**，CPU/内存要求低： | 用量级别 | 容量估算 | 推荐设备 | |---|---|---| | 个人 / 实验室 | 万篇 paper ≈ 10 GB PDF + 50 GB images = 60 GB | 任意 NAS / SBC / 廉价 VPS | | 团队 | 10 万篇 ≈ 600 GB | 中等 NAS / 中端 VPS | | 多边缘共享 | 取决于增长曲线，建议 EC 起步 | 高 IO NAS / 专属对象存储 VPS | CPU 2 核 + 内存 2 GB 已绰绰有余（HDD 场景见现有 `RUSTFS_DRIVE_TIMEOUT_PROFILE=high_latency` 优化）。 ## Required env vars | Var | Example | Notes | | -------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------- | | `QATLAS_S3_ENDPOINT` | `http://:9000` | Must include scheme. Production prefers mesh-direct (avoids edge-Caddy self-loop). | | `QATLAS_S3_BUCKET` | `qatlas-raw` | Must exist; bootstrap script creates it idempotently. | | `QATLAS_S3_ACCESS_KEY_ID` | `CNEDAZ2HQDU9TX8A2BUO` | Service-account key (`qatlasd` IAM user). Never use root keys here. | | `QATLAS_S3_SECRET_ACCESS_KEY` | `…` | Secret printed once by bootstrap; copy directly into `.env` (mode 600). | | `QATLAS_S3_PUBLIC_ENDPOINT` (可选) | `https://` | 公网入口，给 client presigned URL 用；留空 = 单 endpoint 模式（仅适合 dev）| ### Dual-endpoint mode 生产部署里 server↔RustFS 走**mesh / 内网**（省一跳反代 + TLS 终结），但 server 内部若需给 client 签 presign URL 拿对象，必须**公网可达**。两者用同一份 endpoint 显然不行——所以 qatlas server 支持 dual-endpoint： | 用途 | 走哪个 endpoint | |---|---| | server 内部 Put/Get/Stat/List | `QATLAS_S3_ENDPOINT`（internal） | | presign URL（dev / 调试场景）| `QATLAS_S3_PUBLIC_ENDPOINT`（public） | 启用方法：在 `.env` 同时设两个： ```bash QATLAS_S3_ENDPOINT=http://:9000 # mesh 内网 QATLAS_S3_PUBLIC_ENDPOINT=https:// # 公网（独立子域） ``` **公网入口必须反代到内网 RustFS 端口，且 `preserve Host header`**——SigV4 把 Host 算进 canonical request，反代改 Host 会让 RustFS 报 `SignatureDoesNotMatch`。最小 Caddy 模板： ```caddy { reverse_proxy :9000 { header_up Host {host} } } ``` 详见 [反向代理](reverse-proxy.md)。 启动 log 区分两种模式： ``` raw store: S3 backend http://:9000/qatlas-raw (presign via https://) ``` 少了 `(presign via ...)` 那段就是单 endpoint 模式。 每台边缘各自配自己的 public endpoint，**不共享**： - Edge A: `https://`（举例：LE 真证书 + 独立子域） - Edge B: `https://:`（举例：`tls internal` 自签，client 必须 `-k`） ## IAM policy: `qatlas-raw-rw` The `qatlasd` IAM user is bound to this policy (created by `scripts/rustfs_bootstrap.sh`): ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:GetObjectVersion", "s3:DeleteObjectVersion" ], "Resource": "arn:aws:s3:::qatlas-raw/*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:ListBucketVersions", "s3:GetBucketLocation", "s3:GetBucketVersioning", "s3:PutBucketVersioning" ], "Resource": "arn:aws:s3:::qatlas-raw" } ] } ``` What each permission is for: | Action | Why qatlas needs it | | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `s3:GetObject` / `s3:PutObject` | Routine PDF / markdown / JSON I/O via the upload handlers. | | `s3:DeleteObject` | Soft-delete via the (currently unimplemented) `DELETE /api/papers/*` route + admin cleanup. | | `s3:GetObjectVersion` | Reading a specific past version (for future rollback CLI; not yet exposed in HTTP). | | `s3:DeleteObjectVersion` | **Required by `qatlasd storage prune --yes`** — versioned deletes are a separate AWS perm from `s3:DeleteObject`. | | `s3:ListBucket` / `s3:GetBucketLocation` | minio-go probes the endpoint and walks prefixes (e.g. enumerate-needs-mineru). | | `s3:ListBucketVersions` | Powers `ObjectVersion`-aware listing — backs `qatlasd storage prune` enumeration. | | `s3:GetBucketVersioning` / `s3:PutBucketVersioning` | Lets qatlas self-manage versioning at boot (see "Versioning" below). | **Deliberately not granted** (re-test before adding): - `s3:DeleteBucket`, `s3:PutBucketPolicy`, `s3:PutBucketAcl` — bucket destruction / ACL change should stay root-only ops; qatlas has no use case. - `s3:GetLifecycleConfiguration`, `s3:PutLifecycleConfiguration` — **RustFS 1.0.0-beta.5 rejects these action names** with `invalid action`. Re-test when bumping RustFS; until then qatlas doesn't use lifecycle rules anyway (see "Why no auto-expiration" below). ## Bucket layout Object keys are constructed by `internal/paperassets.AssetKey` as ``` //v. ``` with `` being the first 4 chars of the YYMM segment (e.g. `2501` → `pdf/2501/2501.00010v1.pdf`) so a flat list of papers shards naturally into year-month folders, keeping individual prefix listings manageable. | Kind | Path | Content-Type | | ---------- | ------------------------------------- | ------------------------------------ | | `pdf` | `pdf//v.pdf` | `application/pdf` | | `json` | `json//v.json` | `application/json` | | `markdown` | `markdown//v.md` | `text/markdown; charset=utf-8` | User metadata always includes `x-amz-meta-sha256` (lowercase) with the hex digest of the bytes — see [upload-api.md](upload-api.md). This is the field `qatlasd storage prune` and the upload handler both rely on for idempotency / dedup. ## Versioning: qatlas self-manages `internal/objstore/s3.go::EnsureVersioning` is called once at server boot, right after `initRawStore`. Pattern: ``` GetBucketVersioning(bucket) if Status == "Enabled" → log "already enabled", no-op else → EnableVersioning(bucket), log "enabled (was: )" ``` This is **idempotent** and **monotonic**: qatlas only ever transitions to `Enabled`, never to `Suspended`. Even if an operator manually suspends versioning via mc, the next qatlas restart re-enables it. Rationale: losing the ability to recover an over-written PDF is a much bigger correctness hazard than the (small) extra storage cost. Boot log lines you should always see (in this order): ``` raw store: S3 backend http://:9000/qatlas-raw bucket versioning: enabled (was: "") ← first boot ever bucket versioning: already enabled ← every subsequent boot Server started at http://127.0.0.1:4200 ``` Failure mode: if the IAM user lacks `s3:Put/GetBucketVersioning`, EnsureVersioning logs `WARN bucket versioning: reconcile failed; …` and the server **continues to serve**. Uploads still work; you only lose overwrite-rollback safety until perms are fixed. This is a deliberate warn-and-continue choice — bouncing the whole server because of a non-critical config drift is worse than degrading. ## Why no auto-expiration (lifecycle) We **deliberately do not install an S3 lifecycle rule** to auto-expire noncurrent versions. The model is "Synology Snapshot / Time Machine": keep everything by default, prune on demand. Reasoning: - sha256 dedup already short-circuits identical re-uploads (no wasted version), so the noncurrent versions we accumulate are real content changes — worth holding onto for rollback. - Auto-expiration windows are operationally fraught: pick 30d and you regret it the day someone needs to restore a 6-week-old draft; pick 365d and the cost picture matters again. - The ops side has full visibility + control via `qatlasd storage prune` (see next section), so manual policy is just as good in our scale regime. When (if ever) the bucket grows past a few hundred GB of noncurrent versions, revisit. RustFS may by then support the standard `s3:*LifecycleConfiguration` actions and we can add a rule. ## `qatlasd storage prune` The on-server CLI for manual cleanup. Lives in `cmd/qatlasd/storage_cmd.go`; runs against whatever the server's own env vars say (`QATLAS_S3_*` from the same `.env` qatlas reads at boot). ``` qatlasd storage prune [--prefix P] [--older-than DUR] [--keep-last N] [--yes] [--json] [--dry-run] # default true ``` Flags: | Flag | Effect | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--prefix` | scope to keys under this prefix (e.g. `pdf/2511/2511.00010v1.pdf` for a single paper, `pdf/2511/` for a cohort). Default empty = whole bucket. | | `--older-than` | only versions older than this duration are eligible. Accepts Go duration syntax (`24h`, `720h`) plus operator-friendly `d` / `w` / `y` units (`30d`, `4w`, `1y`). Default empty = no age cap. | | `--keep-last N` | per object key, keep the N most-recent noncurrent versions, only delete those beyond that count. Current version is ALWAYS kept regardless. Default 0 = no per-key cap. | | `--yes` | actually delete. Without it, the run is forced dry (regardless of `--dry-run`). | | `--json` | emit one JSON object per row on stdout (audit-log friendly). | | `--dry-run` | preview only. Defaults to true; `--yes` is the only way to actually delete. | Hard safety invariants (enforced by `planPruneCandidates` + unit tested in `cmd/qatlasd/storage_cmd_test.go`): - **Current (latest) versions are NEVER deleted.** No flag combination can override this. - **Latest delete markers are NEVER deleted.** Deleting one would resurrect the prior version, which is almost never what the operator wants. - Filters compose. A version must satisfy BOTH `--older-than` and fall outside `--keep-last` to be pruned. So `--older-than 90d --keep-last 5` means "keep at least 5 most-recent noncurrent per key, plus drop anything younger than 90d even if it's beyond the keep-last cap". ### Recipes ```bash # Audit pass: list every noncurrent in the bucket (no deletes) sudo -u $TARGET storage prune # Cohort cleanup: drop all noncurrent for one paper, keep current sudo -u $TARGET storage prune \ --prefix pdf/2501/2501.00010v1.pdf --yes # Tightening retention: per paper, keep at most 5 noncurrent sudo -u $TARGET storage prune --keep-last 5 --yes # Age-based: drop anything noncurrent for > 1 year sudo -u $TARGET storage prune --older-than 1y --yes # Machine-readable for an audit log sudo -u $TARGET storage prune --json | tee prune-$(date +%F).log ``` `$TARGET` = the qatlas binary (`/home//.local/bin/qatlasd` on the production deploy). Run as the `` service user (the systemd unit's `User=`) so the env / file paths resolve identically to the running server. ### Output format Plain dry-run / preview: ``` KEY VERSION_ID SIZE AGE ACTION pdf/2511/2511.88888v1.pdf 5f14251f-8b00-4be4-a0d1-e5ff592a8f89 92826 20m7s DELETE_PLANNED pdf/2511/2511.88888v1.pdf 69537cbf-2035-4aa2-8ec3-4fc8dca357a6 92812 20m15s DELETE_PLANNED --- candidates: 2 versions, 0.18 MiB total dry-run only — pass --yes to delete the listed versions ``` `--yes` adds per-row deletion lines: ``` pdf/2511/2511.88888v1.pdf @5f14251f-8b00-4be4-a0d1-e5ff592a8f89 DELETED pdf/2511/2511.88888v1.pdf @69537cbf-2035-4aa2-8ec3-4fc8dca357a6 DELETED --- deleted: 2, failed: 0, freed: 0.18 MiB ``` ## Bootstrap (initial RustFS setup) `scripts/rustfs_bootstrap.sh` is idempotent and creates everything the server expects: bucket `qatlas-raw`, IAM user `qatlasd`, policy `qatlas-raw-rw`, and one fresh service-account key pair. ```bash export RUSTFS_ENDPOINT=https:// # public, root-creds path export RUSTFS_ROOT_ACCESS_KEY= export RUSTFS_ROOT_SECRET_KEY= bash scripts/rustfs_bootstrap.sh ``` Last few lines of stdout print the new access key + secret. Copy into the server's `.env` immediately — they are NEVER persisted to disk by the script. Bootstrapping a second time creates an *additional* service-account key (existing keys are not rotated / deleted) — useful for key rotation, see the script's own comments. Local variable naming: the script uses `IAM_USER` (not `USER`) internally. `$USER` is auto-set in every interactive shell to the login user, so `${USER:-qatlasd}` would never fall through to the default. Setting `IAM_USER=…` from the environment if you want to bootstrap a non-default IAM user. ## Troubleshooting ### "Access Denied" on upload but versioning works at boot Probably the IAM user record got deleted (RustFS quirk) while the service-account key remained. Symptoms: - `mc admin user info qatlas qatlasd` → "user does not exist" - `mc admin user svcacct ls qatlas qatlasd` → still shows your key - Server boots fine (versioning Get/Put succeed somehow) - Upload returns `500 {"detail": "stat …: objstore: stat …: Access Denied."}` Recovery: ```bash RAND_PW=$(openssl rand -base64 24) mc admin user add qatlas qatlasd "$RAND_PW" mc admin policy attach qatlas qatlas-raw-rw --user qatlasd # verify mc admin user info qatlas qatlasd # should now show PolicyName ``` Existing service-account keys re-associate with the recreated user record. You do NOT need to regenerate credentials or restart qatlas. ### `policy create` succeeded but svcacct still 403 Cache. RustFS 1.0.0-beta.5 has a short policy-eval cache. Wait ~30s and retry. If still 403, double-check policy JSON via `mc admin policy info qatlas qatlas-raw-rw` — sometimes mc reports "created" but the JSON didn't apply (we hit this with `s3:*Lifecycle*` action names, see "Deliberately not granted" above). ### `storage prune --yes` fails with "Access Denied" on delete The policy is missing `s3:DeleteObjectVersion` (versioned delete is a different AWS perm from `s3:DeleteObject`). Update the policy via mc + re-run prune. Bootstrap script already grants it correctly since 2026-05-28. ### Boot log says `bucket versioning: reconcile failed` The IAM user lacks `s3:Put/GetBucketVersioning`. Fix the policy (see "IAM policy" section). Server continues to run without rollback safety until the policy is fixed and the server restarts (or EnsureVersioning runs again on next boot). ### `s3:GetLifecycleConfiguration` errors with "invalid action" Known RustFS 1.0.0-beta.5 limitation. Don't grant lifecycle perms to the IAM user. We don't use lifecycle anyway (see "Why no auto-expiration"). Revisit when bumping RustFS. ## Recovery walk-through: rolling back an overwritten PDF ```bash # Find versions of the paper mc ls --versions qatlas/qatlas-raw/pdf/2501/2501.00010v1.pdf # Output: # [2026-05-28 14:27:33 +08] 90KiB STANDARD v2 PUT 2501.00010v1.pdf # [2026-05-28 14:27:14 +08] 689KiB STANDARD v1 PUT 2501.00010v1.pdf # Restore v1 by copying it as the new current mc cp --version-id \ qatlas/qatlas-raw/pdf/2501/2501.00010v1.pdf \ qatlas/qatlas-raw/pdf/2501/2501.00010v1.pdf ``` The server's next GET for that key serves the restored bytes. No restart needed. The over-written v2 becomes noncurrent (but is still recoverable until `storage prune` decides otherwise). ## 写入留痕 audit sink (T10) **问题**：S3 svcacct key 一旦泄露，持有者能绕过 `qatlasd` API 直连桶 写/删对象。我们要能在日志里**看到**这种直连，并区分它和正规 server 写， 且**跨 edge 一致**（两台 edge 共享 RustFS，审计要落在一处）。 **方案**：RustFS **notify webhook**（per-bucket subscribe，PUT/DELETE 推到 sink） → NAS docker 内 **Fluent Bit sidecar**（HTTP input + S3 output）→ 批量写入 `qatlas-s3-events` 桶。每条事件带 `userIdentity.principalId`（SigV4 `accessKey`）、 `requestParameters.sourceIPAddress`、`userAgent`、`eventName`（`s3:ObjectCreated:*` / `s3:ObjectRemoved:*`）、`s3.bucket.name`、`s3.object.key` 等。sink 刻意选**通用、 零后端约定**的日志转发器（Fluent Bit，CNCF Graduated 项目）作为 sidecar，**不碰 我们的 binary**——dumb 存储层不该被后端演进中的约定（事件 JSON 解析、桶布局、 过滤逻辑）绑死；Go server 按人工审核的 `v` Git tag 发版，正式版本仅从该 tag 派生，也不该逼 NAS 跟着换 sink 镜像。Go server 唯一参与的是 `QATLAS_EDGE_NAME` 打的 UA 标（见下，纯辅助标识）。 **为什么不用 RustFS 原生 audit**：audit 子系统在 1.0.0-beta.5 上有 `has_any_audit_targets` 门控 bug——env 把 target 摆进 config view 但 `start_audit_system` 读的是 persisted store（默认空），判定无 target → 跳过 activation → target 永远 `status=offline` / `not_loaded_in_runtime`；console 改 target 又被「env-source 锁定」挡掉（`audit target '' is managed by environment variables and cannot be modified from the console`）。死循环，beta.5 上没有干净 出口。notify webhook（per-bucket subscribe）路径经实测可靠，于是改用这条。 ### 取证判定（主键 = accessKey，不是 UA） - `accessKey` = root（部署期取的 root 别名，例如 `admin`）→ 直接点名误用 root（SigV4 绑定，不可伪造，强信号）。 - `accessKey` ≠ 任何预期 svcacct（既非 edge 写 key、也非 sink 自己）→ 有人拿别的 key 直连。 - `sourceIPAddress` 非预期网段 → 佐证。 - **UA 只作辅助提示，绝不作判定主键**——UA 可伪造，靠 UA 判定的话攻击者把 UA 伪装成 `qatlasd/*` 就隐身了。`QATLAS_EDGE_NAME` 打的 UA 标 （`qatlasd//`）只是让正规写在事件流里"一眼可读"，不是安全边界。 注意：两台 edge **共享同一把 svcacct key**，光看 `accessKey` 分不出是哪台 edge 写的——这正是 UA edge 标唯一的用处（要它生效得在每台 edge `.env` 设 `QATLAS_EDGE_NAME`）。 ### 自循环陷阱（源头不订阅 > Fluent Bit filter drop） 如果给 `qatlas-s3-events` 桶也加 notify subscription，sink 写入事件对象本身 又触发 PUT 事件 → sink 再写 → 无限循环。**解法：只订阅 5 个资产桶** （`qatlas-raw` / `qatlas-pdf` / `qatlas-md` / `qatlas-images` / `qatlas-openalex`），**不订阅 `qatlas-s3-events`**——从源头不产生事件，比 Fluent Bit `grep` filter drop 更干净（filter drop 仍有 RustFS→sink 一次 HTTP 投递的开销，源头不订阅是零开销，也少一处可能配错的逻辑）。 > 警告：**sink 仍用独立 svcacct（`qatlas-s3-events-writer`），不复用 edge 的 > `QATLAS_S3_ACCESS_KEY_ID`**——理由是**最小权限 + 审计不可变**：sink 只拿 > `qatlas-s3-events` 桶的 Get/Put/List，**没有 Delete**（审计落了删不掉），也碰 > 不到 5 个资产桶。复用 edge key 既越权、又会污染分析（"是 sink 自己写的还是 > 谁直连写的？"分不开）。 ### 两个必守的配置坑（实测踩出来的真因） 1. **`QUEUE_DIR` 必须可写**：RustFS notify webhook 自带磁盘队列（投递失败时缓冲 重放），默认 `/opt/rustfs/events` 在 container 内**不可写** → target 创建直接 失败、状态 `not_loaded_in_runtime`。改成 `/data/.notify-events`（在 `rustfs_data` named volume 内）就行。 2. **ARN 必须小写**：notify webhook env 后缀 `_QATLAS` 会被 RustFS 内部小写化 成 `account_id="qatlas"`，所以 `mc event add` 的 ARN 必须写 `arn:rustfs:sqs::qatlas:webhook`——**大写 ARN（如 `::QATLAS:webhook`）静默 丢弃所有事件**（bucket→target 解析失败，event 直接丢，没报错也没日志）。 ### probe 失败 ≠ 永久放弃 RustFS notify webhook 启动时会跑一次 sink endpoint probe；probe 失败**不会** 让 target 永久 disable，而是落盘到 `/*.event.snappy`。sink 起来后 会自动 replay 队列，**不需要** docker compose 加 `depends_on.condition: service_healthy` 起停顺序——RustFS 容器可以先起，Fluent Bit 慢几秒起来也不 丢事件，简单 `depends_on:` 即可。 ### 供给（用户持 root 跑一次） ```bash # RustFS root key 在 NAS compose env 里，agent 不持有。用户跑： export RUSTFS_ENDPOINT=http://:9000 read -rs RUSTFS_ROOT_ACCESS_KEY; export RUSTFS_ROOT_ACCESS_KEY # = compose RUSTFS_ACCESS_KEY read -rs RUSTFS_ROOT_SECRET_KEY; export RUSTFS_ROOT_SECRET_KEY # = compose RUSTFS_SECRET_KEY bash scripts/rustfs_notify_bootstrap.sh ``` 脚本幂等： 1. 建 `qatlas-s3-events` 桶（无 versioning，事件对象 write-once）； 2. 建 `qatlas-s3-events-rw` policy（Get/Put/ListBucket，**故意不给 Delete** = 审计不可变）+ `qatlas-s3-events-writer` user/svcacct； 3. 建 `qatlas-s3-events-ro` 只读 policy 挂到现有 `qatlasd` 父用户（edge svcacct 自动继承读，给未来 Go 侧对账/扫描预留只读）； 4. **5 个资产桶逐一绑定**小写 ARN `arn:rustfs:sqs::qatlas:webhook`（`qatlas-raw`、 `qatlas-pdf`、`qatlas-md`、`qatlas-images`、`qatlas-openalex`），`qatlas-s3-events` **不绑**（断自循环）。绑定持久化在 RustFS 数据卷中，跨重启/recreate 不丢； 只有 wipe `rustfs_data` 才需重跑此脚本的 `[6/7]` 段。 **只打印 sink 的 access/secret**，root 不落盘——跟 `rustfs_bootstrap.sh` 供给 edge svcacct 同款套路（agent 全程只见 scoped key，没见过 root）。 ### NAS compose（RustFS notify + Fluent Bit sidecar） 完整 compose 模板见 [`deploy/nas-rustfs-compose.example.yaml`](https://github.com/IAI-USTC-Quantum/QuantumAtlas/blob/main/deploy/nas-rustfs-compose.example.yaml) （含占位符 + 配置坑注释）。结构： - **rustfs**：5 个资产桶的 PUT/DELETE 事件通过 notify webhook 推到 sidecar。 target 命名 `QATLAS`（env 后缀；RustFS 内部小写化为 `account_id=qatlas`、 ARN `arn:rustfs:sqs::qatlas:webhook`），endpoint `http://fluent-bit:9880/` （docker 网络内 service 名解析，零公网暴露），queue dir `/data/.notify-events`，queue limit 100k（≈ 8 个月日上传量）。 - **fluent-bit**：HTTP input :9880 收事件 → S3 output 批量写入 `qatlas-s3-events`（endpoint `http://rustfs:9000`、`use_put_object=On`、 `s3_key_format=/%Y/%m/%d/%H-%M-%S-$UUID.json`）。sink 凭据走 `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` env（S3 SDK 标准；值 = bootstrap 输出的 `qatlas-s3-events-writer` svcacct）。**不需要 grep filter 断自循环**——`qatlas-s3-events` 本来就不在 subscription 列表里。 - **桶→target 绑定**：notify env 只声明 target，**bucket→target 订阅必须用 `mc event add` 或 S3 `PutBucketNotificationConfiguration`**——见上面 bootstrap 脚本 `[6/7]` 段，5 个资产桶各绑一次（小写 ARN）。绑定持久化在 RustFS 数据卷中，跨重启/recreate 不丢；只有 wipe rustfs_data 才需重跑。 durability 两层兜底：RustFS notify webhook 自带 `QUEUE_DIR` 磁盘队列（sink 挂 时缓冲重放，limit 100k）；Fluent Bit S3 output 自带 filesystem buffer （`store_dir`，RustFS 写挂时缓冲）。 > 警告：NAS 是 Synology DSM，compose 编辑 + 容器 down/up **只能在 DSM GUI 完成** > （ssh 用户不在 docker 组、sudo 要交互密码）。agent 写好 compose 片段交用户 > 在 DSM 里粘贴 + down/up。 ### 对象布局 Fluent Bit S3 output 把多条事件**批量**攒成时间分区的 JSON 对象 （`/%Y/%m/%d/%H-%M-%S-.json`），每次 upload 是一个**全新不可变对象**—— S3 无 append，但这里根本不需要 append（不是 read-modify-write 同一文件，没有 并发丢行问题；Fluent Bit 的 disk buffer 负责攒批 + 崩溃重放）。读取： `mc cat qatlas-s3-events///
/*.json | jq`。 ## Related docs - [upload-api.md](upload-api.md) — request/response shape, sha256 semantics, in-transit guard from the client's perspective. - [storage-architecture.md](../concepts/storage-architecture.md) — wider architecture (why Raw / Metadata / Graph are separate layers). - [deployment.md](operations.md) — systemd unit, .env layout.