Maity: Board

ops: two hub replicas, pool metrics, 5xx alerts, read access to pg_stat_activity

tsk-ftnk7d1p5117 in progress Maity OPS Team (maity-ops-team)

Step 3 of the overload follow-ups (req-v10nsyz26p78), after the load test: two hub instances instead of one, response times in the log, pool metrics, alerts on 5xx, and read access to the active DB connections for diagnosis (it was blocked during the 2026-10-26 incident and cost about 30 minutes). Needs the ops agent and probably Oliver for the cluster and secrets.

Notes

  1. Maity Coder Claude 005 (maity-coder-claude-005) ·

    Plan (oz-agent-f, decided by oz-agent-a 2026-10-29). One pod at launch; replicas after the launch.

    Why not just `replicas: 2`: the hub is one process on purpose (hub/context/kubernetes.md, docs/architecture.md §8). In the process: long-poll notifier and "one long-poll per session", the request-signing nonce window, login/join/webhook rate limits, caches, and every background worker (the mail worker is the riskiest: a double send). The 2026-10-26 incident was CPU-bound on one vCPU, so two pods on that node would not help; the load test (tsk-ryas52q9k53z) decides.

    Now, before the load test, two small separate PRs:

    1. Request log with ms= and the route pattern; every 60 s one line with the pool state (checked_out, overflow, waiters) and the 5xx count since the last line. Plain key=value lines on stdout for kubectl logs (k3s, no log stack), not JSON.
    2. application_name=maity-hub on every connection, so pg_stat_activity shows who holds them.

    After the launch (PRs 3-7):

    1. Workers only in a leader: an advisory lock per worker (pg_try_advisory_lock; MariaDB GET_LOCK); another pod takes it on the next tick.
    2. Nonce window in the database (unique insert, expiry).
    3. Long-poll "one per session": the newest wins (to be decided by the lead when this PR comes).
    4. Login rate limit as a database counter; the other limits stay per pod and are documented as such.
    5. Chart: replicas as a value with a schema bound, RollingUpdate with maxSurge: 1, a PodDisruptionBudget, preStop. Chart is ops's; proposed only.

    Not hub code (ops, probably Oliver): read access to pg_stat_activity (role pg_monitor on a diagnostic role); 5xx alerts, to be settled by ops once the log lines exist; a node with more than one vCPU.

    Not checked line by line yet: which workers already claim their rows atomically.

  2. Maity OPS Team (maity-ops-team) ·

    S3 for attachments (B, 2026-10-31, contract docs/api.md 8.7; store choice still open for the maintainer): settings MAITY_ATTACHMENT_STORE=s3, MAITY_ATTACHMENT_S3_ENDPOINT (https; plain http only to loopback or cluster host with MAITY_ATTACHMENT_S3_ALLOW_HTTP=1), _BUCKET (required), _PREFIX, _REGION, _ACCESS_KEY_ID and _SECRET_ACCESS_KEY (Secret). Rights: Put/Get/DeleteObject under the prefix and ListBucket on it, no bucket or policy rights. The hub streams objects (no presigned URLs): the endpoint only has to be reachable from the pods. Bucket versioning from the start, replication to a second location if possible (replaces the attachment tar). Path-style vs virtual-hosted not decided yet. No provider details in the public repo.

  3. Maity OPS Team (maity-ops-team) ·

    OVH stage 1 prepared outside the repo, /workspace/tmp/a/ovh/: traefik-values.yaml (chart traefik/traefik 42.1.3, PROXY protocol v2 from the load balancer, externalTrafficPolicy Local, LB timeouts 120 s, 2 replicas, redirect to https; renders with helm template), hub-values-ovh.yaml (live values with Traefik namespace for the NetworkPolicy and a placeholder for config.trustedProxies; renders with the chart), issuer-dns01.yaml + cert-prod.yaml (DNS-01 at Cloudflare, token as Secret), RUNBOOK-stufe1.md (9 steps, the maintainer executes). Open placeholders: LB egress IPs, Traefik pod CIDR, image digest. Not run anywhere. Watch: Traefik access log keeps request paths, so single-use join tokens (/<token>, /i/<token>) appear in it; short retention or a path filter should be decided before prod logs are shipped anywhere.