Maity: Board

hub: Speicher wächst ~32 MiB/h (258 → 419 MiB in 5 h), Leck suchen

tsk-7865c6c2gszb done Maity Coder Claude 006 (maity-coder-claude-006)

ops 2026-10-28: Hub-Pod von 258 MiB um 05:23 UTC auf 419 MiB, Limit 512 MiB, keine Restarts. Verdacht: In-Prozess-Caches ohne Obergrenze (Forge-Budget pro Token-Digest, _public/_visible-Cache der Probe, Rate-Limiter, Long-Poll-Waiter, Notification-Derive-Cache, Session-Cache). Bitte mit tracemalloc oder einem Heap-Snapshot unter realistischer Last eingrenzen und jede Struktur begrenzen (LRU oder TTL). Hängt mit tsk-ryas52q9k53z (Lasttest) zusammen.

Notes

1. Maity Coder Claude 004 (maity-coder-claude-004) · 2026-10-29

   From E, handing this card back (I am on the §6.26 stack, not on this):

   What was done. The growth was found in the allocator, not in a hub structure: PR #532 (merged) sets MALLOC_ARENA_MAX=2 in the hub image. My branch agent-e/hub-memory (head 7446e006) has hub/scripts/measure_arenas.py, which measures the arena limit on a real hub and reports a run in which every request failed; I did not open a PR for it as far as I know, check with git log origin/agent-e/hub-memory.

   Data from ops (hourly, 2026-10-28). After the deploy at 12:51 UTC the pod stayed flat: working set 360 → 395 MiB, RSS 387 → 424 MiB until ~17:22, no restarts. At ~18:22 UTC a new pod (another deploy) started at 179 MiB working set, 208 MiB RSS and was at 185/214 MiB at 19:27. So no growth of ~32 MiB/h after the change. No heap snapshot exists: ops does not exec into the production pod without Oliver.

   What is left. Re-measure over at least 24 hours on the new pod; if it is flat (or only sawtooths below the 512 MiB limit), close the card. If it grows again, tracemalloc or py-spy in the pod needs Oliver's go, and the candidates in the card body (forge budget per token digest, the probe caches, rate limiters, long-poll waiters, the notification derive cache, the session cache) still need a bound each; I did not bound any of them.

2. Maity OPS Team (maity-ops-team) · 2026-10-29

   Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 1 samples, working set min 198 max 198 Mi; last: 2026-10-29T22:39,maity-maity-hub-69bc5b4f79-cgkk6,198,227,0,57. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

3. Maity OPS Team (maity-ops-team) · 2026-10-29

   Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 13 samples, working set min 197 max 198 Mi; last: 2026-10-29T23:40,maity-maity-hub-69bc5b4f79-cgkk6,197,226,0,47. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

4. Maity OPS Team (maity-ops-team) · 2026-10-30

   Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 25 samples, working set min 197 max 198 Mi; last: 2026-10-30T00:40,maity-maity-hub-69bc5b4f79-cgkk6,197,226,0,62. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

5. Maity OPS Team (maity-ops-team) · 2026-10-30

   Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 37 samples, working set min 193 max 198 Mi; last: 2026-10-30T01:40,maity-maity-hub-69bc5b4f79-cgkk6,193,222,0,30. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

6. Maity OPS Team (maity-ops-team) · 2026-10-30

   Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 49 samples, working set min 192 max 198 Mi; last: 2026-10-30T02:41,maity-maity-hub-69bc5b4f79-cgkk6,193,222,0,39. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

7. Maity OPS Team (maity-ops-team) · 2026-10-30

   Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 61 samples, working set min 192 max 198 Mi; last: 2026-10-30T03:41,maity-maity-hub-69bc5b4f79-cgkk6,193,222,0,42. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

8. Maity OPS Team (maity-ops-team) · 2026-10-30

   Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 73 samples, working set min 192 max 198 Mi; last: 2026-10-30T04:41,maity-maity-hub-69bc5b4f79-cgkk6,193,222,0,83. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

9. Maity OPS Team (maity-ops-team) · 2026-10-30

   Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 85 samples, working set min 186 max 198 Mi; last: 2026-10-30T05:42,maity-maity-hub-69bc5b4f79-cgkk6,189,218,0,53. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

10. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 97 samples, working set min 121 max 198 Mi; last: 2026-10-30T06:42,maity-maity-hub-55bf9d4b6-9n7ms,139,168,0,65. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

11. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 109 samples, working set min 121 max 198 Mi; last: 2026-10-30T07:42,maity-maity-hub-55bf9d4b6-9n7ms,154,181,0,113. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

12. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 121 samples, working set min 121 max 198 Mi; last: 2026-10-30T08:42,maity-maity-hub-55bf9d4b6-9n7ms,156,183,0,227. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

13. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 133 samples, working set min 130 max 198 Mi; last: 2026-10-30T09:43,maity-maity-hub-5fdc59b468-xn9bm,145,174,0,182. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

14. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 145 samples, working set min 130 max 198 Mi; last: 2026-10-30T10:43,maity-maity-hub-5fdc59b468-xn9bm,194,223,0,218. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

15. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:39 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:39 was not sampled, earlier series ended 2026-10-28 19:58): 157 samples, working set min 130 max 198 Mi; last: 2026-10-30T11:43,maity-maity-hub-5fdc59b468-xn9bm,194,223,0,263. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

16. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:40 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:40 was not sampled, earlier series ended 2026-10-28 19:58): 164 samples, working set min 130 max 197 Mi; last: 2026-10-30T13:21,maity-maity-hub-5fdc59b468-xn9bm,194,223,0,65. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

17. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:40 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:40 was not sampled, earlier series ended 2026-10-28 19:58): 169 samples, working set min max 197 Mi; last: 2026-10-30T14:13,,,223,,. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

18. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:40 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:40 was not sampled, earlier series ended 2026-10-28 19:58): 181 samples, working set min 198 max 200 Mi; last: 2026-10-30T15:13,maity-maity-hub-5fdc59b468-xn9bm,199,227,0,47. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

19. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:40 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:40 was not sampled, earlier series ended 2026-10-28 19:58): 193 samples, working set min 126 max 200 Mi; last: 2026-10-30T16:14,maity-maity-hub-68f99fc87c-d8m9k,134,168,0,95. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

20. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:40 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:40 was not sampled, earlier series ended 2026-10-28 19:58): 205 samples, working set min 126 max 200 Mi; last: 2026-10-30T17:14,maity-maity-hub-68f99fc87c-d8m9k,155,183,0,111. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

21. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:40 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:40 was not sampled, earlier series ended 2026-10-28 19:58): 217 samples, working set min 126 max 200 Mi; last: 2026-10-30T18:14,maity-maity-hub-68f99fc87c-d8m9k,161,190,0,101. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

22. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:40 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:40 was not sampled, earlier series ended 2026-10-28 19:58): 229 samples, working set min 126 max 200 Mi; last: 2026-10-30T19:14,maity-maity-hub-68f99fc87c-d8m9k,160,188,0,104. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

23. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:40 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:40 was not sampled, earlier series ended 2026-10-28 19:58): 241 samples, working set min 126 max 200 Mi; last: 2026-10-30T20:15,maity-maity-hub-68f99fc87c-d8m9k,166,195,0,69. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

24. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:40 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:40 was not sampled, earlier series ended 2026-10-28 19:58): 253 samples, working set min 126 max 200 Mi; last: 2026-10-30T21:15,maity-maity-hub-68f99fc87c-d8m9k,170,198,0,70. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.

25. Maity OPS Team (maity-ops-team) · 2026-10-30

    Hub memory since 2026-10-29T22:40 UTC (upgrade 39 was 2026-10-29 17:58 UTC; history before 2026-10-29T22:40 was not sampled, earlier series ended 2026-10-28 19:58): 265 samples, working set min 126 max 200 Mi; last: 2026-10-30T22:15,maity-maity-hub-68f99fc87c-d8m9k,170,198,0,143. Columns utc,pod,working_set_Mi,rss_Mi,restarts,cpu_m; full series /workspace/tmp/a/hub-mem.csv (every 5 min). Read-only, no exec into the pod.