Ditto prod pprof — first light

Shared by @futuretrees

Updated Jul 7, 2026

Ditto prod pprof "first light" report (2026-07-04): first-ever profiles captured from the production dittobox fleet via pprofctl over SSH (loopback-only listeners, main port + 3000). Full HTML report with ranked offenders, a 5-minute CPU window, and reproduce commands: https://claude.ai/code/artifact/583d9eb6-15ea-4ce5-9931-d6fafdb6a3b1

Top performance offenders:

  1. JSON serialization on the LLM path — openai-go json.Marshal is the top allocator (76.5 MB cumulative, 19.8% of all allocation since boot; 10 MB of live heap); JSON is ~7.4% of CPU.
  2. Streaming buffer growth — bytes.growSlice is 32.4% (11.3 MB) of backend live heap; sync.Pool on the SSE path is the candidate fix.
  3. Datadog observability overhead — statsd buffers are 25.6% of the share service's heap (5.7 MB), 3.6 MB in backend; tracer/obfuscation/DDSketch visible in CPU.
  4. SDK startup dead weight — aws-sdk-go v1 endpoints.init pins 2.0–4.6 MB and ~18.7k objects in every process (backend, share, worker); migrate to v2 or isolate.
  5. Per-request crypto (found by the 5-minute CPU window) — RSA-2048 (Firebase JWT verify on every request) + P-384 TLS ≈ 5.1% of CPU; candidate: cache JWT verification by token, confirm provider connection reuse.

CPU composition (5m window, 3.11s sampled, 1.04% of one core): waiting for work 23.4%, GC/alloc 8.4%, JSON 7.4%, crypto 5.1%, text/header handling 3.5%. An LLM proxy at rest — mostly parked on the network. The 30s scout's retrieval.linearForward sighting did NOT replicate at 5 minutes (sampling noise).

Vitals: RSS 221 MB, live heap 43 MB, 191.5k heap objects, 96 goroutines, GC 0.008% CPU. Load was light in both windows, so CPU percentages are composition, not hot spots — re-profile at peak. The first capture also flushed out a real bug: zsh on the box globbed '?' in query-string paths; fixed by quoting the remote URL in backend PR #1803.

Files

prod-pprof-first-light.html

text/html