Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
Guides expert-level Go performance optimization: pprof profiling, escape analysis, allocation optimization, GOMAXPROCS tuning, and benchmarking with testing.B.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
476 lines · 4,741 words
Use this skill when the user asks about:
Profiling a Go application with pprof -- CPU profiles, heap profiles, goroutine profiles, mutex profiles, or block profiles
Escape analysis output from go build -gcflags="-m"
Guides expert-level Go performance optimization: pprof profiling, escape analysis, allocation optimization, GOMAXPROCS tuning, and benchmarking with testing.B.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the Go Performance skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
What it covers
When to Use
Process
Output Format
Rules
Edge Cases
Example
Go Performance Analysis: HTTP Request JSON Decoding
Reducing heap allocations in hot paths -- sync.Pool, value semantics, pre-allocated slices, avoiding interface boxing
GOMAXPROCS tuning for specific deployment environments (containers, NUMA hardware, latency-sensitive services)
Writing and interpreting Go benchmarks with testing.B -- b.ReportAllocs(), b.SetBytes(), sub-benchmarks, benchmark profiling
GC pressure reduction -- tuning GOGC, GOMEMLIMIT, understanding GC pacing, reducing live heap size
Memory layout optimization -- struct field ordering, cache line alignment, false sharing between goroutines
Compiler optimizations -- inlining decisions, bounds check elimination, dead code elimination
Do NOT use this skill when:
The user asks about Go idioms, naming, or code style -- use go-idioms
The user asks about goroutines, channels, select, or concurrency patterns -- use go-concurrency-patterns
The user asks about general load testing, k6, wrk, or HTTP benchmarking -- use performance-testing
The user asks about Go module management, versioning, or dependency resolution -- use go-modules
The user asks about debugging logical bugs, nil pointer panics, or runtime errors (not performance-related) -- use go-debugging
The user asks about database query optimization or ORM performance -- use database-optimization
The user asks about network I/O performance independent of Go runtime behavior -- use network-optimization
Before touching a single line of code, capture quantitative ground truth.
Enable pprof endpoints for long-running services by importing _ "net/http/pprof" and exposing an HTTP server on a debug port (e.g., :6060). This registers handlers at /debug/pprof/ automatically.
For batch programs or CLIs, use pprof.StartCPUProfile(f) / pprof.StopCPUProfile() and pprof.WriteHeapProfile(f) around the hot section.
Collect the right profile type for the symptom:
High CPU usage -- CPU profile (/debug/pprof/profile?seconds=30)
High memory or frequent GC -- heap profile (/debug/pprof/heap)
Goroutine leaks or stalls -- goroutine profile (/debug/pprof/goroutine)
Lock contention -- mutex profile (requires runtime.SetMutexProfileFraction(1)) and block profile (requires runtime.SetBlockProfileRate(1))
Scheduler latency -- trace (/debug/pprof/trace?seconds=5) analyzed with go tool trace
Use go tool pprof to analyze profiles interactively: go tool pprof -http=:8080 cpu.prof opens a browser UI with flame graphs, top functions, and source annotations.
Record wall-clock baselines before any change: p50, p95, p99 latency, requests-per-second, and runtime.ReadMemStats values (HeapAlloc, NumGC, PauseTotalNs).
Do not guess. Let the profiling data drive classification.
CPU-bound if a function appears at the top of the pprof top output consuming >20% of CPU, especially in tight loops or string/slice manipulation. Look for runtime.mallocgc in the CPU profile -- its presence indicates allocation cost is masquerading as CPU cost.
Allocation-bound (GC pressure) if runtime.mallocgc and runtime.gcBgMarkWorker consume significant CPU shares, or if NumGC in MemStats is >1 GC per second under load, or if heap profile shows a single function responsible for >30% of allocations.
Memory-bound if working set exceeds L3 cache (typically 8--32 MB on server hardware) and the CPU profile shows high time in cache-miss-prone code. Use perf stat or GODEBUG=gccheckmark=1 as supplemental signals.
Goroutine contention if the mutex or block profile shows a lock held for >100 microseconds at high frequency. Look for sync.Mutex, sync.RWMutex, and channel send/receive as blocking sites.
Scheduler latency if go tool trace shows long goroutine runqueue wait times (>1ms) or stop-the-world GC pauses visible in the trace view.
I/O-bound if pprof shows time dominated by syscall.Read, syscall.Write, or network poll functions. In this case, the optimization path is buffering, batching, or connection pooling -- not Go-level tuning.
Allocation reduction is the highest-leverage optimization in most Go programs because every allocation eventually costs GC time.
Run escape analysis: go build -gcflags="-m=2" ./.... Variables that "escape to heap" are the candidates. Variables that do NOT escape are stack-allocated and free.
Understand the main escape triggers:
Returning a pointer to a local variable
Storing a value in an interface (interface boxing always allocates if the concrete type is >1 pointer word and not a scalar)
Appending to a slice whose capacity is unknown at compile time
Closures capturing outer variables
Values passed to fmt.Sprintf, fmt.Println (interface varargs)
Replace fmt.Sprintf in hot paths with strconv.AppendInt, strconv.AppendFloat, or manual byte-slice building to avoid interface{} boxing and the format string parser overhead.
Pre-allocate slices when the final size is known: make([]T, 0, expectedLen) avoids repeated doubling reallocations. Each reallocation copies the entire backing array.
Use sync.Pool for objects that are frequently allocated and discarded (e.g., bytes.Buffer, large temporary slices, decoder structs). Pool objects are reclaimed between GC cycles, not within a request. Always reset pooled objects before returning them to the pool. Never store pointers to Pool-sourced objects across GC cycles.
Prefer value receivers and value semantics for small structs (<=3 pointer-sized fields). Passing a small struct by value keeps it on the stack; passing a pointer may cause it to escape.
Use []byte instead of string concatenation in loops. String concatenation with + allocates a new string each iteration. Use strings.Builder or bytes.Buffer instead, then call .String() once at the end.
Once allocations are under control, optimize actual computation.
Verify inlining decisions with go build -gcflags="-m". Functions are NOT inlined if they contain: loops, closures, function literals assigned to variables, calls to recover(), or if their AST node budget exceeds ~80 nodes (the exact threshold is runtime-version dependent). Restructure hot functions to fall under the inlining budget if the inlining gain is worth the readability cost.
Eliminate bounds checks in tight loops by assigning the slice length to a local variable before the loop: n := len(s); for i := 0; i < n; i++ { ... }. The compiler can then prove the index is in bounds and elide the check. Verify with go build -gcflags="-d=ssa/check_bce/debug=1".
Use unsafe.Slice and unsafe.String (Go 1.17+) for zero-copy conversion between []byte and string in hot serialization paths -- but only when the lifetime of the result does not outlive the source, and document with a safety comment.
Profile branch prediction impact with go tool trace or perf. Avoid highly unpredictable branches inside tight loops -- sorting input data or restructuring conditionals can improve branch prediction rates.
Avoid interface dispatch in hot loops. Calling a method through an interface involves two pointer dereferences (itab lookup + method call). In a loop executing 10^8 times, this is measurable. Use concrete types or generics (Go 1.18+) to eliminate dynamic dispatch.
Consider SIMD via assembly only after all other optimizations are exhausted, and only if the function is genuinely compute-intensive (e.g., hashing, encoding, matrix operations). Use golang.org/x/sys/cpu to detect CPU features and provide a pure-Go fallback.
The Go GC is a concurrent, tricolor mark-and-sweep collector. Its behavior is tunable.
GOGC (default: 100): Sets the GC target as a percentage. GOGC=100 means GC triggers when heap size doubles from the last collection. Increase GOGC (e.g., GOGC=200) to reduce GC frequency at the cost of higher peak memory. Decrease it (e.g., GOGC=50) to reduce peak memory at the cost of more frequent GC. For latency-sensitive services, try GOGC=off combined with GOMEMLIMIT (Go 1.19+).
GOMEMLIMIT (Go 1.19+): Sets a soft memory limit for the Go runtime (e.g., GOMEMLIMIT=512MiB). This prevents OOM kills in containerized environments by aggressively triggering GC before the limit is hit. Set this to ~90% of the container memory limit. Combine with GOGC=off to eliminate periodic GC and rely solely on memory pressure to trigger collection -- this is effective for throughput-focused services with predictable memory usage.
Reduce live heap size: GC pause time and frequency scale with live heap. Eliminate long-lived caches, reduce buffering, and expire data aggressively.
Use runtime.GC() strategically after one-time startup operations that generate temporary allocations (loading config, parsing templates) to return heap to steady state before serving traffic.
Monitor GC metrics: Use runtime.ReadMemStats or expvar to track NumGC, PauseTotalNs, GCCPUFraction, and HeapInuse. A GCCPUFraction above 0.05 (5%) indicates GC overhead is significant.
GOMAXPROCS controls how many OS threads run Go code simultaneously.
Default behavior: Since Go 1.5, GOMAXPROCS defaults to the number of logical CPUs reported by the OS. This is correct for bare-metal deployments.
Container environments: Containers often have CPU limits set via cgroups (e.g., --cpus=2.0), but the Go runtime reads /proc/cpuinfo or sysconf(_SC_NPROCESSORS_ONLN), which reports the host's total CPU count. This causes over-scheduling. Use uber-go/automaxprocs or runtime.GOMAXPROCS(runtime.NumCPU()) combined with reading cgroup CPU quota from /sys/fs/cgroup/cpu/cpu.cfs_quota_us and /sys/fs/cgroup/cpu/cpu.cfs_period_us. The automaxprocs package handles this automatically.
Latency-sensitive services: Setting GOMAXPROCS to less than the available CPUs can reduce scheduler thrashing and improve p99 latency by reducing context switching. Benchmark both directions.
CPU-pinning and NUMA: For NUMA-sensitive workloads (e.g., HPC, real-time processing), use numactl --cpunodebind=0 to restrict the process to a single NUMA node, and set GOMAXPROCS to the number of cores on that node.
GODEBUG=schedtrace=1000: Prints scheduler state every 1000ms. Use it to detect goroutine runqueue buildup, which indicates either too few threads (increase GOMAXPROCS) or lock contention (reduce contention).
Benchmarks are only useful if they measure the right thing with statistical rigor.
Basic benchmark structure: Functions must be func BenchmarkFoo(b *testing.B) in _test.go files. The loop body runs b.N times, where the testing framework auto-scales b.N until the total time exceeds 1 second.
Always call b.ReportAllocs() to surface allocations-per-operation and bytes-per-operation. This is equivalent to using -benchmem but applies per benchmark regardless of flag.
Use b.SetBytes(n) when benchmarking throughput (encoding, hashing, parsing): it causes the output to report MB/s, making comparison across implementations intuitive.
Prevent dead-code elimination: Assign results to a package-level var sink interface{} or use runtime.KeepAlive(result) to prevent the compiler from optimizing away the entire computation.
Use b.ResetTimer() after expensive setup code (reading test fixtures, building large data structures) so setup time does not inflate per-operation measurements.
Use sub-benchmarks (b.Run("size=1000", func(b *testing.B) {...})) to test across input sizes and generate scaling data. This reveals whether an optimization is O(1), O(n), or has a hidden quadratic component.
Use benchstat (from golang.org/x/perf/cmd/benchstat) to compare two sets of benchmark runs statistically. Run each benchmark at least 10 times (go test -bench=. -count=10) and pass both output files to benchstat old.txt new.txt. It reports p-values and confidence intervals to distinguish real improvements from noise.
Profile benchmarks directly: go test -bench=BenchmarkFoo -cpuprofile=cpu.prof generates a pprof profile for the benchmark run. This combines measurement and profiling in one step.
Optimization without regression prevention is temporary.
Run the full test suite (go test ./... -race) after every optimization. The race detector catches data races introduced by concurrency changes made during optimization.
Add the benchmark to CI using benchstat comparisons against a stored baseline. Alert on regressions >5% in throughput or >10% in allocation count.
Document the optimization with a comment that includes: the problem observed, the profiling evidence (e.g., "heap profile showed 40% of allocations from parseToken"), the technique applied, and the before/after metric.
Commit benchmark outputs alongside code changes. Store benchstat baseline files in testdata/benchmarks/ so future reviewers can reproduce comparisons.
Set GOGC and GOMEMLIMIT values in deployment configuration with comments explaining the values, not buried in code. Changes to these values should go through the same review process as code changes.
When analyzing a Go performance problem, structure the response as:
## Go Performance Analysis: [component or function name]
### Symptom Summary
- Observed behavior: [e.g., "p99 latency spikes to 200ms under load"]
- Environment: Go [version], [OS/arch], [deployment context]
- Workload: [RPS, concurrent goroutines, data sizes]
### Profiling Command Used
[Exact commands run to collect the profile]
### Profile Findings
| Rank | Function | Flat% | Cum% | Profile Type |
|------|----------|-------|------|--------------|
| 1 | [func] | [n%] | [n%] | CPU / Heap |
| 2 | [func] | [n%] | [n%] | CPU / Heap |
| 3 | [func] | [n%] | [n%] | CPU / Heap |
### Bottleneck Classification
- Primary type: [CPU-bound | Allocation-bound | GC-pressure | Goroutine contention | I/O-bound]
- Root cause: [Specific diagnosis, e.g., "interface boxing of []byte in fmt.Sprintf call in hot path"]
- Evidence: [pprof line reference, escape analysis output, or MemStats value]
### Optimization Plan
| Step | Technique | Expected Gain | Risk |
|------|-----------|---------------|------|
| 1 | [technique] | [e.g., 30% alloc reduction] | [Low/Med/High] |
| 2 | [technique] | [e.g., 2x throughput] | [Low/Med/High] |
### Code Change
[Before code snippet]
[After code snippet]
### Benchmark ResultsBenchmarkFoo/before 1000000 1245 ns/op 512 B/op 8 allocs/op BenchmarkFoo/after 1000000 312 ns/op 64 B/op 1 allocs/op
benchstat: -75% ns/op (p=0.001), -87.5% B/op
### Memory Stats Comparison
| Metric | Before | After | Change |
|----------------|-----------|-----------|----------|
| HeapAlloc | 450 MB | 120 MB | -73% |
| NumGC (per min)| 48 | 12 | -75% |
| GCCPUFraction | 0.12 | 0.03 | -75% |
| PauseTotalNs | 180ms | 45ms | -75% |
### Configuration Changes
- GOGC: [old value] → [new value] (rationale: ...)
- GOMEMLIMIT: [old value] → [new value] (rationale: ...)
- GOMAXPROCS: [old value] → [new value] (rationale: ...)
### Regression Prevention
- Benchmark added: [yes/no, benchmark name]
- Test suite: PASS (go test ./... -race)
- Baseline stored: [testdata/benchmarks/foo_baseline.txt]Never optimize without a profile. "This looks slow" is not evidence. The Go compiler and runtime make surprising optimization decisions. Code that looks like it allocates often does not; code that looks cheap often is not. Only pprof output constitutes valid evidence for optimization.
Never interpret escape analysis output without the -m=2 flag. The single -m flag shows decisions but not reasons. -m=2 shows the full reasoning chain (e.g., "parameter leaks to result" vs. "assigned to interface"). Understanding the reason is required to fix it correctly.
Never use sync.Pool for objects with pointer-containing fields that reference external resources (file handles, network connections, database cursors). Pool objects are released between GC cycles without finalizers running. Use sync.Pool only for pure data buffers.
Never benchmark with b.N = 1 loops that contain only a single operation without a sink variable. The compiler's dead-code elimination and escape analysis will optimize away the computation, making the benchmark measure zero real work. Always store results in a package-level var Sink variable.
Never set GOMAXPROCS below 1 or above the available logical CPU count without documented justification. Setting it above the logical CPU count does not improve throughput and increases scheduler overhead. Setting it to 1 eliminates parallelism and is only correct for single-core targets or debugging.
Never recommend GOGC=off without simultaneously setting GOMEMLIMIT. Disabling GC without a memory limit causes unbounded heap growth and OOM kills. The two settings are a pair: GOGC=off + GOMEMLIMIT=<90% of container limit>.
Never use unsafe for performance without measuring the actual gain. unsafe operations (pointer arithmetic, unsafe.Slice, unsafe.String) eliminate safety guarantees. They are only justified when profiling shows the safe version is the measured bottleneck AND the unsafe version shows a statistically significant improvement in benchstat output.
Never micro-optimize struct field order without verifying cache line behavior. Reordering struct fields to minimize padding (use fieldalignment from golang.org/x/tools/go/analysis/passes/fieldalignment) can reduce struct size, but if the struct is accessed concurrently from multiple goroutines, placing hot fields in the same cache line causes false sharing. Measure both memory usage and mutex/atomic contention together.
Always run benchmarks with -count=10 minimum before using benchstat. A single benchmark run is not statistically meaningful -- Go benchmarks have variance from GC timing, OS scheduling, and CPU frequency scaling. Fewer than 5 runs makes benchstat p-values unreliable.
Never conflate heap profile "inuse_space" with "alloc_space". inuse_space shows current live allocations -- what is consuming memory right now. alloc_space shows cumulative allocations since program start -- where allocation pressure is coming from. High alloc_space with low inuse_space indicates objects are being allocated and freed rapidly (GC pressure). High inuse_space indicates a memory leak or large long-lived cache. Use the correct view for the problem being diagnosed.
In Kubernetes or Docker, a pod with resources.limits.cpu: "2" has a cgroup CPU quota, but the Go runtime by default reads the host's logical CPU count (e.g., 64 cores on a large node) and sets GOMAXPROCS=64. This causes 64 OS threads competing for 2 CPU shares, resulting in severe scheduler thrashing, elevated p99 latency, and CPU throttling visible in container metrics as container_cpu_cfs_throttled_seconds_total. Fix: add uber-go/automaxprocs as the first import in main.go (import _ "go.uber.org/automaxprocs"). It reads cgroup CPU quota and sets GOMAXPROCS correctly at startup. Verify with GOMAXPROCS logged at startup.
When a service receives a burst of requests, it allocates a large amount of memory quickly. The GC triggers when heap doubles, causing a concurrent mark phase that competes with the request-handling goroutines for CPU. The symptom is p99/p999 latency spikes that correlate with NumGC spikes in metrics. The fix is not to tune GOGC lower (that makes it worse). Instead: (1) reduce allocation volume in the hot path using the techniques in Step 3; (2) set GOMEMLIMIT to trigger GC based on absolute memory pressure rather than heap doubling; (3) pre-warm the pool before serving traffic by calling runtime.GC() after startup initialization to return the heap to steady state.
sync.Pool Causing Latency Jittersync.Pool objects are cleared at every GC cycle. In services with frequent GC, this means pooled objects are frequently evicted, and pool Get() calls frequently allocate new objects -- eliminating the pool's benefit. Diagnosis: add a counter to the pool New function and monitor it. If New is called at the same rate as Get, the pool is not helping. Fix: reduce GC frequency first (via allocation reduction or GOGC tuning). Alternatively, for objects that survive across GC cycles, consider a hand-rolled free list protected by a mutex, or a channel-based pool with a fixed capacity.
A common pattern that causes hidden allocations: a function accepts interface{} or any parameters (e.g., a logging function, a metrics recorder, a middleware handler). Every call to that function boxes the concrete argument onto the heap. Diagnosis: escape analysis shows "argument escapes to heap" at the call site, and the heap profile shows the logging/metrics call as a top allocator. Fixes in order of preference: (1) use typed parameters where the interface is not necessary; (2) use generics (Go 1.18+) to specialize the function for common types; (3) use zap-style structured logging with typed fields (zap.Int("count", n)) instead of fmt.Sprintf-style; (4) add a fast path that skips the expensive operation when a condition is false (e.g., if !logger.Enabled(level) { return }).
When two goroutines on different cores concurrently access different fields of the same struct, and those fields land in the same 64-byte cache line, every write by one goroutine invalidates the cache line for the other. This causes cache coherence traffic that does not show up as lock contention in pprof -- it appears as mysterious CPU overhead in atomic operations. Diagnosis: use perf c2c on Linux to detect cache-to-cache transfers. Fix: pad hot fields to 64-byte boundaries using a [64]byte padding field, or separate per-goroutine state into independent cache-line-aligned structs. Example: type padded struct { value int64; _ [56]byte } ensures value occupies its own cache line.
JSON encoding and HTTP header handling frequently convert between string and []byte. Each conversion in the safe model allocates a new backing array. In a high-throughput HTTP service, this can account for 20--40% of all allocations. Diagnosis: heap profile shows []byte allocations in serialization code. Fixes: (1) use json.Marshal into a pre-allocated bytes.Buffer obtained from sync.Pool; (2) use a faster JSON library that avoids intermediate allocations (e.g., one that uses code generation to serialize directly to a writer); (3) where correctness can be guaranteed (the []byte is not mutated and does not outlive the string), use the unsafe conversion pattern: s := *(*string)(unsafe.Pointer(&b)) -- but document the safety invariant explicitly in a comment.
A benchmark shows 5x improvement, but production p99 latency improves by only 10%. Common causes: (1) the benchmark is CPU-isolated and does not model GC interruptions -- add runtime.GC() calls within the benchmark loop to simulate GC pressure; (2) the benchmark uses small, cache-hot data while production uses large, cache-cold data -- increase input size to exceed L3 cache; (3) the optimized path is not the actual bottleneck in the production request -- re-profile production under real load to confirm the optimized function is still in the top 5 of the CPU profile; (4) the optimization introduced lock contention under concurrent load -- benchmark with b.RunParallel to expose concurrency behavior.
Input: "Our Go HTTP service handles JSON decoding of incoming requests. Under load, we see p99 latency of 180ms (target: 20ms) and heap usage of 800MB on a pod with a 1GB limit. The service runs on Kubernetes with cpu: 2, but the node has 32 cores. How do I diagnose and fix this?"
Output:
Observed behavior: p99 latency 180ms (9x above 20ms target); heap 800MB on 1GB pod limit
Environment: Go 1.21, linux/amd64, Kubernetes pod (2 CPU limit, 32-core node)
Workload: ~5,000 RPS, JSON decode on every request, average payload 4KB
# Step 1: Enable pprof in main.go (already present via net/http/pprof import)
# Step 2: Collect profiles under load
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile?seconds=30
go tool pprof -http=:8081 http://localhost:6060/debug/pprof/heap
# Step 3: Collect scheduler trace to inspect GC pause and GOMAXPROCS impact
curl http://localhost:6060/debug/pprof/trace?seconds=5 -o trace.out
go tool trace trace.out
# Step 4: Check MemStats via expvar endpoint
curl http://localhost:6060/debug/vars | jq '.memstats'CPU Profile (30-second sample):
| Rank | Function | Flat% | Cum% | Profile Type |
|---|---|---|---|---|
| 1 | runtime.gcBgMarkWorker | 28% | 28% | CPU |
| 2 | encoding/json.(*decodeState).object | 18% | 31% | CPU |
| 3 | runtime.mallocgc | 14% | 42% | CPU |
| 4 | encoding/json.indirect | 9% | 51% | CPU |
| 5 | runtime.memmove | 7% | 58% | CPU |
Heap Profile (alloc_space -- cumulative allocations):
| Rank | Function | alloc_space% | Profile Type |
|---|---|---|---|
| 1 | encoding/json.(*decodeState).object | 44% | Heap |
| 2 | bytes.(*Buffer).WriteString | 12% | Heap |
| 3 | fmt.Sprintf (in request logger) | 11% | Heap |
| 4 | http.Header.Set (string conversion) | 8% | Heap |
MemStats at time of capture:
HeapAlloc: 780 MB
NumGC: 61 per minute (>1 per second)
GCCPUFraction: 0.28 ← 28% of CPU is GC -- critical
PauseTotalNs: 420ms accumulated over 30s
GOMAXPROCS: 32 ← reading host CPUs, not cgroup limitPrimary type: Allocation-bound (GC pressure)
Secondary type: GOMAXPROCS misconfiguration causing scheduler thrashing
Root cause: encoding/json decoder allocates a new map[string]interface{} and a new string for every JSON key and value. With 5,000 RPS at 4KB average payload, this produces ~500 MB/s of allocation throughput. GC cannot keep up, using 28% of CPU. Simultaneously, GOMAXPROCS=32 causes 32 OS threads competing for 2 CPU shares, inflating all latencies.
Evidence: gcBgMarkWorker at 28% CPU + NumGC=61/min + GCCPUFraction=0.28 confirm GC is the primary cost. GOMAXPROCS=32 on a 2-CPU cgroup pod is confirmed by runtime.GOMAXPROCS(0) returning 32 at startup.
| Step | Technique | Expected Gain | Risk |
|---|---|---|---|
| 1 | Fix GOMAXPROCS via automaxprocs | 20--40% latency reduction | Low |
| 2 | Set GOMEMLIMIT=900MiB + GOGC=off | Eliminate periodic GC, prevent OOM | Low |
| 3 | Decode JSON into typed structs, not map[string]interface{} | 60--80% alloc reduction in decoder | Low |
| 4 | Pool json.Decoder via sync.Pool | 10--20% alloc reduction | Low |
| 5 | Replace fmt.Sprintf in logger hot path | 11% heap reduction | Low |
Fix 1: GOMAXPROCS (main.go)
// Before: GOMAXPROCS defaults to 32 (host CPUs), ignoring cgroup limit
import (
"net/http"
_ "net/http/pprof"
)
// After: automaxprocs reads cgroup quota and sets GOMAXPROCS=2
import (
"net/http"
_ "net/http/pprof"
_ "go.uber.org/automaxprocs" // sets GOMAXPROCS=2 from cgroup cpu quota
)Fix 2: GOMEMLIMIT + GOGC (deployment config or main.go init)
// In main() before serving traffic:
import (
"runtime/debug"
)
func main() {
// 900 MiB = ~90% of 1 GiB pod limit
// GOGC=off: rely on GOMEMLIMIT pressure, not heap doubling
debug.SetMemoryLimit(900 * 1024 * 1024)
debug.SetGCPercent(-1) // equivalent to GOGC=off
// Force a GC after initialization to return startup allocations
runtime.GC()
startServer()
}Fix 3: Typed struct decoding instead of map[string]interface{}
// Before: decodes into generic map, allocates a string for every key and value
func handleRequest(r *http.Request) error {
var payload map[string]interface{}
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
return err
}
// access payload["user_id"], payload["action"], etc.
}
// After: decodes into typed struct -- compiler-known layout, no map allocations
type RequestPayload struct {
UserID int64 `json:"user_id"`
Action string `json:"action"`
Timestamp int64 `json:"timestamp"`
// ... other fields
}
var decoderPool = sync.Pool{
New: func() interface{} {
return json.NewDecoder(nil)
},
}
func handleRequest(r *http.Request) error {
var payload RequestPayload
dec := decoderPool.Get().(*json.Decoder)
dec.Reset(r.Body)
err := dec.Decode(&payload)
decoderPool.Put(dec)
if err != nil {
return err
}
// access payload.UserID, payload.Action directly
}Fix 5: Replace fmt.Sprintf in logger hot path
// Before: fmt.Sprintf boxes all arguments into []interface{}, allocates result string
func logRequest(method, path string, status int, duration time.Duration) {
msg := fmt.Sprintf("method=%s path=%s status=%d duration=%s",
method, path, status, duration)
logger.Info(msg)
}
// After: use structured logging with typed fields (zero allocations for scalar types)
func logRequest(method, path string, status int, duration time.Duration) {
logger.Info("request",
zap.String("method", method),
zap.String("path", path),
zap.Int("status", status),
zap.Duration("duration", duration),
)
}# Run with: go test -bench=BenchmarkDecode -benchmem -count=10 > after.txt
# Compare: benchstat before.txt after.txt
BenchmarkDecode/map_interface 1200 982450 ns/op 18432 B/op 312 allocs/op
BenchmarkDecode/typed_struct 8800 136100 ns/op 2048 B/op 14 allocs/op
BenchmarkDecode/typed_pool 12000 98300 ns/op 512 B/op 3 allocs/op
benchstat: typed_struct vs map_interface:
ns/op: -86.2% (p=0.000)
B/op: -88.9% (p=0.000)
allocs/op: -95.5% (p=0.000)| Metric | Before | After | Change |
|---|---|---|---|
| HeapAlloc | 780 MB | 95 MB | -88% |
| NumGC (per min) | 61 | 0* | -100% |
| GCCPUFraction | 0.28 | 0.02* | -93% |
| PauseTotalNs | 420ms/30s | 12ms/30s | -97% |
| GOMAXPROCS | 32 | 2 | fixed |
| p99 latency | 180ms | 14ms | -92% |
*GC now triggered only by GOMEMLIMIT pressure, not periodically. Occasional GC runs when heap approaches 900MiB limit; this occurs ~2 times per minute at peak load.
GOMAXPROCS: 32 → 2 via automaxprocs (matches cgroup cpu quota of 2.0)
GOMEMLIMIT: unset → 900 MiB (90% of 1GiB pod limit, prevents OOM kill)
GOGC: 100 → -1 (disabled, memory pressure from GOMEMLIMIT drives GC instead)
Benchmarks added: BenchmarkDecode/typed_struct, BenchmarkDecode/typed_pool
Baseline stored: testdata/benchmarks/decode_baseline.txt
Test suite: go test ./... -race -- PASS (0 races detected)
CI check: benchstat alert configured for >10% regression in ns/op or allocs/op
Startup log: GOMAXPROCS=2 (from cgroup quota 2.0) confirmed in pod logs
SKILL.md
476 lines · 4,741 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
Go Performance bot
Waiting in your sidebar once you say go.
No account needed · Runs on the AI plan you already pay for