Why Go's encoding/csv Burns 540MB on 5M Rows (and How I Fixed It)

TL;DR: Go's built-in encoding/csv.Read() allocates string slices and strings for every single row, generating 10M+ allocations and burning 540 MB on a 5M-row file. I built go-zerocsv to replace this with in-place typed scanning (Record.Scan) and a compacting 4 KB buffer, keeping memory flat at ~5 KB with zero allocations per record on the hot path. If you parse or write large CSV files in Go, you've probably run into the standard library's memory appetite. A few days ago, I was profiling a batch pipeline processing a 5-million-row CSV export. The Go process was consuming hundreds of megabytes of RAM, and the garbage collector was eating a noticeable chunk of CPU time. When you fire up go tool pprof on a heavy encoding/csv run, you immediately see the culprits dominating the allocation graph: runtime.makeslice and runtime.rawstring. The standard library's API is fundamentally designed around allocating new heap objects for every single field and row. To solve this, I wrote go-zerocsv — a zero-dependency CSV parser and writer that runs with 0 heap allocations per record on the hot path and keeps memory usage at a constant ~5 KB, whether you're streaming 1,000 rows or 10,000,000 rows. Here is a breakdown of why encoding/csv allocates so much, how go-zerocsv eliminates those allocations, and the engineering decisions behind it. Why encoding/csv Allocates So Much Look at the reader's method signature in encoding/csv: func (r *Reader) Read() (record []string, err error) Every time Read() is called, two things happen: It allocates a new []string slice for the row (runtime.makeslice). It copies each parsed byte slice into a newly allocated heap string (runtime.rawstring). If you read a 5,000,000-row CSV file with 6 columns: 10,000,000+ allocations occur just from slice headers and string conversions. ~540 MB of cumulative heap allocations are pushed onto the garbage collector. Memory Model Comparison encoding/csv (Millions of individual heap objects): Each row allocates a slice, and each cell allocates a heap string: Row 1 -> []string{"1", "Alice", "98.5", "true"} (4 heap allocations) Row 2 -> []string{"2", "Bob", "85.2", "false"} (4 heap allocations) Result: Over 10,000,000 pointer objects created across the run. go-zerocsv (One reusable 4 KB buffer): A single internal buffer holds raw bytes and is compacted between records: [ 1,Alice,98.5,true\n ] -> In-place Scan(&id, &name, &score, &active) Result: The buffer is reused for the next row. Total heap usage stays flat at ~5 KB. The writer has a similar problem. Because Writer.Write([]string) only accepts strings, writing typed values (int, float64, time.Time, bool) forces you to call strconv.Itoa, strconv.FormatFloat, or fmt.Sprintf first — each creating temporary throwaway strings for every cell in every row. The Benchmark Numbers Here is a direct comparison running on an AMD Ryzen 5 8400F (linux/amd64, Go 1.26): Reading a 5,000,000-Row CSV (6 columns per row) Package Time Throughput Memory (B/op) Allocations encoding/csv (stdlib) 492 ms 292 MB/s 540 MB 10,000,014 go-zerocsv 302 ms 474 MB/s 5.0 KB 12 go-zerocsv parses the entire 5M-row file in ~300ms while holding just 5 KB in RAM. Writing a 5,000,000-Row CSV (mixed types + RFC3339 timestamp) Package Time Memory (B/op) Allocations encoding/csv (stdlib) 1,119 ms 272 MB 19,999,917 go-zerocsv 708 ms 4.3 KB 6 Before vs. After: Side-by-Side Reading & Parsing Rows // ❌ BEFORE (encoding/csv): Allocates strings on Read(), then allocates during parsing for { record, err := r.Read() // Allocates []string + strings if err == io.EOF { break } id, _ := strconv.Atoi(record[0]) score, _ := strconv.ParseFloat(record[2], 64) active, _ := strconv.ParseBool(record[3]) process(id, record[1], score, active) } // ✅ AFTER (go-zerocsv): 0 heap allocations on the hot path for { rec, err := r.Read() if err == io.EOF { break } if rec.IsFirst() { continue } // Header helper var id int var name string var score float64 var active bool // Scans in-place directly from reader buffer if err := rec.Scan(&id, &name, &score, &active); err != nil { log.Fatal(err) } process(id, name, score, active) } Full Code Examples 1. High-Performance Reading with Record.Scan Read() returns a lightweight Record struct. You can scan fields directly into variables in-place, mirroring the familiar database/sql pattern: package main import ( "fmt" "io" "strings" zerocsv "github.com/fikrimohammad/go-zerocsv" ) func main() { data := `id,name,score,active 1,Alice,98.5,true 2,Bob,85.2,false ` r := zerocsv.NewReader(strings.NewReader(data)) for { rec, err := r.Read() if err == io.EOF { break } if err != nil { panic(err) } if rec.IsFirst() { continue // skip header } var ( id int name string score float64 active bool ) // Parses numbers, booleans, and strings in-place (0 allocs) if err := rec.Scan(&id, &name, &score, &active); err != nil { panic(err) } fmt.Printf("id=%d name=%s score=%.1f active=%t\n", id, name, score, active) } } If you don't need Scan, you can access fields with rec.String(i) or copy field bytes into a caller-owned slice using rec.Bytes(i, dst). 2. Zero-Allocation Writing zerocsv.Writer writes tagged value-type Column structs instead of []string. Reusing a single slice across loop iterations makes the write loop allocation-free: package main import ( "os" "time" zerocsv "github.com/fikrimohammad/go-zerocsv" ) func main() { w := zerocsv.NewWriter(os.Stdout) defer w.Flush() // Reuse this slice across all rows row := make([]zerocsv.Column, 5) for i := 0; i < 100_000; i++ { row[0] = zerocsv.ColumnInt(i) row[1] = zerocsv.ColumnString("Alice") row[2] = zerocsv.ColumnFloat64(98.5) row[3] = zerocsv.ColumnBool(true) row[4] = zerocsv.ColumnString(time.Now().Format(time.RFC3339)) if err := w.Write(row...); err != nil { panic(err) } } } 3. Custom Types without Reflection If you have custom domain types (like dates or status enums), you don't need reflection or any interface boxing. Implement FieldScanner (for reading) and FieldValuer (for writing): type Date time.Time // ScanCSV parses raw bytes into Date with 0 allocations func (d *Date) ScanCSV(field []byte) error { t, err := time.Parse("2006-01-02", string(field)) if err != nil { return err } *d = Date(t) return nil } // AppendCSV formats Date directly into the writer's internal buffer func (d Date) AppendCSV(dst []byte) ([]byte, error) { return time.Time(d).AppendFormat(dst, "2006-01-02"), nil } Usage: // Reading: var birthday Date err := rec.Scan(&id, &name, &birthday) // Writing: err := w.Write( zerocsv.ColumnString("Alice"), zerocsv.ColumnValuer(birthday), ) How It Works Internally 1. Reusable Compacting Buffer Instead of reading the entire file into memory or slicing new buffers constantly, the reader keeps a single reusable buffer (default 4 KB). As records are parsed, processed bytes are shifted and new chunks are read in. If a single CSV line exceeds 4 KB, the buffer grows to accommodate it, and trims back to 4 KB once that line is done. 2. In-Place Quote Decoding When a CSV field contains escaped quotes (e.g. "He said ""hello"""), standard parsers allocate a clean string. go-zerocsv decodes escaped quotes in-place inside the raw byte buffer and adjusts slice bounds, avoiding intermediate strings. 3. 48-Byte Column Value Struct The Column struct is designed to fit in 48 bytes on 64-bit architecture: type Column struct { valuer FieldValuer // 16 bytes (interface for custom types) s string // 16 bytes (string or zero-copy byte view) n uint64 // 8 bytes (ints, uints, floats, bools) kind ColumnKind // 1 byte (+ 7 bytes padding) } Numbers, booleans, and floats are stored directly in n using bit manipulation (e.g. math.Float64bits), so creating a ColumnInt or ColumnFloat64 never allocates on the heap. 4. Zero Allocations for Formatted Writing The writer maintains a small reusable 32-byte scratch buffer. When ColumnInt(12345) is passed, it uses strconv.AppendInt(scratch[:0], 12345, 10) and streams the bytes straight into the bufio.Writer — no intermediate string created. Strict Conformance with encoding/csv A fast parser is useless if it misparses edge cases. We fuzzed go-zerocsv against Go's encoding/csv over 3,000,000+ iterations using Go native fuzzing: Handles standard RFC 4180 rules, CRLF vs LF, multi-line quoted fields, and lazy quotes (WithLazyQuotes). Matches encoding/csv's non-fatal ErrFieldCount behavior. Validates delimiters to 7-bit ASCII (WithDelimiter) to prevent multi-byte UTF-8 stream corruption. Includes WithMaxBuffer(n) to guard against denial-of-service from malformed single-line files. When Is Standard encoding/csv Still Fine? To be fair: if you are writing a quick one-off CLI script, parsing a small 50-row country code lookup table, or loading a mock fixture in a unit test, encoding/csv is completely fine. The few dozen allocations will complete in a fraction of a millisecond and won't noticeably affect your application. go-zerocsv is built for scenarios where throughput and memory predictability matter: High-volume ETL and batch ingestion pipelines Streaming database dumps or generating multi-gigabyte CSV exports Microservices processing concurrent user CSV uploads under strict memory limits (e.g., small Kubernetes pods) Real-time event logging where GC pause spikes directly degrade p99 latencies Summary If your Go service processes CSVs at scale, go-zerocsv offers an easy drop-in performance win: ~60% faster reads (~475 MB/s). Flat 5 KB memory regardless of file size. Zero allocations on the hot path. No external dependencies (standard library only, Go 1.18+). The code and benchmarks are available on GitHub: Repo: github.com/fikrimohammad/go-zerocsv Docs: pkg.go.dev/github.com/fikrimohammad/go-zerocsv Feedback, issues, and PRs are welcome!