Memlogger Architecture

The memlogger is the core component that enables efficient, production-ready debug logging without performance penalties.

Overview

Memlogger implements a production-ready logger that:

Buffers logs in memory
Compresses them asynchronously with gzip
Persists to a Write-Ahead Log (WAL) with sidecar index for efficient replay

Design Principles

1. Non-Blocking Operation

The memlogger never blocks the consensus or state machine:

State Change → Buffer (instant) → Async Compression → Disk Write
     ↓
  Continue

State changes are buffered in memory instantly
Compression happens in background goroutines
Disk writes are asynchronous
Consensus continues unaffected

2. Zero-Allocation Hot Path

Uses object pooling to minimize garbage collection:

// Compression buffer pool
var bufferPool = sync.Pool{
    New: func() interface{} {
        return new(bytes.Buffer)
    },
}
 
// Reuse buffers across compression operations
buf := bufferPool.Get().(*bytes.Buffer)
defer bufferPool.Put(buf)

Benefits:

Reduced GC pressure
Consistent memory usage
Predictable performance

3. Configurable Flushing

Two triggers for flushing buffer to disk:

Time-Based:

interval = "2s"  # Flush every 2 seconds

Size-Based:

memory-bytes = 50000000  # Flush at 50MB

Or both simultaneously (whichever triggers first).

Components

Buffer Manager

Responsibility: Manage in-memory log buffer

Operations:

Append log messages to buffer
Track buffer size
Trigger compression when threshold reached
Reset buffer after successful flush

Thread Safety: Uses mutex for concurrent access

Compression Engine

Responsibility: Compress log data using gzip

Features:

Asynchronous operation (goroutine per compression)
Configurable compression level
Object pooling for buffers
Error handling with fallback

Compression Ratio: Typically 90%+ reduction in size

Example:

Raw logs:     10 MB
Compressed:    0.8 MB
Ratio:        92% reduction

Message Filter

Responsibility: Filter logs by consensus-critical events

Filter Configuration:

filter = true  # Enable filtering

Allow-List (when filter=true):

State change events
Consensus messages
ABCI events
Block commit info
Validator updates

Dropped (when filter=true):

General debug messages
Info-level logs not related to state
Verbose module logs

Impact: Reduces log volume by 70-80% while keeping critical data

WAL Writer

Responsibility: Persist compressed data to disk

Features:

Segment-based files (sequential numbering)
Sidecar index files for efficient replay
Atomic writes with fsync
Automatic segment rotation
Platform-optimized I/O

File Naming:

seg-000001.wal.gz    # Compressed log data
seg-000001.wal.idx   # Index for efficient seeking

Operation Flow

Normal Operation

1. Log Message Generated
   ↓
2. Filter Check (if enabled)
   ↓ (pass)
3. Append to Buffer
   ↓
4. Check Thresholds
   ├─ Time expired?
   └─ Size exceeded?
   ↓ (yes to either)
5. Spawn Compression Goroutine
   ├─ Get buffer from pool
   ├─ Compress data (gzip)
   └─ Return buffer to pool
   ↓
6. Write to WAL
   ├─ Write compressed data
   ├─ Write index entry
   └─ Fsync to disk
   ↓
7. Reset Buffer

Flush Triggers

Time-Based Flush:

ticker := time.NewTicker(flushInterval)
for {
    select {
    case <-ticker.C:
        flush()
    }
}

Size-Based Flush:

if currentBufferSize >= memoryBytes {
    flush()
}

Compression Process

func compress(data []byte) ([]byte, error) {
    // Get buffer from pool
    buf := bufferPool.Get().(*bytes.Buffer)
    buf.Reset()
    defer bufferPool.Put(buf)
 
    // Create gzip writer
    gw := gzip.NewWriter(buf)
 
    // Write and compress
    _, err := gw.Write(data)
    if err != nil {
        return nil, err
    }
 
    // Flush and close
    if err := gw.Close(); err != nil {
        return nil, err
    }
 
    // Return compressed data
    return buf.Bytes(), nil
}

Failure Handling

Graceful Degradation

The memlogger is designed to fail gracefully:

Compression Failure:

Log message → Compression fails → Drop message → Continue logging

Disk Full:

WAL write fails → Log error → Drop buffer → Continue operation

Memory Pressure:

Buffer size limit → Force flush → Clear space → Continue

Ordering Guarantee

Despite async processing, chronological order is preserved:

Messages buffered in arrival order
Buffers compressed sequentially
WAL segments numbered sequentially
Index maintains event ordering

Drop Policy: On failure, entire buffer is dropped (never partial writes)

This ensures:

No corrupted state in logs
Clear gap detection (missing segment numbers)
Consistent replay

Performance Characteristics

Memory Usage

Baseline: ~10-20 MB for memlogger structures

Buffer: Configurable (via memory-bytes)

Default (time-only): Varies with log rate, typically 5-30 MB
With size limit: Capped at configured value

Compression buffers: Pooled, ~1-2 MB total

Total: ~15-50 MB typically

CPU Usage

Compression: 1-5% CPU (depends on log volume)

Runs in background goroutines
Does not compete with consensus

Buffer management: < 0.1% CPU

Simple append operations
Minimal locking contention

Filtering: < 0.1% CPU

Simple string matching
Optimized hot path

Disk I/O

Write frequency: Every interval (e.g., 2 seconds)

Write size: Compressed buffer size (typically 0.5-5 MB)

IOPS impact: Minimal (1 write per interval)

Throughput: 0.25-2.5 MB/s average (depending on log rate)

Configuration Trade-offs

Short Interval (e.g., “1s”)

Pros:

Lower memory usage
More frequent updates
Faster detection of issues

Cons:

More frequent disk writes
Slightly lower compression ratio
More WAL segments

Long Interval (e.g., “5s”)

Pros:

Better compression ratio
Fewer disk writes
Fewer WAL segments

Cons:

Higher memory usage
Less frequent updates
Longer delay before logs available

Recommended: “2s”

Balanced configuration for most use cases:

Moderate memory usage (10-30 MB)
Good compression ratio (90%+)
Reasonable update frequency
Manageable number of segments

Advanced Features

Platform-Optimized Fsync

Different fsync strategies per platform:

Linux: fdatasync() - metadata not required macOS: F_FULLFSYNC - required for crash safety Others: Standard fsync()

Benefits:

Optimal durability guarantees
Best performance for each platform
Crash recovery support

Automatic Segment Rotation

Segments rotate automatically:

Per day (new directory created)
Per size threshold (configurable)
On process restart

Benefits:

Bounded file sizes
Easy archival
Efficient seeking

Sidecar Indexing

Each .wal.gz has a .wal.idx:

Index Entry:
- Offset in compressed file
- Timestamp of events
- Number of events
- Checksum

Benefits:

Fast seeking to specific time
Efficient replay
Integrity verification

Comparison with Standard Logging

Feature	Standard Logging	Memlogger
Level	Info/Warn/Error	Debug
Format	Plain text	JSON (compressed)
Volume	Low	High (filtered)
Storage	Uncompressed	Gzip compressed
Overhead	Minimal	Minimal
State tracking	No	Yes
Async	Usually buffered	Fully async
Filtering	By level	By message type

Implementation Details

Directory Structure

$CHAIN_DIR/data/log.wal/
└── node-<node-id>/          # Unique per node
    ├── 2025-11-23/           # Daily rotation
    │   ├── seg-000001.wal.gz
    │   ├── seg-000001.wal.idx
    │   ├── seg-000002.wal.gz
    │   └── seg-000002.wal.idx
    └── 2025-11-24/
        └── seg-000001.wal.gz
        └── seg-000001.wal.idx

Segment Lifecycle

1. Create: seg-NNNNNN.wal.gz.tmp
2. Write: Append compressed data
3. Index: Write seg-NNNNNN.wal.idx
4. Finalize: Rename to seg-NNNNNN.wal.gz
5. Ship: Picked up by analyzer-shipper

Temporary suffix prevents shipping incomplete files.

Debugging Memlogger

Enable Verbose Logging

log_level = "debug"

Look for memlogger-specific messages:

"memlogger flushed" - Successful flush
"memlogger compression" - Compression stats
"memlogger error" - Any errors

Check WAL Directory

# List recent segments
ls -lth $CHAIN_DIR/data/log.wal/node-*/$(date +%Y-%m-%d)/
 
# Check segment sizes
du -sh $CHAIN_DIR/data/log.wal/node-*/$(date +%Y-%m-%d)/

Monitor Memory

# Check process memory
ps aux | grep chaind
 
# Monitor over time
watch -n 5 'ps aux | grep chaind'

Next Steps

Log Format & Storage - Understand WAL structure
Setup Guide - Configure memlogger for your chain

Overview Log Format & Storage