Memory Tuning - Wyatt's Notes

DDR5 represents a fundamental shift in memory architecture. The comparison is not merely about speed — the power delivery model, signal integrity, and error correction mechanisms are all different.

DDR5 moves the power management IC (PMIC) from the motherboard onto the DIMM itself. This means each DIMM manages its own power regulation, reducing the burden on the motherboard VRM. It also means you Cannot adjust VDDQ/VPP independently per DIMM from the motherboard — the DIMM”s PMIC handles this Internally.

DDR5 doubles the bank group count (from 4 to 8) and doubles the burst length (from BL8 to BL16). This means each activation can transfer 64 bytes per bank group instead of 32 bytes, improving Efficiency for sequential access patterns. However, the double burst length means DDR5 has higher CAS latency in absolute nanoseconds at equivalent CAS latencies in clock cycles.

Clock cycle latency alone is misleading. What matters is actual access time in nanoseconds:

DDR5-6000 CL30 matches DDR4-3200 CL14 in actual latency while delivering nearly double the Bandwidth. This is why DDR5-6000 is considered the sweet spot for AMD Zen 4 and Intel 13th/14th Gen.

The four primary timings define the basic memory access pattern:

These four timings are commonly expressed as CL-tRCD-tRP-tRAS (e.g., 16-18-18-38).

The primary timings are not independent. They are related by the DRAM’s electrical characteristics:

This is the minimum constraint. In practice, tRAS is set to tRCD + tRP + a small margin (2–8 Cycles). Setting tRAS too low relative to tRCD + tRP can cause data corruption because the memory Cell does not have enough time to hold a charge before precharge.

The total access latency for a random read (worst case, row miss) is:

Secondary timings have a smaller but measurable impact on performance:

Tertiary timings have a small impact but can be the difference between stability and instability at Tight settings:

A full timing string for DDR5 might look like:

30-36-36-76-58 (CL-tRCD-tRP-tRAS-tRC)

Or more detailed:

30-38-38-78-58-2-56-56-76-52 (CL-tRCD-tRP-tRAS-tRC-tRFC-tFAW-tRRD_L-tRRD_S-tREFI)

XMP is an Intel-developed standard for storing pre-validated overclocking profiles in the SPD (Serial Presence Detect) EEPROM on the DIMM. When you enable XMP in BIOS, the motherboard reads the Profile and applies the specified frequency, timings, and voltages.

XMP supports two profiles (Profile 1 and Profile 2), allowing vendors to store a “safe” and an “aggressive” profile. For example:

EXPO is AMD’s equivalent of XMP. Functionally identical — it stores validated profiles in the SPD Chip. The main difference is branding: EXPO profiles are validated on AMD platforms. In practice, Most EXPO kits work fine on Intel and vice versa, as the memory ICs are the same.

JEDEC defines the standard (non-overclocked) operating parameters. Every DDR5 DIMM ships with a JEDEC profile that guarantees operation at the specified speed, timings, and voltage. JEDEC DDR5 Speeds include 4800, 5200, 5600, and 6000 MT/s.

When you do not enable XMP/EXPO, your memory runs at the JEDEC base speed. For high-performance DIMMs, this can mean running at 4800 MT/s instead of the rated 6000+ MT/s. Always enable XMP/EXPO to Get the performance you paid for.

XMP/EXPO profiles are validated by the RAM manufacturer on a reference platform with a single DIMM. Real-world stability depends on:

1. Motherboard memory routing quality. Trace length matching, impedance control, and layer stackup vary between boards.
2. CPU memory controller quality. The silicon lottery applies to the memory controller too. Some CPUs can drive high-speed memory with two DIMMs populated; others cannot.
3. DIMM population. Two DIMMs is harder than one; four DIMMs is harder still. The memory controller must drive twice or four times the electrical load.
4. Temperature. DRAM is temperature-sensitive. Higher temperatures require more voltage or relaxed timings.

Modern desktop platforms support dual-channel memory. The memory controller has two independent 64-bit channels, each with its own address/command bus and data bus. Dual-channel operation doubles The theoretical peak bandwidth:

For dual-channel operation, populate slots according to the motherboard manual. :

• 2 DIMMs: Slots A2 and B2 (the second slot from each channel, color-coded)
• 4 DIMMs: Slots A2, B2, A1, B1

Populating the wrong slots can result in single-channel operation, halving your bandwidth. Verify With CPU-Z (Windows) or lshw -class memory (Linux).

DRAM modules can be single-rank (1R) or dual-rank (2R). A rank is a set of DRAM chips that share the Same address/command bus. A 2R DIMM has two independent sets of banks that can be accessed in an Interleaved fashion.

2R DIMMs have higher bank-level parallelism, which means:

• Better performance for random access patterns (more banks to interleave)
• Higher capacity per DIMM
• Harder to overclock (the memory controller drives two ranks instead of one)

For maximum overclocking headroom, 1R DIMMs are preferred. For maximum capacity and mixed-workload Performance, 2R DIMMs are better. The practical difference at the same speed and timings is 3–8% in gaming workloads, favoring 2R.

The fundamental trade-off in memory tuning is between bandwidth (frequency) and latency (timings). Higher frequency provides more bandwidth but often comes with looser timings, which increases Latency.

The answer depends on the workload:

• Gaming: Latency matters more than bandwidth for most titles. DDR5-6000 CL30 is often faster than DDR5-7200 CL34 in games, despite the lower bandwidth.
• Compression/encoding: Bandwidth matters more. These workloads stream large amounts of data sequentially and benefit from higher transfer rates.
• Scientific computing: Depends on the access pattern. Dense matrix operations benefit from bandwidth; sparse operations benefit from low latency.
• Database workloads: Latency is critical. Each query involves many small random reads.

On AMD Zen 4 (Ryzen 7000 series), the memory controller runs at half the DDR5 frequency (1:2 mode) Up to 6000 MT/s. Above 6000 MT/s, it switches to 1:2.67 mode (UCLK = DDR5 / 2.67), which introduces Additional latency. This makes DDR5-6000 the optimal frequency for Zen 4 in most workloads.

Intel’s memory controller handles DDR5 up to ~7200 MT/s comfortably. The sweet spot is DDR5-6400–6800 CL32-CL34. Beyond 7200 MT/s, stability with two DIMMs becomes increasingly difficult.

Intel’s Gear Mode controls the ratio between the memory bus frequency and the memory controller’s Internal clock:

Gear 1 provides the lowest latency because the memory controller operates at the same frequency as The memory bus. Gear 2 halves the controller frequency, which adds approximately 2–4 ns of latency. For DDR5, most kits above 6000 MT/s require Gear 2.

On AMD platforms, FCLK (Infinity Fabric Clock) is the clock speed of the Infinity Fabric Interconnect that connects the CPU cores to the memory controller and PCIe controller. FCLK has a Significant impact on performance because it determines the speed of inter-core communication and L3 Cache access.

• Synchronous mode (1:1): FCLK = UCLK = MCLK / 2. Maximum FCLK is 2000–2200 MHz (DDR5-4000 to DDR5-4400 equivalent for Zen 3, or DDR5-6000 on Zen 4 with 1:2 UCLK).
• Asynchronous mode: FCLK can be set independently of UCLK. This allows higher memory frequencies but introduces a desynchronization penalty for fabric-crossing traffic.

On Zen 3, the synchronous 1:1 FCLK:UCLK ratio was critical for performance, limiting effective DDR4 Speed to ~3600 MT/s. On Zen 4, the memory controller architecture changed, making asynchronous Operation less penalizing.

1. Start with XMP/EXPO enabled. This gives you the manufacturer’s validated baseline.

2. Establish a stability baseline. Run MemTest86 for 4 passes and TestMem5 for 30 minutes at XMP/EXPO settings. If XMP is not stable at two DIMMs, reduce frequency by one step or increase voltage.

3. Tighten primary timings. Reduce CL by 1 cycle. Test. If stable, reduce tRCD and tRP by 1 each. Test. Continue until unstable, then back off.

4. Tighten secondary timings. Focus on tFAW (reduce by 2–4 cycles) and tRRD_L (reduce by 1–2 cycles). These have a measurable impact with relatively low risk.

5. Tertiary timings (advanced). Reduce tRFC (this is the biggest tertiary timing on DDR5) and increase tREFI. These require extended stability testing.

6. Increase frequency (optional). If timings are tight, try increasing frequency by 200 MT/s. You may need to relax timings slightly to compensate.

7. Voltage adjustments. If you hit instability, increase VDDQ by 0.05 V increments (up to 1.4 V for daily use on DDR5). For extreme tuning, VDD can go to 1.45 V with adequate cooling.

Error-Correcting Code (ECC) memory adds an extra memory chip (or chips) per DIMM that stores parity And syndrome information. For single-bit errors, ECC can detect and correct the error transparently (SEC — Single Error Correction). For multi-bit errors, ECC can detect the error but cannot correct It (DED — Double Error Detection), triggering a machine check exception.

DDR5 introduces “in-band” ECC, where each 128-bit access includes 8 extra ECC bits that allow the DRAM to correct single-bit errors internally. This is separate from traditional ECC — it protects Against bit flips within the DRAM chip itself but does not protect against bus errors or provide the Same level of protection as platform-level ECC.

ECC memory has a small performance penalty (1–3%) due to the extra read-modify-write cycle for Updates and the latency of error checking. On registered ECC (RDIMM), there is an additional latency From the register buffer. On load-reduced DIMMs (LRDIMM), the latency penalty is larger (~5–10%) but Capacity is significantly higher.

• Servers and workstations with important data: ECC is strongly recommended. The cost premium is small relative to the value of the data.
• ZFS NAS systems: ECC is highly recommended. ZFS relies on data integrity; silent bit flips in memory can propagate to disk and corrupt your pool.
• Gaming and desktop use: ECC is generally unnecessary. Consumer platforms rarely support ECC anyway (Intel Z-series chipsets do not; AMD AM5 does).

The Linux kernel’s vm.swappiness parameter controls the tendency to swap anonymous memory (application data) versus dropping file cache pages. The default value is 60, which is a poor Default for systems with sufficient RAM.

For desktops with 16 GB or more RAM:

## Check current value
cat /proc/sys/vm/swappiness

## Set to 10 for desktop use
echo 10 | sudo tee /proc/sys/vm/swappiness

# Make permanent
echo "vm.swappiness=10" | sudo tee -a /etc/sysctl.d/99-swappiness.conf

THP allows the kernel to allocate 2 MB pages instead of the standard 4 KB pages, reducing TLB misses For workloads with large memory footprints. THP is enabled by default (madvise mode on many Distributions, always on some).

# Check THP mode
cat /sys/kernel/mm/transparent_hugepage/enabled

# Set to madvise (only applications that request it get huge pages)
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

# Set to always (kernel aggressively uses huge pages)
echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

Static huge pages are pre-allocated at boot time and cannot be swapped out. They are used by Applications like databases and DPDK that require pinned, non-swappable memory.

# Configure 1024 huge pages (each 2 MB = 2 GB total)
echo 1024 | sudo tee /proc/sys/vm/nr_hugepages

# Verify
grep -i huge /proc/meminfo

# Persistent configuration: add to kernel parameters
# default_hugepagesz=2M hugepagesz=2M hugepages=1024

On NUMA systems, you can control memory allocation policies:

# Interleave allocations across all NUMA nodes
numactl --interleave=all ./my_app

# Prefer allocating on node 0, with fallback to node 1
numactl --preferred=0 ./my_app

# Bind exclusively to node 0
numactl --cpunodebind=0 --membind=0 ./my_app

Memory tuning is like organizing a workshop. If your tools (frequently accessed data) are close at hand (in cache), you work faster. If you have to walk to the garage (RAM) or the warehouse (disk), you waste time. Memory tuning ensures your most-used tools are always within reach.

RAM timings are like the speed of a librarian. Lower latency means the librarian finds your book faster. Higher frequency means the librarian can process more requests per second. The best performance comes from a librarian who is both fast and handles many requests.

Two identical-looking RAM kits from different production batches can have different memory ICs (e.g., Samsung B-die vs. Micron Rev.E). When mixed, the system must use the timings of the slower Kit, and the memory controller may struggle with the combined electrical load. Always buy a single Kit with the total capacity you need.

A system that boots and runs benchmarks is not necessarily stable. Memory instability can cause Silent data corruption — the system continues running but produces incorrect results. This is Particularly dangerous for ZFS pools, databases, and compilation workloads. Always run MemTest86 and TestMem5 after any timing or frequency change.

DDR5 has much higher tRFC values than DDR4 ( 500–700 cycles vs. 300–350 cycles). TRFC Represents dead time where the memory cannot service requests while refreshing. Tightening tRFC is One of the most impactful tertiary timing adjustments on DDR5, but it is also one of the most likely To cause instability if set too aggressively.

Populating all four DIMM slots significantly increases the electrical load on the memory controller. Most kits rated for 6000+ MT/s are only validated for two-DIMM configurations. With four DIMMs, you Will likely need to reduce frequency by one or two steps (e.g., from 6000 to 5200 or 5600 MT/s) or Increase voltage. Check your motherboard’s QVL (Qualified Vendor List) for four-DIMM validated Speeds.

DDR memory speeds are measured in MT/s (mega-transfers per second), not MHz. DDR5-6000 transfers 6000 million times per second, but the actual clock frequency is 3000 MHz because DDR transfers data On both edges of the clock signal (double data rate). When you see “6000 MHz” in a BIOS or review, It means 6000 MT/s.

High-speed DDR5 modules generate significant heat — 5–8 W per DIMM under sustained load. Without Adequate airflow over the DIMMs, temperatures can exceed 60 °C, which forces the memory controller To insert wait states or can cause instability. Ensure your case has intake airflow that passes over The DIMM area, or use aftermarket DRAM heatsinks and fans.

DDR5 operates at significantly higher speeds than DDR4, which makes signal integrity critical. Key Electrical parameters:

At DDR5 speeds (4800+ MT/s), the signal degrades significantly over the PCB traces. DDR5 introduces A Decision Feedback Equalizer (DFE) in the memory controller and on the DIMM to compensate for Inter-symbol interference (ISI):

• Transmit DFE (TX DFE): On the memory controller side, compensates for signal degradation on the write path.
• Receive DFE (RX DFE): On the DIMM side, compensates for signal degradation on the read path.

DFE training occurs during system boot (JEDEC training sequence) and can be a source of Compatibility issues — some CPU/memory combinations fail DFE training at high speeds.

On boot, the memory controller performs a calibration sequence:

1. ZQ Calibration: Calibrates the on-die termination (ODT) resistors by comparing against a precision external resistor (RZQ). Takes approximately 256 clock cycles.
2. Read Leveling: Adjusts the DQS read strobe timing to align with the center of the data eye. The controller sweeps the DQS delay and finds the optimal sampling point.
3. Write Leveling: Adjusts the DQS write strobe timing from the controller’s perspective.
4. VREF Training: Calibrates the reference voltage for the receiver to optimize the data eye opening.
5. DFE Training: (DDR5 only) Trains the decision feedback equalizer taps.

This entire sequence takes 1–3 seconds. If any training step fails, the system falls back to a lower Speed or fails to boot.

Intel’s memory controller is integrated into the CPU die and connects to the DDR5 DIMMs via the Processor’s DDR5 interface:

• One memory controller per CPU (not per CCD like AMD).
• Supports 2 channels with 1 DIMM per channel for DDR5-6400+ stable operation.
• Gear mode: Controls the ratio between the memory bus and the internal clock. Gear 2 (1:2) is required above DDR5-5600.

The Intel memory controller is generally more forgiving than AMD’s when it comes to running two DIMMs at high speeds. Most Intel 13th/14th Gen systems can run two DIMMs at DDR5-6800 with relaxed Timings.

AMD’s Zen 4 memory controller is a significant improvement over Zen 3:

• 1:2 UCLK ratio at DDR5-6000, providing optimal latency.
• Up to DDR5-5200 in 1:1 UCLK mode (FCLK = UCLK = MCLK/2).
• Infinity Fabric clock can run asynchronously at higher speeds, reducing the penalty of 1:2.67 mode.

The Zen 4 memory controller’s sweet spot is DDR5-6000 CL30. Going above this requires 1:2.67 mode, Which adds approximately 3–5 ns of latency.

After tightening primary timings, secondary timings provide the next level of optimization:

1. tFAW (Four Activate Window): Start at the JEDEC default and reduce by 2 cycles at a time. For dual-rank DDR5, tFAW is 24–32. Try reducing to 20–24.

• Lower tFAW allows more activate commands in a given window, improving row access parallelism.
• Too low causes data corruption because the DRAM cells do not have enough time to settle.

1. tRRD_S and tRRD_L:

• tRRD_S (Same Bank Group): Start at 4, try reducing to 3.
• tRRD_L (Different Bank Group): Start at 6, try reducing to 4.
• These affect the time between activating different rows. Lower values improve interleaved access performance.

1. tCWL (CAS Write Latency): CL - 1 or CL. Try setting to CL - 1 if it is currently at CL. Write latency has less impact on most workloads than read latency.

2. tWR (Write Recovery Time): Try reducing by 1 cycle. This is the time after a write before the row can be precharged. Related to tRTP (Read to Precharge).

Tertiary timings are risky to tune but can provide measurable improvements:

1. tRFC (Refresh Cycle Time): This is the biggest tertiary timing on DDR5. Default values are often very conservative. Try reducing by 20–40 cycles at a time.

• DDR5-6000 CL30 has tRFC around 500–550. Many kits can run at 460–480.
• Reducing tRFC directly improves performance because the DRAM is unavailable for a shorter period during refreshes.

1. tREFI (Refresh Interval): Default is 31200 for DDR5 (at normal temperature). Increasing tREFI delays refresh cycles, improving performance but reducing data retention margin.

• Try increasing to 62400 or 93600.
• At higher tREFI values, the DRAM may lose data if the ambient temperature is high or if there are long periods without access.

1. tRDWR (Read to Write Turnaround): Default is 5–7 cycles. Try reducing by 1.

2. tWRRD (Write to Read Turnaround): Default is 6–8 cycles. Try reducing by 1.

Tertiary timing instability is often intermittent and may not appear in short tests:

1. Run TestMem5 with Anta777 config for 2+ hours.
2. Run Karhu RAM Test for 4+ cycles (if available).
3. Run y-cruncher for 1+ hour.
4. Run MemTest86 for 8+ passes.
5. Use the system normally for 24 hours and check for any unexpected crashes or corruption.

THP works by the kernel’s khugepaged daemon scanning process memory and collapsing contiguous Standard (4 KB) pages into huge pages (2 MB). This process involves:

1. Scanning: khugepaged periodically scans process address spaces looking for contiguous 4 KB pages that could be collapsed into a single 2 MB page.
2. Copying: When eligible pages are found, khugepaged allocates a 2 MB page, copies the data from the constituent 4 KB pages, and updates the page tables.
3. Collapsing: The original 4 KB pages are freed and replaced with the single 2 MB page.

This process consumes CPU and memory bandwidth. Under memory pressure, the cost of the collapsing Operation can cause latency spikes.

# Check current huge page settings
cat /proc/meminfo | grep -i huge

# Allocate 1024 huge pages at boot (2 GB total)
# Add to kernel parameters:
# default_hugepagesz=2M hugepagesz=2M hugepages=1024

# Or allocate dynamically
echo 1024 | sudo tee /proc/sys/vm/nr_hugepages

# Verify allocation
grep -i huge /proc/meminfo

# Configure libhugetlbfs for transparent usage
# LD_PRELOAD=/usr/lib64/libhugetlbfs.so

# Create a swap file
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Make permanent
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# Verify
swapon --show
free -h

• CPU Fundamentals - How memory controllers and IMC settings interact with CPU performance
• Stress Testing - How memory stress tests validate XMP/EXPO profile stability
• Undervolting and Overclocking - How memory voltage and timing adjustments affect system stability