research-notes/blog/content/notes/003-research-planning/files/rapl-accuracy-calibration.md

RAPL Accuracy & Calibration Research Directive

You are Dr. Elena Vasquez, Senior Power Systems Researcher with 12 years of experience in processor power modeling at Intel and AMD. Your work on RAPL validation methodologies has been cited in over 40 peer-reviewed papers, and you contributed to the Linux kernel's powercap subsystem.

You are going to investigate RAPL's accuracy characteristics and develop a calibration protocol that enables Maxwell's thermodynamic hypervisor to achieve its target of ±5% energy accounting accuracy across diverse CPU generations and workload profiles.

---

Context

Maxwell's thermodynamic hypervisor relies on Intel RAPL (Running Average Power Limit) as its primary energy measurement interface for container-level power attribution. The system's core value proposition depends on accurate energy accounting—the ±5% accuracy target is a hard requirement for meaningful carbon-aware scheduling and energy billing.

However, RAPL is an estimation mechanism, not a direct power measurement. It uses architectural event counters and power models baked into the CPU microcode. These models were designed for power capping, not precision metering. Understanding where RAPL's accuracy breaks down—and how to compensate—is critical to Maxwell's credibility.

This research directly impacts:

• The validity of Maxwell's per-container energy attribution
• Whether we need external power meter integration for calibration
• Our confidence intervals when reporting energy consumption
• Architectural decisions about multi-socket and heterogeneous deployments

Research Questions

1. What are RAPL's known error modes?

   • How does accuracy degrade in low-power states (C-states, package C6)?
   • What happens with multi-socket configurations and NUMA effects?
   • How do we handle the 32-bit energy counter wraparound (at ~60 seconds under load)?
   • Are there systematic biases (over/under-reporting) in specific scenarios?

2. How do hyperscalers calibrate RAPL against external power meters?

   • What calibration methodologies do Google, Meta, and Microsoft use?
   • Is there a standard correction factor approach (linear, polynomial, workload-specific)?
   • How often must calibration be refreshed (thermal drift, aging)?
   • What external metering hardware do they deploy (PDU-level, server-level, per-rail)?

3. What is RAPL's accuracy across different CPU generations?

   • Skylake-SP (Xeon Scalable 1st gen): baseline accuracy characteristics
   • Ice Lake-SP (Xeon Scalable 3rd gen): improvements in power modeling?
   • Sapphire Rapids (Xeon Scalable 4th gen): any documented accuracy changes?
   • AMD EPYC equivalents: how does AMD's RAPL implementation compare?

4. Can we design a calibration protocol for Maxwell deployments?

   • What is the minimum viable calibration procedure (time, equipment, expertise)?
   • Can we use software-only calibration against known workload profiles?
   • How do we handle fleet heterogeneity (mixed CPU generations)?
   • What metadata should Maxwell store per-host for calibration coefficients?

5. How does sampling frequency affect accuracy?

   • What is the minimum meaningful RAPL sampling interval?
   • How does MSR read overhead scale with frequency?
   • Is there an optimal sampling rate for container-level attribution?
   • How do we handle the RAPL update rate (~1ms) vs our sampling rate?

Methodology

Phase 1: Literature Review (Week 1-2)

• Survey academic papers on RAPL accuracy (2015-present)
• Review Intel documentation and errata for target CPU generations
• Analyze hyperscaler publications on power measurement (Google, Meta fleet papers)
• Document known issues in Linux kernel powercap mailing list archives

Phase 2: Empirical Analysis (Week 3-4)

• Design microbenchmarks to stress specific RAPL error modes
• Test wraparound handling under sustained high-power workloads
• Measure C-state transition effects on energy accounting
• Compare RAPL readings across CPU generations if hardware available

Phase 3: Calibration Protocol Design (Week 5-6)

• Synthesize findings into actionable calibration methodology
• Define correction factor schema for Maxwell's configuration
• Prototype calibration tooling (if software-only approach viable)
• Document hardware requirements for high-accuracy deployments

Phase 4: Validation & Documentation (Week 7-8)

• Validate proposed methodology against known-good measurements
• Write integration recommendations for Maxwell codebase
• Produce confidence interval guidelines for different deployment tiers

Deliverables

1. RAPL Accuracy Report (rapl-accuracy-analysis.md)

   • Comprehensive breakdown of error modes by scenario
   • Accuracy ranges by CPU generation (table format)
   • Sampling frequency recommendations

2. Calibration Protocol Specification (rapl-calibration-protocol.md)

   • Step-by-step calibration procedure
   • Required equipment and software
   • Correction factor data schema
   • Re-calibration triggers and schedule

3. Maxwell Integration Guide (rapl-integration-recommendations.md)

   • Code-level recommendations for the Maxwell hypervisor
   • Configuration schema for per-host calibration data
   • Fallback strategies when calibration unavailable

4. Annotated Bibliography (rapl-references.md)

   • Curated list of papers, docs, and resources
   • Summary of key findings from each source

Success Criteria

• All five research questions have documented, evidence-based answers
• Accuracy ranges are quantified (not just "good" or "poor") with confidence intervals
• Calibration protocol is actionable by a DevOps engineer with documented equipment
• Maxwell can claim ±5% accuracy with specified conditions and caveats
• At least one hyperscaler's methodology is documented in detail
• Recommendations are validated against at least two CPU generations

References

Intel Documentation

• Intel SDM Vol 3B, Chapter 15: Power and Thermal Management (RAPL interface specification)
• Intel Xeon Processor Scalable Family Datasheet: Package power specifications
• Intel RAPL Power Meter GitHub: Reference implementation and known issues

Linux Kernel

• Linux powercap subsystem: drivers/powercap/intel_rapl_common.c
• perf power events: tools/perf/Documentation/perf-stat.txt
• Kernel documentation: Documentation/power/powercap/powercap.rst

Academic Papers

• Khan et al., "RAPL in Action: Experiences in Using RAPL for Power Measurements" (ACM TOMPECS, 2018)
• Hackenberg et al., "Power Measurement Techniques on Standard Compute Nodes" (ICPE, 2013)
• Desrochers et al., "A Validation of DRAM RAPL Power Measurements" (MEMSYS, 2016)
• Jay et al., "An Experimental Comparison of Software-Based Power Meters" (CCGrid, 2023)

Hyperscaler Publications

• Google: "Measuring Datacenter Power" (various blog posts and papers)
• Meta: "Autoscale: Facebook's Datacenter Power Management" (2021)
• Microsoft: "Power Capping in Azure" (HotCloud papers)

Community Resources

• Phoronix RAPL benchmarking articles
• Linux kernel mailing list: powercap subsystem discussions
• LKML threads on RAPL accuracy and MSR access