DEV Community

compilersutra
compilersutra

Posted on Originally published at compilersutra.com

Phase Comparison: A Checklist for Honest PR Reviews

#codereview #csperf #performance #engineering

CompilerSutraPerf episode cover

Cover — real hardware measurement series.

Why this lesson exists

In a fast‑moving codebase, performance pull requests often get approved or blocked without a clear, evidence‑based process. A reviewer might see a single line of output or a vague claim and decide, "Looks good," or “We can’t afford to risk a slowdown.” This episode gives you a concrete, csperf‑driven checklist so every PR is judged on the same, reproducible metrics.

Recap — where we are in the series

The misconception

Many teams treat a single csperf run as a definitive verdict. Without a side‑by‑side diff, reviewers can’t tell if a change truly improves or degrades performance, or if the observed difference is noise.

What problem csperf solves (this episode's slice)

csperf’s diff command produces a side‑by‑side CSV that lists every measured metric before and after a change. By feeding that CSV into a checklist, reviewers can systematically verify that every claim in the PR is backed by data.

Mental model

  1. Baseline – Run the current code and capture results/before.json.
  2. Change – Apply the PR, build, and run again to get results/after.json.
  3. Diff – csperf diff results/before.json results/after.json --csv results/pr.csv creates a CSV that shows the delta for each metric.
  4. Checklist – Inspect the CSV against the four criteria below.

Lab: install and first commands 

pip install compilersutra-perf
csperf doctor
Enter fullscreen mode Exit fullscreen mode

Lab: what we ran on this machine 

# Baseline
csperf run --workload matrix_traversal.cpp --output results/before.json
# After PR
csperf run --workload matrix_traversal.cpp --output results/after.json
# Diff
csperf diff results/before.json results/after.json --csv results/pr.csv
Enter fullscreen mode Exit fullscreen mode

Results (real numbers only)

Trial wall times from this csperf run

Graph: trial wall_time_ms from this run's artifacts (not invented).

Artifacts missing for this episode — numbers withheld.

How to read the artifacts

  • execution_time_ms – The median wall‑time of the workload.
  • trial wall_time_ms – Each individual run’s wall‑time.
  • timing_summary – Aggregated statistics (min, max, median).
  • metadata – CPU, OS, compiler version, and environment variables.

When you open results/pr.csv, look for columns like execution_time_ms_before, execution_time_ms_after, and delta_ms. A positive delta indicates a slowdown; a negative delta indicates a speedup.

Common mistakes (teacher checklist)

  1. Skipping the baseline – Always run before.json on the same machine.
  2. Ignoring metadata – Verify that CPU and compiler versions match.
  3. Overlooking noise – Look at the trial wall_time_ms spread; a wide spread means the delta may be insignificant.
  4. Assuming any delta is meaningful – Apply a threshold (e.g., 5 % change) before marking a PR as performance‑critical.

Try this next (homework)

Do this tonight — Episode 23 starts by assuming you did.

Closing

A disciplined, artifact‑driven review process turns performance claims from opinion into data. By adopting this checklist, you’ll reduce false positives, avoid regressions, and build trust in your performance pipeline.

The series so far

References

Top comments (0)