Skip to main content

Lab vs. field data: understanding your performance metrics

You can measure web performance with two types of data: field data (real user data) and lab data (synthetic tests). Field data captures performance metrics from actual users visiting your site across devices, networks, and locations. Lab data comes from running a single, controlled test in a consistent environment as a detailed performance snapshot. Keep in mind that your analytics and Real User Monitoring (RUM) data include only users who successfully loaded the page. Users on very slow connections or old devices might have abandoned the page before your analytics script fired, creating survivorship bias. Your "average" user is often faster than your actual average user.

Anchor to Field dataField data

Use field data to understand what real users experience. It's the right tool for:

  • Establishing your performance baseline.
  • Understanding what problems real users are facing.
  • Making strategic decisions about what to optimize.
  • Verifying that optimizations improved the user experience.

Field data is available through:

  • Shopify Web Performance Dashboard: Your store's private RUM data.
  • Chrome User Experience Report (CrUX): Public RUM data from Chrome users, available in PageSpeed Insights and tools like TREO.
  • Custom RUM providers, such as SpeedCurve or New Relic.

Anchor to Lab dataLab data

Use lab data to debug and understand specific issues. It's the right tool for:

  • Debugging specific issues identified in RUM.
  • Testing a change before and after to confirm it has the desired effect.
  • Understanding why a metric is poor on a technical level.
  • Analyzing the resource loading waterfall.

Lab data is available through:

  • WebPageTest: The preferred tool for performance engineers, offering detailed analysis.
  • Chrome DevTools: For local development and debugging.
  • Lighthouse and PageSpeed Insights: Good for high-level audits and identifying opportunities.

Anchor to Compare the toolsCompare the tools

ToolData typeBest forShopify context
Shopify Web Performance DashboardField (private RUM)Baselines, regression detection, and segmenting by page type, device, and regionAvailable in the Shopify admin for your store only. Data comes from Chrome users who opted into sharing performance metrics. Hydrogen storefronts need analytics integration configured before data arrives.
CrUX and PageSpeed InsightsField (public RUM)Comparing your store against competitors, and checking the same data Google Search Console reportsOrigin-level and URL-level data for any public domain, so you can benchmark other stores. Reports the 75th percentile over a 28-day rolling window, so a fix takes weeks to show fully. Low-traffic pages might have no data.
LighthouseLab (single synthetic run)Quick audits, catching regressions in CI, and the opportunity listShopify evaluates Theme Store submissions with a Lighthouse benchmark, so theme developers can run the same test on a development store. Its default mobile run applies network and CPU throttling, which won't match your real visitor mix.
WebPageTestLab (configurable synthetic runs)Waterfall analysis, filmstrips, and testing a specific device, connection, or locationThe right tool for diagnosing a slow storefront request chain, including app scripts, third-party domains, and proxy hops in front of the Shopify CDN. Run tests from a location where you have real customers.
Third-party RUM, such as SpeedCurve or New RelicField (private RUM)Custom dimensions, alerting, longer retention, and correlating performance with business metricsAdds a script to your storefront, so budget for its own weight. Useful when you need segments the Shopify Web Performance Dashboard doesn't provide, such as by theme version or by campaign.

CrUX and your own RUM measure the same users but not the same population. CrUX includes only Chrome users who opted in, aggregates to the 75th percentile, and lags by weeks. Your own RUM fires on whatever browsers your script supports, and you choose the percentile and the window. Treat a difference between the two as expected, and use one of them consistently when you're tracking a change.

Anchor to Using them togetherUsing them together

Use field data to identify a problem, use lab data to debug and fix it, then use field data again to verify the fix worked for real users. Compare metrics between lab and field data to identify discrepancies.

A short decision rule:

  • You don't know what's slow yet: start with field data.
  • You know what's slow but not why: switch to lab data.
  • You shipped a change: go back to field data, and wait for the reporting window to catch up.

Anchor to When lab and field disagreeWhen lab and field disagree

Disagreement is normal, and the cause is usually a difference in what's being measured rather than a broken tool. Before you act, work out which population each number describes.

Anchor to Lab looks good, field looks badLab looks good, field looks bad

  • Device mix: Your lab test runs on one device profile. Real traffic spans years of phone hardware, and JavaScript execution and rendering cost scale with CPU. A storefront that's fast on a recent flagship can miss the LCP threshold on a budget phone. Segment your RUM by device category first.
  • Geography: A test from a datacenter near your customers hides the latency your distant customers pay. Check the regional breakdown in your RUM data, then re-run the lab test from the slowest region.
  • Caching state: Lab tools typically report a cold, first-view load, but they can also be configured for a warm repeat view. Real traffic is a mix of both, plus visitors arriving with a partly warm CDN and DNS cache. If your lab run is warm and your visitors are cold, then the lab number is optimistic.
  • Real-world variability: Real users scroll, tap, switch tabs, and load your page inside in-app browsers with injected scripts. INP in particular has no lab equivalent, because it depends on actual interactions.

Anchor to Field looks good, lab looks badField looks good, lab looks bad

  • Test parameters: Confirm the throttling, device emulation, and location match your real traffic instead of a worst case you don't serve.
  • Bot and synthetic traffic: Uptime monitors, scrapers, and your own synthetic checks don't appear in RUM, because RUM needs a real browser session. The reverse also matters: if a monitoring tool loads one URL every minute, don't read its numbers as a customer experience.
  • Single-run noise: One lab run is one sample. Take the median of several runs before you believe a regression.
  • Sample window: Field data is an aggregate over a window, so it's slow to move. A regression you shipped yesterday can be invisible in a 28-day CrUX window, and a fix can look like it did nothing. Compare like-for-like windows, and prefer your own RUM for anything you need to see quickly.

If both sources agree that a metric is poor, then fix it. If only one does, then find the population difference before you spend engineering time.

Anchor to ReferencesReferences

Was this page helpful?