I currently have limited availability - please get in touch if you'd like to work with me.

How long does an HTTP request take?

Orde Saunders' avatar by Orde Saunders

Statistical analysis of page load speed data collected using the Navigation Timing API shows that an HTTP request can be reasonably approximated to 0.5 seconds.

Introduction

Whilst newer browsers implement a timing API that can give the time for a request including network latency, this is not available in all browsers. In browsers that do not support the timing API it is only possible, using on page JavaScript, to measure the time taken from the start of the DOM loading to a subsequent event - such as DOM ready. Using information from browsers that support the timing API it is possible to measure both the time from request to DOM ready and the time from DOM start to DOM ready and establish a relationship such that request_time = multiplier x dom_time + constant. Once this relationship is known it can be applied when only the DOM loading to ready time is available to estimate the full request time.

Method

From visitors that were using a browser that had support for the Navigation Timing API a random 1% sample was taken over the course of a week on a site with a very high level of traffic. For the sampled visitors two measurements were sent as query string parameters on a tracking pixel located on a dedicated subdomain. The code to insert the tracking pixels was similar to the following:

window.onload = function() {
  var body,
      img,
      t_dom,
      t_request,
      tracking_pixel = 'http://subdomain.example.com/tracking.gif';
  if (typeof window.performance !== "undefined" && typeof window.performance.timing !== "undefined") {
    if (!Math.floor(Math.random()*100)) {
      t_dom = window.performance.timing.domInteractive - window.performance.timing.domLoading;
      t_request = window.performance.timing.domInteractive - window.performance.timing.requestStart;
      body = document.getElementsByTagName('body')[0];
      img = document.createElement("img");
      img.src = tracking_pixel + '?dom=' + t_dom + '&request=' + t_request;
      img.alt = '';
      img.height = 1;
      img.width = 1;
      body.insertBefore(img, body.lastChild);
    }
  }
};

The tracking pixel returned a 204 (No content) status code.

The server logs for the test period were filtered for references to the tracking pixel, concatenated and passed through a Python data processing script. This performs a liner regression using least squares to find the best fit to the equation: request_time = multiplier x dom_time + constant.

In order to prevent outlier results adversely affecting the result a cut off was placed on both the request time and the DOM time: if either is greater than six seconds then both are discarded.

In order to validate that the performance timing API provides equivalent data to manually setting in-page timers a page similar to the following was set up:

<!DOCTYPE html>
<html>
  <head>
    <script>var t_head = new Date();</script> <!-- Must be the very first item in the head. -->
  </head>
  <body>
    ...
    <script>
      if (typeof window.performance !== "undefined" && typeof window.performance.timing !== "undefined" && typeof console.log !== "undefined" && typeof document.addEventListener !== "undefined") {
        document.addEventListener("DOMContentLoaded", function () {
          var jitter,
              t_dom = new Date();
          jitter = (window.performance.timing.domInteractive - window.performance.timing.domLoading) - (t_dom - t_head);
          console.log(jitter);
        });
      }
    </script>
  </body>
</html>

Results

The following graph is the output from the data processing script:

Request time vs DOM time

Based on 1641 data points, the average request time for a page was 1.475 seconds.

The discrepancy between the in-page timers and the performance timing API was found to be consistently less than 50ms which is an error of ~0.05% on the average measured DOM load time and so could be effectively ignored. It also shows that using in-page timers is a valid substitute for the the timing API.

The measured values for the formula request_time = multiplier x dom_time + constant give a multiplier of 0.997 and a constant of 0.442 seconds.

Conclusion

It is important to note that, at the time of conducting the experiment, no common mobile browsers supported the timing API so it is reasonable to suppose that the majority of this data was gathered on fast connections where we would expect the overhead of a request to be minimal.

The browser breakdown is as follows:

Internet Exporer 9
46%
Chrome
29%
Firefox
21%

Based on this research is it reasonable to use the approximation of adding 0.5 seconds to the time measured to load the DOM using in-page timers to estimate the time taken for the full request.

Related