CopyDisable

Saturday, 11 August 2018

HTTP2 Push benchmarking


To check the performance of HTTP2 push feature I used a very basic HTML page with some static content like images and to load some CSS and two heavy JavaScript files.

This HTML page will be served by Nginx webserver version 1.15.2 (since Nginx version 1.13.9, HTTP2 push is supported) on Ubuntu 16.04.

HTML file used for testing:

<!DOCTYPE html>

<html lang="en">

<head>

<title>My HTTP2 Test</title>

<script src="js/angular.js"></script>

<script src="js/jquery-latest.js"></script>

<link href="css/RatingStars.css" rel="stylesheet" type="text/css" />

<link href="css/speech.css" rel="stylesheet" type="text/css" />

<link href="css/style.css" rel="stylesheet" type="text/css" />

<link href="css/track.css" rel="stylesheet" type="text/css" />

<link href="css/speech1.css" rel="stylesheet" type="text/css" />

<link href="css/speech2.css" rel="stylesheet" type="text/css" />

<link href="css/speech3.css" rel="stylesheet" type="text/css" />

<link href="css/speech4.css" rel="stylesheet" type="text/css" />

<link href="css/speech5.css" rel="stylesheet" type="text/css" />

<link href="css/speech6.css" rel="stylesheet" type="text/css" />

<link href="css/speech7.css" rel="stylesheet" type="text/css" />

<link href="css/speech8.css" rel="stylesheet" type="text/css" />

<link href="css/speech9.css" rel="stylesheet" type="text/css" />

<link href="css/speech10.css" rel="stylesheet" type="text/css" />

</head>

<body>

<div>

<h1>Images</h1>

<div><img src="images/1.jpg" /></div>

<div><img src="images/2.jpg" /></div>

<div><img src="images/3.jpg" /></div>

<div><img src="images/4.jpg" /></div>

<div><img src="images/5.jpg" /></div>

<div><img src="images/6.jpg" /></div>

<div><img src="images/7.jpg" /></div>

<div><img src="images/8.jpg" /></div>

<div><img src="images/9.jpg" /></div>

<div><img src="images/10.jpg" /></div>

</div>

</body>

Nginx configuration:

For simple static file testing I used the http2_push directive in Nginx configuration file.

Note: By default Nginx limits number of concurrent push requests in a connection to 10. As the number of objects that we will be pushing is more than 10 for this test, so I will be changing the http2_max_concurrent_pushes directive to some desired value. For this test I am setting this to say 50. Set this in the http section of the nginx config file
http2_max_concurrent_pushes 50;

Test Scenarios

I took four different scenarios for this benchmarking test

1. Webserver will serve the page over simple HTTP using protocol HTTP/1.1

server {

listen 80 default_server;

listen [::]:80 default_server;

root /var/www/html;

index index.html index.htm index.nginx-debian.html;

server_name _;

location /index.html {

try_files $uri $uri/ =404;

}

}

2. Webserver will serve the page over HTTPS using protocol HTTP/1.1

server {

listen 80 default_server;

listen [::]:80 default_server;

listen 443 ssl default_server;

listen [::]:443 ssl default_server;

ssl on;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_certificate /etc/nginx/ssl/star_mkcl_org.crt;

ssl_certificate_key /etc/nginx/ssl/mkcl-org.key;

ssl_stapling on;

ssl_stapling_verify on;

root /var/www/html;

index index.html index.htm index.nginx-debian.html;

server_name _;

location /index.html {

try_files $uri $uri/ =404;

}

}

3. Webserver will serve the page over HTTPS using protocol HTTP/2

server {

listen 80 default_server;

listen [::]:80 default_server;

listen 443 ssl http2 default_server;

listen [::]:443 ssl http2 default_server;

ssl on;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_certificate /etc/nginx/ssl/star_mkcl_org.crt;

ssl_certificate_key /etc/nginx/ssl/mkcl-org.key;

ssl_stapling on;

ssl_stapling_verify on;

root /var/www/html;

index index.html index.htm index.nginx-debian.html;

server_name _;

location /index.html {

try_files $uri $uri/ =404;

}

}

4. Webserver will serve the page using protocol HTTP2 with Push feature.
server {

listen 80 default_server;

listen [::]:80 default_server;

listen 443 ssl http2 default_server;

listen [::]:443 ssl http2 default_server;

ssl on;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_certificate /etc/nginx/ssl/star_mkcl_org.crt;

ssl_certificate_key /etc/nginx/ssl/mkcl-org.key;

ssl_stapling on;

ssl_stapling_verify on;

root /var/www/html;

index index.html index.htm index.nginx-debian.html;

server_name _;

location /index.html {

http2_push /css/RatingStars.css;

http2_push /css/speech.css;

http2_push /css/style.css;

http2_push /css/track.css;

http2_push /css/speech1.css;

http2_push /css/speech2.css;

http2_push /css/speech3.css;

http2_push /css/speech4.css;

http2_push /css/speech5.css;

http2_push /css/speech6.css;

http2_push /css/speech7.css;

http2_push /css/speech8.css;

http2_push /css/speech9.css;

http2_push /css/speech10.css;

http2_push /js/angular.js;

http2_push /js/jquery-latest.js;

http2_push /images/1.jpg;

http2_push /images/2.jpg;

http2_push /images/3.jpg;

http2_push /images/4.jpg;

http2_push /images/5.jpg;

http2_push /images/6.jpg;

http2_push /images/7.jpg;

http2_push /images/8.jpg;

http2_push /images/9.jpg;

http2_push /images/10.jpg;

try_files $uri $uri/ =404;

}

}

Tests over internet connections are not that reliable. As I found two same test results gave big variation for the same configuration (e.g. one test gave 32 second load time and another test gave 7 second load time for the same scenario using internet connection).

So I decided to run the tests in two different environments:

· Hosted the above HTML page in public server and run the page load tests from Google Chrome and WebPageTest.org

· Hosted the HTML file in virtualbox and run the tests using Google chrome. Here I can get stable and reliable network link between the browser and the webserver. To simulate over the internet scenario, I used the tool Wondershaper to limit the incoming and outgoing bandwidth of the virtualbox VM to 1024kbps.

Test results:

Test run on Public Server using WebPageTest.org tool:
clip_image002

I have selected the Test Location Ireland and Browser as Chrome for my tests.

HTTP2 with Push

I run this test for 7 times and got the below results:

Performance Results (Median Run)

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 1)

6.925s

0.777s

7.000s

7000

6.925s

28

3,325 KB

7.144s

29

3,326 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

7.032s

0.882s

7.100s

7100

> 7.077s

7.032s

28

3,325 KB

7.241s

29

3,326 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

6.857s

0.697s

6.900s

6900

> 6.898s

6.857s

28

3,325 KB

7.053s

29

3,326 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 3)

7.077s

0.848s

7.100s

7100

> 7.122s

7.077s

28

3,325 KB

7.279s

29

3,326 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

6.900s

0.751s

7.000s

7000

> 6.945s

6.900s

28

3,325 KB

7.089s

29

3,326 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 1)

7.223s

0.753s

7.200s

7200

> 7.275s

7.223s

28

3,325 KB

7.421s

29

3,326 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

7.049s

0.762s

7.100s

7100

> 7.098s

7.049s

28

3,325 KB

7.258s

29

3,326 KB

$$$$$

Considering the Fully loaded time of the webpage, we will calculate the average time: 7.212

HTTP2 without PUSH

I run this test for 3 times and got the below results:

Performance Results (Median Run)

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

7.119s

0.740s

7.100s

7100

> 7.168s

7.119s

28

3,327 KB

7.326s

29

3,329 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 1)

7.172s

0.739s

7.200s

7200

> 7.224s

7.172s

28

3,327 KB

7.384s

29

3,329 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

7.118s

0.702s

7.200s

7200

> 7.166s

7.118s

28

3,327 KB

7.315s

29

3,329 KB

$$$$$

Considering the Fully loaded time of the webpage, we will calculate the average time: 7.342

HTTPS without HTTP2

I run this test for 3 times and got the below results:

Performance Results (Median Run)

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 3)

7.170s

0.720s

7.200s

7200

> 7.217s

7.170s

28

3,327 KB

7.385s

29

3,329 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 1)

7.196s

0.801s

7.200s

7200

> 7.251s

7.196s

28

3,327 KB

7.407s

29

3,329 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 3)

7.143s

0.722s

7.200s

7200

> 7.187s

7.143s

28

3,327 KB

7.342s

29

3,329 KB

$$$$$

Considering the Fully loaded time of the webpage, we will calculate the average time: 7.378

HTTP 1.1 without HTTPS

I run this test for 8 times and got the below results:

Performance Results (Median Run)

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

8.409s

0.442s

3.800s

3800

3.806s

8.409s

27

3,327 KB

8.605s

28

3,328 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

6.760s

0.410s

3.400s

3413

3.365s

6.760s

27

3,327 KB

6.963s

28

3,328 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

6.567s

0.428s

4.100s

4100

4.110s

6.567s

27

3,327 KB

6.768s

28

3,328 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 1)

6.601s

0.407s

4.000s

4000

3.963s

6.601s

27

3,327 KB

6.796s

28

3,328 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 1)

9.671s

0.416s

6.700s

6700

6.740s

9.671s

27

3,327 KB

9.870s

28

3,328 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 1)

6.765s

0.443s

3.800s

3800

3.806s

6.765s

27

3,327 KB

6.969s

28

3,328 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

10.586s

0.421s

5.000s

5000

5.055s

10.586s

27

3,327 KB

10.781s

28

3,328 KB

$$$$$

Document Complete

Fully Loaded

Load Time

First Byte

Start Render

Speed Index

First Interactive (beta)

Time

Requests

Bytes In

Time

Requests

Bytes In

Cost

First View (Run 2)

6.553s

0.409s

3.800s

3800

3.761s

6.553s

27

3,327 KB

6.765s

28

3,328 KB

$$$$$

Considering the Fully loaded time of the webpage, we will calculate the average time: 7.9395

Final result of WebPageTest.org tool:

Here I considered the Full page load time returned by the tool.

For each test scenarios, I run the tool minimum 3 times and maximum 8 times and calculated the average of the results that I got.

The results are:

Scenario

Fully loaded time (secs)

HTTP1.1 without SSL

7.9395

HTTP1.1 with SSL

7.378

HTTP2.0 with SSL

7.342

HTTP2.0 with SSL and Push

7.212

clip_image004

Conclusion: From the above results that we got from the WebPageTest.org tool, we can see that HTTP2 with Push feature has speed advantage over others.

Test run on Public Server using Google Chrome browser and using the developer tool:

First I tried to run the tests over our broadband connections, as broadband connections are not that reliable and I started to get weird results because of network link fluctuations. So I decided to use mobile phone 4G network for this test.

· HTTP1.1 without SSLclip_image006

· HTTP1.1 with SSL
clip_image008

· HTTP2.0 with SSL
clip_image010

· HTTP2.0 with SSL and Pushclip_image012

Test1

Requests

Data Transfer (MB)

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

28

3.3

10.91

8.36

10.79

HTTP1.1 with SSL

28

3.3

32.19

32.06

32.07

HTTP2.0 with SSL

28

3.3

17.04

11.56

17.31

HTTP2.0 with SSL and Push

28

3.3

7.74

7.63

7.64

Test2

Requests

Data Transfer (MB)

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

28

3.3

16.90

16.76

16.77

HTTP1.1 with SSL

28

3.3

7.23

5.82

7.12

HTTP2.0 with SSL

28

3.3

10.70

6.85

10.60

HTTP2.0 with SSL and Push

28

3.3

5.78

4.59

5.68

Test3

Requests

Data Transfer (MB)

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

28

3.3

10.12

7.64

10

HTTP1.1 with SSL

28

3.3

10.39

10.25

10.26

HTTP2.0 with SSL

28

3.3

7.25

5.65

7.16

HTTP2.0 with SSL and Push

28

3.3

5.79

5.65

5.66

Test 4

Requests

Data Transfer (MB)

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

28

3.3

7.42

7.02

7.31

HTTP1.1 with SSL

28

3.3

10.32

10.18

10.19

HTTP2.0 with SSL

28

3.3

9.15

6.43

9.03

HTTP2.0 with SSL and Push

28

3.3

12.06

8.69

11.96

Averaging the results of the above 4 tests and selecting the Finish, DOMContentLoaded and Load times, we get the following results:

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

11.3375

9.945

11.2175

HTTP1.1 with SSL

15.0325

14.5775

14.91

HTTP2.0 with SSL

11.035

7.6225

11.025

HTTP2.0 with SSL and Push

7.8425

6.64

7.735

clip_image014

clip_image016

clip_image018

Conclusion: Here also we can conclude that HTTP2 with Push feature has loaded the page faster than others.

Test run on webpage hosted on virtualbox VM and requests sent using Google chrome.

In ideal scenario, the network should be reliable to get the proper results for benchmarks. But this is hardly possible in public network over internet. So I tried to simulate the scenario, using a tool named Wondershaper in my Ubuntu virtualbox VM. With the help of this tool I was able to limit the incoming and outgoing bandwidth of the virtualbox VM to 1024kbps, which is close to what we get in normal slow internet connections.

Results:

Test1

Requests

Data Transfer (MB)

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

28

3.3

30.95

30.89

30.91

HTTP1.1 with SSL

28

3.3

30.25

30.50

30.51

HTTP2.0 with SSL

28

3.3

30.59

13.92

30.59

HTTP2.0 with SSL and Push

28

3.3

30.70

15.80

30.70

Test2

Requests

Data Transfer (MB)

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

28

3.3

31.15

31.11

31.13

HTTP1.1 with SSL

28

3.3

30.89

31.16

31.18

HTTP2.0 with SSL

28

3.3

31.84

16.68

31.84

HTTP2.0 with SSL and Push

28

3.3

30.57

15.52

30.58

Test3

Requests

Data Transfer (MB)

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

28

3.3

30.79

26.62

30.75

HTTP1.1 with SSL

28

3.3

30.85

31.18

31.19

HTTP2.0 with SSL

28

3.3

30.91

15.71

30.92

HTTP2.0 with SSL and Push

28

3.3

30.72

15.90

30.72

Test4

Requests

Data Transfer (MB)

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

28

3.3

31.27

31.19

31.20

HTTP1.1 with SSL

28

3.3

30.51

30.75

30.77

HTTP2.0 with SSL

28

3.3

30.03

30.24

30.26

HTTP2.0 with SSL and Push

28

3.3

31.27

16.41

31.28

Test5

Requests

Data Transfer (MB)

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

28

3.3

30.99

30.96

30.97

HTTP1.1 with SSL

28

3.3

31.35

31.55

31.58

HTTP2.0 with SSL

28

3.3

30.65

15.41

30.65

HTTP2.0 with SSL and Push

28

3.3

30.44

15.31

30.44

Averaging the results of the above 5 tests and selecting the Finish, DOMContentLoaded and Load times, we get the following results:

Finish (Secs)

DOMContentLoaded (Secs)

Load (Secs)

HTTP1.1 without SSL

31.03

30.154

30.992

HTTP1.1 with SSL

30.77

31.028

31.046

HTTP2.0 with SSL

30.804

18.392

30.852

HTTP2.0 with SSL and Push

30.74

15.788

30.744

clip_image020

clip_image022

clip_image024

Conclusion: Here also we can see that HTTP2 Push shown speed improvement over the all the others.

Final Conclusion:

From all the above three types of tests, we can see that HTTP2 Push clearly had improved page load time. Also it reduces the time the browser spends waiting on the network.