CopyDisable

Saturday, 11 August 2018

HTTP2 Push benchmarking


To check the performance of HTTP2 push feature I used a very basic HTML page with some static content like images and to load some CSS and two heavy JavaScript files.
This HTML page will be served by Nginx webserver version 1.15.2 (since Nginx version 1.13.9, HTTP2 push is supported) on Ubuntu 16.04.

HTML file used for testing:

<!DOCTYPE html>
<html lang="en">
<head>
<title>My HTTP2 Test</title>
<script src="js/angular.js"></script>
<script src="js/jquery-latest.js"></script>
<link href="css/RatingStars.css" rel="stylesheet" type="text/css" />
<link href="css/speech.css" rel="stylesheet" type="text/css" />
<link href="css/style.css" rel="stylesheet" type="text/css" />
<link href="css/track.css" rel="stylesheet" type="text/css" />
<link href="css/speech1.css" rel="stylesheet" type="text/css" />
<link href="css/speech2.css" rel="stylesheet" type="text/css" />
<link href="css/speech3.css" rel="stylesheet" type="text/css" />
<link href="css/speech4.css" rel="stylesheet" type="text/css" />
<link href="css/speech5.css" rel="stylesheet" type="text/css" />
<link href="css/speech6.css" rel="stylesheet" type="text/css" />
<link href="css/speech7.css" rel="stylesheet" type="text/css" />
<link href="css/speech8.css" rel="stylesheet" type="text/css" />
<link href="css/speech9.css" rel="stylesheet" type="text/css" />
<link href="css/speech10.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div>
<h1>Images</h1>
<div><img src="images/1.jpg" /></div>
<div><img src="images/2.jpg" /></div>
<div><img src="images/3.jpg" /></div>
<div><img src="images/4.jpg" /></div>
<div><img src="images/5.jpg" /></div>
<div><img src="images/6.jpg" /></div>
<div><img src="images/7.jpg" /></div>
<div><img src="images/8.jpg" /></div>
<div><img src="images/9.jpg" /></div>
<div><img src="images/10.jpg" /></div>
</div>
</body>

Nginx configuration:

For simple static file testing I used the http2_push directive in Nginx configuration file.
Note: By default Nginx limits number of concurrent push requests in a connection to 10. As the number of objects that we will be pushing is more than 10 for this test, so I will be changing the http2_max_concurrent_pushes directive to some desired value. For this test I am setting this to say 50. Set this in the http section of the nginx config file
http2_max_concurrent_pushes 50;

Test Scenarios

I took four different scenarios for this benchmarking test
1. Webserver will serve the page over simple HTTP using protocol HTTP/1.1
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location /index.html {
try_files $uri $uri/ =404;
}
}
2. Webserver will serve the page over HTTPS using protocol HTTP/1.1
server {
listen 80 default_server;
listen [::]:80 default_server;
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_certificate /etc/nginx/ssl/star_mkcl_org.crt;
ssl_certificate_key /etc/nginx/ssl/mkcl-org.key;
ssl_stapling on;
ssl_stapling_verify on;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location /index.html {
try_files $uri $uri/ =404;
}
}
3. Webserver will serve the page over HTTPS using protocol HTTP/2
server {
listen 80 default_server;
listen [::]:80 default_server;
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_certificate /etc/nginx/ssl/star_mkcl_org.crt;
ssl_certificate_key /etc/nginx/ssl/mkcl-org.key;
ssl_stapling on;
ssl_stapling_verify on;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location /index.html {
try_files $uri $uri/ =404;
}
}
4. Webserver will serve the page using protocol HTTP2 with Push feature.
server {
listen 80 default_server;
listen [::]:80 default_server;
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_certificate /etc/nginx/ssl/star_mkcl_org.crt;
ssl_certificate_key /etc/nginx/ssl/mkcl-org.key;
ssl_stapling on;
ssl_stapling_verify on;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location /index.html {
http2_push /css/RatingStars.css;
http2_push /css/speech.css;
http2_push /css/style.css;
http2_push /css/track.css;
http2_push /css/speech1.css;
http2_push /css/speech2.css;
http2_push /css/speech3.css;
http2_push /css/speech4.css;
http2_push /css/speech5.css;
http2_push /css/speech6.css;
http2_push /css/speech7.css;
http2_push /css/speech8.css;
http2_push /css/speech9.css;
http2_push /css/speech10.css;
http2_push /js/angular.js;
http2_push /js/jquery-latest.js;
http2_push /images/1.jpg;
http2_push /images/2.jpg;
http2_push /images/3.jpg;
http2_push /images/4.jpg;
http2_push /images/5.jpg;
http2_push /images/6.jpg;
http2_push /images/7.jpg;
http2_push /images/8.jpg;
http2_push /images/9.jpg;
http2_push /images/10.jpg;
try_files $uri $uri/ =404;
}
}
Tests over internet connections are not that reliable. As I found two same test results gave big variation for the same configuration (e.g. one test gave 32 second load time and another test gave 7 second load time for the same scenario using internet connection).
So I decided to run the tests in two different environments:
· Hosted the above HTML page in public server and run the page load tests from Google Chrome and WebPageTest.org
· Hosted the HTML file in virtualbox and run the tests using Google chrome. Here I can get stable and reliable network link between the browser and the webserver. To simulate over the internet scenario, I used the tool Wondershaper to limit the incoming and outgoing bandwidth of the virtualbox VM to 1024kbps.

Test results:

Test run on Public Server using WebPageTest.org tool:



I have selected the Test Location Ireland and Browser as Chrome for my tests.
HTTP2 with Push
I run this test for 7 times and got the below results:
Performance Results (Median Run)
Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexTimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 1)6.925s0.777s7.000s70006.925s283,325 KB7.144s293,326 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)7.032s0.882s7.100s7100> 7.077s7.032s283,325 KB7.241s293,326 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)6.857s0.697s6.900s6900> 6.898s6.857s283,325 KB7.053s293,326 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 3)7.077s0.848s7.100s7100> 7.122s7.077s283,325 KB7.279s293,326 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)6.900s0.751s7.000s7000> 6.945s6.900s283,325 KB7.089s293,326 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 1)7.223s0.753s7.200s7200> 7.275s7.223s283,325 KB7.421s293,326 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)7.049s0.762s7.100s7100> 7.098s7.049s283,325 KB7.258s293,326 KB$$$$$

Considering the Fully loaded time of the webpage, we will calculate the average time: 7.212
HTTP2 without PUSH
I run this test for 3 times and got the below results:

Performance Results (Median Run)
Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)7.119s0.740s7.100s7100> 7.168s7.119s283,327 KB7.326s293,329 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 1)7.172s0.739s7.200s7200> 7.224s7.172s283,327 KB7.384s293,329 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)7.118s0.702s7.200s7200> 7.166s7.118s283,327 KB7.315s293,329 KB$$$$$

Considering the Fully loaded time of the webpage, we will calculate the average time: 7.342
HTTPS without HTTP2
I run this test for 3 times and got the below results:
Performance Results (Median Run)
Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 3)7.170s0.720s7.200s7200> 7.217s7.170s283,327 KB7.385s293,329 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 1)7.196s0.801s7.200s7200> 7.251s7.196s283,327 KB7.407s293,329 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 3)7.143s0.722s7.200s7200> 7.187s7.143s283,327 KB7.342s293,329 KB$$$$$

Considering the Fully loaded time of the webpage, we will calculate the average time: 7.378
HTTP 1.1 without HTTPS
I run this test for 8 times and got the below results:
Performance Results (Median Run)
Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)8.409s0.442s3.800s38003.806s8.409s273,327 KB8.605s283,328 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)6.760s0.410s3.400s34133.365s6.760s273,327 KB6.963s283,328 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)6.567s0.428s4.100s41004.110s6.567s273,327 KB6.768s283,328 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 1)6.601s0.407s4.000s40003.963s6.601s273,327 KB6.796s283,328 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 1)9.671s0.416s6.700s67006.740s9.671s273,327 KB9.870s283,328 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 1)6.765s0.443s3.800s38003.806s6.765s273,327 KB6.969s283,328 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)10.586s0.421s5.000s50005.055s10.586s273,327 KB10.781s283,328 KB$$$$$


Document CompleteFully Loaded
Load TimeFirst ByteStart RenderSpeed IndexFirst Interactive (beta)TimeRequestsBytes InTimeRequestsBytes InCost
First View (Run 2)6.553s0.409s3.800s38003.761s6.553s273,327 KB6.765s283,328 KB$$$$$

Considering the Fully loaded time of the webpage, we will calculate the average time: 7.9395
Final result of WebPageTest.org tool:
Here I considered the Full page load time returned by the tool.
For each test scenarios, I run the tool minimum 3 times and maximum 8 times and calculated the average of the results that I got.
The results are:
ScenarioFully loaded time (secs)
HTTP1.1 without SSL7.9395
HTTP1.1 with SSL7.378
HTTP2.0 with SSL7.342
HTTP2.0 with SSL and Push7.212






Conclusion: From the above results that we got from the WebPageTest.org tool, we can see that HTTP2 with Push feature has speed advantage over others.

Test run on Public Server using Google Chrome browser and using the developer tool:

First I tried to run the tests over our broadband connections, as broadband connections are not that reliable and I started to get weird results because of network link fluctuations. So I decided to use mobile phone 4G network for this test.




· HTTP1.1 without SSL
· HTTP1.1 with SSL



· HTTP2.0 with SSL




· HTTP2.0 with SSL and Push
Test1
RequestsData Transfer (MB)Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL283.310.918.3610.79
HTTP1.1 with SSL283.332.1932.0632.07
HTTP2.0 with SSL283.317.0411.5617.31
HTTP2.0 with SSL and Push283.37.747.637.64
Test2
RequestsData Transfer (MB)Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL283.316.9016.7616.77
HTTP1.1 with SSL283.37.235.827.12
HTTP2.0 with SSL283.310.706.8510.60
HTTP2.0 with SSL and Push283.35.784.595.68
Test3
RequestsData Transfer (MB)Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL283.310.127.6410
HTTP1.1 with SSL283.310.3910.2510.26
HTTP2.0 with SSL283.37.255.657.16
HTTP2.0 with SSL and Push283.35.795.655.66
Test 4
RequestsData Transfer (MB)Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL283.37.427.027.31
HTTP1.1 with SSL283.310.3210.1810.19
HTTP2.0 with SSL283.39.156.439.03
HTTP2.0 with SSL and Push283.312.068.6911.96

Averaging the results of the above 4 tests and selecting the Finish, DOMContentLoaded and Load times, we get the following results:
Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL11.33759.94511.2175
HTTP1.1 with SSL15.032514.577514.91
HTTP2.0 with SSL11.0357.622511.025
HTTP2.0 with SSL and Push7.84256.647.735





Conclusion: Here also we can conclude that HTTP2 with Push feature has loaded the page faster than others.

Test run on webpage hosted on virtualbox VM and requests sent using Google chrome.

In ideal scenario, the network should be reliable to get the proper results for benchmarks. But this is hardly possible in public network over internet. So I tried to simulate the scenario, using a tool named Wondershaper in my Ubuntu virtualbox VM. With the help of this tool I was able to limit the incoming and outgoing bandwidth of the virtualbox VM to 1024kbps, which is close to what we get in normal slow internet connections.
Results:
Test1
RequestsData Transfer (MB)Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL283.330.9530.8930.91
HTTP1.1 with SSL283.330.2530.5030.51
HTTP2.0 with SSL283.330.5913.9230.59
HTTP2.0 with SSL and Push283.330.7015.8030.70
Test2
RequestsData Transfer (MB)Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL283.331.1531.1131.13
HTTP1.1 with SSL283.330.8931.1631.18
HTTP2.0 with SSL283.331.8416.6831.84
HTTP2.0 with SSL and Push283.330.5715.5230.58
Test3
RequestsData Transfer (MB)Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL283.330.7926.6230.75
HTTP1.1 with SSL283.330.8531.1831.19
HTTP2.0 with SSL283.330.9115.7130.92
HTTP2.0 with SSL and Push283.330.7215.9030.72
Test4
RequestsData Transfer (MB)Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL283.331.2731.1931.20
HTTP1.1 with SSL283.330.5130.7530.77
HTTP2.0 with SSL283.330.0330.2430.26
HTTP2.0 with SSL and Push283.331.2716.4131.28
Test5
RequestsData Transfer (MB)Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL283.330.9930.9630.97
HTTP1.1 with SSL283.331.3531.5531.58
HTTP2.0 with SSL283.330.6515.4130.65
HTTP2.0 with SSL and Push283.330.4415.3130.44

Averaging the results of the above 5 tests and selecting the Finish, DOMContentLoaded and Load times, we get the following results:
Finish (Secs)DOMContentLoaded (Secs)Load (Secs)
HTTP1.1 without SSL31.0330.15430.992
HTTP1.1 with SSL30.7731.02831.046
HTTP2.0 with SSL30.80418.39230.852
HTTP2.0 with SSL and Push30.7415.78830.744







Conclusion: Here also we can see that HTTP2 Push shown speed improvement over the all the others.

Final Conclusion:

From all the above three types of tests, we can see that HTTP2 Push clearly had improved page load time. Also it reduces the time the browser spends waiting on the network.