To check the performance of HTTP2 push feature I used a very basic HTML page with some static content like images and to load some CSS and two heavy JavaScript files.
This HTML page will be served by Nginx webserver version 1.15.2 (since Nginx version 1.13.9, HTTP2 push is supported) on Ubuntu 16.04.
HTML file used for testing:
<!DOCTYPE html><html lang="en">
<head>
<title>My HTTP2 Test</title>
<script src="js/angular.js"></script>
<script src="js/jquery-latest.js"></script>
<link href="css/RatingStars.css" rel="stylesheet" type="text/css" />
<link href="css/speech.css" rel="stylesheet" type="text/css" />
<link href="css/style.css" rel="stylesheet" type="text/css" />
<link href="css/track.css" rel="stylesheet" type="text/css" />
<link href="css/speech1.css" rel="stylesheet" type="text/css" />
<link href="css/speech2.css" rel="stylesheet" type="text/css" />
<link href="css/speech3.css" rel="stylesheet" type="text/css" />
<link href="css/speech4.css" rel="stylesheet" type="text/css" />
<link href="css/speech5.css" rel="stylesheet" type="text/css" />
<link href="css/speech6.css" rel="stylesheet" type="text/css" />
<link href="css/speech7.css" rel="stylesheet" type="text/css" />
<link href="css/speech8.css" rel="stylesheet" type="text/css" />
<link href="css/speech9.css" rel="stylesheet" type="text/css" />
<link href="css/speech10.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div>
<h1>Images</h1>
<div><img src="images/1.jpg" /></div>
<div><img src="images/2.jpg" /></div>
<div><img src="images/3.jpg" /></div>
<div><img src="images/4.jpg" /></div>
<div><img src="images/5.jpg" /></div>
<div><img src="images/6.jpg" /></div>
<div><img src="images/7.jpg" /></div>
<div><img src="images/8.jpg" /></div>
<div><img src="images/9.jpg" /></div>
<div><img src="images/10.jpg" /></div>
</div>
</body>
Nginx configuration:
For simple static file testing I used the http2_push directive in Nginx configuration file.Note: By default Nginx limits number of concurrent push requests in a connection to 10. As the number of objects that we will be pushing is more than 10 for this test, so I will be changing the http2_max_concurrent_pushes directive to some desired value. For this test I am setting this to say 50. Set this in the http section of the nginx config file
http2_max_concurrent_pushes 50;
Test Scenarios
I took four different scenarios for this benchmarking test1. Webserver will serve the page over simple HTTP using protocol HTTP/1.1
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location /index.html {
try_files $uri $uri/ =404;
}
}
2. Webserver will serve the page over HTTPS using protocol HTTP/1.1
server {
listen 80 default_server;
listen [::]:80 default_server;
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_certificate /etc/nginx/ssl/star_mkcl_org.crt;
ssl_certificate_key /etc/nginx/ssl/mkcl-org.key;
ssl_stapling on;
ssl_stapling_verify on;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location /index.html {
try_files $uri $uri/ =404;
}
}
3. Webserver will serve the page over HTTPS using protocol HTTP/2
server {
listen 80 default_server;
listen [::]:80 default_server;
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_certificate /etc/nginx/ssl/star_mkcl_org.crt;
ssl_certificate_key /etc/nginx/ssl/mkcl-org.key;
ssl_stapling on;
ssl_stapling_verify on;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location /index.html {
try_files $uri $uri/ =404;
}
}
4. Webserver will serve the page using protocol HTTP2 with Push feature.
server {
listen 80 default_server;
listen [::]:80 default_server;
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_certificate /etc/nginx/ssl/star_mkcl_org.crt;
ssl_certificate_key /etc/nginx/ssl/mkcl-org.key;
ssl_stapling on;
ssl_stapling_verify on;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location /index.html {
http2_push /css/RatingStars.css;
http2_push /css/speech.css;
http2_push /css/style.css;
http2_push /css/track.css;
http2_push /css/speech1.css;
http2_push /css/speech2.css;
http2_push /css/speech3.css;
http2_push /css/speech4.css;
http2_push /css/speech5.css;
http2_push /css/speech6.css;
http2_push /css/speech7.css;
http2_push /css/speech8.css;
http2_push /css/speech9.css;
http2_push /css/speech10.css;
http2_push /js/angular.js;
http2_push /js/jquery-latest.js;
http2_push /images/1.jpg;
http2_push /images/2.jpg;
http2_push /images/3.jpg;
http2_push /images/4.jpg;
http2_push /images/5.jpg;
http2_push /images/6.jpg;
http2_push /images/7.jpg;
http2_push /images/8.jpg;
http2_push /images/9.jpg;
http2_push /images/10.jpg;
try_files $uri $uri/ =404;
}
}
Tests over internet connections are not that reliable. As I found two same test results gave big variation for the same configuration (e.g. one test gave 32 second load time and another test gave 7 second load time for the same scenario using internet connection).
So I decided to run the tests in two different environments:
· Hosted the above HTML page in public server and run the page load tests from Google Chrome and WebPageTest.org
· Hosted the HTML file in virtualbox and run the tests using Google chrome. Here I can get stable and reliable network link between the browser and the webserver. To simulate over the internet scenario, I used the tool Wondershaper to limit the incoming and outgoing bandwidth of the virtualbox VM to 1024kbps.
Test results:
Test run on Public Server using WebPageTest.org tool:
HTTP2 with Push
I run this test for 7 times and got the below results:Performance Results (Median Run)
| Document Complete | Fully Loaded | ||||||||||
| Load Time | First Byte | Start Render | Speed Index | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 1) | 6.925s | 0.777s | 7.000s | 7000 | 6.925s | 28 | 3,325 KB | 7.144s | 29 | 3,326 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 7.032s | 0.882s | 7.100s | 7100 | > 7.077s | 7.032s | 28 | 3,325 KB | 7.241s | 29 | 3,326 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 6.857s | 0.697s | 6.900s | 6900 | > 6.898s | 6.857s | 28 | 3,325 KB | 7.053s | 29 | 3,326 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 3) | 7.077s | 0.848s | 7.100s | 7100 | > 7.122s | 7.077s | 28 | 3,325 KB | 7.279s | 29 | 3,326 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 6.900s | 0.751s | 7.000s | 7000 | > 6.945s | 6.900s | 28 | 3,325 KB | 7.089s | 29 | 3,326 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 1) | 7.223s | 0.753s | 7.200s | 7200 | > 7.275s | 7.223s | 28 | 3,325 KB | 7.421s | 29 | 3,326 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 7.049s | 0.762s | 7.100s | 7100 | > 7.098s | 7.049s | 28 | 3,325 KB | 7.258s | 29 | 3,326 KB | $$$$$ |
Considering the Fully loaded time of the webpage, we will calculate the average time: 7.212
HTTP2 without PUSH
I run this test for 3 times and got the below results:Performance Results (Median Run)
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 7.119s | 0.740s | 7.100s | 7100 | > 7.168s | 7.119s | 28 | 3,327 KB | 7.326s | 29 | 3,329 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 1) | 7.172s | 0.739s | 7.200s | 7200 | > 7.224s | 7.172s | 28 | 3,327 KB | 7.384s | 29 | 3,329 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 7.118s | 0.702s | 7.200s | 7200 | > 7.166s | 7.118s | 28 | 3,327 KB | 7.315s | 29 | 3,329 KB | $$$$$ |
Considering the Fully loaded time of the webpage, we will calculate the average time: 7.342
HTTPS without HTTP2
I run this test for 3 times and got the below results:Performance Results (Median Run)
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 3) | 7.170s | 0.720s | 7.200s | 7200 | > 7.217s | 7.170s | 28 | 3,327 KB | 7.385s | 29 | 3,329 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 1) | 7.196s | 0.801s | 7.200s | 7200 | > 7.251s | 7.196s | 28 | 3,327 KB | 7.407s | 29 | 3,329 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 3) | 7.143s | 0.722s | 7.200s | 7200 | > 7.187s | 7.143s | 28 | 3,327 KB | 7.342s | 29 | 3,329 KB | $$$$$ |
Considering the Fully loaded time of the webpage, we will calculate the average time: 7.378
HTTP 1.1 without HTTPS
I run this test for 8 times and got the below results:Performance Results (Median Run)
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 8.409s | 0.442s | 3.800s | 3800 | 3.806s | 8.409s | 27 | 3,327 KB | 8.605s | 28 | 3,328 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 6.760s | 0.410s | 3.400s | 3413 | 3.365s | 6.760s | 27 | 3,327 KB | 6.963s | 28 | 3,328 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 6.567s | 0.428s | 4.100s | 4100 | 4.110s | 6.567s | 27 | 3,327 KB | 6.768s | 28 | 3,328 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 1) | 6.601s | 0.407s | 4.000s | 4000 | 3.963s | 6.601s | 27 | 3,327 KB | 6.796s | 28 | 3,328 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 1) | 9.671s | 0.416s | 6.700s | 6700 | 6.740s | 9.671s | 27 | 3,327 KB | 9.870s | 28 | 3,328 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 1) | 6.765s | 0.443s | 3.800s | 3800 | 3.806s | 6.765s | 27 | 3,327 KB | 6.969s | 28 | 3,328 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 10.586s | 0.421s | 5.000s | 5000 | 5.055s | 10.586s | 27 | 3,327 KB | 10.781s | 28 | 3,328 KB | $$$$$ |
| Document Complete | Fully Loaded | |||||||||||
| Load Time | First Byte | Start Render | Speed Index | First Interactive (beta) | Time | Requests | Bytes In | Time | Requests | Bytes In | Cost | |
| First View (Run 2) | 6.553s | 0.409s | 3.800s | 3800 | 3.761s | 6.553s | 27 | 3,327 KB | 6.765s | 28 | 3,328 KB | $$$$$ |
Considering the Fully loaded time of the webpage, we will calculate the average time: 7.9395
Final result of WebPageTest.org tool:
Here I considered the Full page load time returned by the tool.
For each test scenarios, I run the tool minimum 3 times and maximum 8 times and calculated the average of the results that I got.
The results are:
| Scenario | Fully loaded time (secs) |
| HTTP1.1 without SSL | 7.9395 |
| HTTP1.1 with SSL | 7.378 |
| HTTP2.0 with SSL | 7.342 |
| HTTP2.0 with SSL and Push | 7.212 |
Conclusion: From the above results that we got from the WebPageTest.org tool, we can see that HTTP2 with Push feature has speed advantage over others.
Test run on Public Server using Google Chrome browser and using the developer tool:
First I tried to run the tests over our broadband connections, as broadband connections are not that reliable and I started to get weird results because of network link fluctuations. So I decided to use mobile phone 4G network for this test.
· HTTP1.1 without SSL
· HTTP1.1 with SSL
· HTTP2.0 with SSL
· HTTP2.0 with SSL and Push
| Test1 | |||||
| Requests | Data Transfer (MB) | Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 28 | 3.3 | 10.91 | 8.36 | 10.79 |
| HTTP1.1 with SSL | 28 | 3.3 | 32.19 | 32.06 | 32.07 |
| HTTP2.0 with SSL | 28 | 3.3 | 17.04 | 11.56 | 17.31 |
| HTTP2.0 with SSL and Push | 28 | 3.3 | 7.74 | 7.63 | 7.64 |
| Test2 | |||||
| Requests | Data Transfer (MB) | Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 28 | 3.3 | 16.90 | 16.76 | 16.77 |
| HTTP1.1 with SSL | 28 | 3.3 | 7.23 | 5.82 | 7.12 |
| HTTP2.0 with SSL | 28 | 3.3 | 10.70 | 6.85 | 10.60 |
| HTTP2.0 with SSL and Push | 28 | 3.3 | 5.78 | 4.59 | 5.68 |
| Test3 | |||||
| Requests | Data Transfer (MB) | Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 28 | 3.3 | 10.12 | 7.64 | 10 |
| HTTP1.1 with SSL | 28 | 3.3 | 10.39 | 10.25 | 10.26 |
| HTTP2.0 with SSL | 28 | 3.3 | 7.25 | 5.65 | 7.16 |
| HTTP2.0 with SSL and Push | 28 | 3.3 | 5.79 | 5.65 | 5.66 |
| Test 4 | |||||
| Requests | Data Transfer (MB) | Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 28 | 3.3 | 7.42 | 7.02 | 7.31 |
| HTTP1.1 with SSL | 28 | 3.3 | 10.32 | 10.18 | 10.19 |
| HTTP2.0 with SSL | 28 | 3.3 | 9.15 | 6.43 | 9.03 |
| HTTP2.0 with SSL and Push | 28 | 3.3 | 12.06 | 8.69 | 11.96 |
Averaging the results of the above 4 tests and selecting the Finish, DOMContentLoaded and Load times, we get the following results:
| Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 11.3375 | 9.945 | 11.2175 |
| HTTP1.1 with SSL | 15.0325 | 14.5775 | 14.91 |
| HTTP2.0 with SSL | 11.035 | 7.6225 | 11.025 |
| HTTP2.0 with SSL and Push | 7.8425 | 6.64 | 7.735 |
Conclusion: Here also we can conclude that HTTP2 with Push feature has loaded the page faster than others.
Test run on webpage hosted on virtualbox VM and requests sent using Google chrome.
In ideal scenario, the network should be reliable to get the proper results for benchmarks. But this is hardly possible in public network over internet. So I tried to simulate the scenario, using a tool named Wondershaper in my Ubuntu virtualbox VM. With the help of this tool I was able to limit the incoming and outgoing bandwidth of the virtualbox VM to 1024kbps, which is close to what we get in normal slow internet connections.Results:
| Test1 | |||||
| Requests | Data Transfer (MB) | Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 28 | 3.3 | 30.95 | 30.89 | 30.91 |
| HTTP1.1 with SSL | 28 | 3.3 | 30.25 | 30.50 | 30.51 |
| HTTP2.0 with SSL | 28 | 3.3 | 30.59 | 13.92 | 30.59 |
| HTTP2.0 with SSL and Push | 28 | 3.3 | 30.70 | 15.80 | 30.70 |
| Test2 | |||||
| Requests | Data Transfer (MB) | Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 28 | 3.3 | 31.15 | 31.11 | 31.13 |
| HTTP1.1 with SSL | 28 | 3.3 | 30.89 | 31.16 | 31.18 |
| HTTP2.0 with SSL | 28 | 3.3 | 31.84 | 16.68 | 31.84 |
| HTTP2.0 with SSL and Push | 28 | 3.3 | 30.57 | 15.52 | 30.58 |
| Test3 | |||||
| Requests | Data Transfer (MB) | Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 28 | 3.3 | 30.79 | 26.62 | 30.75 |
| HTTP1.1 with SSL | 28 | 3.3 | 30.85 | 31.18 | 31.19 |
| HTTP2.0 with SSL | 28 | 3.3 | 30.91 | 15.71 | 30.92 |
| HTTP2.0 with SSL and Push | 28 | 3.3 | 30.72 | 15.90 | 30.72 |
| Test4 | |||||
| Requests | Data Transfer (MB) | Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 28 | 3.3 | 31.27 | 31.19 | 31.20 |
| HTTP1.1 with SSL | 28 | 3.3 | 30.51 | 30.75 | 30.77 |
| HTTP2.0 with SSL | 28 | 3.3 | 30.03 | 30.24 | 30.26 |
| HTTP2.0 with SSL and Push | 28 | 3.3 | 31.27 | 16.41 | 31.28 |
| Test5 | |||||
| Requests | Data Transfer (MB) | Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 28 | 3.3 | 30.99 | 30.96 | 30.97 |
| HTTP1.1 with SSL | 28 | 3.3 | 31.35 | 31.55 | 31.58 |
| HTTP2.0 with SSL | 28 | 3.3 | 30.65 | 15.41 | 30.65 |
| HTTP2.0 with SSL and Push | 28 | 3.3 | 30.44 | 15.31 | 30.44 |
Averaging the results of the above 5 tests and selecting the Finish, DOMContentLoaded and Load times, we get the following results:
| Finish (Secs) | DOMContentLoaded (Secs) | Load (Secs) | |
| HTTP1.1 without SSL | 31.03 | 30.154 | 30.992 |
| HTTP1.1 with SSL | 30.77 | 31.028 | 31.046 |
| HTTP2.0 with SSL | 30.804 | 18.392 | 30.852 |
| HTTP2.0 with SSL and Push | 30.74 | 15.788 | 30.744 |
Conclusion: Here also we can see that HTTP2 Push shown speed improvement over the all the others.
