nginx和haproxy代理时对后端的长连接分析

由于代理服务器上开启了长连接后，可以减少每次与后端服务器的三次握手，提高后端服务器的效率。所以nginx在1.1.4版本中终于增加了upstream对keepalive的支持，而haproxy原本对后端是支持长连接的（但是这个时候由于不能把client的IP加到header转发给后端服务器造成会丢失客户端的IP）。今天简单测试了一下。基本的结构是A:Client –> nginx(proxy) –>nginx(server)B:Client –> haproxy(proxy) –>nginx(server)

对比的结构

Client –>nginx(server)

先测试一下client直接和server端能否保持住长连接，我是直接在server端抓包观察，确定OK后再进行nginx和haproxy的测试。

nginx proxy的配置如下：

server {

location / {root   html;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_intercept_errors on;
proxy_set_header Host      $http_host;
proxy_set_header ORIG_CLIENT_IP  $remote_addr;
index index.html,index.htm;
proxy_pass http://httpd;
}
0172.189.085.161.50970-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50970: HTTP/1.1 304 Not Modified0172.189.085.161.50971-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50971: HTTP/1.1 404 Not Found0172.189.085.161.50972-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50972: HTTP/1.1 304 Not Modified0172.189.085.161.50973-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50973: HTTP/1.1 404 Not Found0172.189.085.161.50974-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50974: HTTP/1.1 304 Not Modified0172.189.085.161.50975-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50975: HTTP/1.1 404 Not Found0172.189.085.161.50976-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50976: HTTP/1.1 304 Not Modified0172.189.085.161.50977-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50977: HTTP/1.1 404 Not Found0172.189.085.161.50978-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50978: HTTP/1.1 304 Not Modified0172.189.085.161.50979-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50979: HTTP/1.1 404 Not Found0172.189.085.161.50980-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50980: HTTP/1.1 304 Not Modified0172.189.085.161.50981-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50981: HTTP/1.1 404 Not Found0172.189.085.161.50982-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50982: HTTP/1.1 304 Not Modified0172.189.085.161.50983-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50983: HTTP/1.1 404 Not Found0172.189.085.161.50984-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50984: HTTP/1.1 304 Not Modified0172.189.085.161.50985-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50985: HTTP/1.1 404 Not Found0172.189.085.161.50986-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50986: HTTP/1.1 304 Not Modified0172.189.085.161.50987-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50987: HTTP/1.1 404 Not Found0172.189.085.161.50988-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50988: HTTP/1.1 304 Not Modified0172.189.085.161.50989-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50989: HTTP/1.1 404 Not Found0172.189.085.161.50990-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50990: HTTP/1.1 304 Not Modified0172.189.085.161.50991-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50991: HTTP/1.1 404 Not Found0172.189.085.161.50992-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50992: HTTP/1.1 304 Not Modified0172.189.085.161.50993-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50993: HTTP/1.1 404 Not Found0172.189.085.161.50994-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50994: HTTP/1.1 304 Not Modified0172.189.085.161.50995-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50995: HTTP/1.1 404 Not Found0172.189.085.161.50996-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50996: HTTP/1.1 304 Not Modified0172.189.085.161.50997-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50997: HTTP/1.1 404 Not Found
server {
```bash
location / {root   html;proxy_set_header Connection "";proxy_http_version 1.1;proxy_intercept_errors on;proxy_set_header Host      $http_host;proxy_set_header ORIG_CLIENT_IP  $remote_addr;index index.html,index.htm;proxy_pass http://httpd;}

0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified
然后我又测试了一下haproxy。主要是之前对haproxy的理解有点误区。一方面doc里面说了

```bash
A last improvement in the communications is the pipelining mode. It still useskeep-alive, but the client does not wait for the first response to send thesecond request. This is useful for fetching large number of images composing apage :

[CON] [REQ1] [REQ2] … [RESP1] [RESP2] [CLO] …

This can obviously have a tremendous benefit on performance because the networklatency is eliminated between subsequent requests. Many HTTP agents do notcorrectly support pipelining since there is no way to associate a response withthe corresponding request in HTTP. For this reason, it is mandatory for theserver to reply in the exact same order as the requests were received.

不过抓包后直接看到了当访问1689端口的时候对后端服务器的连接是一直保持的，但是显示不出客户端的IP（没有转发ORIG_CLIENT_IP）,0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.10172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.10172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found 访问1690端口，即是配置了httpclose的时候发现能显示客户端的IP，但是每个连接都在请求完成后就关闭了，抓包的记录就不贴了。看来有问题还是需要自己抓包来实际分析才知道究竟是怎么个情况。

后记：

今天统计了一下做代理的nginx似乎每个链接对后端请求处理100个后就断掉了，仔细查了一下wiki发现这个实际上是后端nginx的默认设置的问题。可以吧后端的nginx的配置改成keepalive_requests 1000,这样代理服务器向后端的连接能每个处理1000个再中断。