nginx和haproxy代理时对后端的长连接分析

由于代理服务器上开启了长连接后，可以减少每次与后端服务器的三次握手，提高后端服务器的效率。所以nginx在1.1.4版本中终于增加了upstream对keepalive的支持，而haproxy原本对后端是支持长连接的（但是这个时候由于不能把client的IP加到header转发给后端服务器造成会丢失客户端的IP）。今天简单测试了一下。基本的结构是
A:
Client –> nginx(proxy) –>nginx(server)
B:
Client –> haproxy(proxy) –>nginx(server)

对比的结构

Client –>nginx(server)

先测试一下client直接和server端能否保持住长连接，我是直接在server端抓包观察，确定OK后再进行nginx和haproxy的测试。

nginx proxy的配置如下：

[text]
server {

location / {
root html;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_intercept_errors on;
proxy_set_header Host $http_host;
proxy_set_header ORIG_CLIENT_IP $remote_addr;
index index.html,index.htm;
proxy_pass http://httpd;
}

}
upstream httpd {
server 172.189.85.156:1080;
keepalive 4;
least_conn;
}

[/text]
在server端tcpflow抓包后可以看到
[text]
0172.189.085.161.50970-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50970: HTTP/1.1 304 Not Modified
0172.189.085.161.50971-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50971: HTTP/1.1 404 Not Found
0172.189.085.161.50972-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50972: HTTP/1.1 304 Not Modified
0172.189.085.161.50973-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50973: HTTP/1.1 404 Not Found
0172.189.085.161.50974-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50974: HTTP/1.1 304 Not Modified
0172.189.085.161.50975-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50975: HTTP/1.1 404 Not Found
0172.189.085.161.50976-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50976: HTTP/1.1 304 Not Modified
0172.189.085.161.50977-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50977: HTTP/1.1 404 Not Found
0172.189.085.161.50978-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50978: HTTP/1.1 304 Not Modified
0172.189.085.161.50979-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50979: HTTP/1.1 404 Not Found
0172.189.085.161.50980-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50980: HTTP/1.1 304 Not Modified
0172.189.085.161.50981-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50981: HTTP/1.1 404 Not Found
0172.189.085.161.50982-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50982: HTTP/1.1 304 Not Modified
0172.189.085.161.50983-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50983: HTTP/1.1 404 Not Found
0172.189.085.161.50984-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50984: HTTP/1.1 304 Not Modified
0172.189.085.161.50985-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50985: HTTP/1.1 404 Not Found
0172.189.085.161.50986-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50986: HTTP/1.1 304 Not Modified
0172.189.085.161.50987-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50987: HTTP/1.1 404 Not Found
0172.189.085.161.50988-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50988: HTTP/1.1 304 Not Modified
0172.189.085.161.50989-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50989: HTTP/1.1 404 Not Found
0172.189.085.161.50990-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50990: HTTP/1.1 304 Not Modified
0172.189.085.161.50991-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50991: HTTP/1.1 404 Not Found
0172.189.085.161.50992-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50992: HTTP/1.1 304 Not Modified
0172.189.085.161.50993-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50993: HTTP/1.1 404 Not Found
0172.189.085.161.50994-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50994: HTTP/1.1 304 Not Modified
0172.189.085.161.50995-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50995: HTTP/1.1 404 Not Found
0172.189.085.161.50996-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50996: HTTP/1.1 304 Not Modified
0172.189.085.161.50997-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50997: HTTP/1.1 404 Not Found

[/text]
从上面可以看到代理服务器每次连接后端服务器使用的端口都是不同的，这个时候我就郁闷了，难道nginx对后端开启不了长连接？
测试了一两个小时我终于想到是不是加了最小连接数的那个算法导致的。果然把那行去掉后就OK了。
[最新的状况是把 keepalive 4;设置放在least_conn之后就可以了 ]

[text]
server {

}
upstream httpd {
server 172.189.85.156:1080;
keepalive 4;
}

[/text]

[text]
0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified
0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found
0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified
0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found
0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified
0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found
0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified
0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found
0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified

[/text]

然后我又测试了一下haproxy。主要是之前对haproxy的理解有点误区。一方面doc里面说了

A last improvement in the communications is the pipelining mode. It still uses
keep-alive, but the client does not wait for the first response to send the
second request. This is useful for fetching large number of images composing a
page :

[CON] [REQ1] [REQ2] … [RESP1] [RESP2] [CLO] …

This can obviously have a tremendous benefit on performance because the network
latency is eliminated between subsequent requests. Many HTTP agents do not
correctly support pipelining since there is no way to associate a response with
the corresponding request in HTTP. For this reason, it is mandatory for the
server to reply in the exact same order as the requests were received.

By default HAProxy operates in a tunnel-like mode with regards to persistent
connections: for each connection it processes the first request and forwards
everything else (including additional requests) to selected server. Once
established, the connection is persisted both on the client and server
sides. Use “option http-server-close” to preserve client persistent connections
while handling every incoming request individually, dispatching them one after
another to servers, in HTTP close mode. Use “option httpclose” to switch both
sides to HTTP close mode. “option forceclose” and “option
http-pretend-keepalive” help working around servers misbehaving in HTTP close
mode.
另外一方面就是大家都知道haproxy对后端是没有keepalive的，目前连测试版也不支持。。。这个就让我觉得很是疑惑。
今天其实主要是为了测试一下haproxy，配置文件如下

frontend httpf
bind *:1689
default_backend httpd
backend httpd
mode http
option forwardfor header ORIG_CLIENT_IP
balance roundrobin
server ser2 172.189.85.156:1080

frontend httpf2
bind *:1690
default_backend httpd2
backend httpd2
mode http
option httpclose
balance roundrobin
option forwardfor header ORIG_CLIENT_IP
server ser2 172.189.85.156:1080

不过抓包后直接看到了当访问1689端口的时候对后端服务器的连接是一直保持的，但是显示不出客户端的IP（没有转发ORIG_CLIENT_IP）,
[text]
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified
0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found
0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified
0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found
0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified
0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found
0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified
0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found
0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified
0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1
0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found

[/text]

访问1690端口，即是配置了httpclose的时候发现能显示客户端的IP，但是每个连接都在请求完成后就关闭了，抓包的记录就不贴了。
看来有问题还是需要自己抓包来实际分析才知道究竟是怎么个情况。

后记：

今天统计了一下做代理的nginx似乎每个链接对后端请求处理100个后就断掉了，仔细查了一下wiki发现这个实际上是后端nginx的默认设置的问题。
可以吧后端的nginx的配置改成keepalive_requests 1000,这样代理服务器向后端的连接能每个处理1000个再中断。

nginx和haproxy代理时对后端的长连接分析

发表回复取消回复

近期文章

近期评论

归档

分类

其他操作

blog

友情链接

有用的网站

nginx和haproxy代理时对后端的长连接分析

发表回复 取消回复

近期文章

近期评论

归档

分类

其他操作

blog

友情链接

有用的网站

发表回复取消回复