LVS负载均衡之tun模式

lvs常用的模式就三种,分别是DR、TUN和NAT。其中DR模式的性能最好,但需要Director和RS至少能有在同一VLAN下直接连接,比较适合一个CDN节点下的使用,作为顶层的负载设备对haproxy集群进行负载均衡,haproxy集群通过url hash提高缓存的命中率。NAT模式因为进出的流量都要通过Director,所以如果不使用万兆网卡的本身的网络是瓶颈,而且NAT也会比较耗性能一些,还需要把RS的网关指向Director,实用的价值不是太大,不过现在淘宝做的fullnat还比较好,把部署的架构难度降低了,但是官方的内核和keepalived都还没有合并进去,而且也只有2.6.32 rhel版本内核才能跑,广泛实用性也不是很大。TUN模式其实是从DR模式演化来的,主要是解决了Director和RS跨网段的情况。

 

 

其结构比较简单,当用户发出来包达到Director的时候,会把请求的包封装进一个IPIP包,然后发给一个RS,RS接受到包后解包还原成原始的包,然后再进行进一步的处理。需要注意的是Director上不是用内核的ipip处理函数进行标准的封转。

LVS-Tun is an LVS original. It is based on LVS-DR. The LVS code encapsulates the original packet (CIP->VIP) inside an ipip packet of DIP->RIP, which is then put into the OUTPUT chain, where it is routed to the realserver. (There is no tunl0 device on the director; ip_vs() does its own encapsulation and doesn’t use the standard kernel ipip code. This possibly is the reason why PMTU on the director does not work for LVS-Tun – see MTU.) The realserver receives the packet on a tunl0 device (seeneed tunl0 device) and decapsulates the ipip packet, revealing the original CIP->VIP packet.

以上是LVS   HOWTO上的原文。

简单的配置一下tun模式的双机互备结构,如果机器不够就把备机撤掉。

1.先进行简单的安装,在2台服务器上编译keepalived安装。

wget http://www.keepalived.org/software/keepalived-1.2.7.tar.gz
tar zxvf keepalived-1.2.7.tar.gz
cd keepalived-1.2.7
./configure –prefix=/opt/keepalived && make -j 10 && make install

2.主备 Director的配置

主机的配置如下:

! Configuration File for keepalived

!global_defs {
!  notification_email {
!  
acassen@firewall.loc
!     failover@firewall.loc
!     sysadmin@firewall.loc
!   }
!   notification_email_from
Alexandre.Cassen@firewall.loc
!   smtp_server 192.168.200.1
!   smtp_connect_timeout 30
!   router_id LVS_DEVEL
!}

vrrp_instance VI_1 {
    state MASTER
    interface eth1 #vrrp通信使用的网卡,一定是内部可以和其他服务器通信的网卡。
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
 10.253.85.250 dev eth1 label eth1:1 #如果有双网卡这里也可以写    10.253.85.250 dev eth0 label eth0:1

    }
}

virtual_server  10.253.85.250 80 {
    delay_loop 6
    lb_algo rr #随机轮询
    lb_kind TUN #DR模式
    nat_mask 255.255.255.0
    persistence_timeout 0  #为了方便测试故意关闭会话的保持,根据自己需要来设置
    protocol TCP
    alpha #alpah模式是新增的RS都是down,检测后再开启
    ha_suspend #非MASTER不进行健康检查。
    real_server  10.253.26.1 80{
        weight 1
  TCP_CHECK {
     connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server  10.253.26.2 80{
        weight 1
  TCP_CHECK {
     connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server  10.253.26.3 80{
        weight 1
  TCP_CHECK {
     connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

备机的配置:

! Configuration File for keepalived

!global_defs {
!  notification_email {
!   acassen@firewall.loc
!     failover@firewall.loc
!     sysadmin@firewall.loc
!   }
!   notification_email_from Alexandre.Cassen@firewall.loc
!   smtp_server 192.168.200.1
!   smtp_connect_timeout 30
!   router_id LVS_DEVEL
!}

vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
 10.253.85.250 dev eth1 label eth1:1
    }
}

virtual_server  10.253.85.250 80 {
    delay_loop 6
    lb_algo rr
    lb_kind TUN
    nat_mask 255.255.255.0
    persistence_timeout 0
    protocol TCP
    alpha
    ha_suspend
    real_server  10.253.26.1 80{
        weight 1
  TCP_CHECK {
     connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server  10.253.26.2 80{
        weight 1
  TCP_CHECK {
     connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server  10.253.26.3 80{
        weight 1
  TCP_CHECK {
     connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}
主备都直接启动起来:

 /opt/keepalived/sbin/keepalived -f /opt/keepalived/etc/keepalived/keepalived_tun.conf -D

3. 三台RS上绑定好VIP,脚本如下

#!/bin/bash
VIP=’10.253.85.250′
case $1 in
start)
modprobe -r ipip
modprobe ipip
ip link set tunl0 up
ip link set tunl0 arp off
for IP in $VIP
do
        NO=$((NO+1))
        ip addr add $IP/32 br $IP label tunl0:$NO dev tunl0
        ip route add $IP/32 dev tunl0
done
echo 1 > /proc/sys/net/ipv4/conf/tunl0/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/tunl0/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
;;
stop)
echo 0 > /proc/sys/net/ipv4/conf/tunl0/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/tunl0/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
modprobe -r ipip
;;
*)
echo “$0: Usage: $0 {start|stop|status}”
exit 1
;;
esac

 

4. 停止keepalived 需要向主进程发送TERM信号。

ps -C keepalived u |awk ‘{print $2}’|sort|head -n 1|xargs kill -TERM

可以先把备机启动,然后启动主机的时候看到VIP被备机踢出,并在主机上绑定,当主机挂掉的时候也会再切换到备机上面。

如果不希望主机恢复后VIP从备机切换回主机,可以做成双备的模式,都是BACKUP模式,然后设置  nopreempt。  nopreempt的说明如下:

           # VRRP will normally preempt a lower priority
           # machine when a higher priority machine comes
           # online.  “nopreempt” allows the lower priority
           # machine to maintain the master role, even when
           # a higher priority machine comes back online.
           # NOTE: For this to work, the initial state of this
           # entry must be BACKUP.
  

我分别对RS和VIP进行了一下压测。

直接压测RS的结果如下:

Webbench – Simple Web Benchmark 1.5
Copyright (c) Radim Kolar 1997-2004, GPL Open Source Software.

Benchmarking: GET http://10.253.26.2/
1000 clients, running 30 sec.

Speed=465536 pages/min, 1838409 bytes/sec.
Requests: 232710 susceed, 58 failed.

直接压测VIP的结果如下:

Webbench – Simple Web Benchmark 1.5
Copyright (c) Radim Kolar 1997-2004, GPL Open Source Software.

Benchmarking: GET http://10.253.85.250/
1000 clients, running 30 sec.

Speed=1637384 pages/min, 6467578 bytes/sec.
Requests: 818692 susceed, 0 failed.

简单的看似乎是通过VIP的QPS和三个RS的和基本差不多。

 

此条目发表在keepalived分类目录。将固定链接加入收藏夹。

发表评论