tengine+keepalived搭建高可用负载均衡服务器 tengine+keepalived搭建高可用负载

一、tengine安装过程 1.1 下载
http://tengine.taobao.org/download.html找到下载包并且下载（Tengine-2.2.1.tar.gz）
1.2 解压
tar zxvf Tengine-2.2.1.tar.gz
1.3 配置检查
进入解压后的目录 ./configure
异常1：

[root@bogon tengine]# ./configure checking for OS + Linux 3.10.0-327.el7.x86_64 x86_64 checking for C compiler ... not found ./configure: error: C compiler cc is not found

也就是c编译器 gcc找不到
安装gcc，执行：

yum -y install gcc

【tengine+keepalived搭建高可用负载均衡服务器】继续检查
异常2：

./configure checking for PCRE library ... not found checking for PCRE library in /usr/local/ ... not found checking for PCRE library in /usr/include/pcre/ ... not found checking for PCRE library in /usr/pkg/ ... not found checking for PCRE library in /opt/local/ ... not found ./configure: error: the HTTP rewrite module requires the PCRE library. You can either disable the module by using --without-http_rewrite_module

缺少pcre和pcre-devel 安装命令：

yum -y install pcre pcre-devel

这里安装好后可以通过
rpm -qa | grep pcre找到所有pcre的包
rpm -ql 完整包名查看安装的路径
异常3：

./configure

checking for OpenSSL library ... not found

缺少openssl和openssl-devel 执行安装命令：

yum -y install openssl openssl-devel

继续检测成功，可以在日志看到需要pcre openssl zlib（安装openssl自动安装）的库，可以看出现在只差jemalloc库不可用了，如果没有需要优化tengine内存的话，可以不用安装。

Configuration summary + using system PCRE library + using system OpenSSL library + md5: using OpenSSL library + sha1: using OpenSSL library + using system zlib library + jemalloc library is disablednginx path prefix: "/usr/local/nginx" nginx binary file: "/usr/local/nginx/sbin/nginx" nginx configuration prefix: "/usr/local/nginx/conf" nginx configuration file: "/usr/local/nginx/conf/nginx.conf" nginx pid file: "/usr/local/nginx/logs/nginx.pid" nginx error log file: "/usr/local/nginx/logs/error.log" nginx http access log file: "/usr/local/nginx/logs/access.log" nginx http client request body temporary files: "client_body_temp" nginx dso module path: "/usr/local/nginx/modules/" nginx http proxy temporary files: "proxy_temp" nginx http fastcgi temporary files: "fastcgi_temp" nginx http uwsgi temporary files: "uwsgi_temp" nginx http scgi temporary files: "scgi_temp"

其中jemalloc库若需要的话，也可以安装下，下载 jemalloc-3.6.0.tar.bz2包，解压

tar jvxf jemalloc-3.6.0.tar.bz2

最后将设置tengine的安装路径为/usr/local/tengine，jemalloc模块的路径指定为~/jemalloc-3.6.0/，也就是jemalloc解压的路径，执行下面命令：

./configure --prefix=/usr/local/tengine --user=venky --group=venky --with-http_gzip_static_module --with-http_realip_module --with-http_stub_status_module --with-http_concat_module --with-jemalloc=~/jemalloc-3.6.0/

1.4 安装
配置完成后，执行命令：

make && make install

安装成功后，会在/usr/local/tengine下看到tengine的安装文件。
1.5 配置
两台虚拟tengine主机都按照如下配置。

#usernobody; worker_processes1; events { worker_connections1024; } http { includemime.types; default_typeapplication/octet-stream; sendfileon; keepalive_timeout65; upstream myServer{ server 192.168.216.128:9080; server 192.168.216.135:9080; } server { listen9080; server_namelocalhost; server_name_in_redirect off; proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr; proxy_set_header REMOTE-HOST $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; location / { proxy_passhttp://myServer; } error_page500 502 503 504/50x.html; location = /50x.html { roothtml; } } }

1.6 执行命令
进入/usr/local/tengine/sbin/，执行命令：

./nginx -V

可以看到tengine安装的所有的模块信息。
tengine常用命令以及参数：

nginx -m 显示所有加载的模块 nginx -l 显示所有可以使用的指令 nginx -t 检查nginx的配置文件是否正确 nginx -s 启动nginx nginx -s reload 重启nginx nginx -s stop 停止nginx

二、Keepalived安装与配置 keepalived的高可用分为两种模式，一种是抢占模式，一种是非抢占模式。抢占模式即MASTER从故障中恢复后，会将VIP从BACKUP节点中抢占过来。非抢占模式即MASTER恢复后不抢占BACKUP升级为MASTER后的VIP。下面分别介绍Red Hat 6.5下抢占模式和非抢占模式的配置方式
2.1 安装keepalived
安装的服务环境是Red Hat Enterprise Linux Server release 6.5，使用yum方式安装

# yum install -y keepalived

查看安装的keepalived版本

# keepalived -v Keepalived v1.2.13 (03/19,2015)

2.2 tengine监控脚本
该脚本的作用是检查tengine节点是否已经停止，如果停止尝试启动tengine，如果无法启动则停止keepalived服务，让其他服务器接管。两台虚拟主机配置相同。

vim /etc/keepalived/check_nginx.sh

#!/bin/bash counter=$(ps -C nginx --no-heading|wc -l) if [ "${counter}" = "0" ]; then /usr/local/bin/nginx sleep 2 counter=$(ps -C nginx --no-heading|wc -l) if [ "${counter}" = "0" ]; then /etc/init.d/keepalived stop fi fi

2.3 keepalived配置
2.3.1 抢占模式配置主机192.168.216.128配置：

! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_script chk_nginx { script "/etc/keepalived/check_nginx.sh" #tengine监控脚本 interval2 weight -5 fall 3 rise 2 } vrrp_instance VI_1 { state MASTER #设置为主节点 interface eth0 virtual_router_id 51 priority 100 #优先级 advert_int 1 #不争抢 #nopreempt authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { #192.168.200.16 #192.168.200.17 192.168.216.188 } track_script { chk_nginx } }

主机192.168.216.135配置：

! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL } cript chk_nginx { script "/etc/keepalived/check_nginx.sh" interval2 weight -5 fall 3 rise 2 } vrrp_instance VI_1 { state BACKUP #设置为从节点 interface eth0 virtual_router_id 51 priority 90 #优先级要小于主节点 advert_int 1 #nopreempt authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { #192.168.200.16 #192.168.200.17 192.168.216.188 } track_script { chk_nginx } }

两台虚拟主机配置了同一个vip，都是192.168.216.188。
启动keepalived服务

# service keepalived restart # ps -ef | grep keepalived root5362610 14:02 ?00:00:02 /usr/sbin/keepalived -D root53628536260 14:02 ?00:00:11 /usr/sbin/keepalived -D root53629536260 14:02 ?00:00:30 /usr/sbin/keepalived -D

如果看到如上进程信息，表示keepalived已经启动成功。下面用ip add命令查看vip绑定的情况，如下图所示：

文章图片
查看vip地址绑定
从上图可以看出，vip地址192.168.216.188绑定在MASTER（192.168.216.128）的eth0网卡上。
测试主机发生故障
停止在128机器上的主节点的keepalived节点服务。

# service keepalived stop # ip addr

可以发现，停止主节点后vip的地址立刻从128主机飘到135主机上。

文章图片
查看vip绑定地址
再次启动128上的keepalived服务。

# service keepalived start # ip addr

可以发现，VIP又再次回到128的主机上。

文章图片
128主机vpi绑定 2.3.2 非抢占模式配置
非抢占模式和抢占模式的不同的地方是，非抢占模式master从故障中恢复后，不会抢占备份节点的vip。

! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_script chk_nginx { script "/etc/keepalived/check_nginx.sh" #tengine监控脚本 interval2 weight -5 fall 3 rise 2 } vrrp_instance VI_1 { state BACKUP #设置为从节点 interface eth0 virtual_router_id 51 priority 100 #优先级 advert_int 1 #不争抢 nopreempt authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { #192.168.200.16 #192.168.200.17 192.168.216.188 } track_script { chk_nginx } }

和非抢占模式的配置相比，只改了两个地方：

在vrrp_instance块下两个节点各增加了nopreempt指令，表示不争抢vip
节点的state都为BACKUP
两个keepalived节点都启动后，默认都是BACKUP状态，双方在发送组播信息后，会根据优先级来选举一个MASTER出来。由于两者都配置了nopreempt，所以MASTER从故障中恢复后，不会抢占vip。这样会避免VIP切换可能造成的服务延迟。

不懂运维的程序员不是好的吉他手o(￣︶￣)o