Linux之在CentOS上一次艰难的木马查杀过程网络安全

今天朋友说他一台要准备上线的生产服务器被挂马，特征ps命令找不到进程，top能看到负载最高的一个程序是一个随机的10位字母的东西，kill掉之后自动再次出现一个随机10位字母的进程。
我让他关闭这个机器的外网，内网放开，在局域网中给我一个跳板。等我拿到权限之后进入机器，先按照朋友说的验证了一遍，果然是那样，木马有自我保护自我恢复。
这时候我想到一个问题居然是能自我开机启动，要么/etc/rc.d/{init.d,rc{1,2,3,4,5}.d}/下有启动脚本，要么有cron计划任务。
于是我发现crontab -l是正常，来到/etc/rc.d/init.d下发现了异常有10位字母的启动脚本，脚本内容如下

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 [root@Xd9BdoAkG ~] # cat /etc/rc.d/init.d/fregonnzkq #!/bin/sh # chkconfig: 12345 90 90 # description: fregonnzkq ### BEGIN INIT INFO # Provides:fregonnzkq # Required-Start: # Required-Stop: # Default-Start:1 2 3 4 5 # Default-Stop: # Short-Description:fregonnzkq ### END INIT INFO case $1 in start) /usr/bin/fregonnzkq ; ; stop) ; ; *) /usr/bin/fregonnzkq ; ; esac

看到这我真是佩服这帮人单用户启动模式都不放过啊，尼玛，你这是赶尽杀绝啊。。。。。。。
到了这里我天真的删除了几个这样的启动脚本，然后重启服务器，问题依旧。。。。。。。。。你妹啊。。。不带这样玩的。。。。
到了这时候我多想了下，是不是还有启动脚本？仔细核查发现/etc/rc.d/rc3.d/下还有问题

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 [root@Xd9BdoAkG rc3.d] # ls -lt total 0 lrwxrwxrwx1 root root 20 Sep8 13:21 S90eviykluziy -> .. /init .d /eviykluziy lrwxrwxrwx1 root root 20 Sep8 12:49 S90yuurxgembh -> .. /init .d /yuurxgembh lrwxrwxrwx1 root root 20 Sep8 10:37 S90fregonnzkq -> .. /init .d /fregonnzkq lrwxrwxrwx. 1 root root 15 Sep7 14:08 S85nginx -> .. /init .d /nginx lrwxrwxrwx. 1 root root 24 Sep7 13:55 S99libvirt-guests -> .. /init .d /libvirt-guests lrwxrwxrwx. 1 root root 19 Sep7 13:55 S26haldaemon -> .. /init .d /haldaemon lrwxrwxrwx. 1 root root 19 Sep7 13:54 K10saslauthd -> .. /init .d /saslauthd lrwxrwxrwx. 1 root root 20 Sep7 13:54 S22messagebus -> .. /init .d /messagebus lrwxrwxrwx. 1 root root 14 Sep7 13:51 S55sshd -> .. /init .d /sshd lrwxrwxrwx. 1 root root 18 Sep7 13:51 K15svnserve -> .. /init .d /svnserve lrwxrwxrwx. 1 root root 17 Sep7 13:44 S10network -> .. /init .d /network lrwxrwxrwx. 1 root root 17 Sep7 13:44 S12rsyslog -> .. /init .d /rsyslog lrwxrwxrwx. 1 root root 15 Sep7 13:44 S90crond -> .. /init .d /crond lrwxrwxrwx. 1 root root 19 Sep7 13:44 K75udev-post -> .. /init .d /udev-post lrwxrwxrwx. 1 root root 17 Sep7 13:44 K30postfix -> .. /init .d /postfix lrwxrwxrwx. 1 root root 15 Sep7 13:44 K75netfs -> .. /init .d /netfs lrwxrwxrwx. 1 root root 19 Sep7 13:44 K85mdmonitor -> .. /init .d /mdmonitor lrwxrwxrwx. 1 root root 22 Sep7 13:44 K99lvm2-monitor -> .. /init .d /lvm2-monitor lrwxrwxrwx. 1 root root 15 Sep7 13:44 K80kdump -> .. /init .d /kdump lrwxrwxrwx. 1 root root 18 Sep7 13:44 K92iptables -> .. /init .d /iptables lrwxrwxrwx. 1 root root 19 Sep7 13:44 K92ip6tables -> .. /init .d /ip6tables lrwxrwxrwx. 1 root root 20 Sep7 13:44 K90eyshcjdmzg -> .. /init .d /eyshcjdmzg lrwxrwxrwx. 1 root root 26 Sep7 13:44 K75blk-availability -> .. /init .d /blk-availability lrwxrwxrwx. 1 root root 16 Sep7 13:44 K88auditd -> .. /init .d /auditd lrwxrwxrwx. 1 root root 17 Sep7 13:37 K75ntpdate -> .. /init .d /ntpdate lrwxrwxrwx. 1 root root 20 Sep7 12:15 K50netconsole -> .. /init .d /netconsole lrwxrwxrwx. 1 root root 11 Sep7 12:15 S99local -> .. /rc . local lrwxrwxrwx. 1 root root 15 Sep7 12:15 K89rdisc -> .. /init .d /rdisc lrwxrwxrwx. 1 root root 21 Sep7 12:15 K87restorecond -> .. /init .d /restorecond

看到后我只想把这个木马的作者找到然后说收我做徒弟吧。。。。。。。。
于是我又很傻很天真的删除了这些启动脚本，并且kill了相关进程，希望的太阳没有升起，沉重的打击再次来临，木马再次自我复制自我运行了。。。。启动脚本再次出现了。。
我知道我进入了误区，重新想思路。。。
不知道为什么我瞬间想到了我遗漏了一个地方，cron，对。。。我是crontab -l 来查看的，还有个地方的cron任务不会在这个命令下出现/etc/cron.*

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 [root@Xd9BdoAkG ~] # cat /etc/crontab SHELL= /bin/bash PATH= /sbin : /bin : /usr/sbin : /usr/bin MAILTO=root HOME=/ # For details see man 4 crontabs # Example of job definition: # .---------------- minute (0 - 59) # |.------------- hour (0 - 23) # ||.---------- day of month (1 - 31) # |||.------- month (1 - 12) OR jan,feb,mar,apr ... # ||||.---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat # ||||| # ***** user-name command to be executed * /3 * * * * root /etc/cron .hourly /gcc .sh

你妹啊啊啊啊，不带这样玩的

1 2 3 4 5 6 [root@Xd9BdoAkG ~] # cat /etc/cron.hourly/gcc.sh #!/bin/sh PATH= /bin : /sbin : /usr/bin : /usr/sbin : /usr/local/bin : /usr/local/sbin : /usr/X11R6/bin for i in ` cat /proc/net/dev | grep :| awk -F: { 'print $1' }`; do ifconfig $i up& done cp /lib/libudev .so /lib/libudev .so.6 /lib/libudev .so.6

1 2 3 4 5 6 [root@Xd9BdoAkG ~] # cat /proc/net/dev|grep :|awk -F: {'print $1'} lo em1 em2 em3 em4

我擦，看到这，再次shit，你还知道主动启动网络和外面联系啊。。。。

1 2	`[root@Xd9BdoAkG ~]` `# file /lib/libudev.so` `/lib/libudev` `.so: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked,` `for` `GNU` `/Linux` `2.6.9, not stripped`

居然不是脚本什么的，想看文件具体内容暂时是没时间了。。。。。。。
到了这里我们可以确认有几个地方有问题/lib/libudev.so /etc/cron.hourly/gcc.sh /etc/crontab /etc/rc.d/init.d/ /etc/rc.d/rc3.d/
由于对方发送大量数据包，所以开始采取iptables来封禁，发现这玩意直接把output是 state 为new的drop掉。。。。。。。。不想说了，已经被他玩够了，不在乎多一次。。
通过排查可以肯定/lib/libudev.so是主体。其他是协助运行和自我保护自我复制的实现。既然你是个程序还在系统上，我有root，还搞不定么。为了不再多拖时间，直接查杀了。。

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 [root@Xd9BdoAkG ~] # chmod 0000 /lib/libudev.so && rm -rf /lib/libudev.so && chattr +i /lib/ #在一步一点要这样运行，不然赶不上木马的自我恢复速度。。。。。我在这里坑了很长时间。 [root@Xd9BdoAkG ~] # ls /lib/ cppfirmwarekbdmodulessecurityterminfoudev [root@Xd9BdoAkG ~] # ls /lib/ cppfirmwarekbdmodulessecurityterminfoudev [root@Xd9BdoAkG ~] # ls /lib/ cppfirmwarekbdmodulessecurityterminfoudev [root@Xd9BdoAkG ~] # ls /lib/ cppfirmwarekbdmodulessecurityterminfoudev [root@Xd9BdoAkG ~] # ls /lib/ cppfirmwarekbdmodulessecurityterminfoudev [root@Xd9BdoAkG ~] # ls /lib/ cppfirmwarekbdmodulessecurityterminfoudev ##貌似主体被控制住了，不能再次自我恢复了。。。 [root@Xd9BdoAkG ~] # ls /etc/rc.d/rc3.d/ | awk '$7>=8 && $NF~/^K90/{print $NF}' | egrep '^.{10}$'| xargs -i rm -rf /etc/rc.d/rc3.d/{} chmod 0000 /etc/rc .d /rc3 .d/ && chmod 0000 /etc/rc .d /init .d && chattr +i /etc/rc .d /rc3 .d/ && chattr +i /etc/rc .d /init .d #删除启动脚本 awk '$7>=8 && $NF~/^K90/{print $NF}' 这里的8是当天的日期8号的意思，写的不严紧，勿喷~~~ #且控制目录不能写东西了

1 2	`[root@Xd9BdoAkG ~]` `# sed '/gcc.sh/d' /etc/crontab && chmod 0000 /etc/crontab && chattr +i /etc/crontab` `#删除计划任务且控制计划任务不能写东西`

到了这里基本就差不多了。。。。。现在去重启服务器，

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 [root@Xd9BdoAkG ~] # top -b -n1 | head top - 18:13:47 up 0 min,1 user,load average: 0.11, 0.03, 0.01 Tasks: 178 total,2 running, 176 sleeping,0 stopped,0 zombie Cpu(s):1.4%us,1.6%sy,0.0%ni, 95.7% id ,1.3%wa,0.0%hi,0.0%si,0.0%st Mem:32827160k total,486308k used, 32340852k free ,6864k buffers Swap: 16482300k total,0k used, 16482300k free ,28312k cached PID USERPRNIVIRTRESSHR S %CPU %MEMTIME+COMMAND 1 root200 19232 1512 1224 S0.00.00:01.26 init 2 root200000 S0.00.00:00.00 kthreadd 3 rootRT0000 S0.00.00:00.00 migration /0[root@Xd9BdoAkG ~] # top -b -n1 | head top - 18:13:51 up 0 min,1 user,load average: 0.10, 0.03, 0.01 Tasks: 178 total,1 running, 177 sleeping,0 stopped,0 zombie Cpu(s):1.3%us,1.5%sy,0.0%ni, 96.0% id ,1.2%wa,0.0%hi,0.0%si,0.0%st Mem:32827160k total,486136k used, 32341024k free ,6872k buffers Swap: 16482300k total,0k used, 16482300k free ,28344k cached PID USERPRNIVIRTRESSHR S %CPU %MEMTIME+COMMAND 1414 root200 15020 1232872 R2.00.00:00.01 top 1 root200 19232 1512 1224 S0.00.00:01.26 init 2 root200000 S0.00.00:00.00 kthreadd [root@Xd9BdoAkG ~] # top -b -n1 | head top - 18:14:15 up 1 min,1 user,load average: 0.06, 0.03, 0.01 Tasks: 178 total,1 running, 177 sleeping,0 stopped,0 zombie Cpu(s):0.8%us,0.9%sy,0.0%ni, 97.6% id ,0.7%wa,0.0%hi,0.0%si,0.0%st Mem:32827160k total,483360k used, 32343800k free ,6900k buffers Swap: 16482300k total,0k used, 16482300k free ,28360k cached PID USERPRNIVIRTRESSHR S %CPU %MEMTIME+COMMAND 1 root200 19232 1512 1224 S0.00.00:01.26 init 2 root200000 S0.00.00:00.00 kthreadd 3 rootRT0000 S0.00.00:00.00 migration /0

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 [root@Xd9BdoAkG ~] # ls /etc/cron.hourly/ 0anacron [root@Xd9BdoAkG ~] # ls -lt /etc/rc.d/rc3.d/ total 0 lrwxrwxrwx. 1 root root 15 Sep7 14:08 S85nginx -> .. /init .d /nginx lrwxrwxrwx. 1 root root 24 Sep7 13:55 S99libvirt-guests -> .. /init .d /libvirt-guests lrwxrwxrwx. 1 root root 19 Sep7 13:55 S26haldaemon -> .. /init .d /haldaemon lrwxrwxrwx. 1 root root 19 Sep7 13:54 K10saslauthd -> .. /init .d /saslauthd lrwxrwxrwx. 1 root root 20 Sep7 13:54 S22messagebus -> .. /init .d /messagebus lrwxrwxrwx. 1 root root 14 Sep7 13:51 S55sshd -> .. /init .d /sshd lrwxrwxrwx. 1 root root 18 Sep7 13:51 K15svnserve -> .. /init .d /svnserve lrwxrwxrwx. 1 root root 17 Sep7 13:44 S10network -> .. /init .d /network lrwxrwxrwx. 1 root root 17 Sep7 13:44 S12rsyslog -> .. /init .d /rsyslog lrwxrwxrwx. 1 root root 15 Sep7 13:44 S90crond -> .. /init .d /crond lrwxrwxrwx. 1 root root 19 Sep7 13:44 K75udev-post -> .. /init .d /udev-post lrwxrwxrwx. 1 root root 17 Sep7 13:44 K30postfix -> .. /init .d /postfix lrwxrwxrwx. 1 root root 15 Sep7 13:44 K75netfs -> .. /init .d /netfs lrwxrwxrwx. 1 root root 19 Sep7 13:44 K85mdmonitor -> .. /init .d /mdmonitor lrwxrwxrwx. 1 root root 22 Sep7 13:44 K99lvm2-monitor -> .. /init .d /lvm2-monitor lrwxrwxrwx. 1 root root 15 Sep7 13:44 K80kdump -> .. /init .d /kdump lrwxrwxrwx. 1 root root 18 Sep7 13:44 K92iptables -> .. /init .d /iptables lrwxrwxrwx. 1 root root 19 Sep7 13:44 K92ip6tables -> .. /init .d /ip6tables lrwxrwxrwx. 1 root root 26 Sep7 13:44 K75blk-availability -> .. /init .d /blk-availability lrwxrwxrwx. 1 root root 16 Sep7 13:44 K88auditd -> .. /init .d /auditd lrwxrwxrwx. 1 root root 17 Sep7 13:37 K75ntpdate -> .. /init .d /ntpdate lrwxrwxrwx. 1 root root 20 Sep7 12:15 K50netconsole -> .. /init .d /netconsole lrwxrwxrwx. 1 root root 11 Sep7 12:15 S99local -> .. /rc . local lrwxrwxrwx. 1 root root 15 Sep7 12:15 K89rdisc -> .. /init .d /rdisc lrwxrwxrwx. 1 root root 21 Sep7 12:15 K87restorecond -> .. /init .d /restorecond