logstach对nginx/tcpudp日志收集 _收集

著论准过秦，作赋拟子虚。这篇文章主要讲述logstach对nginx/tcpudp日志收集相关的知识，希望能为你提供帮助。
1.nginx日志转json格式

log_format json "@timestamp":"$time_iso8601",
"host":"$server_addr",
"service":"nginxTest",
"trace":"$upstream_http_ctx_transaction_id",
"log":"log",
"clientip":"$remote_addr",
"remote_user":"$remote_user",
"request":"$request",
"http_user_agent":"$http_user_agent",
"size":$body_bytes_sent,
"responsetime":$request_time,
"upstreamtime":"$upstream_response_time",
"upstreamhost":"$upstream_addr",
"http_host":"$host",
"url":"$uri",
"domain":"$host",
"xff":"$http_x_forwarded_for",
"referer":"$http_referer",
"status":"$status";
access_log /var/log/nginx/access.log json ;

root@ubuntu:/data# vim /etc/nginx/nginx.conf

root@ubuntu:/data# cat /etc/logstash/conf.d/nginx.conf

input
file
path => "/var/log/nginx/access.log"
start_position => "beginning"
stat_interval => 3
type => "nginx-access-log"
codec => "json"

output
if [type] == "nginx-access-log"
elasticsearch
hosts => ["192.168.47.106:9200"]
index => "nginx-%+YYYY.MM.dd"

stdout
codec => "rubydebug"

root@ubuntu:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx.conf

2.收集TCP/UDP日志
通过logstash的tcp/udp插件收集日志，通常用于在向elasticsearch日志补录丢失的部分日志,可以将丢失的日志写到一个文件,然后通过TCP日志收集方式直接发送给logstash然后再写入到elasticsearch服务器
参考：??https://www.elastic.co/guide/en/logstash/5.6/input-plugins.html??
root@ubuntu:/data# vim /etc/logstash/conf.d/tcp.conf

input
tcp
port => 9889
type => "tcplog"
mode => "server"

output
stdout
codec => rubydebug

root@ubuntu:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp.conf

在其他主机测试：

echo 11 |nc -q 1 192.168.47.109 9889
或者
root@ela2:~# apt-get install nmap
root@ela2:~# echo 22 |ncat 192.168.47.109 9889

通过伪设备的方式发送消息
在类Unix操作系统中,块设备有硬盘、内存的硬件,但是还有设备节点并不一定要对应物理设备,我们把没有这种对应关系的设备是伪设备,比如/dev/null,/dev/zero,/dev/random以及/dev/tcp和/upd等,Linux操作系统使用这些伪设备提供了多种不通的功能,tcp通信只是dev下面众多伪设备当中的一种设备。