ELK 收集 Java 后台日志_王清欢Randy_elk java

大大的周 02-07 6298

01 Java 日志样式

Java日志的特点在于输出信息非常多，通常需要将多行日志信息拼成一个事件，所以需要多行匹配模式。由于Elasticsearch本身就是使用Java开发的，所以Java日志收集实例就直接收集ES的日志。

如下所示是Elasticsearch的几条日志目录，可以看到这些日志条目通过第一个中括号中的时间戳进行区分，第二个日志条目中有多行Java日志，这多行日志组成了一个事件，怎么使用Filebeat采集这种多行日志呢？

[2021-08-02T07:14:18,201][INFO ][o.e.x.s.c.f.PersistentCache] [master] persistent cache index loaded [2021-08-02T07:14:28,351][ERROR][o.e.b.Bootstrap ] [master] Exception org.elasticsearch.transport.BindTransportException: Failed to bind to 172.16.255.13:[9300-9400] at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:406) ~[elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:370) ~[elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:120) ~[?:?] at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.doStart(SecurityNetty4Transport.java:85) ~[?:?] at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4ServerTransport.doStart(SecurityNetty4ServerTransport.java:47) ~[?:?] at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:48) ~[elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.transport.TransportService.doStart(TransportService.java:263) ~[elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:48) ~[elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.node.Node.start(Node.java:865) ~[elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:311) ~[elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:406) [elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.13.2.jar:7.13.2] at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.13.2.jar:7.13.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.13.2.jar:7.13.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.13.2.jar:7.13.2] Caused by: java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.bind0(Native Method) ~[?:?] at sun.nio.ch.Net.bind(Net.java:552) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:336) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294) ~[?:?] at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134) ~[?:?] at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491) ~[?:?] at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) ~[?:?] at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248) ~[?:?] at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356) ~[?:?] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:831) ~[?:?] [2021-08-02T07:14:28,357][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [master] uncaught exception in thread [main] 02 配置 Filbeat 多行匹配收集多行日志

多行匹配配置参考官方文档：多行日志收集配置

参考多行日志配置指导，配置Filebeat采集Java日志的输入如下

# ------------------------------Elasticsearch-Java---------------------------------- - type: log enabled: true paths: # - /var/log/tomcat8/localhost_access_log.2021-08-02.log - /var/log/elasticsearch/elasticsearch.log tags: ["es-java"] # 多行日志配置一下四行内容 multiline.type: pattern multiline.pattern: '^\[' multiline.negate: true multiline.match: after 03 测试 Filbeat 收集多行日志

先启动Filebeat让其一直收集ES中的Java日志，然后修改ES的配置文件使其产生多行错误日志，最后修复ES配置文件并查看日志采集结果

# 修改配置文件并重启Filebeat root@master:/etc/filebeat$ vim /etc/filebeat/filebeat.yml root@master:/etc/filebeat$ systemctl restart filebeat # 修改ES的配置文件（可以通过修改IP地址制作错误），使其启动失败产生多行输出的错误日志 root@master:/etc/filebeat$ vim /etc/elasticsearch/elasticsearch.yml root@master:/etc/filebeat$ systemctl restart elasticsearch Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details. # 修复ES的配置文件，并重新启动查看多行错误日志是否被正确收集 root@master:/etc/filebeat$ vim /etc/elasticsearch/elasticsearch.yml root@master:/etc/filebeat$ systemctl restart elasticsearch

查看ES-head，是否成功采集生成对应索引

使用Kibana查看是否正确收集多行Java日志