【ELK日志采集】filebeat日志采集及错误Provided Grok expressions do not match field value解决_one

未知 02-07 7715

1、文档使用工具：

? ? ? ? 图形化：kibana_6.7.1

????????数据存储：elasticsearch-6.7.1

????????日志采集：filebeat-6.7.1?

以下模拟日志数据采集使用工具是：

? ? Kibana? ?>> Dev Tools >> Console

2、两种样例数据

样例1为异常数据格式

2022-02-16 18:15:11 10.10.11.2 os[7028]:? 2022 RAC:root login? from 172.17.199.200

样例2为正常需求数据格式

2022-02-19 10:56:37 10.10.80.18 Severity:? Informational, Category: Storage, MessageID: CTL37, Message: A Patrol Read operation started for RAID Controller in Slot 7

3、初始定义pipeline规则 PUT /_ingest/pipeline/idrac-pipeline_v2?pretty { "description" : "Pipeline for parsing idrac logs.", "processors" : [ { "grok" : { "field" : "_source.message", "patterns" : [ "%{MY_DATETIME:time} %{IPORHOST:server_ip} Severity: %{DATA:Severity}, Category: %{DATA:Category}, MessageID: %{DATA:MessageID}, Message: %{GREEDYDATA:Message}" ], "pattern_definitions" : { "MY_DATE" : "%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}", "MY_TIME" : "[0-9][0-9]:[0-9][0-9]:[0-9][0-9]", "MY_DATETIME" : "%{MY_DATE} %{MY_TIME}" } } }, { "date" : { "field" : "time", "target_field" : "@timestamp", "formats" : [ "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||yyyy/MM/dd||yyyy/MM/dd HH:mm:ss||EEE MMM dd HH:mm:ss zzz yyyy||strict_date_optional_time||date_optional_time||basic_date_time" ], "timezone" : "Asia/Shanghai" } }, { "remove" : { "field" : "_source.message" } }, { "set" : { "field" : "_type", "value" : "idrac_v2" } } ] } 4、模拟测试请求数据结果（正常） POST _ingest/pipeline/idrac-pipeline_v2/_simulate { "docs": [ { "_source": { "message": "2022-02-19 10:56:37 10.10.80.18 Severity: Informational, Category: Storage, MessageID: CTL37, Message: A Patrol Read operation started for RAID Controller in Slot 7" } } ] }

5、模拟测试请求数据结果（异常） POST _ingest/pipeline/idrac-pipeline_v2/_simulate?verbose { "docs": [ { "_source": { "message": "2022-02-16 18:15:11 10.10.11.2 os[7028]: 2022 RAC:root login from 172.17.199.200" } } ] }

错误数据请求过程

java.lang.IllegalArgumentException: Provided Grok expressions do not match field value

6、解决方式1：

增加异常输出规则，不符合规则日志格式到错误索引

? ? "on_failure": [ ?? ??? ?{ ?? ??? ??? ?"set": { ?? ??? ??? ??? ?"field": "_index", ?? ??? ??? ??? ?"value": "idrac_error" ?? ??? ??? ?} ?? ??? ?} ?? ?]

PUT /_ingest/pipeline/idrac-pipeline_v2?pretty { "description" : "Pipeline for parsing idrac logs.", "processors" : [ { "grok" : { "field" : "_source.message", "patterns" : [ "%{MY_DATETIME:time} %{IPORHOST:server_ip} Severity: %{DATA:Severity}, Category: %{DATA:Category}, MessageID: %{DATA:MessageID}, Message: %{GREEDYDATA:Message}" ], "pattern_definitions" : { "MY_DATE" : "%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}", "MY_TIME" : "[0-9][0-9]:[0-9][0-9]:[0-9][0-9]", "MY_DATETIME" : "%{MY_DATE} %{MY_TIME}" } } }, { "date" : { "field" : "time", "target_field" : "@timestamp", "formats" : [ "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||yyyy/MM/dd||yyyy/MM/dd HH:mm:ss||EEE MMM dd HH:mm:ss zzz yyyy||strict_date_optional_time||date_optional_time||basic_date_time" ], "timezone" : "Asia/Shanghai" } }, { "remove" : { "field" : "_source.message" } }, { "set" : { "field" : "_type", "value" : "idrac_v2" } } ], "on_failure": [ { "set": { "field": "_index", "value": "idrac_error" } } ] }

模拟测试请求数据结果（正常）

POST _ingest/pipeline/idrac-pipeline_v2/_simulate { "docs": [ { "_source": { "message": "2022-02-16 18:15:11 10.10.11.2 os[7028]: 2022 RAC:root login from 172.17.199.200" } } ] }

请求结果

注意：异常结果录入到："_index" : "idrac_error",索引中。

7、解决方式2：

1、删除异常输出规则，不符合规则日志格式到错误索引

? ? "on_failure": [ ?? ??? ?{ ?? ??? ??? ?"set": { ?? ??? ??? ??? ?"field": "_index", ?? ??? ??? ??? ?"value": "idrac_error" ?? ??? ??? ?} ?? ??? ?} ?? ?]

2、增加新的patterns规则格式：

???????? "%{MY_DATETIME:time} %{IPORHOST:server_ip} os%{DATA:Message}"

PUT /_ingest/pipeline/idrac-pipeline_v2?pretty { "description" : "Pipeline for parsing idrac logs.", "processors" : [ { "grok" : { "field" : "_source.message", "patterns" : [ "%{MY_DATETIME:time} %{IPORHOST:server_ip} Severity: %{DATA:Severity}, Category: %{DATA:Category}, MessageID: %{DATA:MessageID}, Message: %{GREEDYDATA:Message}", "%{MY_DATETIME:time} %{IPORHOST:server_ip} os%{DATA:Message}" ], "pattern_definitions" : { "MY_DATE" : "%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}", "MY_TIME" : "[0-9][0-9]:[0-9][0-9]:[0-9][0-9]", "MY_DATETIME" : "%{MY_DATE} %{MY_TIME}" } } }, { "date" : { "field" : "time", "target_field" : "@timestamp", "formats" : [ "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||yyyy/MM/dd||yyyy/MM/dd HH:mm:ss||EEE MMM dd HH:mm:ss zzz yyyy||strict_date_optional_time||date_optional_time||basic_date_time" ], "timezone" : "Asia/Shanghai" } }, { "remove" : { "field" : "_source.message" } }, { "set" : { "field" : "_type", "value" : "idrac_v2" } } ] }

模拟测试请求数据结果（正常）

POST _ingest/pipeline/idrac-pipeline_v2/_simulate { "docs": [ { "_source": { "message": "2022-02-16 18:15:11 10.10.11.2 os[7028]: 2022 RAC:root login from 172.17.199.200" } } ] }

至此异常日志格式入ES问题解决

8、filebeat_idrac_v2收集日志配置文件

cat /usr/local/filebeat_idrac_v2/filebeat.yml? filebeat.inputs: - type: log ? enabled: true ? paths: ? ? - /opt/log_server/*/*.log? ? exclude_files: ['127\.0\.0']

? barvester_buffer_size: 163840 setup.template.name: "idrac-template_v2" setup.template.pattern: "idrac_v2-*" xpack.monitoring: ? enabled: true

output.elasticsearch: ? hosts: ["172.17.80.103:9200"] ? pipeline: "idrac-pipeline_v2" ? index: "idrac_v2-%{+yyyy.MM.dd}" ? username: "admin" ? password: "admin" ? worker: 2 ?

9、最终日志收集结果：