一、Filebeat概述

Filebeat主要功能：

• 【主动采集】：采集文件内容信息（读文件，日志文件）；
• 将文件内容进行简单处理，并发送给指定收集机系统；
• 一般架构分为：
  • 普通需求架构：【Filebeat数据收集】-【ElasticSearch集群存储】
  • 特殊需求架构：【Filebeat数据收集】-【Logstash数据二次处理】-【ElasticSearch集群存储】
  • 大数据架构：【Filebeat数据收集】-【kafka削峰填谷】-【Logstash数据二次处理】-【ElasticSearch集群存储】

二、Filebeat安装部署

注意：

• 为了方便学习，节约同学们的物理机资源；
• 本次学习部署在kibana服务器上；

1，下载filebeat安装包

https://www.elastic.co/

# 页面向下滑动，找到【Beats】顶级进入

# 进入页面后，再次滑动页面，找到：【Filebeat】，点击进入

# 找到历史版本入口，点击进入【View past releases】

# 选择要下载的版本

# 点击【Download】进入下载页面

# 选择系统架构类型，点击开始下载

# 等待下载完成

2，控制台部署调试filebeat

· 上传安装包

[root@kibana ~ ]# rz -E
[root@kibana ~ ]# ls -l
-rw-r--r-- 1 root root  64361096 Dec  4 05:21 filebeat-8.18.8-linux-x86_64.tar.gz

· 创建安装目录

[root@kibana ~ ]# mkdir -p /tools/

· 软件包解压到代码目录

[root@kibana ~ ]# tar xf filebeat-8.18.8-linux-x86_64.tar.gz -C /tools/

[root@kibana ~ ]# ls -l /tools/
total 12
drwxr-xr-x  5 root   root   4096 Dec  5 04:52 filebeat-8.18.8-linux-x86_64
drwxr-xr-x 10 kibana kibana 4096 Nov 27 05:03 kibana-7.17.29-linux-x86_64
drwxr-xr-x 11 kibana kibana 4096 Dec  3 08:47 kibana-8.18.8

· 编辑配置文件(调试模式)

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
# 数据从哪里来？
filebeat.inputs:
  # 【stdin】表示控制台标准输入 
- type: stdin
  # 启动这个标准输入的配置
  enabled: true

# 数据到哪里去？
#【output.console】表示控制台输出；
output.console:
  # 启动这个控制台输出
  enabled: true
  # json格式输出
  pretty: true

· 检查配置文件格式

[root@kibana ~ ]# cd /tools/filebeat-8.18.8-linux-x86_64
[root@kibana /tools/filebeat-8.18.8-linux-x86_64 ]# ./filebeat test config
Config OK

· 启动测试

参数	含义说明
-e	输出到stderr（标准错误输出），就是输出到控制台，开发调试时使用；
-c	指定配置文件
-path.data	指定数据存储目录(默认当前目录下的./data)
-path.logs	指定日志存储路径(默认当前目录下的./logs)

[root@kibana ~ ]# cd /tools/filebeat-8.18.8-linux-x86_64/
[root@kibana /tools/filebeat-8.18.8-linux-x86_64 ]# ./filebeat -e -c filebeat.yml

· 控制台输入输出测试

1，控制台输入

你好，我是蛙老师

2，控制台输出

{
  "@timestamp": "2025-12-05T09:32:28.925Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.18.8"
  },
  "host": {
    "name": "kibana"
  },
  "agent": {
    "type": "filebeat",
    "version": "8.18.8",
    "ephemeral_id": "6de1f032-b77a-40eb-b568-c1bfee510e11",
    "id": "78a12b85-14fc-40bc-94d1-2e72af95995a",
    "name": "kibana"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": ""
    }
  },
  "message": "你好，我是蛙老师",
  "input": {
    "type": "stdin"
  },
  "ecs": {
    "version": "8.0.0"
  }
}
{"log.level":"error","@timestamp":"2025-12-05T09:32:38.926Z","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/file.(*States).CleanupWith","file.name":"file/states.go","file.line":125},"message":"State for  should have been dropped, but couldn't as state is not finished.","service.name":"filebeat","ecs.version":"1.6.0"}

3，文件的输入输出配置与system管理

· 准备测试数据

[root@kibana ~ ]# mkdir /test
[root@kibana ~ ]# touch /test/1.txt
[root@kibana ~ ]# touch /test/2.txt

· 编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
# 数据的来源设置
filebeat.inputs:
- type: log
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"
    - "/test/2.txt"

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件最大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

· 编辑system启动文件

[root@kibana ~ ]# vim /lib/systemd/system/filebeat.service
[Unit]
Description=Filebeat sends log files to Logstash or Elasticsearch.
Documentation=https://www.elastic.co/beats/filebeat
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/tools/filebeat-8.18.8-linux-x86_64/filebeat -c /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

· 启动

[root@kibana ~ ]# systemctl daemon-reload
[root@kibana ~ ]# systemctl enable filebeat
[root@kibana ~ ]# systemctl start filebeat

· 输入文件写入内容测试

# 写入内容
[root@kibana ~ ]# echo "张三，你好" >> /test/1.txt 
[root@kibana ~ ]# echo "李四，你好" >> /test/2.txt

# 查看是否创建输出文件
[root@kibana ~ ]# ls -l /test/
total 12
-rw-r--r-- 1 root root  16 Dec  5 09:40 1.txt
-rw-r--r-- 1 root root  16 Dec  5 09:40 2.txt
-rw-r--r-- 1 root root 838 Dec  5 09:40 result.json-20251205.ndjson

# 查看输出内容
[root@kibana ~ ]# cat /test/result.json-20251205.ndjson 
{"@timestamp":"2025-12-05T09:40:26.530Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"ecs":{"version":"8.0.0"},"log":{"file":{"path":"/test/1.txt"},"offset":0},"message":"张三，你好","input":{"type":"log"},"host":{"name":"kibana"},"agent":{"type":"filebeat","version":"8.18.8","ephemeral_id":"870b64d4-579f-4cc6-a85c-87ec7984b230","id":"78a12b85-14fc-40bc-94d1-2e72af95995a","name":"kibana"}}
{"@timestamp":"2025-12-05T09:40:36.531Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"log":{"offset":0,"file":{"path":"/test/2.txt"}},"input":{"type":"log"},"ecs":{"version":"8.0.0"},"host":{"name":"kibana"},"agent":{"name":"kibana","type":"filebeat","version":"8.18.8","ephemeral_id":"870b64d4-579f-4cc6-a85c-87ec7984b230","id":"78a12b85-14fc-40bc-94d1-2e72af95995a"},"message":"李四，你好"}

三、filebeat的数据输入输出方式

输入/输出	常用的类型
input	【stdin】：控制台手动输入 【log】：读取本地文件（7.16版本之前主要使用） 【tcp】：接收远程tcp传输的数据； 【filestream】：读取本地文件（7.16版本之后主要使用） 【docker】：收集docker容器的日志 【container】：收集containerd容器的日志
output	【console】：输出到控制台，不存储； 【file】：写入到本机的文件中； 【elasticsearch】：写入到ES集群中； 【Logstash】：写入到Logstash组件中； 【kafka】：写入到kafka集群环境中；

1，TCP写入filebeat

· 编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
# 数据的来源设置
filebeat.inputs:
  # 指定tcp类型
- type: tcp
  enabled: true
  host: "0.0.0.0:5044"       # 监听地址和端口
  max_message_size: 10MiB    # 最大消息大小
  timeout: 300s              # 连接超时时间

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

· 测试语法

[root@kibana /tools/filebeat-8.18.8-linux-x86_64 ]# ./filebeat test config
Config OK

· 重启服务

[root@kibana ~ ]# systemctl restart filebeat.service 

[root@kibana ~ ]# ss -tnulp
......
tcp   LISTEN 0      511               0.0.0.0:5044     0.0.0.0:*    users:(("node",pid=141627,fd=95))

· 随便一台机器链接tcp写入数据

[root@kibana ~ ]# nc 10.0.0.110 5044
111
222
333
444
555
666
777
888

· 查看输出文件内容

[root@kibana ~ ]# tail -1  /test/result.json-20251207-1.ndjson 
{
	"@timestamp":"2025-12-07T11:00:01.398Z",
	"@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},
	"message":"888",
	"log":{"source":{"address":"10.0.0.210:44228"}},
	"input":{"type":"tcp"},
	"ecs":{"version":"8.0.0"},
	"host":{"name":"kibana"},
	"agent":{
		"ephemeral_id":"42fc2e75-9788-4315-afc8-7ec73964e032",
		"id":"926eb22d-85eb-4c28-9f8c-a9b7c8d28133",
		"name":"kibana",
		"type":"filebeat",
		"version":"8.18.8"
	}
}

2，filestream读取本地文件

与log类型大致一样

· 修改配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
# 数据的来源设置
filebeat.inputs:
  # 指定filestream类型
- type: filestream
  # 比log类型多一个自定义的id字段
  id: bakwite_01
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"
    - "/test/2.txt"

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

· 输入文件写入测试数据

[root@kibana ~ ]# echo "123" >> /test/1.txt

· 查看输出文件

[root@kibana ~ ]# tail -1  /test/result.json-20251207-2.ndjson 
{"@timestamp":"2025-12-07T11:26:10.937Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"message":"123","input":{"type":"filestream"},"host":{"name":"kibana"},"agent":{"id":"926eb22d-85eb-4c28-9f8c-a9b7c8d28133","name":"kibana","type":"filebeat","version":"8.18.8","ephemeral_id":"1f1042ee-0b66-4eef-9bb9-bde4c22525b3"},"ecs":{"version":"8.0.0"},"log":{"offset":742,"file":{"path":"/test/1.txt","device_id":"64768","inode":"4587526"}}}

{
	"@timestamp":"2025-12-07T11:26:10.937Z",
	"@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},
	"message":"123",
	"input":{"type":"filestream"},
	"host":{"name":"kibana"},
	"agent":{
		"id":"926eb22d-85eb-4c28-9f8c-a9b7c8d28133",
		"name":"kibana",
		"type":"filebeat",
		"version":"8.18.8",
		"ephemeral_id":"1f1042ee-0b66-4eef-9bb9-bde4c22525b3"
	},
	"ecs":{"version":"8.0.0"},
	"log":{"offset":742,"file":{"path":"/test/1.txt","device_id":"64768","inode":"4587526"}}}

3，输出到ES集群

· 复制ES的CA根证书

1，创建证书目录

[root@kibana ~ ]# mkdir /tools/filebeat-8.18.8-linux-x86_64/certs/

2，复制ES根证书到证书目录

[root@kibana ~ ]# scp 10.0.0.101:/tools/es8/elasticsearch-8.18.8/config/certs/ca.crt /tools/filebeat-8.18.8-linux-x86_64/certs/
#查看证书
[root@kibana ~ ]# ls -l /tools/filebeat-8.18.8-linux-x86_64/certs/
-rw-r--r--  1 root root      2029 Dec  5 06:22 ca.crt

· 编辑配置文件

目标：

• 启动两个收集系统，标记两个id，
• 不同的id，创建、写入不同的索引中；

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
# 数据的来源设置
filebeat.inputs:
  # 第一个filestream类型的收集器
- type: filestream
  # 比log类型多一个自定义的id字段
  id: bakwite_01
  # 启用这个filestream收集方式
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"
    - "/test/2.txt"


# 数据的出口设置；
output.elasticsearch:
  # ES集群主机地址（自动负载均衡）
  hosts: 
    - "https://10.0.0.101:10200"
    - "https://10.0.0.102:10200"
    - "https://10.0.0.103:10200"
  # ES的用户密码
  username: "elastic"
  password: "123456"
  # ES的SSl证书验证
  ssl:
    # 启动这个ssl证书验证
    enabled: true
    # 验证证书的有效性
    verification_mode: certificate
    # 验证ES公钥的CA证书；
    certificate_authorities: 
      # ES的ca根证书（复制过来再启动filebeat）；
      - "/tools/filebeat-8.18.8-linux-x86_64/certs/ca.crt"
  # 定义索引名称
  index: "wazong-%{+yyyy.MM.dd}"

# ============================================================= #
# ========== 索引模板配置[企业中，不会再filebeat中创建模板] ========= #
# ========== [了解即可] ======================================== #
# ============================================================= #
setup.ilm.enabled: false               # 关闭滚动更新
setup.template:
  enabled: flase                        # 开启索引模板配置功能
  
#下面的内容不写入进去-----------------------------------------------------
  name: "wazong-index-template"            # 索引模板名称
  pattern: "wazong-*"                      # 索引模板匹配模式
  overwrite: true                      # 如果ES已经存在这个模板，则覆盖
  settings:                            # 模板的settings字段设置
    index:
      number_of_shards: 3              # 主分片
      number_of_replicas: 1            # 副本分片

· 重启服务

[root@kibana ~ ]# systemctl restart filebeat.service

· 写入数据测试

[root@kibana ~ ]# echo 123 >> /test/1.txt 
[root@kibana ~ ]# echo 123 >> /test/1.txt 
[root@kibana ~ ]# echo 123 >> /test/1.txt 
[root@kibana ~ ]# echo 123 >> /test/1.txt 
[root@kibana ~ ]# echo 123 >> /test/1.txt 
[root@kibana ~ ]# echo 123 >> /test/2.txt 
[root@kibana ~ ]# echo 123 >> /test/2.txt 
[root@kibana ~ ]# echo 123 >> /test/2.txt 
[root@kibana ~ ]# echo 123 >> /test/2.txt

· kibana查看索引

# 查看索引

# 查看模板

· ES的API查看验证

1，查看ES集群索引列表

[root@kibana ~ ]# curl -k -u elastic:123456 https://10.0.0.102:10200/_cat/indices

2，查看filebeat日志

[root@kibana ~ ]# ls -l /tools/filebeat-8.18.8-linux-x86_64/logs/
total 640
-rw------- 1 root root 493993 Dec  5 07:52 filebeat-20251205-1.ndjson
-rw------- 1 root root  73480 Dec  5 07:57 filebeat-20251205-2.ndjson
-rw------- 1 root root  40926 Dec  5 08:03 filebeat-20251205-3.ndjson
-rw------- 1 root root  29010 Dec  5 06:56 filebeat-20251205.ndjson

# 安装jq工具
[root@kibana ~ ]# apt -y install jq

# 查看日志
[root@kibana ~ ]# tail -1 /tools/filebeat-8.18.8-linux-x86_64/logs/filebeat-20251207-137.ndjson | jq
......
      "filebeat": {
        "harvester": {
          "open_files": 0,
          "running": 0
        }
......

4，多端输出到不同ES索引

· 编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
# 数据的来源设置
filebeat.inputs:
  # 第1个filestream类型的收集器
- type: filestream
  # 比log类型多一个自定义的id字段[bakwite_01]
  id: bakwite_01
  # 启用这个filestream收集方式
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"
  # 第2个filestream类型的收集器
- type: filestream
  # 比log类型多一个自定义的id字段[bakwite_02]
  id: bakwite_02
  # 启用这个filestream收集方式
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/2.txt"


# 数据的出口设置；
output.elasticsearch:
  # ES集群主机地址（自动负载均衡）
  hosts: 
    - "https://10.0.0.101:10200"
    - "https://10.0.0.102:10200"
    - "https://10.0.0.103:10200"
  # ES的用户密码
  username: "elastic"
  password: "123456"
  # ES的SSl证书验证
  ssl:
    # 启动这个ssl证书验证
    enabled: true
    # 验证证书的有效性
    verification_mode: certificate
    # 验证ES公钥的CA证书；
    certificate_authorities: 
      # ES的ca根证书（复制过来再启动filebeat）；
      - "/tools/filebeat-8.18.8-linux-x86_64/certs/ca.crt"
  # 配置写入索引
  indices:
    # 定义索引名称
    - index: "test02-wa01%{+yyyy.MM.dd}"
      when:
        # [equals]表示完全精准匹配
        # [contains]表示模糊匹配
        equals:
          # 【提示】：这个字段是从输出内容字段中匹配的；
          log.file.path: "/test/1.txt"
    # 定义索引名称
    - index: "test02-wa02%{+yyyy.MM.dd}"
      when:
        equals:
          log.file.path: "/test/2.txt"
        

# ============================================================= #
# ========== 索引模板配置[企业中，不会再filebeat中创建模板] ========= #
# ========== [了解即可] ======================================== #
# ============================================================= #
setup.ilm.enabled: false               # 关闭滚动更新
setup.template:
  enabled: false                        # 开启索引模板配置功能
  
#下面的内容不写入进去-----------------------------------------------------
  name: "test02-index-template"            # 索引模板名称
  pattern: "test02-*"                      # 索引模板匹配模式
  overwrite: true                      # 如果ES已经存在这个模板，则覆盖
  settings:                            # 模板的settings字段设置
    index:
      number_of_shards: 3              # 主分片
      number_of_replicas: 1            # 副本分片

· 重启服务

[root@kibana ~ ]# systemctl restart filebeat.service

· 写入数据

[root@kibana ~ ]# echo 123 >> /test/1.txt 
[root@kibana ~ ]# echo 123 >> /test/2.txt

· 查看kibana结果

# 索引模板

# 索引

5，收集容器日志输出到文件:star:

· 准备容器环境

# 下载
[root@kibana ~ ]# wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.24.tgz

# 配置内核转发
[root@kibana ~ ]# echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
[root@kibana ~ ]# sysctl -p
net.ipv4.ip_forward = 1

# 解压安装包
[root@kibana ~ ]# tar xf docker-20.10.24.tgz 

# 授权
[root@kibana ~ ]# chown root.root docker/*

# 移动至PATH路径
[root@kibana ~ ]# cp docker/* /usr/bin/

# 编辑启动文件
[root@kibana ~ ]# vim /lib/systemd/system/docker.service
[Unit]
Description=bakwite docker 
Documentation=https://www.bakwite.com
After=network-online.target containerd.service   
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
LimitNOFILE=infinity
LimitNPROC=infinity
[Install]
WantedBy=multi-user.target

# 编辑仓库设置国内的仓库地址
[root@docker01 ~ 13:28:26]# vim /etc/docker/daemon.json
{
  "registry-mirrors" : ["https://do.nark.eu.org",
       "https://dc.j8.work",
       "https://docker.m.daocloud.io",
       "https://dockerproxy.com",
       "https://docker.mirrors.ustc.edu.cn",
       "https://docker.nju.edu.cn",
       "https://docker.mirrors.ustc.edu.cn",
       "https://docker.registry.cyou",
       "https://docker-cf.registry.cyou",
       "https://dockercf.jsdelivr.fyi",
       "https://docker.jsdelivr.fyi",
       "https://dockertest.jsdelivr.fyi",
       "https://mirror.aliyuncs.com",
       "https://dockerproxy.com",
       "https://mirror.baidubce.com",
       "https://docker.m.daocloud.io",
       "https://docker.nju.edu.cn",
       "https://docker.mirrors.sjtug.sjtu.edu.cn",
       "https://docker.mirrors.ustc.edu.cn",
       "https://mirror.iscas.ac.cn",
       "https://docker.rainbond.cc",
       "https://do.nark.eu.org",
       "https://docker.shootchat.top",
       "https://registry.docker-cn.com",
       "https://reg.bakwite.com"]
}


# 启动
[root@kibana ~ ]# systemctl daemon-reload 
[root@kibana ~ ]# systemctl enable --now docker.service


# 拉群nginx镜像
[root@kibana ~ ]# docker pull nginx:1.20.1-alpine

# 启动两个nginx容器
[root@kibana ~ ]# docker run -d --name web-nginx-01 -p 99:80 nginx:1.20.1-alpine
[root@kibana ~ ]# docker run -d --name web-nginx-02 -p 999:80 nginx:1.20.1-alpine
[root@kibana ~ ]# docker ps 

· 编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
filebeat.inputs:
# container类型的输入
- type: container
  # 启动这个收集
  enabled: true
  # 容器标准输出的日志地址（docker固定的位置）
  paths:
    - '/var/lib/docker/containers/*/*.log'
  
# 添加和删除字段的功能（后面会讲）
processors:
  # 添加docker的元数据字段
  - add_docker_metadata:
      # 元数据字段来源：docker的套接字文件
      host: "unix:///var/run/docker.sock"
  # 删除事件（后面会讲）
  - drop_event:
      # 当
      when:
        # 数据中若[不是]则删除，不收集
        not:
          # 模糊匹配
          regexp:
            # 匹配container.name这个字段；(注意：正则表达式，必须是单引号)
            container.name: "^web*"
# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

· 重启filebeat

[root@kibana ~ ]# systemctl restart filebeat.service

· 测试访问2个容器

[root@kibana ~ ]# curl 10.0.0.210:99
[root@kibana ~ ]# curl 10.0.0.210:99

· 查看输出文件内容

[root@kibana /tools/filebeat-8.18.8-linux-x86_64 ]# cat /test/result.json-20251208-1.ndjson 
[root@kibana /tools/filebeat-8.18.8-linux-x86_64 ]# cat /test/result.json-20251208.ndjson 
{
	"@timestamp":"2025-12-08T08:56:37.131Z",
	"@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},
	"ecs":{"version":"8.0.0"},
	"agent":{"id":"d91d4900-f4d9-4246-b9c2-abcc023f23f8",
	"name":"kibana",
	"type":"filebeat",
	"version":"8.18.8",
	"ephemeral_id":"98ac9387-c214-432b-9c17-38ba7ea71a37"},
	"container":{"id":"82ffff731d948378a37e8186237ff97ea7dac85ba0ab62a1b51bed69c643424d","labels":{"maintainer":"NGINX Docker Maintainers <docker-maint@nginx.com>"},
	"image":{"name":"nginx:1.20.1-alpine"},"name":"web-nginx-01"},"log":{"offset":5680,"file":{"path":"/var/lib/docker/containers/82ffff731d948378a37e8186237ff97ea7dac85ba0ab62a1b51bed69c643424d/82ffff731d948378a37e8186237ff97ea7dac85ba0ab62a1b51bed69c643424d-json.log"}},"stream":"stdout","message":"10.0.0.210 - - [08/Dec/2025:08:56:37 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.81.0\" \"-\"","host":{"name":"kibana"},"input":{"type":"container"}}

6，k8s容器日志收集（k8s课程中讲解）:star:

# 未完待续...

四、Filebeat读文件原理

1，filebeat默认单行读取

· 读取文件原理

1，先清空输入输出文件

[root@kibana ~ ]# echo "" > /test/1.txt 
[root@kibana ~ ]# echo "" > /test/2.txt
[root@kibana ~ ]# echo "" > /test/result.json-20251205.ndjson

2，编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
# 数据的来源设置
filebeat.inputs:
  # 指定filestream类型
- type: filestream
  # 比log类型多一个自定义的id字段
  id: bakwite_01
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"
    - "/test/2.txt"

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

3，重启

[root@kibana ~ ]# systemctl restart filebeat.service

4，监控输出文件

[root@kibana ~ ]# tail -f /test/result.json-20251205.ndjson

5，输入文件写入内容不带换行符

[root@kibana ~ ]# echo -n "zhagsan" >> /test/1.txt 

6，查看监控输出结果得出结论

# 只要文件输入的内容，没有换行符，则filebeat不读取

7，输入文件写入内容带换行符

[root@kibana ~ ]# echo  "lisi" >> /test/1.txt

8，查看监控输出结果得出结论

结论：filebeat读取文件是按照两个换行符之间的数据进行读取的；

[root@kibana ~ ]# tail -f /test/result.json-20251205.ndjson 

{"@timestamp":"2025-12-06T05:29:23.375Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"host":{"name":"kibana"},"log":{"offset":1,"file":{"path":"/test/1.txt"}},"message":"zhagsanlisi","input":{"type":"log"},"agent":{"id":"78a12b85-14fc-40bc-94d1-2e72af95995a","name":"kibana","type":"filebeat","version":"8.18.8","ephemeral_id":"870b64d4-579f-4cc6-a85c-87ec7984b230"},"ecs":{"version":"8.0.0"}}

· 读文件偏移量offset

1，查看输入文件的位置

查看文件大小：【13字节】

[root@kibana ~ ]# ls -l /test/1.txt 
-rw-r--r-- 1 root root 13 Dec  6 05:29 /test/1.txt

2，输出文件记录的offset字段

{
	"@timestamp":"2025-12-06T05:29:23.375Z",
	"@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},
	"host":{"name":"kibana"},
	"log":{
	    // 可以看到，是从offset=1的位置开始读取的
		"offset":1,
		"file":{"path":"/test/1.txt"}},
	"message":"zhagsanlisi",
	"input":{"type":"log"},
	"agent":{
		"id":"78a12b85-14fc-40bc-94d1-2e72af95995a",
		"name":"kibana",
		"type":"filebeat",
		"version":"8.18.8",
		"ephemeral_id":"870b64d4-579f-4cc6-a85c-87ec7984b230"
	},
	"ecs":{"version":"8.0.0"}
}

3，输入文件写入内容

[root@kibana ~ ]# echo  "wangwu" >> /test/1.txt

4，查看监控输入文件内容

{
	"@timestamp":"2025-12-06T05:38:03.553Z",
	"@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},
	"host":{"name":"kibana"},
	"agent":{
		"ephemeral_id":"870b64d4-579f-4cc6-a85c-87ec7984b230",
		"id":"78a12b85-14fc-40bc-94d1-2e72af95995a",
		"name":"kibana",
		"type":"filebeat",
		"version":"8.18.8"
	},
	"message":"wangwu",
	"log":{
		// 可以看到，这次是从offset=13开始读取的
		"offset":13,
		"file":{"path":"/test/1.txt"}},
	"input":{"type":"log"},
	"ecs":{"version":"8.0.0"}
}

· filebeat记录offset位置

1，查看被读取文件当前大小

# 可以看到是20字节
[root@kibana ~ ]# ls -l /test/1.txt 
-rw-r--r-- 1 root root 20 Dec  6 05:37 /test/1.txt

2，查看filebeat记录读文件日志

# 可以看到，读文件1.txt的offset记录正好是【20】
[root@kibana ~ ]# tail -3  /tools/filebeat-8.18.8-linux-x86_64/data/registry/filebeat/log.json 

{"k":"filebeat::logs::native::4587522-64768","v":{"ttl":-1,"FileStateOS":{"inode":4587522,"device":64768},"identifier_name":"native","id":"native::4587522-64768","prev_id":"","source":"/test/1.txt","offset":20,"timestamp":[630556245,1764999788],"type":"log"}}
{"op":"set","id":214}
{"k":"filebeat::logs::native::4587523-64768","v":{"prev_id":"","offset":1,"timestamp":[393335673,1764998948],"id":"native::4587523-64768","source":"/test/2.txt","ttl":-1,"type":"log","FileStateOS":{"device":64768,"inode":4587523},"identifier_name":"native"}}

· 恢复从零开始读取

就是清空offset记录文件

[root@kibana ~ ]# echo "" > /tools/filebeat-8.18.8-linux-x86_64/data/registry/filebeat/log.json 

2，filebeat多行匹配配置

· 基于行数匹配

1，模拟多行输入文件

假设我们需要输入的文件中内容，需要多行代表一条数据；

[root@kibana ~ ]# vim /test/1.txt
{
	"name": "张三",
	"age": "18",
	"sex": "女",
	"tags": ["跳舞","化妆","骑行"]
}
{
	"name": "李四",
	"age": "27",
	"sex": "男",
	"tags": ["荒野求生","滑行伞","蹦极"]
}

2，清空数据

[root@kibana ~ ]# echo "" > /tools/filebeat-8.18.8-linux-x86_64/data/registry/filebeat/log.json
[root@kibana ~ ]# rm -rf /test/result.json-20251207*

3，重启服务

[root@kibana ~ ]# systemctl restart filebeat.service

4，查看监控输入文件内容

可以看到：

• 我们文件中的6行为【一组数据】，
• 被拆分成了6条数据输出到了目标文件中；

[root@kibana ~ ]# tail -f /test/result.json-20251205.ndjson 

5，编辑配置文件基于多行匹配

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
# 数据的来源设置
filebeat.inputs:
- type: filestream
  id: bakwite_02
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"
    - "/test/2.txt"
  # log类型没有[parsers]这个字段，filestream类型需要写这个字段
  parsers:
    # 读取文件方式配置（如果是log类型，没有）
  - multiline:
      # 【count】表示基于自定义行数匹配
      type: count
      # 设置多少行为一条数据；
      count_lines: 6

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

5，清空数据

[root@kibana ~ ]# echo "" > /tools/filebeat-8.18.8-linux-x86_64/data/registry/filebeat/log.json
[root@kibana ~ ]# rm -rf /test/result.json-20251207*

6，重启filebeat

[root@kibana ~ ]# systemctl restart filebeat.service

8，查看监控输出结果

可以看到：每6行一组的数据，被当成一条数据输出到了文件中

[root@kibana ~ ]# cat /test/result.json-20251207.ndjson 
{
......
"message":"{\n\t\"name\": \"张三\",\n\t\"age\": \"18\",\n\t\"sex\": \"女\",\n\t\"tags\": [\"跳舞\",\"化妆\",\"骑行\"]\n}",
......
"message":"{\n\t\"name\": \"李四\",\n\t\"age\": \"27\",\n\t\"sex\": \"男\",\n\t\"tags\": [\"荒野求生\",\"滑行伞\",\"蹦极\"]\n}",
......

· 基于模式（正则）匹配行

1，先清空所有数据

# 清空输入文件
[root@kibana ~ ]# echo "" > /test/1.txt 
[root@kibana ~ ]# echo "" > /test/2.txt 

# 清空输出文件
[root@kibana ~ ]# echo "" > /test/result.json-20251206.ndjson 

# 清空offset记录
[root@kibana ~ ]# echo "" > /tools/filebeat-8.18.8-linux-x86_64/data/registry/filebeat/log.json

2，配置文件编辑说明

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml 
......
  # 读取文件方式配置
  multiline:
    # 【pattern】表示基于模式匹配
    type: pattern
    # 设置要匹配的内容
    pattern: '正则表达式'
    # 表示：符合正则匹配规则的，算作新数据，还是算作当前数据？
    # 【true】: 表示算作新的一条数据
    # 【false】: 表示算作当前数据
    negate: true/false
    # 匹配到结果，存储的时候，放到这条数据的前面还是后面（注意，不是下一条数据。这里的设置是存放位置的设置）
    match: after/before
    # 遇到什么（正则匹配）算作一条数据的结束;【很少用】
    flush_pattern: ""
......

案例1：假设我们的输入文件如下；

[root@kibana ~ ]# vim /test/1.txt
{
	"name": "张三",
	"age": "18",
}
{
	"name": "李四",
	"age": "27",
	"sex": "男",
	"tags": ["荒野求生","滑行伞","蹦极"]
}
{
	"name": "李四",
	"age": "27",
	"sex": "男",
	"tags": ["荒野求生","滑行伞","蹦极"],
	"chengji": {
		"yishu": "98",
		"shuxue": "77",
		"english": "65"
	}
}

3，编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml 
# 数据的来源设置
filebeat.inputs:
- type: filestream
  id: bakwite_03
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"

  parsers:
  - multiline:
      type: pattern
      # 匹配以【{左花括号】开头的，作为匹配的目标；(注意：正则表达式，必须是单引号)
      pattern: '^\{'
      # 表示取反，不以【{左花括号】开头的就续行，表示为一条数据；
      negate: true
      # 放到这条数据的后面
      match: after

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

4，重启filebeat

[root@kibana ~ ]# systemctl restart filebeat.service

5，查看监控输出文件结果

[root@kibana ~ ]# cat /test/result.json-20251207.ndjson 
{"@timestamp":"2025-12-07T11:47:34.880Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"log":{"offset":35,"file":{"path":"/test/1.txt","device_id":"64768","inode":"4587526"},"flags":["multiline"]},"message":"{\n\t\"name\": \"张三\",\n\t\"age\": \"18\",\n}","input":{"type":"filestream"},"ecs":{"version":"8.0.0"},"host":{"name":"kibana"},"agent":{"ephemeral_id":"427b2ca5-511e-44b1-afe4-c841a91f81cc","id":"926eb22d-85eb-4c28-9f8c-a9b7c8d28133","name":"kibana","type":"filebeat","version":"8.18.8"}}
{"@timestamp":"2025-12-07T11:47:34.880Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"host":{"name":"kibana"},"log":{"flags":["multiline"],"offset":134,"file":{"path":"/test/1.txt","device_id":"64768","inode":"4587526"}},"message":"{\n\t\"name\": \"李四\",\n\t\"age\": \"27\",\n\t\"sex\": \"男\",\n\t\"tags\": [\"荒野求生\",\"滑行伞\",\"蹦极\"]\n}","input":{"type":"filestream"},"agent":{"id":"926eb22d-85eb-4c28-9f8c-a9b7c8d28133","name":"kibana","type":"filebeat","version":"8.18.8","ephemeral_id":"427b2ca5-511e-44b1-afe4-c841a91f81cc"},"ecs":{"version":"8.0.0"}}
{"@timestamp":"2025-12-07T11:47:34.880Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"input":{"type":"filestream"},"ecs":{"version":"8.0.0"},"host":{"name":"kibana"},"agent":{"type":"filebeat","version":"8.18.8","ephemeral_id":"427b2ca5-511e-44b1-afe4-c841a91f81cc","id":"926eb22d-85eb-4c28-9f8c-a9b7c8d28133","name":"kibana"},"log":{"offset":304,"file":{"path":"/test/1.txt","device_id":"64768","inode":"4587526"},"flags":["multiline"]},"message":"{\n\t\"name\": \"李四\",\n\t\"age\": \"27\",\n\t\"sex\": \"男\",\n\t\"tags\": [\"荒野求生\",\"滑行伞\",\"蹦极\"],\n\t\"chengji\": {\n\t\t\"yishu\": \"98\",\n\t\t\"shuxue\": \"77\",\n\t\t\"english\": \"65\"\n\t}\n}"}

五、processors控制输出字段:x:

1，当前输出数据

字段	解释说明
@timestamp	文档记录的时间
@metadata	filebeat的元数据信息
host	当前主机名
agent	写入数据的客户端信息
log	日志信息：读取的偏移量，读取的文件，处理的模块
input	输入的类型log、stdin、tcp等等
ecs	文档写入的规范版本，用于协调与其他组件读写文件的一致性；
message	我们真实写入的数据信息

{
	"@timestamp":"2025-12-06T10:43:06.402Z",
	"@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},
	"host":{"name":"kibana"},
	"agent":{
		"type":"filebeat",
		"version":"8.18.8",
		"ephemeral_id":"7477e477-7239-48b7-aa3b-8c83dc653880",
		"id":"78a12b85-14fc-40bc-94d1-2e72af95995a",
		"name":"kibana"
	},
	"log":{
		"offset":0,
		"file":{"path":"/test/1.txt"},
		"flags":["multiline"]
	},
	"message":"{\n\t\"name\": \"张三\",\n\t\"age\": \"18\",\n}",
	"input":{"type":"log"},
	"ecs":{"version":"8.0.0"}
}

2，调整message字段级别

· 清空数据

[root@kibana ~ ]# echo "" > /test/1.txt 
[root@kibana ~ ]# echo "" > /test/2.txt 
[root@kibana ~ ]# rm -rf /test/result.json*
[root@kibana ~ ]# rm -rf /tools/filebeat-8.18.8-linux-x86_64/data/*

· 编辑配置文件

这里注意：

• filebeat的设计缺陷；
• multiline的匹配多行，与json解析器，二者有冲突；
• 所以配置中，只能配置二者中的一项；
• 也就是说，当你使用json解析器的时候，日志文件中，必须一行一条数据；不能分行；

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml 
# 数据的来源设置
filebeat.inputs:
- type: filestream
  id: bakwite_03
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"

  parsers:
  # 调用json解析器
  - ndjson:
      message_key: "message"   # 指定数据字段名（message）
      keys_under_root: true    # 把JSON字段提升到根级
      add_error_key: true      # 可选：解析失败时添加错误

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

· 重启filebeat

[root@kibana ~ ]# systemctl restart filebeat.service

· 测试输出内容

1，写入测试数据

[root@kibana ~ ]# vim /test/1.txt
{"name": "张三","age": "18"}

2，查看输出文件内容

可以看到：

• name和age字段已经跑到了一级字段中；

[root@kibana ~ ]# cat /test/result.json-20251207.ndjson 
{
	"@timestamp":"2025-12-07T11:52:21.023Z",
	"@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},
	"log":{"offset":0,"file":{"device_id":"64768","inode":"4587525","path":"/test/1.txt"}},
	"error":{"message":"Key 'message' not found","type":"json"},
	"name":"张三",
	"input":{"type":"filestream"},
	"ecs":{"version":"8.0.0"},
	"host":{"name":"kibana"},
	"agent":{"ephemeral_id":"4b82584e-c59a-4761-9951-1051d9c00356","id":"ca275eb0-8ff2-4c00-a08c-d668c91667e2","name":"kibana","type":"filebeat","version":"8.18.8"},
	"age":"18"
}

3，添加与删除字段

· 编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml 
# 数据的来源设置
filebeat.inputs:
- type: filestream
  id: bakwite_03
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"

  parsers:
  # 调用json解析器
  - ndjson:
    message_key: "message"   # 指定数据字段名（message）
    keys_under_root: true    # 把JSON字段提升到根级
    #add_error_key: true      # 调试时使用，可选：解析失败时添加错误
# 字段处理器
processors:
  # 要删除的字段的动作
  - drop_fields:
      # 要删除哪些字段？
      fields: 
          # 【@metadata】字段无法删除，这里测试看看结果
        - "@metadata"   
        - "agent"
        - "input"
        - "host"
        - "log.offset"
        - "log.flags"
        - "ecs"
      # 删除的字段如果不存在，不报错；
      ignore_missing: true
  # 要添加的字段的动作
  - add_fields:
      # 添加的字段，放到哪个字段下？【""】表示放到一级字段；
      target: ""
      fields:
        bakwite: "测试添加字段"
        xjsw: "心机之蛙"


# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

· 清空数据测试

[root@kibana ~ ]# echo "" > /test/1.txt 
[root@kibana ~ ]# echo "" > /test/2.txt 
[root@kibana ~ ]# rm -rf /tools/filebeat-8.18.8-linux-x86_64/data/*
[root@kibana ~ ]# rm -rf /test/result.json*

· 重启filebeat

[root@kibana ~ ]# systemctl restart filebeat.service

· 输入文件写入数据

[root@kibana ~ ]# vim /test/1.txt
{"name": "张三","age": "18"}

· 查看输入出文件

[root@kibana ~ ]# cat /test/result.json-20251207.ndjson 
{
	"@timestamp":"2025-12-07T11:55:40.255Z",
	"@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},
	"log":{"file":{"path":"/test/1.txt","device_id":"64768","inode":"4587524"}},
	"age":"18",
	"name":"张三",
	"bakwite":"测试添加字段",
	"xjsw":"心机之蛙"
}

4，修改字段名称与数据类型

· 编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml 
# 数据的来源设置
filebeat.inputs:
- type: filestream
  id: bakwite_03
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"

  parsers:
  # 调用json解析器
  - ndjson:
    message_key: "message"   # 指定数据字段名（message）
    keys_under_root: true    # 把JSON字段提升到根级
    #add_error_key: true      # 调试时使用，可选：解析失败时添加错误
# 字段处理器
processors:
  # 要删除的字段的动作
  - drop_fields:
      # 要删除哪些字段？
      fields: 
          # 【@metadata】字段无法删除，这里测试看看结果
        - "@metadata"   
        - "agent"
        - "input"
        - "host"
        - "log.offset"
        - "log.flags"
        - "ecs"
      # 删除的字段如果不存在，不报错；
      ignore_missing: true
  # 要添加的字段的动作
  - add_fields:
      # 添加的字段，放到哪个字段下？【""】表示放到一级字段；
      target: ""
      fields:
        bakwite: "测试添加字段"
        xjsw: "心机之蛙"
  # 字段改名
  - rename:
      fields:
        - from: "name"     # 将name字段
          to: "username"   # 改为username字段
  
  # 2. 改名+改类型
  - convert:
      fields:
        - from: "age"      # 将字段name的数据类型
          to: "age"        # 名称不变
          type: "long"     # 具体类型；

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

· 清空数据

[root@kibana ~ ]# echo "" > /test/1.txt 
[root@kibana ~ ]# echo "" > /test/2.txt 
[root@kibana ~ ]# rm -rf /tools/filebeat-8.18.8-linux-x86_64/data/*
[root@kibana ~ ]# rm -rf /test/result.json*

· 重启filebeat

[root@kibana ~ ]# systemctl restart filebeat.service

· 输入文件写入数据

[root@kibana ~ ]# vim /test/1.txt
{"name": "张三","age": "18"}

· 查看输出文件

[root@kibana ~ ]# cat /test/result.json-20251207.ndjson
......
{
	"@timestamp":"2025-12-07T07:45:32.005Z",
	"@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},
	"bakwite":"测试添加字段",
	"xjsw":"心机之蛙",
	"log":{"file":{"path":"/test/1.txt"}},
	"username":"张三",
	"age":18,
	"message":""
}

5，过滤输入的字段数据:x:

匹配方式	说明
regexp	正则匹配
contains	包含模糊匹配
equals	精准匹配

· 准备测试数据

[root@kibana ~ ]# vim /test/1.txt
{"user_ip": "101.101.1.101","username": "张三","age": 18,"sex": "女","xueli": "硕士","techang": ["游泳","跳水","花样滑冰","芭蕾"],"chengji": {"语文": "97","数学": "105","英语": "77","物理": "94","化学": "90","生物": "99"}}
{"user_ip": "101.101.2.101","username": "李四","age": 22,"sex": "男","xueli": "本科","techang": ["跳伞","爬山","计算机","滑雪"],"chengji": {"语文": "66","数学": "144","英语": "30","物理": "60","化学": "49","生物": "100"}}
{"user_ip": "101.101.3.101","username": "王五","age": 20,"sex": "女","xueli": "大专","techang": ["跳伞","跳远","跳高","跳绳"],"chengji": {"语文": "44","数学": "10","英语": "70","物理": "9","化学": "100","生物": "100"}}

· 精准匹配删除事件

目标：只收集本科及以上学历

1，清空数据

[root@kibana ~ ]# echo "" > /test/1.txt 
[root@kibana ~ ]# echo "" > /test/2.txt 
[root@kibana ~ ]# rm -rf /tools/filebeat-8.18.8-linux-x86_64/data/*
[root@kibana ~ ]# rm -rf /test/result.json*

2，编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml 
# 数据的来源设置
filebeat.inputs:
- type: filestream
  id: bakwite_03
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"

  parsers:
  # 调用json解析器
  - ndjson:
    message_key: "message"   # 指定数据字段名（message）
    keys_under_root: true    # 把JSON字段提升到根级
    #add_error_key: true      # 调试时使用，可选：解析失败时添加错误
# 字段处理器
processors:
  # 要删除的字段的动作
  - drop_fields:
      # 要删除哪些字段？
      fields: 
          # 【@metadata】字段无法删除，这里测试看看结果
        - "@metadata"   
        - "agent"
        - "input"
        - "host"
        - "log.offset"
        - "log.flags"
        - "ecs"
      # 删除的字段如果不存在，不报错；
      ignore_missing: true
  # [删除事件]学历过滤
  - drop_event:
      # 当
      when:
        # 数据中若[不是]则删除，不收集
        not:
          # 下面的条件是或者的关系
          or:
              # 精确匹配。必须完全相等；
            - equals:
                # 学历字段是"本科"
                xueli: "本科"
              # 精确匹配。必须完全相等；
            - equals:
                # 学历字段是"硕士"
                xueli: "硕士"

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件最大大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

3，重启

[root@kibana ~ ]# systemctl restart filebeat.service

4，数据写入输入文件

[root@kibana ~ ]# vim /test/1.txt
{"user_ip": "101.101.1.101","username": "张三","age": 18,"sex": "女","xueli": "硕士","techang": ["游泳","跳水","花样滑冰","芭蕾"],"chengji": {"语文": "97","数学": "105","英语": "77","物理": "94","化学": "90","生物": "99"}}
{"user_ip": "101.101.2.101","username": "李四","age": 22,"sex": "男","xueli": "本科","techang": ["跳伞","爬山","计算机","滑雪"],"chengji": {"语文": "66","数学": "144","英语": "30","物理": "60","化学": "49","生物": "100"}}
{"user_ip": "101.101.3.101","username": "王五","age": 20,"sex": "女","xueli": "大专","techang": ["跳伞","跳远","跳高","跳绳"],"chengji": {"语文": "44","数学": "10","英语": "70","物理": "9","化学": "100","生物": "100"}}

5，查看输出文件

结论：值写入了我们过滤的两条数据；学历史大专的没有写入

[root@kibana ~ ]# cat /test/result.json-20251207.ndjson 
{"@timestamp":"2025-12-07T09:24:19.567Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"xueli":"硕士","techang":["游泳","跳水","花样滑冰","芭蕾"],"username":"张三","message":"","user_ip":"101.101.1.101","log":{"file":{"path":"/test/1.txt"}},"chengji":{"生物":"99","语文":"97","数学":"105","英语":"77","物理":"94","化学":"90"},"age":18,"sex":"女"}
{"@timestamp":"2025-12-07T09:24:19.567Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"sex":"男","xueli":"本科","chengji":{"语文":"66","数学":"144","英语":"30","物理":"60","化学":"49","生物":"100"},"message":"","log":{"file":{"path":"/test/1.txt"}},"age":22,"techang":["跳伞","爬山","计算机","滑雪"],"username":"李四","user_ip":"101.101.2.101"}

· 正则匹配删除事件

目标：只收集固定网段的数据

1，清空数据

[root@kibana ~ ]# echo "" > /test/1.txt 
[root@kibana ~ ]# echo "" > /test/2.txt 
[root@kibana ~ ]# rm -rf /tools/filebeat-8.18.8-linux-x86_64/data/*
[root@kibana ~ ]# rm -rf /test/result.json*

2，编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml 
# 数据的来源设置
filebeat.inputs:
- type: filestream
  id: bakwite_03
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"

  parsers:
  # 调用json解析器
  - ndjson:
    message_key: "message"   # 指定数据字段名（message）
    keys_under_root: true    # 把JSON字段提升到根级
    #add_error_key: true      # 调试时使用，可选：解析失败时添加错误
# 字段处理器
processors:
  # 要删除的字段的动作
  - drop_fields:
      # 要删除哪些字段？
      fields: 
          # 【@metadata】字段无法删除，这里测试看看结果
        - "@metadata"   
        - "agent"
        - "input"
        - "host"
        - "log.offset"
        - "log.flags"
        - "ecs"
      # 删除的字段如果不存在，不报错；
      ignore_missing: true
  # [删除事件]ip过滤
  - drop_event:
      # 当
      when:
        # 数据中若[不是]则删除，不收集
        not:
          # 基于正则匹配
          regexp:
            # 匹配user_id这个字段；(注意：正则表达式，必须是单引号)
            user_ip: '^101\.101\.1\.\d+$'

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

3，重启

[root@kibana ~ ]# systemctl restart filebeat.service

4，写入测试数据

[root@kibana ~ ]# vim /test/1.txt
{"user_ip": "101.101.1.101","username": "张三","age": 18,"sex": "女","xueli": "硕士","techang": ["游泳","跳水","花样滑冰","芭蕾"],"chengji": {"语文": "97","数学": "105","英语": "77","物理": "94","化学": "90","生物": "99"}}
{"user_ip": "101.101.2.101","username": "李四","age": 22,"sex": "男","xueli": "本科","techang": ["跳伞","爬山","计算机","滑雪"],"chengji": {"语文": "66","数学": "144","英语": "30","物理": "60","化学": "49","生物": "100"}}
{"user_ip": "101.101.3.101","username": "王五","age": 20,"sex": "女","xueli": "大专","techang": ["跳伞","跳远","跳高","跳绳"],"chengji": {"语文": "44","数学": "10","英语": "70","物理": "9","化学": "100","生物": "100"}}

5，查看输出文件结果

结论：只收集了ip是101.101.1.0网段的数据

[root@kibana ~ ]# cat /test/result.json-20251207.ndjson 
{"@timestamp":"2025-12-07T09:42:42.956Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"techang":["游泳","跳水","花样滑冰","芭蕾"],"log":{"file":{"path":"/test/1.txt"}},"username":"张三","sex":"女","message":"","user_ip":"101.101.1.101","xueli":"硕士","chengji":{"生物":"99","语文":"97","数学":"105","英语":"77","物理":"94","化学":"90"},"age":18}

· 模糊匹配删除事件

目标：只收集字段中包含特定字符串的数据

1，清空数据

[root@kibana ~ ]# echo "" > /test/1.txt 
[root@kibana ~ ]# echo "" > /test/2.txt 
[root@kibana ~ ]# rm -rf /tools/filebeat-8.18.8-linux-x86_64/data/*
[root@kibana ~ ]# rm -rf /test/result.json*

2，编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml 
# 数据的来源设置
filebeat.inputs:
- type: filestream
  id: bakwite_03
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"

  parsers:
  # 调用json解析器
  - ndjson:
    message_key: "message"   # 指定数据字段名（message）
    keys_under_root: true    # 把JSON字段提升到根级
    #add_error_key: true      # 调试时使用，可选：解析失败时添加错误
# 字段处理器
processors:
  # 要删除的字段的动作
  - drop_fields:
      # 要删除哪些字段？
      fields: 
          # 【@metadata】字段无法删除，这里测试看看结果
        - "@metadata"   
        - "agent"
        - "input"
        - "host"
        - "log.offset"
        - "log.flags"
        - "ecs"
      # 删除的字段如果不存在，不报错；
      ignore_missing: true
  # [删除事件]特长过滤
  - drop_event:
      # 当
      when:
        # 数据中若[不是]则删除，不收集
        not:
          # 基于模糊匹配
          contains:
            # 匹配techang这个字段；
            techang: '跳伞'

# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

3，重启

[root@kibana ~ ]# systemctl restart filebeat.service

4，写入测试数据

[root@kibana ~ ]# vim /test/1.txt
{"user_ip": "101.101.1.101","username": "张三","age": 18,"sex": "女","xueli": "硕士","techang": ["游泳","跳水","花样滑冰","芭蕾"],"chengji": {"语文": "97","数学": "105","英语": "77","物理": "94","化学": "90","生物": "99"}}
{"user_ip": "101.101.2.101","username": "李四","age": 22,"sex": "男","xueli": "本科","techang": ["跳伞","爬山","计算机","滑雪"],"chengji": {"语文": "66","数学": "144","英语": "30","物理": "60","化学": "49","生物": "100"}}
{"user_ip": "101.101.3.101","username": "王五","age": 20,"sex": "女","xueli": "大专","techang": ["跳伞","跳远","跳高","跳绳"],"chengji": {"语文": "44","数学": "10","英语": "70","物理": "9","化学": "100","生物": "100"}}

5，查看输出文件结果

结论：凡是包含"跳伞"的，都被收集了；

[root@kibana ~ ]# cat /test/result.json-20251207.ndjson 
{"@timestamp":"2025-12-07T09:54:14.292Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"log":{"file":{"path":"/test/1.txt"}},"xueli":"本科","techang":["跳伞","爬山","计算机","滑雪"],"chengji":{"化学":"49","生物":"100","语文":"66","数学":"144","英语":"30","物理":"60"},"username":"李四","sex":"男","message":"","user_ip":"101.101.2.101","age":22}
{"@timestamp":"2025-12-07T09:54:14.292Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"username":"王五","sex":"女","user_ip":"101.101.3.101","log":{"file":{"path":"/test/1.txt"}},"chengji":{"物理":"9","化学":"100","生物":"100","语文":"44","数学":"10","英语":"70"},"message":"","age":20,"xueli":"大专","techang":["跳伞","跳远","跳高","跳绳"]}

· 数值范围删除事件

目标：只收集数值范围内的数据

1，清空数据

[root@kibana ~ ]# echo "" > /test/1.txt 
[root@kibana ~ ]# echo "" > /test/2.txt 
[root@kibana ~ ]# rm -rf /tools/filebeat-8.18.8-linux-x86_64/data/*
[root@kibana ~ ]# rm -rf /test/result.json*

2，编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml 
# 数据的来源设置
filebeat.inputs:
- type: filestream
  id: bakwite_03
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"

  parsers:
  # 调用json解析器
  - ndjson:
    message_key: "message"   # 指定数据字段名（message）
    keys_under_root: true    # 把JSON字段提升到根级
    #add_error_key: true      # 调试时使用，可选：解析失败时添加错误
# 字段处理器
processors:
  # 要删除的字段的动作
  - drop_fields:
      # 要删除哪些字段？
      fields: 
          # 【@metadata】字段无法删除，这里测试看看结果
        - "@metadata"   
        - "agent"
        - "input"
        - "host"
        - "log.offset"
        - "log.flags"
        - "ecs"
      # 删除的字段如果不存在，不报错；
      ignore_missing: true
  # [删除事件]年龄过滤
  - drop_event:
      # 当
      when:
        # 数据中若[不是]则删除，不收集
        not:
          range:
            age:
              gt: 18
  # [删除事件]学历过滤
  - drop_event:
      # 当
      when:
        # 数据中若[不是]则删除，不收集
        not:
          # 下面的条件是或者的关系
          or:
              # 精确匹配。必须完全相等；
            - equals:
                # 学历字段是"本科"
                xueli: "本科"
              # 精确匹配。必须完全相等；
            - equals:
                # 学历字段是"硕士"
                xueli: "硕士"
# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

3，重启

[root@kibana ~ ]# systemctl restart filebeat.service

4，写入测试数据

[root@kibana ~ ]# vim /test/1.txt
{"user_ip": "101.101.1.101","username": "张三","age": 18,"sex": "女","xueli": "硕士","techang": ["游泳","跳水","花样滑冰","芭蕾"],"chengji": {"语文": "97","数学": "105","英语": "77","物理": "94","化学": "90","生物": "99"}}
{"user_ip": "101.101.2.101","username": "李四","age": 22,"sex": "男","xueli": "本科","techang": ["跳伞","爬山","计算机","滑雪"],"chengji": {"语文": "66","数学": "144","英语": "30","物理": "60","化学": "49","生物": "100"}}
{"user_ip": "101.101.3.101","username": "王五","age": 20,"sex": "女","xueli": "大专","techang": ["跳伞","跳远","跳高","跳绳"],"chengji": {"语文": "44","数学": "10","英语": "70","物理": "9","化学": "100","生物": "100"}}

5，查看输出文件结果

结论：年纪字段大于18岁的全部匹配收集了；

[root@kibana ~ ]# cat /test/result.json-20251207.ndjson 
{"@timestamp":"2025-12-07T10:10:51.916Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"log":{"file":{"path":"/test/1.txt"}},"xueli":"本科","techang":["跳伞","爬山","计算机","滑雪"],"message":"","sex":"男","chengji":{"化学":"49","生物":"100","语文":"66","数学":"144","英语":"30","物理":"60"},"age":22,"user_ip":"101.101.2.101","username":"李四"}
{"@timestamp":"2025-12-07T10:10:51.916Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"message":"","age":20,"username":"王五","chengji":{"数学":"10","英语":"70","物理":"9","化学":"100","生物":"100","语文":"44"},"log":{"file":{"path":"/test/1.txt"}},"xueli":"大专","techang":["跳伞","跳远","跳高","跳绳"],"user_ip":"101.101.3.101","sex":"女"}

· 组合删除事件练习

目标只收集：

• 年龄要大于18，小于等于22；
• 学历是本科及以上的；
• ip地址是：101.101.2.x网段的；

1，清空数据

[root@kibana ~ ]# echo "" > /test/1.txt 
[root@kibana ~ ]# echo "" > /test/2.txt 
[root@kibana ~ ]# rm -rf /tools/filebeat-8.18.8-linux-x86_64/data/*
[root@kibana ~ ]# rm -rf /test/result.json*

2，编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml 
# 数据的来源设置
filebeat.inputs:
- type: filestream
  id: bakwite_03
  enabled: true
  # 采集日志数据的路径位置；
  paths:
    - "/test/1.txt"

  parsers:
  # 调用json解析器
  - ndjson:
    message_key: "message"   # 指定数据字段名（message）
    keys_under_root: true    # 把JSON字段提升到根级
    #add_error_key: true      # 调试时使用，可选：解析失败时添加错误
# 字段处理器
processors:
  # 要删除的字段的动作
  - drop_fields:
      # 要删除哪些字段？
      fields: 
          # 【@metadata】字段无法删除，这里测试看看结果
        - "@metadata"   
        - "agent"
        - "input"
        - "host"
        - "log.offset"
        - "log.flags"
        - "ecs"
      # 删除的字段如果不存在，不报错；
      ignore_missing: true
  # [删除事件]年龄过滤
  - drop_event:
      # 当
      when:
        # 数据中若[不是]则删除，不收集
        not:
          and:
            - range:
                age:
                  gt: 18
            - range:
                age:
                  lte: 22
  # [删除事件]ip过滤
  - drop_event:
      # 当
      when:
        # 数据中若[不是]则删除，不收集
        not:
          # 基于正则匹配
          regexp:
            # 匹配user_id这个字段；(注意：正则表达式，必须是单引号)
            user_ip: '^101\.101\.2\.\d+$'


# 输出到本地文件中
output.file:
  path: "/test/"
  filename: "result.json"
  # 一个文件做大可以存储到少字节数据
  rotate_every_kb: 10240
  # 保留多少个最近的文件
  number_of_files: 5
  # 文件的权限
  permissions: 0644

3，重启

[root@kibana ~ ]# systemctl restart filebeat.service

4，写入测试数据

[root@kibana ~ ]# vim /test/1.txt
{"user_ip": "101.101.1.101","username": "张三","age": 18,"sex": "女","xueli": "硕士","techang": ["游泳","跳水","花样滑冰","芭蕾"],"chengji": {"语文": "97","数学": "105","英语": "77","物理": "94","化学": "90","生物": "99"}}
{"user_ip": "101.101.2.101","username": "李四","age": 22,"sex": "男","xueli": "本科","techang": ["跳伞","爬山","计算机","滑雪"],"chengji": {"语文": "66","数学": "144","英语": "30","物理": "60","化学": "49","生物": "100"}}
{"user_ip": "101.101.3.101","username": "王五","age": 20,"sex": "女","xueli": "大专","techang": ["跳伞","跳远","跳高","跳绳"],"chengji": {"语文": "44","数学": "10","英语": "70","物理": "9","化学": "100","生物": "100"}}

5，查看输出文件结果

结果：满足条件的只有一条数据被过滤成功

[root@kibana ~ ]# cat /test/result.json-20251207.ndjson 
{"@timestamp":"2025-12-07T10:25:03.580Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.18.8"},"chengji":{"化学":"49","生物":"100","语文":"66","数学":"144","英语":"30","物理":"60"},"user_ip":"101.101.2.101","age":22,"techang":["跳伞","爬山","计算机","滑雪"],"message":"","username":"李四","sex":"男","xueli":"本科","log":{"file":{"path":"/test/1.txt"}}}

六、生产环境日志采集案例

要求：

• 采集一台机器上的tomcat和nginx日志
• 分别写入两个ES索引中；
• 要求采集的数据仅包含日志本身，不包含filebeat的其他字段；且日志字段在一级字段；

1，准备环境

为了学习方便：

• 我们模拟网站在kibana服务器上；

• 将Filebeat也部署到kibana服务器上收集网站的日志信息；

· nginx网站环境

[root@kibana ~ ]# apt -y install nginx
[root@kibana ~ ]# systemctl start nginx

· tomcat网站环境

1，下载安装JDK

JDK版本分类：

• 开源版OpenJdk【yum/apt安装】
• Oracle官方版【二进制安装或者下载rpm/deb包】

https://www.oracle.com/cn/java/technologies/downloads/

#JDK8是国内使用率最高的一个版本

#下载到本地

# 上传解压软件包
[root@kibana ~ ]# rz -E
[root@kibana ~ ]# tar xf jdk-8u461-linux-x64.tar.gz -C /tools/
[root@kibana ~ ]# ls -l /tools/
......
drwxr-xr-x  8 root   root   4096 Dec  5 05:37 jdk1.8.0_461

#配置环境变量
[root@kibana ~ ]# vim /etc/profile
......
export JAVA_HOME="/tools/jdk1.8.0_461"
export PATH="$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin"
export CLASSPATH="$CLASSPATH:$JAVA_HOME/lib/:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar"

# 生效环境变量
[root@kibana ~ ]# source  /etc/profile

# 检查
[root@kibana ~ ]# java -version
java version "1.8.0_461"
Java(TM) SE Runtime Environment (build 1.8.0_461-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.461-b11, mixed mode)

2，下载安装tomcat

https://tomcat.apache.org

# 上传解压软件包
[root@kibana ~ ]# rz -E
[root@kibana ~ ]# ls -l
-rw-r--r-- 1 root root  13043951 Dec  5 05:43 apache-tomcat-9.0.112.tar.gz
[root@kibana ~ ]# tar xf apache-tomcat-9.0.112.tar.gz -C /tools/

# 编辑JDK环境变量文件
[root@kibana ~ ]# vim /tools/apache-tomcat-9.0.112/jdk
JAVA_HOME=/tools/jdk1.8.0_461/
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
CLASSPATH=$CLASSPATH:$JAVA_HOME/lib/:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar

# 编辑system文件
[root@kibana ~ ]# vim /lib/systemd/system/tomcat.service

[Unit]
Description=Tomcat server daemon
After=network.target

[Service]
Type=forking
EnvironmentFile=/tools/apache-tomcat-9.0.112/jdk
ExecStart="/tools/apache-tomcat-9.0.112/bin/startup.sh"
ExecStop="/tools/apache-tomcat-9.0.112/bin/shutdown.sh"

[Install]
WantedBy=multi-user.target

# 启动tomcat
[root@kibana ~ ]# systemctl daemon-reload
[root@kibana ~ ]# systemctl start tomcat.service

# 访问测试
10.0.0.110:8080

3，查看两个服务访问日志

[root@kibana ~ ]# cat /tools/apache-tomcat-9.0.112/logs/localhost_access_log.2025-12-05.txt 
10.0.0.1 - - [05/Dec/2025:05:51:58 +0000] "GET / HTTP/1.1" 200 11232
10.0.0.1 - - [05/Dec/2025:05:51:58 +0000] "GET /tomcat.css HTTP/1.1" 200 5584
10.0.0.1 - - [05/Dec/2025:05:51:58 +0000] "GET /tomcat.svg HTTP/1.1" 200 67795
10.0.0.1 - - [05/Dec/2025:05:51:58 +0000] "GET /bg-upper.png HTTP/1.1" 200 3103
10.0.0.1 - - [05/Dec/2025:05:51:58 +0000] "GET /asf-logo-wide.svg HTTP/1.1" 200 7089
10.0.0.1 - - [05/Dec/2025:05:51:58 +0000] "GET /bg-middle.png HTTP/1.1" 200 1918
10.0.0.1 - - [05/Dec/2025:05:51:58 +0000] "GET /bg-button.png HTTP/1.1" 200 713
10.0.0.1 - - [05/Dec/2025:05:51:58 +0000] "GET /bg-nav.png HTTP/1.1" 200 1401
10.0.0.1 - - [05/Dec/2025:05:51:58 +0000] "GET /favicon.ico HTTP/1.1" 200 21630
10.0.0.1 - - [05/Dec/2025:05:53:57 +0000] "GET /index.jsp HTTP/1.1" 200 11232


[root@kibana ~ ]# cat /var/log/nginx/access.log 
10.0.0.1 - - [05/Dec/2025:05:22:02 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36"
10.0.0.1 - - [05/Dec/2025:05:22:02 +0000] "GET /favicon.ico HTTP/1.1" 404 197 "http://10.0.0.210/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36"

· 日志格式修改为json

1，修改nginx日志格式为json

字段/变量	含义说明
"@timestamp":"$time_iso8601"	ISO 8601 格式的时间戳：【2025-12-08T10:30:45+08:00】 【@timestamp】：elasticsearch标准字段名称，ELK专用；
$server_addr	当前服务器IP地址
【$http_host】或【$host】或【$domain】	host请求头
$remote_addr	客户端 IP 地址
$body_bytes_sent	响应主体的字节大小
$http_x_forwarded_for	代理链条中的客户端【真实用户ip地址】
$upstream_response_time	上游服务器响应时间/秒
$upstream_addr	上游服务器ip地址
$http_referer	用户从哪里来
$http_user_agent	用户的客户端
$status	响应状态码
$uri	用户请求的资源路径
$request_time	请求处理的时间，【请求】到【相应完成】的时间

[root@kibana ~ ]# vim /etc/nginx/nginx.conf 
......
http {
......
        log_format gongzheng_nginx_json '{"@timestamp":"$time_iso8601",'
                                      '"host":"$server_addr",'
                                      '"clientip":"$remote_addr",'
                                      '"SendBytes":$body_bytes_sent,'
                                      '"responsetime":$request_time,'
                                      '"upstreamtime":"$upstream_response_time",'
                                      '"upstreamhost":"$upstream_addr",'
                                      '"http_host":"$host",'
                                      '"uri":"$uri",'
                                      '"domain":"$host",'
                                      '"xff":"$http_x_forwarded_for",'
                                      '"referer":"$http_referer",'
                                      '"tcp_xff":"$proxy_protocol_addr",'
                                      '"http_user_agent":"$http_user_agent",'
                                      '"status":"$status"}';
        access_log  /var/log/nginx/access.log  gongzheng_nginx_json;
......

2，修改tomcat日志格式为json

符号/变量	含义说明
&quot；	表示双引号【"】
%h	表示客户端ip地址【10.0.0.200】
%l	客户端标识
%u	HTTP认证用户名
%t	访问时间
%r	HTTP请求行
%s	HTTP响应状态码
%b	响应字节大小
%q	请求URL时传递的参数
%{Referer}i	HTTP请求头Referer
%{User-Agent}i	HTTP请求头User-Agent

[root@kibana ~ ]# vim /tools/apache-tomcat-9.0.112/conf/server.xml
......
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common" -->
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="tomcat_access_log" suffix=".txt"
               pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;request&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;http_user_agent&quot;:&quot;%{User-Agent}i&quot;}" />

      </Host>
.........

3，重启服务

[root@kibana ~ ]# systemctl restart nginx.service 
[root@kibana ~ ]# systemctl restart tomcat.service

4，访问测试日志格式

http://10.0.0.210/

http://10.0.0.210:8080/

# 查看nginx日志
[root@kibana ~ ]# tail -1 /var/log/nginx/access.log
{"@timestamp":"2025-12-08T03:38:38+00:00","host":"10.0.0.210","clientip":"10.0.0.1","SendBytes":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.210","uri":"/index.nginx-debian.html","domain":"10.0.0.210","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36","status":"304"}

# 查看tomcat日志
[root@kibana ~ ]# tail -1 /tools/apache-tomcat-9.0.112/logs/tomcat_access_log.2025-12-08.txt 
{"clientip":"10.0.0.1","ClientUser":"-","authenticated":"-","AccessTime":"[08/Dec/2025:03:39:08 +0000]","request":"GET / HTTP/1.1","status":"200","SendBytes":"11232","Query?string":"","partner":"-","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36"}

2，kibana创建索引模板

· 编辑名称与模式

· settings设置

{
  "number_of_shards": "3",
  "number_of_replicas":1
}

· mappings设置

# 主要映射时间、ip、字节

· alias设置

{
"web_logs": {}
}

· 创建模板

# 点击创建即可

3，配置filebeat收集

· 编辑配置文件

[root@kibana ~ ]# vim /tools/filebeat-8.18.8-linux-x86_64/filebeat.yml
filebeat.inputs:
- type: filestream
  id: bakwite_01
  enabled: true
  paths:
    - "/var/log/nginx/access.log"
- type: filestream
  id: bakwite_02
  enabled: true
  paths:
    - "/tools/apache-tomcat-9.0.112/logs/tomcat_access_log.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].txt"

  parsers:
  # 调用json解析器
  - ndjson:
    message_key: "message"   # 指定数据字段名（message）
    keys_under_root: true    # 把JSON字段提升到根级
# 字段处理器
processors:
  # 要删除的字段的动作
  - drop_fields:
      # 要删除哪些字段？
      fields: 
        - "agent"
        - "input"
        - "host"
        - "ecs"
      # 删除的字段如果不存在，不报错；
      ignore_missing: true


# 数据的出口设置；
output.elasticsearch:
  # ES集群主机地址（自动负载均衡）
  hosts: 
    - "https://10.0.0.101:10200"
    - "https://10.0.0.102:10200"
    - "https://10.0.0.103:10200"
  # ES的用户密码
  username: "elastic"
  password: "123456"
  # ES的SSl证书验证
  ssl:
    # 启动这个ssl证书验证
    enabled: true
    # 验证证书的有效性
    verification_mode: certificate
    # 验证ES公钥的CA证书；
    certificate_authorities: 
      # ES的ca根证书（复制过来再启动filebeat）；
      - "/tools/filebeat-8.18.8-linux-x86_64/certs/ca.crt"
  # 配置写入索引
  indices:
    # 定义索引名称
    - index: "elk-wa01%{+yyyy.MM.dd}"
      when:
        # [equals]表示完全精准匹配
        # [contains]表示模糊匹配
        equals:
          # 【提示】：这个字段是从输出内容字段中匹配的；
          log.file.path: "/var/log/nginx/access.log"
    # 定义索引名称
    - index: "elk-wa02%{+yyyy.MM.dd}"
      when:
        # 由于tomcat日志后面是日期切割的，所以我们要模糊匹配；
        contains:
          log.file.path: "/tools/apache-tomcat-9.0.112/logs/tomcat_access_log"

setup.ilm.enabled: false               # 关闭滚动更新

· 重启filebeat

[root@kibana ~ ]# systemctl restart filebeat.service

4，查看kibana页面验证

Discussion

评论

加载中

正在检查登录状态…