网站建设云,雅淇wordpress,西安制作公司网页多少钱,长治网站设计Centos7 随笔记录记录 docker compose 统一管理 granfana loki vector 监控kafka 信息。
当然如果仅仅是想通过 Grafana 监控kafka#xff0c;推荐使用 Grafana Prometheus 通过JMX监控kafka
目录
1. 目录结构
2. 前提已安装Docker-Compose
3. docker-compose 自定义服…Centos7 随笔记录记录 docker compose 统一管理 granfana loki vector 监控kafka 信息。
当然如果仅仅是想通过 Grafana 监控kafka推荐使用 Grafana Prometheus 通过JMX监控kafka
目录
1. 目录结构
2. 前提已安装Docker-Compose
3. docker-compose 自定义服务相关配置文件
3.1 配置 docker-compose-grafana-loki-vector.yml 文件
3.2 创建 loki-config.yaml
3.3 配置loki 服务
4. docker-compose 单独启动指定服务
4.1 单独启动 loki
4.1.1 启动loki
4.1.2 查询 log
4.2 单独启动vector
4.2.1 启动 vector
4.2.2 查询vector log
4.2.2.1 错误log 问题定位
4.2.2.2 解决方法
4.2.2.3 重启vector 容器查看无error log
4.3 单独启动grafana
4.3.1 启动 grafana
4.3.2 查看 grafana log
4.3.2.1 错误log 问题定位
4.3.2.2 解决方法
4.3.2.3 重启grafana 容器查看无error log
4.3.3 访问grafana
4.4 查看loki metrics数据 和loki 运行状态
5. docker-compose 启停所有服务
5.1 docker-compose 启动所有自定义容器服务
5.2 docker-compose 停止所有容器服务
6. 配置grafana
6.1 添加loki 数据源
6.2 修改时区
6.3 添加query
6.4 过滤展示kafka 消息
6.5 优化后 dashboard, 展示kafka 指定字段
6.5.1 添加添加dashboard
6.5.2 优化dashboard, 显示指定字段
6.5.2.1 筛选kafka 原始消息 6.5.2.2 优化表格显示指定字段
7. 其他异常问题
7.1 grafana 页面有时跳转 网址 http://localhost:3000/XXXX 错误
7.2 跳转错误网址原因定位
7.3 解决方法
7.3.1 Grafana 服务的环境变量 GF_SERVER_ROOT_URL 设置为正确的访问地址
7.3.2 为了grafana 有数据必须修改vector.tml 配置参数
7.3.3 使配置生效 7.4 验证 跳转网址正确 1. 目录结构
# 1 创建 4个文件夹grafana loki vector wal
# 2 为 wal 赋777 权限[rootlocalhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1859 Dec 26 10:58 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# tree
.
├── docker-compose-grafana-loki-vector.yml
└── grafana_loki_vector # 创建 grafana folder ├── grafana├── loki # 创建 loki folder│ └── loki-config.yaml├── vector # 创建 vector folder│ └── vector.toml└── wal # 创建 wal folder,并赋 777 权限8 directories, 5 files
[rootlocalhost grafana_loki_vector]## 为避免docker 中权限问题为wal folder 赋权限[rootlocalhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector/grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# chmod 777 wal
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# ll
total 0
drwxr-xr-x. 4 root root 41 Dec 26 10:04 grafana
drwxr-xr-x. 2 root root 44 Dec 26 10:47 loki
drwxr-xr-x. 2 root root 33 Dec 21 15:03 vector
drwxrwxrwx. 3 root root 59 Dec 26 11:19 wal
[rootlocalhost grafana_loki_vector]#2. 前提已安装Docker-Compose
如何安装 Docker-Compose请参考 安装部署Docker-Compose
3. docker-compose 自定义服务相关配置文件
3.1 配置 docker-compose-grafana-loki-vector.yml 文件
# docker-compose 配置参考https://github.com/grafana/loki/blob/master/production/docker-compose.yaml[rootlocalhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1859 Dec 26 10:58 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# cat docker-compose-grafana-loki-vector.ymlversion: 3 # 这是Docker Compose的版本指定该配置文件使用版本3的语法# 网桥loki - 方便相互通讯
networks: # 定义了一个名为loki的网络该网络是为了Loki和其他服务之间相互通信而创建的网桥。loki:services: # 在这个部分定义了要运行的各个服务loki: # Loki服务的配置image: grafana/loki:latest # 使用了grafana/loki:latest镜像container_name: grafana_loki_vector_demo_loki # 定义 容器名restart: unless-stopped # 设置了重启策略ports:- 3100:3100 # 将主机的3100端口映射到Loki容器的3100端口以便能够从主机访问Loki的APIvolumes: # 将两个主机目录与容器内的目录进行了卷映射- ./grafana_loki_vector/loki:/etc/loki - ./grafana_loki_vector/wal:/wal command: -config.file/etc/loki/loki-config.yamlnetworks:- loki# vector 用于对接kafka 消息vector: # vector 服务的配置# 指定使用 timberio/vector的Docker 镜像并使用0.10.0-alpine 版本image: timberio/vector:0.10.0-alpine container_name: grafana_loki_vector_demo_vector # 定义 容器名restart: unless-stopped # 指定容器在退出时重新启动除非显式停止ports:- 80:80 # 容器的 80 端口映射到主机的 80 端口上volumes:# - ./grafana_loki_vector/vector/logs:/var/logs# 本地主机目录./grafana_loki_vector/vector挂载到容器的/etc/vector目录上- ./grafana_loki_vector/vector:/etc/vector# 这是要在容器中运行的命令通过-c /etc/vector/vector.toml指定了配置文件的路径并通过 -v 启用了详细日志输出command: -c /etc/vector/vector.toml -vnetworks:- loki# 用于UI展示grafana:image: grafana/grafana:latestcontainer_name: grafana_loki_vector_demo_grafanarestart: unless-stoppedports:- 3000:3000volumes:- ./grafana_loki_vector/grafana/data:/var/lib/grafana# - ./grafana_promtail_loki/grafana/config:/etc/grafana- ./grafana_loki_vector/grafana/log:/var/log/grafanaenvironment:# GF_EXPLORE_ENABLED: true# 避免有时Grafana 会将跳转 URL 设置为 localhost- GF_SERVER_ROOT_URLhttp://192.168.2.247:3000 networks:- loki
配置参数说明volumes: # 将两个主机目录与容器内的目录进行了卷映射- ./grafana_loki_vector/loki:/etc/loki # ./grafana_loki_vector/loki映射到了/etc/loki- ./grafana_loki_vector/wal_tmp1:/wal # ./grafana_loki_vector/wal_tmp1映射到了/walcommand: -config.file/etc/loki/loki-local-config.yaml
# -config.file/etc/loki/loki-local-config.yaml 指定了Loki容器启动时使用的配置文件路径environment:# 避免有时Grafana 会将跳转 URL 设置为 localhost- GF_SERVER_ROOT_URLhttp://192.168.2.247:3000
3.2 创建 loki-config.yaml
创建loki-local-config.yaml 文件配置loki 服务[rootlocalhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# tree
.
├── docker-compose-grafana-loki-vector.yml
└── grafana_loki_vector├── grafana│ ├── data│ └── log├── loki│ └── loki-config.yaml # 创建loki-config.yaml文件配置loki 服务├── vector│ └── vector.toml└── wal3.3 配置loki 服务
# 这个是Loki的配置文件示例其中包含了各个方面的配置用于自定义和配置Loki日志系统的行为和存储。auth_enabled: false # 设置认证功能是否启用。设为false表示禁用认证即任何用户都可以访问Loki服务server:http_listen_port: 3100 # 配置Loki服务监听的HTTP端口。在这个示例中Loki将监听3100端口ingester: # 配置Loki的ingester摄取器相关参数lifecycler:address: 127.0.0.1ring:kvstore: # 使用内存作为kvstore存储方式store: inmemoryreplication_factor: 1 # 复制因子设置为 1final_sleep: 0schunk_idle_period: 5m # chunk的闲置期 5mchunk_retain_period: 30s # chunk的保留期 30sschema_config: # 配置存储模式参数configs:- from: 2018-04-15 # 从特定日期开始使用的配置store: boltdb # 数据存储方式boltdb文件系统object_store: filesystem # 数据存储方式 (文件系统)schema: v9 # schema版本index: # 索引相关参数prefix: index_period: 168hstorage_config: # 配置存储配置指定Loki数据的持久化存储位置boltdb:directory: /tmp/loki/index # 索引数据将存储在/tmp/loki/index目录filesystem:directory: /tmp/loki/chunks # chunk数据将存储在/tmp/loki/chunks目录# 配置限制参数用于强制限制指标名称、拒绝旧样本数据以及设置拒绝旧样本数据的最大时间范围
limits_config: enforce_metric_name: false # 强制限制指标名称reject_old_samples: true # 拒绝旧样本数据reject_old_samples_max_age: 168h # 设置拒绝旧样本数据的最大时间范围chunk_store_config: # 配置chunk存储参数设置最大回溯时间范围max_look_back_period: 0s # 设置最大回溯时间范围table_manager: # 配置表管理器相关参数chunk_tables_provisioning: # 设置chunk表的吞吐量inactive_read_throughput: 0 inactive_write_throughput: 0provisioned_read_throughput: 0provisioned_write_throughput: 0index_tables_provisioning: # 设置索引表的吞吐量inactive_read_throughput: 0inactive_write_throughput: 0provisioned_read_throughput: 0provisioned_write_throughput: 0retention_deletes_enabled: false # 保留删除是否启用retention_period: 0s # 保留期设置# retention_deletes_enabled: true# retention_period: 168h
4. docker-compose 单独启动指定服务
建议docker-compose 自定义多个服务时先单独启动每个服务调试没问题后再统一启动
4.1 单独启动 loki
4.1.1 启动loki # 启动指定自定义服务 # docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up 服务名 -d# 启动loki[rootlocalhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up loki -d
[] Running 7/7✔ loki 6 layers [⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 24.0s✔ c926b61bad3b Pull complete 12.0s✔ 21a0947e3c21 Pull complete 6.0s✔ 19779106cfff Pull complete 17.5s✔ 9c6ee7a30a41 Pull complete 7.4s✔ 2f9c53357ffd Pull complete 8.7s✔ d8cb9832c63f Pull complete 11.4s
[] Running 1/1✔ Container grafana_loki_vector_demo_loki Started 0.2s
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# docker ps # 查询启动容器
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
58528f1b4782 grafana/loki:latest /usr/bin/loki -conf… 9 seconds ago Up 8 seconds 0.0.0.0:3100-3100/tcp, :::3100-3100/tcp grafana_loki_vector_demo_loki
c4f7b69648a8 wurstmeister/kafka start-kafka.sh 18 hours ago Up 18 hours 0.0.0.0:9092-9092/tcp, :::9092-9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper /bin/sh -c /usr/sb… 18 hours ago Up 18 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181-2181/tcp, :::2181-2181/tcp zookeeper
[rootlocalhost grafana_loki_vector]#4.1.2 查询 log
# 查看 loki 容器 log
# docker logs loki服务容器名[rootlocalhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_loki
levelwarn ts2023-12-26T03:39:33.891828641Z callerloki.go:286 msgper-tenant timeout not configured, using default engine timeout (\5m0s\). This behavior will change in the next major to always use the default per-tenant timeout (\5m\).levelinfo ts2023-12-26T03:44:33.988929326Z callercheckpoint.go:502 msgatomic checkpoint finished oldwal/checkpoint.000006.tmp newwal/checkpoint.000006
[rootlocalhost grafana_loki_vector]#4.2 单独启动vector
4.2.1 启动 vector # 启动指定自定义服务 # docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up 服务名 -d# 启动 vector[rootlocalhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up vector -d
[] Running 6/6✔ vector 5 layers [⣿⣿⣿⣿⣿] 0B/0B Pulled 30.6s✔ 21c83c524219 Pull complete 18.0s✔ 88943f7483ee Pull complete 2.9s✔ 24da72aaf03b Pull complete 23.4s✔ 4145aaf02e3f Pull complete 10.7s✔ b16d307dc057 Pull complete 12.0s
[] Running 1/1✔ Container grafana_loki_vector_demo_vector Started 0.3s
[rootlocalhost grafana_loki_vector]#[rootlocalhost grafana_loki_vector]# docker ps # 查询启动容器
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
776ab907da32 timberio/vector:0.10.0-alpine /usr/local/bin/vect… About a minute ago Up About a minute 0.0.0.0:80-80/tcp, :::80-80/tcp grafana_loki_vector_demo_vector
2a5581eea2eb grafana/loki:latest /usr/bin/loki -conf… 2 minutes ago Up 2 minutes 0.0.0.0:3100-3100/tcp, :::3100-3100/tcp grafana_loki_vector_demo_loki
c4f7b69648a8 wurstmeister/kafka start-kafka.sh 21 hours ago Up 21 hours 0.0.0.0:9092-9092/tcp, :::9092-9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper /bin/sh -c /usr/sb… 21 hours ago Up 21 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181-2181/tcp, :::2181-2181/tcp zookeeper
[rootlocalhost grafana_loki_vector]#4.2.2 查询vector log
# 查看 vector 容器log
# docker logs vector 容器名[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_vector
Dec 26 06:02:01.842 INFO vector: Log level debug is enabled.
Dec 26 06:02:01.857 INFO vector: Loading configs. path[/etc/vector/vector.toml]
Dec 26 06:02:01.862 INFO vector: Vector is starting. version0.10.0 git_versionv0.9.0-377-g0f0311a releasedWed, 22 Jul 2020 20:01:29 0000 archx86_64
Dec 26 06:02:01.876 DEBUG vector::tls::settings: Fetching system root certs.
Dec 26 06:02:01.876 INFO vector::topology: Running healthchecks.
Dec 26 06:02:01.876 INFO vector::topology: Starting source in
Dec 26 06:02:01.876 INFO vector::topology: Starting sink out
Dec 26 06:02:01.876 DEBUG http: vector::sinks::util::http: sending request. urihttp://loki:3100/ready methodGET
Dec 26 06:02:01.884 DEBUG http: vector::sinks::util::http: response. status503 versionHTTP/1.1
Dec 26 06:02:01.885 ERROR vector::topology::builder: Healthcheck: Failed Reason: A non-successful status returned: 503 Service Unavailable # Error Log
Dec 26 06:02:05.978 DEBUG http: vector::sinks::util::http: sending request. urihttp://loki:3100/loki/api/v1/push methodPOST
Dec 26 06:02:06.060 DEBUG http: vector::sinks::util::http: response. status204 versionHTTP/1.1
[rootlocalhost grafana_loki_vector]#4.2.2.1 错误log 问题定位
# 确认 Vector 可以连接到 Loki 服务
在 Vector 配置文件中配置的 Loki 服务的地址应该是可访问的。
可以尝试使用 curl 或类似的工具连接到 Loki 服务# curl http://loki:3100/ready[rootlocalhost grafana_loki_vector]# curl http://loki:3100/ready
curl: (6) Could not resolve host: loki; Unknown error
[rootlocalhost grafana_loki_vector]#总结 curl 请求无法解析 loki 主机名并出现了 Unknown error 错误vector 无法连接loki
4.2.2.2 解决方法
1 检查 DNS 配置确认您的 DNS 配置正确可以解析主机名
# nslookup loki[rootlocalhost grafana_loki_vector]# nslookup loki
Server: 114.114.114.114
Address: 114.114.114.114#53** server cant find loki: NXDOMAIN # loki无法解析为有效的 IP 地址返回了 NXDOMAIN 错误。[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#2添加主机名映射如果 DNS 解析不起作用您可以尝试手动添加主机名到 IP 地址的映射[rootlocalhost grafana_loki_vector]# cat /etc/hosts # 添加前
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# cat /etc/hosts # 添加后
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.247 loki
[rootlocalhost grafana_loki_vector]#3检查vector 是否可以连接loki
# curl http://loki:3100/ready[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# curl http://loki:3100/ready
ready # 可以连接loki
[rootlocalhost grafana_loki_vector]#
4.2.2.3 重启vector 容器查看无error log
# 重启vector 容器 or 重新创建 vector 容器
# 无error log, vector 服务运行正常[rootlocalhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_vector
Dec 26 06:20:58.569 INFO vector: Log level debug is enabled.
Dec 26 06:20:58.583 INFO vector: Loading configs. path[/etc/vector/vector.toml]
Dec 26 06:20:58.588 INFO vector: Vector is starting. version0.10.0 git_versionv0.9.0-377-g0f0311a releasedWed, 22 Jul 2020 20:01:29 0000 archx86_64
Dec 26 06:20:58.602 DEBUG vector::tls::settings: Fetching system root certs.
Dec 26 06:20:58.602 INFO vector::topology: Running healthchecks.
Dec 26 06:20:58.602 INFO vector::topology: Starting source in
Dec 26 06:20:58.602 INFO vector::topology: Starting sink out
Dec 26 06:20:58.602 DEBUG http: vector::sinks::util::http: sending request. urihttp://loki:3100/ready methodGET
Dec 26 06:20:58.606 DEBUG http: vector::sinks::util::http: response. status200 versionHTTP/1.1
Dec 26 06:20:58.607 INFO vector::topology::builder: Healthcheck: Passed.
[rootlocalhost grafana_loki_vector]#4.3 单独启动grafana
4.3.1 启动 grafana # 启动指定自定义服务 # docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up 服务名 -d[rootlocalhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up grafana -d
[] Running 11/11✔ grafana 10 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 106.8s✔ 96526aa774ef Pull complete 2.3s✔ 8107a14e7703 Pull complete 1.4s✔ 41aaa20cd87e Pull complete 2.7s✔ e00aef65f013 Pull complete 5.1s✔ 257ea9b0297f Pull complete 5.5s✔ 599ebbeda3b8 Pull complete 6.1s✔ ca6b21901a74 Pull complete 59.8s✔ b5fb47d1c454 Pull complete 92.5s✔ 4fc0f14a8243 Pull complete 8.0s✔ f638e65e9fc0 Pull complete 13.7s
[] Running 1/1✔ Container grafana_loki_vector_demo_grafana Started 0.3s
[rootlocalhost grafana_loki_vector]#4.3.2 查看 grafana log
# 查看 vector 容器log
# docker logs vector 容器名[rootlocalhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_grafana
GF_PATHS_DATA/var/lib/grafana is not writable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
mkdir: cant create directory /var/lib/grafana/plugins: Permission denied
.....
....[rootlocalhost grafana_loki_vector]#4.3.2.1 错误log 问题定位
这些错误通常表示在容器中的某个目录或文件上没有写入权限。
4.3.2.2 解决方法
# 解决此问题的一种可能方法是确保您在运行容器时具有足够的权限# 确保您以具有适当权限的用户身份运行 Docker 命令
# sudo usermod -aG docker $USER# 更改目标目录的所有者和权限
sudo chown -R 472:472 ./grafana_loki_vector/grafana/data
sudo chmod -R 755 ./grafana_loki_vector/grafana/data# 检查您的目标目录权限并确保它具有足够的写入权限[rootlocalhost grafana_loki_vector]# sudo chmod -R 777 ./grafana_loki_vector/grafana/data
[rootlocalhost grafana_loki_vector]#4.3.2.3 重启grafana 容器查看无error log
# 重启grafana 容器 or 重新创建 grafana 容器
# 无error log, grafana服务运行正常# 此处每次都是
1 停止容器 # docker stop container_ID or container_name
2 删除容器 # docker rm container_ID or container_name
3 删除镜像 # docker rmi image_ID
4) 创建 grafana 容器
5 查看 log[rootlocalhost grafana_loki_vector]# docker stop 839b53f097ee # 停止容器
839b53f097ee
[rootlocalhost grafana_loki_vector]# docker rm 839b53f097ee # 删除容器
839b53f097ee
[rootlocalhost grafana_loki_vector]# docker images # 查询镜像ID
REPOSITORY TAG IMAGE ID CREATED SIZE
grafana/grafana latest 8387f19108f9 7 days ago 399MB
grafana/loki latest c2c89eb81f6e 13 days ago 67.7MB
hello-world latest 9c7a54a9a43c 7 months ago 13.3kB
wurstmeister/kafka latest a692873757c0 19 months ago 468MB
timberio/vector 0.10.0-alpine 2ebd263245dd 3 years ago 80.7MB
wurstmeister/zookeeper latest 3f43f72cb283 4 years ago 510MB
[rootlocalhost grafana_loki_vector]# docker rmi 8387f19108f9 # 删除镜像
Untagged: grafana/grafana:latest
Untagged: grafana/grafanasha256:6b5b37eb35bbf30e7f64bd7f0fd41c0a5b7637f65d3bf93223b04a192b8bf3e2
Deleted: sha256:8387f19108f986254ecb7eeb1124445a9aeda16602114d98348ed5cd49151a7a
Deleted: sha256:06619c984d0071cd4b3a8fbfb377c1ca5b991497f4e1f6c2f280b9f33d402cb4
Deleted: sha256:5fe5dac0845a9eb44e2c5cfed4fb2754a4adf3224209f58c1a46cb7bd250f2e7
Deleted: sha256:91a3b38581f407bb488e03f002566138875732ba1da4a5e91c55b3dec08a2756
Deleted: sha256:18c0364cc25d61f2e9c8ba8582944e1fc6528ba44abc25ffc3e3bc5ed6a8dec9
Deleted: sha256:e335659dfae7632b652b2c796e742f11b2a9f0c0c7def8511b4df29158a50ee5
Deleted: sha256:5f7c41d4921ce413887919068764757d42744f4898c3f790953fee2e1b33b79b
Deleted: sha256:e8df864ddb32cefc11b6a549add118948327816e858570f6fcbb4b393643d196
Deleted: sha256:14b7f68d3d9094af53f80ebf90273ec864b731545874ee30cfb09cae5d097935
Deleted: sha256:baf151d12231a621fead4ca244823c83fb4ae6e01c36501e37c48a964138e724
Deleted: sha256:cc2447e1835a40530975ab80bb1f872fbab0f2a0faecf2ab16fbbb89b3589438
[rootlocalhost grafana_loki_vector]## 创建 grafana 容器 [rootlocalhost grafana_loki_vector]# sudo docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up grafana -d
[] Running 11/11✔ grafana 10 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 113.6s✔ 96526aa774ef Pull complete 12.7s✔ 8107a14e7703 Pull complete 2.3s✔ 41aaa20cd87e Pull complete 2.2s✔ e00aef65f013 Pull complete 10.4s✔ 257ea9b0297f Pull complete 5.5s✔ 599ebbeda3b8 Pull complete 10.5s✔ ca6b21901a74 Pull complete 83.9s✔ b5fb47d1c454 Pull complete 97.5s✔ 4fc0f14a8243 Pull complete 17.4s✔ f638e65e9fc0 Pull complete 28.6s
[] Running 1/1✔ Container grafana_loki_vector_demo_grafana Started 0.5s
[rootlocalhost grafana_loki_vector]# # 查看 log
[rootlocalhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_grafana
loggersettings t2023-12-26T07:25:30.701353225Z levelinfo msgStarting Grafana version10.2.3 commit1e84fede543acc892d2a2515187e545eb047f237 branchHEAD compiled2023-12-18T15:46:07Z
loggersettings t2023-12-26T07:25:30.701604782Z levelinfo msgConfig loaded from file/usr/share/grafana/conf/defaults.ini
......
......
...... loggerngalert.scheduler t2023-12-26T07:26:18.891340858Z levelinfo msgStarting scheduler tickInterval10s
loggerticker t2023-12-26T07:26:18.891454409Z levelinfo msgstarting first_tick2023-12-26T07:26:20Z
loggerhttp.server t2023-12-26T07:26:18.897364437Z levelinfo msgHTTP Server Listen address[::]:3000 protocolhttp subUrl socket
loggerplugins.update.checker t2023-12-26T07:26:19.502508394Z levelinfo msgUpdate check succeeded duration611.312832ms
loggergrafana.update.checker t2023-12-26T07:26:20.504658267Z levelinfo msgUpdate check succeeded duration1.613658029s
loggerinfra.usagestats t2023-12-26T07:26:51.904796949Z levelinfo msgUsage stats are ready to report
[rootlocalhost grafana_loki_vector]# 4.3.3 访问grafana 打开 http://grafana 容器所在主机IP:3000 默认 user/pwd: admin/admin 4.4 查看loki metrics数据 和loki 运行状态 # 查看loki metrics数据 # 访问http://loki 容器所在主机IP:3100/metrics # 查看loki运行状态
# 访问http://loki 容器所在主机IP:3100/ready 到此已经调试完成可以通过docker-compose 统一启动所有服务
5. docker-compose 启停所有服务
5.1 docker-compose 启动所有自定义容器服务
# 启动 定义服务
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d[rootlocalhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
[] Running 3/3✔ Container grafana_loki_vector_demo_loki Started 0.0s✔ Container grafana_loki_vector_demo_vector Started 0.0s✔ Container grafana_loki_vector_demo_grafana Started 0.0s
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# docker ps # 查询启动容器
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
252ca4e81d6f grafana/grafana:latest /run.sh 2 hours ago Up 2 seconds 0.0.0.0:3000-3000/tcp, :::3000-3000/tcp grafana_loki_vector_demo_grafana
7068d0da2577 timberio/vector:0.10.0-alpine /usr/local/bin/vect… 3 hours ago Up 2 seconds 0.0.0.0:80-80/tcp, :::80-80/tcp grafana_loki_vector_demo_vector
2a5581eea2eb grafana/loki:latest /usr/bin/loki -conf… 3 hours ago Up 2 seconds 0.0.0.0:3100-3100/tcp, :::3100-3100/tcp grafana_loki_vector_demo_loki
c4f7b69648a8 wurstmeister/kafka start-kafka.sh 24 hours ago Up 24 hours 0.0.0.0:9092-9092/tcp, :::9092-9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper /bin/sh -c /usr/sb… 24 hours ago Up 24 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181-2181/tcp, :::2181-2181/tcp zookeeper
[rootlocalhost grafana_loki_vector]# 5.2 docker-compose 停止所有容器服务
# docker-compose 停止所有容器服务# docker-compose -f docker-compose 自定义容器服务配置 yml 文件 -p 项目名 down[rootlocalhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1909 Dec 28 17:27 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down
[] Running 4/4✔ Container grafana_loki_vector_demo_vector Removed 0.4s✔ Container grafana_loki_vector_demo_loki Removed 2.0s✔ Container grafana_loki_vector_demo_grafana Removed 0.0s✔ Network grafana_loki_vector_loki Removed 0.2s
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c4f7b69648a8 wurstmeister/kafka start-kafka.sh 3 days ago Up 3 days 0.0.0.0:9092-9092/tcp, :::9092-9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper /bin/sh -c /usr/sb… 3 days ago Up 3 days 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181-2181/tcp, :::2181-2181/tcp zookeeper
b38e9b5b6a2e hello-world /hello 3 weeks ago Exited (0) 3 weeks ago infallible_rosalind
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
grafana/grafana latest 8387f19108f9 9 days ago 399MB
grafana/loki latest c2c89eb81f6e 2 weeks ago 67.7MB
hello-world latest 9c7a54a9a43c 7 months ago 13.3kB
wurstmeister/kafka latest a692873757c0 19 months ago 468MB
timberio/vector 0.10.0-alpine 2ebd263245dd 3 years ago 80.7MB
wurstmeister/zookeeper latest 3f43f72cb283 4 years ago 510MB
[rootlocalhost grafana_loki_vector]#6. 配置grafana
6.1 添加loki 数据源
如果找不到data Source, 可以直接search 查找。 6.2 修改时区 6.3 添加query
click Home - Explore - change loki option 6.4 过滤展示kafka 消息
原始kafka 消息数据{currentIndex:0,header:{headerMap:{35:381166}},mapList:[{10007:1},{37:48,38:300000,44:15007900,48:688021,1301:1,11001:国盾量子,11436:20231227,28000:20231227,28001:1703642600915,28005:2,28101:893786271,43220:金额:[3000000,450416980],数量:[300000,300100],价格涨跌幅:[0.02,0.19934],市场成交量占比:[0.3,0.967752],区间内最高卖出成交价: 1799.8,昨收盘价: 1500,下单数量: 300000,当前交易价格: 1500.79,挂单数量: 0,挂单金额: 0,成交数量: 100,成交金额: 179980,市场成交数量: 10100,64101:2403,64102:1101,64103:2403-4096个账户-4096个证券,65098:1,66002:5,66003:1,66004:0,66005:2,91001:200000,93020:2}]
}
将以上kafka 原始消息数据筛选过滤出指定字段语句如下
# 将以上kafka 原始消息数据筛选过滤出指定字段语句如下 {keymag1225}| |json| line_format {{.message}} | json mapList1mapList[1],ruleIDmapList[1][\64101\],rcSysIdmapList[1][\66002\],logDatemapList[1][\28000\],logTimemapList[1][\28001\],securityExchangemapList[1][\1301\],thresholdmapList[1][\43220\],stockNamemapList[1][\11001\],securityIdmapList[1][\48\],orderIdmapList[1][\37\],orderBsmapList[1][\93020\],tradeSysIdmapList[1][\66003\],riskConCaptionmapList[1][\64103\],riskConTypemapList[1][\64102\]|line_format {{.mapList1}}| __error__ 6.5 优化后 dashboard, 展示kafka 指定字段
{keymag1225}| |json| line_format {{.message}} | json mapList1mapList[1]|line_format {{.mapList1}}| __error__
6.5.1 添加添加dashboard Clieck Add visualizaion Button or click Add- Visualization 6.5.2 优化dashboard, 显示指定字段
6.5.2.1 筛选kafka 原始消息
clieck Query 输入 将 kafka 原始消息数据筛选过滤出指定字段语句 过滤语句
{keymag1225}| |json| line_format {{.message}} | json mapList1mapList[1],ruleIDmapList[1][\64101\],rcSysIdmapList[1][\66002\],logDatemapList[1][\28000\],logTimemapList[1][\28001\],securityExchangemapList[1][\1301\],thresholdmapList[1][\43220\],stockNamemapList[1][\11001\],securityIdmapList[1][\48\],orderIdmapList[1][\37\],orderBsmapList[1][\93020\],tradeSysIdmapList[1][\66003\],riskConCaptionmapList[1][\64103\],riskConTypemapList[1][\64102\]|line_format {{.mapList1}}| __error__注意
keyvector.toml 中sinks.out中的自定义的labels.key[rootlocalhost vector]# cat vector.toml# __ __ __
# \ \ / / / /
# \ V / / /
# \_/ \/
#
# V E C T O R
# Configuration
#
# ------------------------------------------------------------------------------
# Website: https://vector.dev
# Docs: https://vector.dev/docs/
# ------------------------------------------------------------------------------# Note: A full config spec is located at ./vector.spec.toml and examples
# in the ./examples folder.# Vector config kafka Loki[ sources.in ]type kafkabootstrap_servers 192.168.2.247:9092 # kafka 地址group_id kafka_mag_groupId1225 # 消费组idtopics [alarm_warning,kafka_consumer] # kafka_consumerauto_offset_reset smallest # 从最早未处理的消息开始消费# topics [^(prefix1|prefix2)-., topic-1, topic-2] # topic名字支持正则#[sources.in.encoding]#codec json#timestamp_format %Y-%m-%dT%H:%M:%S.%fZ#json.fields [level, message][sinks.out]endpoint http://loki:3100inputs [in] # source.intype lokilabels.key mag1225 # 自定义key
# labels.key {{ event_field }} # event的动态值#[sinks.out.encoding]#codec json# timestamp_format %Y-%m-%dT%H:%M:%S.%fZ# json.fields [level, message]
[rootlocalhost vector]#6.5.2.2 优化表格显示指定字段
# 优化已表格显示指定字段1. Click Transform data -- Add Transformation -- Reformat - Extract fields
2. 添加Extract files --Source:Line --Format:JSON -- 打开 Replacee all fields -- 打开Keep time重名列名-调整顺序-显示隐藏指定列配置如下
3. click Add another transformation -- Reorder and rename -- Organize fields by name 7. 其他异常问题
7.1 grafana 页面有时跳转 网址 http://localhost:3000/XXXX 错误
grafana 容器所在主机IP:192.168.2.247
grafana web address: http://192.168.2.247:3000
grafana web 页面有时操作跳转 网址http://localhost:3000/XXXX,导致页面跳转错误7.2 跳转错误网址原因定位
原因定位Grafana Explore 页面上的 Add to Dashboard 后跳转网址为 http://localhost:3000/... 的问题可能是因为 Grafana 配置中使用了相对 URL 路径导致的
7.3 解决方法
7.3.1 Grafana 服务的环境变量 GF_SERVER_ROOT_URL 设置为正确的访问地址
1. 修改 docker-compose-grafana-loki-vector.yml 文件将 Grafana 服务的环境变量 GF_SERVER_ROOT_URL 设置为正确的访问地址 environment:- GF_SERVER_ROOT_URLhttp://192.168.2.247:3000[rootlocalhost grafana_loki_vector]# cat docker-compose-grafana-loki-vector.yml
# docker-compose配置参考https://github.com/grafana/loki/blob/master/production/docker-compose.yaml
version: 3# 网桥loki - 方便相互通讯
networks:loki:services:# Loki日志系统loki:image: grafana/loki:latestcontainer_name: grafana_loki_vector_demo_lokirestart: unless-stoppedports:- 3100:3100volumes:- ./grafana_loki_vector/loki:/etc/loki- ./grafana_loki_vector/wal:/walcommand: -config.file/etc/loki/loki-config.yamlnetworks:- loki# 用于UI展示grafana:image: grafana/grafana:latestcontainer_name: grafana_loki_vector_demo_grafanarestart: unless-stoppedports:- 3000:3000volumes:- ./grafana_loki_vector/grafana/data:/var/lib/grafana# - ./grafana_promtail_loki/grafana/config:/etc/grafana- ./grafana_loki_vector/grafana/log:/var/log/grafanaenvironment:# GF_EXPLORE_ENABLED: true- GF_SERVER_ROOT_URLhttp://192.168.2.247:3000 # 修改正确网址 grafana 容器所在主机networks:- loki# vector 用于对接kafka 消息vector:image: timberio/vector:0.10.0-alpinecontainer_name: grafana_loki_vector_demo_vectorrestart: unless-stoppedports:- 80:80volumes:# - ./grafana_loki_vector/vector/logs:/var/logs- ./grafana_loki_vector/vector:/etc/vectorcommand: -c /etc/vector/vector.toml -vnetworks:- loki[rootlocalhost grafana_loki_vector]#7.3.2 为了grafana 有数据必须修改vector.tml 配置参数
# 为了grafana 展示kafka 数据必须修改vector.tml 文件中2个参数1. group_id kafka_mag_groupId1228 # 任意值 从 *1225 改为 *12282. labels.key mag1228 # 自定义key 从 *1225 改为 *1228[rootlocalhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1909 Dec 28 17:27 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# cat grafana_loki_vector/vector/vector.toml# __ __ __
# \ \ / / / /
# \ V / / /
# \_/ \/
#
# V E C T O R
# Configuration
#
# ------------------------------------------------------------------------------
# Website: https://vector.dev
# Docs: https://vector.dev/docs/
# ------------------------------------------------------------------------------# Note: A full config spec is located at ./vector.spec.toml and examples
# in the ./examples folder.# Vector config kafka Loki[ sources.in ]type kafkabootstrap_servers 192.168.2.247:9092 # kafka 地址group_id kafka_mag_groupId1228 # 消费组id 修改group_IDtopics [alarm_warning,kafka_consumer] # kafka_consumerauto_offset_reset smallest # 从最早未处理的消息开始消费# topics [^(prefix1|prefix2)-., topic-1, topic-2] # topic名字支持正则#[sources.in.encoding]#codec json#timestamp_format %Y-%m-%dT%H:%M:%S.%fZ#json.fields [level, message][sinks.out]endpoint http://loki:3100inputs [in] # source.intype lokilabels.key mag1228 # 自定义key
# labels.key {{ event_field }} # event的动态值#[sinks.out.encoding]#codec json# timestamp_format %Y-%m-%dT%H:%M:%S.%fZ# json.fields [level, message]
[rootlocalhost grafana_loki_vector]#7.3.3 使配置生效
# 使配置生效
2. docker-compose 关闭项目
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down3. docker-copose 启动项目即可。
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d# docker-compose 关闭项目[rootlocalhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down
[] Running 4/4✔ Container grafana_loki_vector_demo_vector Removed 0.5s✔ Container grafana_loki_vector_demo_loki Removed 1.7s✔ Container grafana_loki_vector_demo_grafana Removed 0.4s✔ Network grafana_loki_vector_loki Removed 0.1s
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]## docker-copose 启动项目[rootlocalhost grafana_loki_vector]#
[rootlocalhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
[] Running 4/4✔ Network grafana_loki_vector_loki Created 0.1s✔ Container grafana_loki_vector_demo_vector Started 0.1s✔ Container grafana_loki_vector_demo_grafana Started 0.1s✔ Container grafana_loki_vector_demo_loki Started 0.1s
[rootlocalhost grafana_loki_vector]# 7.4 验证 跳转网址正确
在新页面打开跳转网址正确 http://192.168.2.247:3000/XXXXX 到此基本完成。
当然如果仅仅是想通过 Grafana 监控kafka推荐使用 Grafana Prometheus 通过JMX监控kafka
