比特币支付网站建设,营销课程,wordpress打赏功能,企业培训有哪些方面部署环境说明
为节省资源直接使用1台测试机模拟3节点elasticsearch服务集群做部署#xff0c;在该主机上同时部署了3个elasticsearch实例、1个logstash实例、1个kibana实例、1个filebeat实例。对于生产环境#xff0c;以上实例服务应该做分布式部署。
ELK-TEST1 192.168.10…部署环境说明
为节省资源直接使用1台测试机模拟3节点elasticsearch服务集群做部署在该主机上同时部署了3个elasticsearch实例、1个logstash实例、1个kibana实例、1个filebeat实例。对于生产环境以上实例服务应该做分布式部署。
ELK-TEST1 192.168.10.11
本方案已通过了以下操作系统环境的验证测试
os: rhel 7.9 openssl:1.1.1wos: rhel 8.8 openssl:3.1.2
操作系统参数调优与配置
主机名IP映射
cat EOF /etc/hosts
192.168.10.11 node-1
192.168.10.11 node-2
192.168.10.11 node-3
192.168.10.11 ELK-TEST1
EOF禁用swap
swapoff -a
sed -i /swap/d /etc/fstab调整系统可用资源限制
文件句柄与最大线程并发数量
cat EOF /etc/security/limits.d/usercustom.conf
* soft nofile 65535
* hard nofile 65535
* soft nproc 4096
* hard nproc 4096
* soft fsize unlimited
* hard fsize unlimited
* soft memlock unlimited
* hard memlock unlimited
EOF虚拟内存及网络连接重连
cat EOF /etc/sysctl.d/98-usercustom.conf
vm.max_map_count262144
net.ipv4.tcp_retries2 5
EOFsysctl -p /etc/sysctl.d/98-usercustom.conf创建es专用的系统用户
useradd elastic检查或配置系统安全配置
生产网不建议关闭系统防火墙可以直接对运行ELK服务的主机节点间做网络访问的全部放行配置。
关闭selinux
setenforce 0
sed -i /SELINUX/s/enforcing/disabled/ /etc/selinux/config重启系统以使上面配置全部生效。
部署ELK安装包
将以下4个安装包上传到主机/opt目录下解压缩 elasticsearch-7.17.18-linux-x86_64.tar.gz filebeat-7.17.18-linux-x86_64.tar.gz kibana-7.17.18-linux-x86_64.tar.gz logstash-7.17.18-linux-x86_64.tar.gz
cd /opt
tar zxf elasticsearch-7.17.18-linux-x86_64.tar.gz
tar zxf filebeat-7.17.18-linux-x86_64.tar.gz
tar zxf kibana-7.17.18-linux-x86_64.tar.gz
tar zxf logstash-7.17.18-linux-x86_64.tar.gz
mkdir soft
mv *.gz soft由于我们是使用1个主机来模拟部署一套ELK服务elasticsearch集群服务需要运行在生产模式下至少有3个es实例。所以对部署目录做以下调整
cd /opt
mv elasticsearch-7.17.18/ elastic-node1
cp -r elastic-node1/ elastic-node2
cp -r elastic-node1/ elastic-node3
mv logstash-7.17.18/ logstash
mv kibana-7.17.18-linux-x86_64/ kibana
mv filebeat-7.17.18-linux-x86_64/ filebeat
chown -R elastic.elastic *最终的/opt部署路径结果如下
elastic-node1 elastic-node2 elastic-node3 filebeat kibana logstash softELK集群服务的初始化配置
以下所有的配置均是使用elastic普通用户执行 我们这里是把所有服务部署在一个主机上了所以以下配置命令均在同一个主机上执行。如果你规划的ELK服务集群使用了多个主机节点请根据每个服务实例实际部署位置选择相应的主机并配置。
制作ELK集群使用的证书密钥
cd /opt/elastic-node1
mkdir makecerts
./bin/elasticsearch-certutil ca --out ./makecerts/elastic-stack-ca.p12 --days 36500 # 签发CA根证书有效期100年
./bin/elasticsearch-certutil cert --ca ./makecerts/elastic-stack-ca.p12 --out ./makecerts/elastic-certificates.p12 --dns node-1,ELK-TEST1,node-2,node-3 --ip 192.168.10.11 --days 36500 # 记录好以上两个证书的密码信息
/opt/elastic-node1/jdk/bin/keytool -keystore ./makecerts/elastic-stack-ca.p12 -list # 查看CA证书
/opt/elastic-node1/jdk/bin/keytool -keystore ./makecerts/elastic-certificates.p12 -list # 查看elasticsearch服务证书签发其他实例服务使用的证书:
./bin/elasticsearch-certutil cert --ca ./makecerts/elastic-stack-ca.p12 --out ./makecerts/logstash.zip --name logstash --dns node-1,ELK-TEST1,node-2,node-3 --ip 192.168.10.11 --pem --days 36500
./bin/elasticsearch-certutil cert --ca ./makecerts/elastic-stack-ca.p12 --out ./makecerts/kibana.zip --name kibana --dns node-1 --ip 192.168.10.11 --pem --days 36500
./bin/elasticsearch-certutil cert --ca ./makecerts/elastic-stack-ca.p12 --out ./makecerts/filebeat-10.11.zip --name filebeat-10.11 --dns node-1,ELK-TEST1,node-2,node-3 --pem --days 36500 ./bin/elasticsearch-certutil cert --ca ./makecerts/elastic-stack-ca.p12 --out ./makecerts/metricbeat.zip --name metricbeat --dns node-1 --ip 192.168.10.11 --pem --days 36500 openssl pkcs12 -nocerts -nodes -in ./makecerts/elastic-stack-ca.p12 -out ./makecerts/private.pem
openssl pkcs12 -clcerts -nokeys -in ./makecerts/elastic-stack-ca.p12 -out ./makecerts/cacert.pem # 生成一份pem格式的ca证书文件
openssl x509 -in ./makecerts/cacert.pem -noout -text # 查看ca pem证书信息将elasticsearch证书密码保存到keystore、truststore中:
./bin/elasticsearch-keystore create
./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
./bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password
./bin/elasticsearch-keystore list # 浏览keystore密钥库中保存的信息
./bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password # 查看在密钥库中保存的指定密码信息解压kibana.zip,logstash.zip,filebeat.zip并进一步制作适配logstash服务的证书文件:
cd /opt/elastic-node1/makecerts
unzip filebeat-10.11.zip
unzip kibana.zip
unzip logstash.zip
rm -rf *.ziplogstash input插件需要使用pkcs8格式的密钥文件output elasticsearch插件需要使用truststore密钥库保存pkcs12格式的ca证书:
cd /opt/elastic-node1/makecerts/logstash
openssl pkcs8 -in logstash.key -topk8 -nocrypt -out logstash.p8
/opt/elastic-node1/jdk/bin/keytool -import -file /opt/elastic-node1/makecerts/cacert.pem -keystore truststore.p12 -storepass ueyf36456fh -noprompt -storetype pkcs12分发各实例需要使用的证书、密钥文件:
cd /opt/elastic-node1/makecerts
cp elastic-certificates.p12 /opt/elastic-node1/config
cp elastic-certificates.p12 /opt/elastic-node2/config
cp elastic-certificates.p12 /opt/elastic-node3/config
cp /opt/elastic-node1/config/elasticsearch.keystore /opt/elastic-node2/config/
cp /opt/elastic-node1/config/elasticsearch.keystore /opt/elastic-node3/config/
cp cacert.pem logstash/* /opt/logstash/config/
cp cacert.pem kibana/kibana.* /opt/kibana/config/
cp cacert.pem filebeat-10.11/* /opt/filebeat/注将makecerts目录打包做好备份tar zcf makecerts.tgz makecerts/
设置elasticsearch实例配置
设置elasticsearch实例的jvm缓存请根据实际情况调整
cat EOF /opt/elastic-node1/config/jvm.options.d/jvm-heap.conf
-Xms4g
-Xmx4g
EOFcat EOF /opt/elastic-node2/config/jvm.options.d/jvm-heap.conf
-Xms4g
-Xmx4g
EOFcat EOF /opt/elastic-node3/config/jvm.options.d/jvm-heap.conf
-Xms4g
-Xmx4g
EOF在本示例的3个es服务实例的elasticsearch.yml配置中只有4个参数值有差别它们是node.name、path.data、path.logs、http.port、transport.port 。如果是使用3个主机节点部署且每个主机上只运行一个elasticsearch实例时每个实例间的配置只有node.name、network.host参数值的差别。
cat EOF /opt/elastic-node1/config/elasticsearch.yml
cluster.name: elk-application
node.name: node-1
node.master: true
node.data: true
path.data: /opt/elastic-node1/data
path.logs: /opt/elastic-node1/logs
bootstrap.memory_lock: true
network.host: 192.168.10.11
http.port: 9200
transport.port: 9300
discovery.seed_hosts: [192.168.10.11:9300, 192.168.10.11:9301, 192.168.10.11:9302]
# cluster.initial_master_nodes参数在第1次启动es服务集群后需要及时注释掉
cluster.initial_master_nodes: [node-1, node-2, node-3]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: ./elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: ./elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: ./elastic-certificates.p12
xpack.security.http.ssl.truststore.path: ./elastic-certificates.p12
EOFcat EOF /opt/elastic-node2/config/elasticsearch.yml
cluster.name: elk-application
node.name: node-2
node.master: true
node.data: true
path.data: /opt/elastic-node2/data
path.logs: /opt/elastic-node2/logs
bootstrap.memory_lock: true
network.host: 192.168.10.11
http.port: 9201
transport.port: 9301
discovery.seed_hosts: [192.168.10.11:9300, 192.168.10.11:9301, 192.168.10.11:9302]
# cluster.initial_master_nodes参数在第1次启动es服务集群后需要及时注释掉
cluster.initial_master_nodes: [node-1, node-2, node-3]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: ./elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: ./elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: ./elastic-certificates.p12
xpack.security.http.ssl.truststore.path: ./elastic-certificates.p12
EOFcat EOF /opt/elastic-node3/config/elasticsearch.yml
cluster.name: elk-application
node.name: node-3
node.master: true
node.data: true
path.data: /opt/elastic-node3/data
path.logs: /opt/elastic-node3/logs
bootstrap.memory_lock: true
network.host: 192.168.10.11
http.port: 9202
transport.port: 9302
discovery.seed_hosts: [192.168.10.11:9300, 192.168.10.11:9301, 192.168.10.11:9302]
# cluster.initial_master_nodes参数在第1次启动es服务集群后需要及时注释掉
cluster.initial_master_nodes: [node-1, node-2, node-3]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: ./elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: ./elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: ./elastic-certificates.p12
xpack.security.http.ssl.truststore.path: ./elastic-certificates.p12
EOF设置logstash服务实例配置
logstash服务配置文件:
cat EOF /opt/logstash/config/logstash.yml
node.name: logstash-10-11
xpack.monitoring.enabled: false
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: hDagwy141d
#xpack.monitoring.elasticsearch.hosts: [https://node-1:9200, https://node-2:9201, https://node-3:9202]
xpack.monitoring.elasticsearch.hosts: [https://node-1:9200]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /opt/logstash/config/cacert.pem
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
EOFlogstash 数据转发配置文件:
cat EOF /opt/logstash/config/logstash.conf
input {beats {id logstash-10-11port 5044ssl truessl_certificate_authorities /opt/logstash/config/cacert.pemssl_certificate /opt/logstash/config/logstash.crtssl_key /opt/logstash/config/logstash.p8ssl_verify_mode force_peer}
}
output {elasticsearch {id elk-applicationhosts [https://node-1:9200, https://node-2:9201, https://node-3:9202]manage_template truetemplate_overwrite trueindex test-logs-%{YYYY.MM.dd}user elasticpassword iwuHBG865ssl_certificate_verification truetruststore /opt/logstash/config/truststore.p12truststore_password ueyf36456fh}
}
EOF设置kibana服务实例配置
cat EOF /opt/kibana/config/kibana.yml
server.host: node-1
server.publicBaseUrl: https://192.168.10.11:5601/
elasticsearch.hosts: [https://192.168.10.11:9200]
elasticsearch.username: kibana_system
elasticsearch.password: hfrr53df64
server.ssl.enabled: true
server.ssl.certificate: /opt/kibana/config/kibana.crt
server.ssl.key: /opt/kibana/config/kibana.key
elasticsearch.ssl.certificateAuthorities: [ /opt/kibana/config/cacert.pem ]
elasticsearch.ssl.verificationMode: certificate
xpack.security.encryptionKey: dfe2435fdsdfg2424wegrcvnjhgfr5678909iju
xpack.security.sessionTimeout: 1800000
xpack.monitoring.elasticsearch.hosts: [ https://192.168.10.11:9200 ]
xpack.monitoring.elasticsearch.ssl.certificateAuthorities: config/cacert.pem
EOF设置filebeat服务配置
filebeat.yml配置文件如下因有特殊字符无法使用cat命令直接写入文件请复制下面内容并替换配置文件内容:
filebeat.inputs:
- type: filestreamid: ELK-TEST1-idenabled: truepaths:- /var/log/test-logs/*.log
filebeat.config.modules:path: ${path.config}/modules.d/*.ymlreload.enabled: false
setup.template.settings:index.number_of_shards: 1
output.logstash:hosts: [node-1:5044]ssl.certificate_authorities: [/opt/filebeat/cacert.pem]ssl.certificate: /opt/filebeat/filebeat-10.11.crtssl.key: /opt/filebeat/filebeat-10.11.key
processors:- add_host_metadata:when.not.contains.tags: forwarded- add_cloud_metadata: ~- add_docker_metadata: ~- add_kubernetes_metadata: ~设置rsyslogd服务配置
很多安全、网络类的设备仅支持将设备日志转发给syslog服务所以我们需要配置一个rsyslogd服务接收这些设备日志。在完成日志落盘后由filebeat负责采集和存储到ELK平台。
检查/etc/rsyslog.conf文件启用以下参数
module(loadimudp) # needs to be done just once
input(typeimudp port514)module(loadimtcp) # needs to be done just once
input(typeimtcp port514)注这里的参数配置方法在rhel7和rhel8上有差别但区别不大找到并启用即可。 创建/etc/rsyslog.d/test-logs.conf 配置文件内容如下
$template remote-incoming-logs,/var/log/test-logs/%fromhost-ip%_%$YEAR%.log
*.* ?remote-incoming-logs~mkdir -p /var/log/test-logs
systemctl restart rsyslog
systemctl status rsyslog注意查看rsyslog服务日志、状态。
启动各个服务组件并观察日志
启动elasticsearch服务集群并设置管理账密信息
注依次启协3个服务实例每启动一个后先观察 elk-application.log 日志输出在前一个实例启动结束后再启动下一次。
cd /opt/elastic-node1
./bin/elasticsearch -dcd /opt/elastic-node2
./bin/elasticsearch -dcd /opt/elastic-node3
./bin/elasticsearch -d观察上述服务启动日志输出和集群显示状态均正常后及时注释掉elasticsearch.yml文件中的cluster.initial_master_nodes参数
执行下面命令设置内建管理用户的密码
./bin/elasticsearch-setup-passwords interactive注这里设置的账号密码需要与前面各种服务的配置文件中会使用的账号、密码信息一致。
2启动kibana服务
cd /opt/kibana
nohup ./bin/kibana 注观察并确认日志输出正常服务运行正常。
访问https://192.168.10.11:5601使用上面创建的elastic管理员用户登录。
3启动logstash服务
cd /opt/logstash
./bin/logstash -f ./config/logstash.conf 4启动filebeat服务 使用root用户操作:
cd /opt/filebeat
chown root.root filebeat.yml
./filebeat -e -c filebeat.yml 注由于我们的使用场景中filebeat会采集/var/log下一些系统日志需要root权限所以这里有上述的权限调整。
登录kibana控制台配置索引管理信息
登录后进入Management界面创建一个index pattern
名称为test-logs-* 在discover界面下就可以检索到已经采集到的日志数据了。
创建索引生命周期管理策略
名称test-logs-policy 启用两个生命阶段即可
hot phase管理30天内的索引文件cold phase: 管理大于180天的索引文件
创建索引模板
进入Dev tools界面执行以下命令
PUT _index_template/test-logs-template?pretty
{index_patterns : [test-logs-*],template : {settings : {index : {lifecycle : {name : test-logs-policy,rollover_alias : test-logs},number_of_shards : 1,number_of_replicas : 2}},aliases: {test-logs: {}}}
}查看索引模板
GET _index_template/test-logs-template?pretty到这里主要配置内容基本结束。
