Collecting Logs from different servers into Elastic Search via Logstash
- Lets try to write a configuration in logstash which would listen for input from beats components and send the output to elastic search after parsing content into multiple fields
- Beats:
- Logstash has a beats framework where it supports different types of beats
- Refer Here for all the beats
- For collecting logs from files we can use filebeat Refer Here
- Filebeat docs Refer Here
- Filebeat installation Refer Here
- Apt based filebeat installation Refer Here
- To configure filebeat we need to change the config file /etc/filebeat/filebeat.yml
- Refer Here for the configuration file to read the logs
- Till this point we are running logstash manually. To configure logstash to run as a service and read multiple configurations. Create .conf files and place them in /etc/logstash/conf.d folder
- Now logstash is running, elasticsearch and kibana are also running
- We need to start filebeat
- Now filebeat is also running
- To see the logs, application should generate logs, to simulate users accessing the application, let me create a simple shell script which will call apache server for every 10 s
“`bash
</ul>
<h1>!/bin/bash</h1>
url="http://34.221.122.49"
while :
do
curl ${url}
echo "press <Ctrl+C> to exit"
sleep 10s
done
“`
* Now execute this script from any machine. Windows users execute from gitbash. Refer Here - Logstash has a beats framework where it supports different types of beats
- Now lets move to kibana to view the logs and do some analysis
- Now we need to create an data view (earlier was referred as index pattern)
- Since the kibana is unable to connect to secured elastic, lets use elastic cloud. This is hosted elastic search and kibana which is free trail for 14 days. Refer Here
- Now the changed logstash configuration is as shown below
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
elasticsearch {
cloud_id => "qtelastictrail:dXMtY2VudHJhbDEuZ2NwLmNsb3VkLmVzLmlvJDMzZmNlYzY0NWI5ZTRhY2ViMDA3MzMyNWI0YmEzOWY2JDczN2VkY2JjMDFhMTRjOTI4ZjcxNzIxNWU1ZDIzYmYx"
index => "apache-%{+yyyy.MM.dd}"
cloud_auth => "elastic:8jnJITEZ0bbZxASxDs8ecUFd"
}
stdout {
}
}
- In any of your existing linux machines install metric beat, donot configure
- Please follow the classroom video for steps.
