LogStash
- Extract , Transform and Load (ETL):
- Extract i.e. read the logs from source
- Transform the log records into queryable fields
- load i.e. store/push/index into elastic search
- Install Logstash: Refer Here
- Logstash performs extract, transform and load with the help of plugins
- Extract => input plugins Refer Here
- Transform => filter plugins Refer Here
- Load => Output plugins Refer Here
Play with Logstash
- Logstash is located generally in
/usr/share/logstash/bin
- Lets search for the options
sudo /usr/share/logstash/bin/logstash --help
- Refer Here for the command line options
- To the logstash we need to provide pipeline as input
input
{
<plugin-name>
{
<option-1> => <value-1>
...
<option-n> => <value-n>
}
}
filter
{
<plugin-name>
{
<option-1> => <value-1>
...
<option-n> => <value-n>
}
}
output
{
<plugin-name>
{
<option-1> => <value-1>
...
<option-n> => <value-n>
}
}
- filter is optional
- Activity 1: Lets create a pipeline which reads from command line (termianl) and shows the output on the terminal. Save this in
~/logstash-pipelines/hello-wrold.conf
input
{
stdin
{
}
}
output
{
stdout
{
}
}
- Now try to run logstash with the following command
sudo ./logstash -f ~/logstash-pipelines/hello-wrold.conf
- Now lets give some input
- Activity 2: Now lets try to create a logstash pipeline which reads from stdin and stores in a file and also shows the output in stdout. save it as activity2.conf
input
{
stdin {}
}
output
{
stdout {}
file
{
path => '/tmp/activity2'
create_if_deleted => true
flush_interval => 0
}
}
- Now run logstash
sudo ./logstash -f ~/logstash-pipelines/activity2.conf
- Activity 3: Create a logstash pipeline which reads the inputs from a file /tmp/messages and writes the output to stdout
input
{
file
{
path => '/tmp/messages'
start_position => 'beginning'
}
}
output
{
stdout
{
}
}
- Next Steps:
- Lets read logs from some applications
