Analyzing the log data with logstash
-
Lets install apache server and read the logs of apache server from logstash pipeline
-
The first basic configuration looks as mentioned over Here
Parse and enrich logs using logstash
- We need to use filter plugins to transform the data.
- Lets use a csv file and parse the data using csv filter
- For this lets use the dataset from grouplens Refer Here
- Lets look at filter plugins for csv Refer Here for all plugins & Refer Here for csv plugin. Refer Here for the changeset
- Movie id is read as string, lets change that to integer, for that we will be using mutate filter Refer Here
- Refer Here for the change set
Using GROK Filters
- Logstash shipps with 120 default grok patterns Refer Here
- Refer Here for general grok patterns
