Now lets use logstash to read the access and error logs and redirect to stdout input {
file{
path => ["/var/log/apache2/access.log", "/var/log/apache2/error.log"]
}
}
output {
stdout {
codec => rubydebug
}
}
Now wait for the logstash to start and send http requests to your apache server
File plugin for reading the logs is used when your organizational setup is to store logs on network file systems
Now as a continuation, lets try to send the output of the apache logs to elastic search. For this we need to configure
At this moment we are able to send the logs from apache running locally to elastic search via Logstash. Typically logdata = timestamp + data. In this data we might have multiple fields which can be aggregated.
The individual access log is as shown below
223.238.77.113 - - [01/Mar/2022:03:20:20 +0000] "GET / HTTP/1.1" 200 3477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
49.205.103.150 - - [01/Mar/2022:03:24:30 +0000] "GET /favicon.ico HTTP/1.1" 404 491 "http://34.209.75.177/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36"
157.45.11.132 - - [01/Mar/2022:03:26:48 +0000] "GET /favicon.ico HTTP/1.1" 404 492 "http://34.209.75.177/" "Mozilla/5.0 (Linux; Android 11; RMX3360) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.101 Mobile Safari/537.36"
49.206.34.54 - - [01/Mar/2022:03:27:21 +0000] "GET /favicon.ico HTTP/1.1" 404 492 "http://34.209.75.177/" "Mozilla/5.0 (iPhone; CPU iPhone OS 15_0_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Mobile/15E148 Safari/604.1"
Rather than considering every thing as text if we can create multiple fields like client ip, datetime, http method, uri, status code, client-device
Next Step:
How to parse a text and create multiple fields to help in log analysis