DevOps Classroom notes – 22 April 2021

Script to import the movielens dataset

• python Refer Here
• bash Refer Here

Searching the elastic search

• using q to perform full text search
• using full elastic search queries

Our Scenario of Elastic Search

• We use Elastic search to help us in searching log files to
  • narrow down on errors
  • anamoly detections
• The above example was on sample movies data where we have decent kind of structured data
• The scenario which we would like to simulate is
• To do this better we need to look into
  • installing and configuring logstash
  • understand logstash plugins
  • try to work with different log formats with logstash

Setting up logstash

• Installing logstash Refer Here
• Logstash is a popular open source data collection engine with real time pipelining capabilities
• Logstash allows us to easily build a pipeline that collecting data from various input sources and parse, enrich and store in wide variety of destinations
• So logstash
  • read the data from various source
  • does the necessary conversions to enrich the data
  • stores the data in the destination
• Simple Scenario:
  • Lets try to configure logstash to read the data from standard input and output the data to standard output
• Next Steps:
  • Lets try to understand the logstash architecture and then see how it can help in parsing logs