DevOps Classroom notes – 22 April 2021

Script to import the movielens dataset

Searching the elastic search

  • using q to perform full text search Preview
  • using full elastic search queries Preview Preview Preview Preview Preview Preview Preview Preview

Our Scenario of Elastic Search

  • We use Elastic search to help us in searching log files to
    • narrow down on errors
    • anamoly detections
  • The above example was on sample movies data where we have decent kind of structured data
  • The scenario which we would like to simulate is Preview
  • To do this better we need to look into
    • installing and configuring logstash
    • understand logstash plugins
    • try to work with different log formats with logstash

Setting up logstash

  • Installing logstash Refer Here
  • Logstash is a popular open source data collection engine with real time pipelining capabilities
  • Logstash allows us to easily build a pipeline that collecting data from various input sources and parse, enrich and store in wide variety of destinations
  • So logstash
    • read the data from various source
    • does the necessary conversions to enrich the data
    • stores the data in the destination
  • Simple Scenario:
    • Lets try to configure logstash to read the data from standard input and output the data to standard output Preview Preview Preview
  • Next Steps:
    • Lets try to understand the logstash architecture and then see how it can help in parsing logs

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

About continuous learner

devops & cloud enthusiastic learner