Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite "stash."
In this story, we are taking source from Mysql database and transfers in to AWS Elasticsearch.
We already setup logstash in our server.
If you want to setup in ubuntu you can follow this tutorial.
You can use below common logstash service commands, after installing logstash in you server or machine.
For start : sudo service logstash start
For restart : service logstash restart
For stop : sudo service logstash stop
For status : sudo service logstash stop
You can write logstash code in below path.
sudo vim /etc/logstash/conf.d/logstash.conf
Sample logstash.conf file:
input {
jdbc {
jdbc_connection_string=>"jdbc:mysql://url:port/dbName"
jdbc_user => "user"
jdbc_password => "password"
jdbc_driver_library => "/path/mysql-connector-java-5.1.46-bin.jar"
jdbc_driver_class =>"com.mysql.jdbc.Driver"
statement => "select * from student"
}
}
filter {
json {
source => ""
target => ""
}
mutate {
add_field => { "field1" => "%{coloumn1}" }
add_field => { "field2" => "%{coloumn2}"
}
}
mutate {
remove_field => [""]
}
output {
amazon_es {
hosts => ["url"]
index => "indexName"
region => "region"
aws_access_key_id =>’aws_access_key_id'
aws_secret_access_key =>’aws_secret_access_key'
document_id => "%{id}"
}
stdout { codec => rubydebug }
}
You can view the logs using below command,
sudo tail -f /var/log/logstash/logstash-plain.log
You can also configure logstash pipeline in logstash.yml, if you want to run multiple logstash files.
sudo vim etc/logstash/logstash.yml
Sample logstash.yml
- pipeline.id: my-pipeline_1
path.config: “/etc/logstash/conf.d/logstash1.conf”
- pipeline.id: my-pipeline_1
path.config: “/etc/logstash/conf.d/logstash2.conf”
