Data Enrichment Using Ruby with Logstash

[@luciangabrielandrei1714]

You can use Python in Logstash, but you should use Ruby to call for its execution, as my Ruby skills are not so strong. I used it to check if baby domains have email servers associated with them.
if [first_seen_days_ago] and [first_seen_days_ago]< 180 {
ruby {
code => 'require "open3"
highest = event.get("highest_registered_domain")
cmd = "python3 /etc/logstash/conf.d/mx.py #{highest}"
stdin, stdout, stderr = Open3.popen3(cmd)
event.set("mx_servers", stdout.read)
err = stderr.read
if err.to_s.empty?
filter_matched(event)
else
event.set("mx_servers_err_msg", err)
end'
remove_field => ["highest"]
}
}
if [mx_servers_err_msg] {
mutate {
add_tag => [ "no_mx_servers" ]
remove_field => ["mx_servers_err_msg"]
}
}
This is part of a longer .conf file used to implement all the goodies you teach us in 555: top 1 million, frequency score ...

[@HASecuritySolutions]

There's also domainstats by Mark Baggett for this. Word of caution, calling another program with ruby is subject to OS command injection. I love python so if I want to use it, I typically use logstash with the http filter to call over to python via an http api call (think python fast_api)

[@rogerhuang2413]

This is awesome. Could you share a new video on how to flatten the json file? That will be interesting😊