Graylog - Pipelines Extractors
Вставка
- Опубліковано 10 січ 2024
- In this video we start to look at pipelines and the reason we use them in graylog. We will show a practical example of creating a pipeline rule that acts like an extractor.
Grok Debugger: grokdebugger.com/
I just built my Graylog server to collect and analyze my OPNsense logs (filterlog for now) and I have literally zero knowledge on how use that tool. I managed to set up sending logs to Greylog and now it's time to make the next steps. Your channel is exactly what I was looking for :)
Excellent introduction! I just updated my graylog server to the latest today, so seeing this video is a nice surprise!
This installment in your Graylog series was worth the wait. I think I finally “get” grok patterns now.
I would love to see how pipelines can parse JSON into key value pairs (idea for future video). I know I can do it with string matching and grok patterns but that seems fragile and inefficient.
Very nice explanation for pipeline. But I have one question: I've did it for my source field with the input is localhost to be changed to a specific IP of the server. But the problem is any servers coming in graylog with the source named localhost were changed to this IP (as the pipeline rule I set). So how do I distinguish different IPs from different servers which coming into graylog with the same name localhost? Tks!
Gl2_remote_ip key ALWAYS has the IP that sends the message.
@@fordayinlife it's weird for my case, I've never seen any messages if I filter Gl2_remote_ip key, so I always have to use source or SourceModuleName or ServiceName to be able to looking up for the messages.
thank for the graylog video :)
I am very new with graylog and start to build demo graylog's architecture to save the log data.
I would like to ask about is it possible to set index name as daily format? (ex. graylog_20240215, graylog_20240216)
The default setting produces index name like 'graylog_0' ...
What issue are you trying to solve? If you want ALL logs from a certain time period I would be using the date selectors when searching. I don't think you can change that format,
@fordayinlife
Thank you for the quick response!
There are no issues, but I would like to know whether I can handle Graylog's index name as I want.
Here is my plan for the graylog system:
1) generating graylogs' index and setting its name as daily format (ex. graylog_20240216...)
2) Making a snapshot for the index and saving it to AWS S3 by opensearch dashboard
3) Delete the old index with certain rules (ex., deleting the old index after 6 months). 4) Restoring the deleted index from AWS S3 by opensearch dashboard when I want. 5) For this purpose, I would like to set the graylog's name as a daily format to pick up and restore it by using file name.
Im running the latest package version on Ubuntu 22.04, Graylog 5.0.13 and I dont have that fancy rule builder.
Introduced in Graylog 5.2... you need to upgrade
@@fordayinlife noooooooooo. haha. Ok thank you. Great video btw.
you seriously did a great job breaking down the basics of pipelines. I can expand on this now and create multiple fields in my messages. Wonderful job !