OTEL and/or Log Shipping can be overwhelming when you have used only traditional console loggers for many years, not to mention they all want you to ship to cloud. This is the simplest and clearest 'get started' walkthrough I've seen to date.VERY appreciated.
Thank you so much for this. I've been investigating integrating OTLP into some new development currently on my plate and this answers a LOT of the questions that I had been having. Appreciate all you do!
Seq is not free, you need to pay for a license to deploy to a server. Serilog (Loki Sink) + Loki + Grafana is the easiest and fastest way to get a free, simple, queryable, structured logging solution up and running. No magic instrumentation, no security headaches with certificates (like with ELK, just let the reverse proxy handle it), no pantheon of collectors and exporters and services. It doesn't have metrics and tracing (though they can be added) but sometimes you just want something simple to start off with and logging is the most crucial of the 3 observability components when starting out.
@@g3ff01 They do work with OTel. Loki can infact also ingest OTel logs, but also has its own sink for Serilog using its own API. Grafana is what is used to build the Dashboards and bring it all together. They also have services to ingest other data like traces into Tempo, or metrics into Prometheus. Seq is more or less and all-in-one solution and I like it for lokal testing on my machine because its smaller. Grafana + Loki + Tempo etc. is a stack of multiple services that you can piece together as you need them. Thats what we run in Prod.
It's a very interesting topic. I think it would be very useful to a video about how to integrate like open telemetry + some scoped logging context/metadata (e.g. if we need to log a tenant ID in a multi-tenant app) + for example, Azure app insights as a logging storage, but in a case of using ILogger from DI. I don't think that in a real-world app, devs will create a new logger manually each time.
Great video. Howerver, albeit not the scope of this video. When configuring the endpoints for OTEL. How should we go about doing that in production? I mean, there is an API key, that we want to keep secret, so perhaps fetching that from i.e HCP vault. Then I suppose we can use an environment.variable for the endpoint URI.
@@atziazas me too. We pretty much use ILogger to app insights and all our stuff is running in apps service is Azure. and I'm wondering now if we should be changing that?
3:10 Do you have a video about the structered Logging Naming Convention for attributes? - This is a new "problem" when doing the structured logging as the name you provide in the template will be name in the logging provider like ElasticSearch. So lets say you were to log the execption ex and you write the structered log like ("There was an error handling the request. The error was {Ex}", ex.Message); - The naming in template could easily be {Exception} or {ExceptionMessage} or just {Message}. If you're the only developer on a project that's not a problem but if you're a team you need to have a standard for this attributes but I haven't found a "standard" for this and might have missed if you're ever mentioned this in any of your videos or talks.
I'd like to have some tips re that too. Even in my own project, I find myself I am not consistent enough - because I don't know what's good and simple vs too short vs too bloated. I catch myself on that often.
If you use mediatr then you can catch exceptions or log the request in one place and define your template there. Alternatively if you use source generated high performance logging you can define templates in a resource file for example
@@TehGM I'm not sure if you are going to find a specific naming convention for that. Every team needs to agree to a format that suits them.(And build guard rails to adhere to the convention)
I am in "big tech" company, OTel is still struggling internally. There are as many ways to emit telemetry as there are organizations in my company. Some services emit almost fully denormalized data - one table with something like 500 columns. My point is - don't do what big tech does, instead do what a very serious tech startup would do
Amen.. I've seen some "Big Tech" companies that don't have any structured logging in place. Customer queries have to be resolved by engineers remoting into various servers and inspecting local log files. I 💩you not.
Id already added Open Telemetry but it can get complicated when configuring a boot logger, Serilog, Open Telemetry, and Application insights (or Grafana etc) - then configuring Tracing and Metrics, adding custom Meters. I managed it but I'm not convinced it's all correct. Maybe a new course idea for Dometrain on Observality...? With Serilog having so many different ways to do the same thing, it's becoming confusing...
I wonder if the exporter runs asynchronously as a background service at the lowest priority, unless a critical exception must not be lost and should be logged immediately before the app crashes.
What if the external logger service goes down? Is there a possibility to also save the logs to the file in a host or something? In case of an event where the logger service is down and we need to investigate something how can we achieve that?
You can log to a file and have an agent that reads that file and sends the content to your log server in otel format, for example. Or you write your logs to different sources (otel cloud, file,…) at the same time.
How would I log transactions or "sagas" from multiple services, that might collaborate to fulfill a task. So that I have logs for all steps and can combine logs from multiple services for exactly one business process?
Hi Nick, Thanks for this amazing and knowledgeable video. I have a question though. Let’s say I want to log to a file or maybe create my own log viewer within the application which I can host with the application and push my logs there using open telemetry. Is that doable?
Great content as usual. One question: how can we use open telemetry in production, where could the logs be exported to. Any ideas or suggestions. Thanks
Hey nick - I was wandering, as a Unity developer, a former validation developer at intel, what courses whould i take to make my c# knowledge better? i mean, i do have 3 years of experience in a varaity of c# useages (not .Net in particular), and i am a computer science major, so taking the "SOLID" course or those kind seem obsolete.
Hey Nick, thank you for this great video. Do you perhaps know of a way how OpenTelemetry can log to files? (This is a requirement from our client we do work for)
Great video. One question though. What is the benefit of adding Serilog to the picture? Never used it, so maybe that's a dumb question with obvious answer ;-)
Serilog, in general, brings many benefits, but in this specific context you can get a more useful formatting as well as the ability to serialize objects as part of the structured logged message. Please, keep in mind this is a very superficial answer. Serilog’s true benefits become more evident as more intricate scenarios come into play.
@@josefromspace ah. Thanks. Just always assumed that Serilog is basically predecessor of Microsoft.Extensions.Abstraction.Logging with (mostly) similar functionalities and just small deviations in interfaces or some obscure functionality. Will take a closer look than.
@@woocaschnowakabsolutely, Serilog’s claim of fame was that by using structured logging, it allocated less memory on the heap than .Net’s counterpart at the moment. As time passed both have grown in functionality, definitely worth looking into it.
"Getting weather for {CityName}" is the same string every time, so you can easily search for this in your logs and go from there. If you used string interpolation, you'd have to search for the common substring which would be "Getting weather for " and that's a bit weird
I'm adding logging into a bunch of apps at the moment, and to be honest this level of logging just isn't needed. Normal structured logging is enough for a lot of smaller applications. It comes down to only logging what you actually need and will look at, all the rest is useless fluff.
there's a few reasons, but structured logging allows you to preserve the parameters in such a way that you could query them later like he's doing in the video. Otherwise, you'd have to use complicated patterns (regex) to extract those parameters again after being joined into one string. Also, each unique log line would allocate a new string further increase in garbage collection pressure and slowing down the app.
Lidl swag makes you a better developer
I have the lidl shorts hahaha
OTEL and/or Log Shipping can be overwhelming when you have used only traditional console loggers for many years, not to mention they all want you to ship to cloud. This is the simplest and clearest 'get started' walkthrough I've seen to date.VERY appreciated.
This video is invaluable for my team and myself and arrives with perfect timing for our necessities. Thank you, thank you, thank you.
Thank you so much for this. I've been investigating integrating OTLP into some new development currently on my plate and this answers a LOT of the questions that I had been having. Appreciate all you do!
First time I’ve ever been able to watch one of Nick’s videos and think ‘we already do that!’ Result!
Hi Nick, request you please make more videos on logging.
Been using SEQ with Serilog for years but this highlighted how easy it is to include open telemetry. Thanks Big Fella. Another great video.
Nick's drip goes so hard
Just a lidl drip
Seq is not free, you need to pay for a license to deploy to a server.
Serilog (Loki Sink) + Loki + Grafana is the easiest and fastest way to get a free, simple, queryable, structured logging solution up and running. No magic instrumentation, no security headaches with certificates (like with ELK, just let the reverse proxy handle it), no pantheon of collectors and exporters and services. It doesn't have metrics and tracing (though they can be added) but sometimes you just want something simple to start off with and logging is the most crucial of the 3 observability components when starting out.
Do these anything to do with OpenTelemetry, or is this a completely separate stack?
@@g3ff01 They do work with OTel. Loki can infact also ingest OTel logs, but also has its own sink for Serilog using its own API. Grafana is what is used to build the Dashboards and bring it all together. They also have services to ingest other data like traces into Tempo, or metrics into Prometheus. Seq is more or less and all-in-one solution and I like it for lokal testing on my machine because its smaller. Grafana + Loki + Tempo etc. is a stack of multiple services that you can piece together as you need them. Thats what we run in Prod.
@@g3ff01 TGeorgie describes a indeed completely different stack, althought Loki Grafana support otlp using the otlphttp exporter
It's a very interesting topic. I think it would be very useful to a video about how to integrate like open telemetry + some scoped logging context/metadata (e.g. if we need to log a tenant ID in a multi-tenant app) + for example, Azure app insights as a logging storage, but in a case of using ILogger from DI. I don't think that in a real-world app, devs will create a new logger manually each time.
serilog implements the ILogger, and you can use the serilog sink package
nice, serilog and seq are the best combination we're using since years!
Great video. Howerver, albeit not the scope of this video. When configuring the endpoints for OTEL. How should we go about doing that in production? I mean, there is an API key, that we want to keep secret, so perhaps fetching that from i.e HCP vault. Then I suppose we can use an environment.variable for the endpoint URI.
Hey Nick thx for a great video. How do you feel about Application Insights and Log Analytics?
Had the same question
@@atziazas me too. We pretty much use ILogger to app insights and all our stuff is running in apps service is Azure. and I'm wondering now if we should be changing that?
3:10 Do you have a video about the structered Logging Naming Convention for attributes? - This is a new "problem" when doing the structured logging as the name you provide in the template will be name in the logging provider like ElasticSearch. So lets say you were to log the execption ex and you write the structered log like ("There was an error handling the request. The error was {Ex}", ex.Message); - The naming in template could easily be {Exception} or {ExceptionMessage} or just {Message}. If you're the only developer on a project that's not a problem but if you're a team you need to have a standard for this attributes but I haven't found a "standard" for this and might have missed if you're ever mentioned this in any of your videos or talks.
I'd like to have some tips re that too. Even in my own project, I find myself I am not consistent enough - because I don't know what's good and simple vs too short vs too bloated. I catch myself on that often.
If you use mediatr then you can catch exceptions or log the request in one place and define your template there.
Alternatively if you use source generated high performance logging you can define templates in a resource file for example
@@Wfmike I don't think that's the concern. We know that. The concern is good old variable naming, but with logging in mind in this case.
@@TehGM I'm not sure if you are going to find a specific naming convention for that. Every team needs to agree to a format that suits them.(And build guard rails to adhere to the convention)
Just post it on whatever chat app your team uses and have your team vote for options. There’s not always going to be a convention for everything.
LIDL 💛💙
I am in "big tech" company, OTel is still struggling internally. There are as many ways to emit telemetry as there are organizations in my company. Some services emit almost fully denormalized data - one table with something like 500 columns. My point is - don't do what big tech does, instead do what a very serious tech startup would do
So what does a very serious startup do?
.
Serious startups spend millions of investor dollars buying storage to store 10GB of log per hour in the cloud provider 😅
Amen.. I've seen some "Big Tech" companies that don't have any structured logging in place. Customer queries have to be resolved by engineers remoting into various servers and inspecting local log files. I 💩you not.
Great vid btw. This is what I wish I had a year ago when I first decided to look into OTEL lol.
the sweater looks total dripp
AWESOME!!!!! Thanks for sharing!
Great video, I was not aware of otlp
Will Dometrain courses be available from the Lidl middle aisle any time soon Nick ?
I'm trying to get them to sponsor but they just want me to advertise their "Greek Week" instead
I've always assumed it was pronounced "seek" as in "sequence".
Ah that's smart actually
Same here
You are correct, I spoke to Nicholas Blumhardt (the guy behind Seq) at NDC Oslo and he did pronounce it "seek" :)
I've been pronouncing it like Nick for many years already. But 'seek' makes so much sense...
Id already added Open Telemetry but it can get complicated when configuring a boot logger, Serilog, Open Telemetry, and Application insights (or Grafana etc) - then configuring Tracing and Metrics, adding custom Meters. I managed it but I'm not convinced it's all correct. Maybe a new course idea for Dometrain on Observality...?
With Serilog having so many different ways to do the same thing, it's becoming confusing...
Is there any video on how to build a custom solution to send and view those logs?
Which is better?
- use serilog sink to push logs
- use fluentbit to collect logs from the app
I wonder if the exporter runs asynchronously as a background service at the lowest priority, unless a critical exception must not be lost and should be logged immediately before the app crashes.
Nice PJs Nick! :D
What if the external logger service goes down? Is there a possibility to also save the logs to the file in a host or something? In case of an event where the logger service is down and we need to investigate something how can we achieve that?
You can log to a file and have an agent that reads that file and sends the content to your log server in otel format, for example.
Or you write your logs to different sources (otel cloud, file,…) at the same time.
How would I log transactions or "sagas" from multiple services, that might collaborate to fulfill a task. So that I have logs for all steps and can combine logs from multiple services for exactly one business process?
Hi there, how would you be able to NUnit Test specfically NSubstitute to test OpenTelemetry logging, thanks
Hi Nick, Thanks for this amazing and knowledgeable video. I have a question though.
Let’s say I want to log to a file or maybe create my own log viewer within the application which I can host with the application and push my logs there using open telemetry. Is that doable?
It feels like you are about starting new trend for .NET developers clothes... with your nice Lidl shirt 😀
Great content as usual. One question: how can we use open telemetry in production, where could the logs be exported to. Any ideas or suggestions. Thanks
You can use the ELK stack or even Seq itself since it is OSS, as well as any other paid offering. I’ve used DataDog in the past
Nick do you have an example with Azure App Insights?
Apologies, if it has been asked before, what font are you using in your video?
Hey nick -
I was wandering, as a Unity developer, a former validation developer at intel, what courses whould i take to make my c# knowledge better?
i mean, i do have 3 years of experience in a varaity of c# useages (not .Net in particular), and i am a computer science major, so taking the "SOLID" course or those kind seem obsolete.
Of course I know seqs, I do the seqs every day
Lidl is the new Gucci 💖
Do you know if the .NET built-in logging provider can deconstruct the objects and log its composing properties using the OpenTelemetry exporter?
Yes it can
I'm not quite sure - how is it different to Application Insights?
Hey Nick, thank you for this great video. Do you perhaps know of a way how OpenTelemetry can log to files? (This is a requirement from our client we do work for)
if the default logger gets the job done, i don't see value of using the additional package , serilog in this case.
... ok but how to add OTEL traceability for microservices subscribed to a service bus?
Nice shirt ❣
I always assumed it was pronounced SEEK (as is seek and you shall find). :)
Question: what is the name of the app you use to test the API ?
It is called Kong Insomnia.
Great video. One question though. What is the benefit of adding Serilog to the picture? Never used it, so maybe that's a dumb question with obvious answer ;-)
Serilog, in general, brings many benefits, but in this specific context you can get a more useful formatting as well as the ability to serialize objects as part of the structured logged message. Please, keep in mind this is a very superficial answer. Serilog’s true benefits become more evident as more intricate scenarios come into play.
@@josefromspace ah. Thanks. Just always assumed that Serilog is basically predecessor of Microsoft.Extensions.Abstraction.Logging with (mostly) similar functionalities and just small deviations in interfaces or some obscure functionality. Will take a closer look than.
@@woocaschnowakabsolutely, Serilog’s claim of fame was that by using structured logging, it allocated less memory on the heap than .Net’s counterpart at the moment. As time passed both have grown in functionality, definitely worth looking into it.
I just install the Seq sink. I've also coded a nuget package that enhances the log properties with the user information, payloads, etc.
Can I get a link?
Nick looking different, cool!
why string interpolation in logs not accepted as standard?
Structured logging makes it easier to analyze logs. For example, if you wanted to see the 95th percentile response time grouped by day.
"Getting weather for {CityName}" is the same string every time, so you can easily search for this in your logs and go from there. If you used string interpolation, you'd have to search for the common substring which would be "Getting weather for " and that's a bit weird
Logging από τα LIDL? :D
Can we use the log instead of events? Can I subscribe on the log and then act on them. For let's say save data persistent?
You shouldn't use logs for anything they are not meant for.
No.
The beaty of using Seq is you can also visualise in it your OTEL traces and spans. Without having to use a seperate software, like Jaeger
That was exactly the thing I expected to see in the video, but didn't see. Thanks for confirming Seq can do that!
I'm adding logging into a bunch of apps at the moment, and to be honest this level of logging just isn't needed. Normal structured logging is enough for a lot of smaller applications. It comes down to only logging what you actually need and will look at, all the rest is useless fluff.
Hello guys how can i have access to Nick's discord server? (I have the 5 dollar subscription)
Have you linked your UA-cam channel to Discord?
@@nickchapsas yes nick, still dont have access! (update: i subscribed to patreon and got access)
I hate that you cannot export Request collections in insomnia
So you're saying that I shouldn't be logging the console to text files and pushing them to a database. 😅
Is that a Lidl tshirt? What the heck?
Why not string interpolated logging?
there's a few reasons, but structured logging allows you to preserve the parameters in such a way that you could query them later like he's doing in the video. Otherwise, you'd have to use complicated patterns (regex) to extract those parameters again after being joined into one string. Also, each unique log line would allocate a new string further increase in garbage collection pressure and slowing down the app.
a friend of mine has a full swaggy lidl outfit guys!!
Is it only me who whenever hears the "Telemetry" word thinks *VERY INTRUDING PRIVACY INVASION* ?
Is that a LIDL tshirt? 😅🤣😂
god damn lidl !
SEQ is pronounced SEEK.
Why English thow?
First comment
After watching this video I feel shame because of logging we are doing in our company. 😢
Big tech uses Java 🤔
Long winded