Would be nice to see a new episode on app promotion across the environments. A solution that considers post deployment health checks and automatically raises a PR with the required changes for the next in line env. E.g. Test -> Stg -> Prod
@@DevOpsToolkit Thanks Viktor, your Todo list must be miles long :). Not sure if you've got some experience in that already! Don't want to mud the water. I'm still researching on native capabilities of FluxCD, ArgoCD and their adjacent projects like GitOps and Kargo. Will post my findings.
Man Man Man I love the work that you are doing. I was blown away by the shift left series, all that is left I guess is determining how to set up a k8 based CI system that can gel well with Argo and bam I'll have a platform that I can use for my personal projects which can scale up later on-demand with me just being able to give hobbyist time to this.
Since I use GitLab and there is no direct way to run a CI job when a merge request, also known as a PR, is merged. GitLab CI works on push changes, so I created a job to run on the base branch, which is usually the main branch. This job calls the GitLab merge requests list, filters by merged status using jq to get the IDs. With these IDs, we can then proceed to run cleanup tasks, whether they involve vclusters or namespaces
Like the combinations of tools. Dunno how can be extrapolated within a company with hundred of developers and projects. It would be more a change of mentality rather than technical limitations. Thanks for share
Most of the challenges are NOT technical. It's much easier to change a tool than to convince everyone that tool is useful. It's similar with processes.
@@DevOpsToolkit Great video, but the gist is now in description. I love how you started with the all tools can change and showed in detail how to start automation on projects.
@@zamboz01 Hmmm... When I go to the description of the video (and click the "more" link), the link to the Gist is in the "Additional Info" section (the first one). Not sure why it's not visible in your case or what to do to make it visible :(
Assuming it would be a big app with many microservices for the frontend, would you only build the images that have changed in the PR and use the existing ones for others? I think it could be quite hard to do. Further, how would you expose the endpoint with an ingress / https certs? It would be a great solution to E2E all of it, but I see it as a challenge for large systems.
Sounds great , but I think PR environment fit in stuff like frontend application , but if I have a stateless app that connect to db and there is a migration changes related to this PR so you can not use one database for PRs because they will conflicting each others and in that case you might use temporary database that will be removed after PR merge that will end up with a very complex situation, in addition to install stuff like traefik or nginx in the virtual cluster every PR and so on , what do you think ?
Yeah. The second part of the story is how to connect apps running in preview envs to those running on permanent ones like staging or reproduction. I'm working on two videos on that subject. They will be released in the upcoming weeks.
creating a load balancer as part of the workflow probably isn't a good idea because what'll happen is if the destruction of that load balancer is dependent on that PR being merged or closed in any way is that load balancer costs will sky rocket; they are quite expensive. what would be better, not to state the obvious, but just in case anyone was thinking through this, would be to use (preferably) an API gateway in which the PR adds, effectively, a route, or similarly, a service mesh (not a fan really), this way you're not wasting a ton of money on limbo lb's. my two cents.
Great video Viktor. I have a question: do I have to build a new image when I merge the PR branch to production or can I use the image built from the PR branch, because the same image would be built on the same once it is merged?
You don't have to build a new image when merging a PR into the mainline. As a matter of fact, it should NOT be a new image. That way you know that what you're deploying to production is what was tested in previous environments. In that case, I would create a new tag based on the PR image mostly so that it's properly versioned (e.g,. semantic versioning).
What is the process you're using for the lifecycle of your applications? Which tools do you prefer? IMPORTANT: For reasons I do not comprehend (and Google support could not figure out), UA-cam tends to delete comments that contain links. Please exclude from the links to ensure that your comments are not removed.
I have some theoretical questions and consideration about dynamic environments and testing based on pull request. 1. Do you prefer PR/branch per user story (which delivers complete feature) or PR per task(which deliver only part of feature/story)? In latter case there is no sense creating environment for manual/uat testing per pull request (because desired feature which you want to test isn’t complete), but only for automated regression testing . Former approach, in my opinion breaks continuous integration rule of short leaved feature branches - potentially the branch is shared by many developers, and can live more than one, two days, which can cause merged conflicts. Do you agree? 2. When you merge your PR to main line, it is tested on open PR, but code which was merged from another PR, in meanwile, could breake something . So you should run all runtime tests (functional, performance, tests , etc on production like env.) also after merge. So you duplicate tests. So why not only run unit tests (which are faster) on creating PR, and runtime tests after merge? In that scenario, you really not need on demand environments. Or maybe there is another way to avoid conflicts between pull rquest?
1. I prefer a feature branch that results in a PR for a minimum deliverable to production. That that is called a story, a task, or something else is not that important. What does matter is that a branch is eventually merged to the mainline and delivered to production. The less time is spent between creating a branch and merging it to the mainline, the better. 2. A PR should always contain the latest code from the mainline. That means that you should merge the mainline into the PR whenever the mainline changes. Assuming that any change to the PR executes a pipeline that runs the tests and whichever other verifications you might have, it should not matter the order in which PRs are merged to the mainline. That also means that you do not have to run tests after merging to the mainline since you run them after merging the mainline to the PR. As for on-demand environments... If you do not have environments for PRs, you cannot run anything but static tests which would make merging them to the mainline, not a good idea, assuming that merge to the mainline is the signal that the process ends with deployment to prod should start. In other words, PRs are where most of the action is happening and when we merge them to the mainline, we're doing that because we are confident that it's ready for production. We can never be 100%, but we should get close it.
There is no need for GitHub Actions or any other tool to connect to the production cluster when using Argo CD or Flux. Those are pull-based tools running inside your clusters and the only thing you need in GitHubActions (or any other pipeline tool) is to make changes to the desired state stored in a Git repo.
I meant more about the pr-open action using vcluster. That command uses the kubeconfig you set as a secret, but it still needs to access the api somehow. Is your cluster just public in this case?
@@jdu1111 Yes. In that specific case, the cluster is accessible from GitHub actions. In a real-life scenario, that would not be the production cluster though. Also, I'm using GitHub Actions only as an example. You could accomplish the same result with, let's say, Tekton or Argo Workflows running inside the cluster used for preview environments. In that case, you would not need to enable access to it since the source (Tekton/Argo) is the same as the destination (it's running in the same cluster).
Hi Viktor, thanks for great video! But I’m interesting in what if I want to use my mainline branch to deploy changes to staging first and promote changes to production only via tag? I’m using Argocd and keep all manifests including app argocd custom resource inside the app repo, and I believe to deploy changes to staging it’s enough to set target revision on Argo app to HEAD, but what I should do when I set tag and my production argo app lives in another cluster and should point only to specific tag? Since argo can monitor only specific tag, I should tell him to what new tag he should sync changes. I see several approaches, one of them get direct access to production argo server and manually set target tag in my CD, but this approach violates gitops principles and I should provide access to cluster to my pipeline. The second way is create another repository to hold Argo app manifest there and new Argo app which will sync changes from separate repository and push new commit from my app repo to change target tag in Argo app. What do you think? Both solutions are not good enough from my perspective, but perhaps you have another recipe how to adopt trunkbased with argocd and releasing from tag? Thanks!
I always use specific (container) tags in Argo CD apps (together with env. specific variables). That way, I know exactly what are env. specific properties associated with each app in each environment. I'm not sure why you think that's not "good enough".
@@DevOpsToolkit not good enough, I mean I don’t like an idea with accessing cluster directly and I don’t like having extra repositories. If you use a tag for Argo target revision then how do you update that tag? If I just change a tag in my app manifest and push to the git, Argo cd will not know about this change cuz it still syncing to the previous tag. We should have some way to tell Argo app to use another target revision and I’m interesting in how exactly this can be done?
@@kostyazgara3481I think we missunderstood each other. I am not suggesting anyone ever accessing any cluster (except in break-glass scenarios). I was referring to image tags defined in Argo CD App manifests stored in Git. As for "extra" repositories... That's up to you. You can have many repos, one repo, or anything in between. When you define an Argo CD App, you have to specify a repo and a directory so you can organize "stuff" within directories as well. > If you use a tag for Argo target revision then how do you update that tag? You can update the manifest using pipelines (e.g., Jenkins, GitHub Actions, Argo Workflows) or you can use more specialized tools like Akuity Kargo. In any case, the point is that, one way or another, you modify the desired state in Git. > If I just change a tag in my app manifest and push to the git, Argo cd will not know about this change cuz it still syncing to the previous tag. I'm not sure why you would use Git tags for that (you have image tags that matter and the mainline should represent production). Nevertheless, if you do use Git tag, you need to reference those tag in App manifests and that means that you modify the manifest and push it back to Git. Remember, GitOps is all about defining the desired state. If you don't want to have the desired state in Git, GitOps is not a good option. All in all, change Argo CD manifests and push them to Git. Do that through pipelines that you use for other types of tasks or through specialized tools that monitor contianer image registries. The end result must be the desired state stored in Git.
@@DevOpsToolkit I’d like to set target revision to git tag, cuz not all changes are present in new image tag. Let’s say I have added new lib which requires new env variable. To inject new env var obviously I need to modify deployment manifest, but this change is not present in new image, right? And this change should be applied by argo. But okay, seems I got you, the only true gitops way is somehow change argo app manifest. And this “somehow” becomes a tricky point to me, but hope I will find a best solution that will fit my needs. Thanks again!
What would you do when applications are not to be tested in isolation? e.g. the changed service in the Pull Request has a dependency with many other services. Would you deploy them all in the preview environment?
I rarely play fighting games, mostly because there's no one in my family to play them with :( The last one I played is Shadow Dancer (en.wikipedia.org/wiki/Shadow_Dancer) and before that Wonder Boy (en.wikipedia.org/wiki/Wonder_Boy). Below those stricks is an arcade machine loaded with 3k arcade games from the 80s and 90s.
Very nice setup! Do you have any hints on how it would work in a microservice architecture so that every PR not only deploys the single application but also the rest of the microservices it uses? Also, instead adding commits programmatically for deploying on production you can extract the kustomize part into a second repo containing all infrastructure part and just udpate that on each prod release. This way your main repo stays clean, you still have all versions in git and pull this infra repo from ArgoCD!
Unless you have only a few small apps, I do not think it's a good idea to deploy everything for every PR. That would cost too much (use too much compute). Nevertheless, if that's what you need to do, I would create a virtual cluster (e.g., vCluster) or a Namespace, fetch manifests that describe production, modify the tag of the app in question, and deploy them all, directly or by pushing changes to Git. In my case, even though for simplicity reasons that's not how I organized the demo, production env repo contains only the references to manifests and env-specific overwrites. The base manifests are in repos of applications. I think that's similar, if not the same, as what you're describing when you said that the main repo should stay clean.
@@DevOpsToolkit how else do you recommend creating a staging environment for a microservices architecture where you need to test the changes for 1 service?
@@DevOpsToolkit for that case, wouldn't you have to ensure that the other staging apps are not running some modified code? I mean for testing my service, I should be sure that the other services are running the latest release branch and not some random branch which is being developed. Maybe one would be using different namespaces for this, right? So a namespace with all the apps mimicking production apps and another namespace where we deploy testing apps. Then testing apps interact with the services in "mimicking" namespace
if i use Argo Workflows instead of GitHub Actions, how would be the integration of Argo Workflows with git repository(say bitbucket) for PR approval process ?
Normally, you'd use Argo Events to notify Argo Workflows (or something else) whenever specific events occur in Git (or anywhere else). In that context, think of Argo Events as the entity that a) creates Git webhooks and b) figures out where to redirect the requests coming from those webhooks. Argo Events is much more than that but, within the context of your question, that's what it does. Is that what you meant or I misunderstood your question?
Would be nice to see a new episode on app promotion across the environments. A solution that considers post deployment health checks and automatically raises a PR with the required changes for the next in line env. E.g. Test -> Stg -> Prod
Adding it to my to-do list...
@@DevOpsToolkit Thanks Viktor, your Todo list must be miles long :). Not sure if you've got some experience in that already! Don't want to mud the water. I'm still researching on native capabilities of FluxCD, ArgoCD and their adjacent projects like GitOps and Kargo. Will post my findings.
@powersurge5576 my to-do list is indeed massive. That subject is something working on a lot so it'll be mostly about putting it all together.
Thanks!
Thanks a ton!
Your videos are simply amazing and a great source of inspiration. Thank you for sharing. 😁😁
Man Man Man I love the work that you are doing. I was blown away by the shift left series, all that is left I guess is determining how to set up a k8 based CI system that can gel well with Argo and bam I'll have a platform that I can use for my personal projects which can scale up later on-demand with me just being able to give hobbyist time to this.
Awesome video, Viktor. Keep them coming! ;)
Thanks for the video Viktor.
Combination of pre-commit hooks, gitsign, cosign, devspace, vcluster, devpod, k3s/k3d, VSCode demo locally would be really helpful for developers!!
very inspirational content!!! thank you so much, man.
Excellent demo, thanks.
Since I use GitLab and there is no direct way to run a CI job when a merge request, also known as a PR, is merged. GitLab CI works on push changes, so I created a job to run on the base branch, which is usually the main branch. This job calls the GitLab merge requests list, filters by merged status using jq to get the IDs. With these IDs, we can then proceed to run cleanup tasks, whether they involve vclusters or namespaces
Yeah. It's a bit more difficult, but doable.
@@DevOpsToolkit Ya I did it already and it works
We need more videos like this
Thanks for the encouragement.
Is there a topic you'd like to suggest for one of the upcoming videos?
Like the combinations of tools. Dunno how can be extrapolated within a company with hundred of developers and projects. It would be more a change of mentality rather than technical limitations. Thanks for share
Most of the challenges are NOT technical. It's much easier to change a tool than to convince everyone that tool is useful. It's similar with processes.
hey Viktor, nice video, I think you missed to put the link to the gist 😜
Thanks for all your work!
Thanks for letting me know. It should be fixed now :)
@@DevOpsToolkit Great video, but the gist is now in description. I love how you started with the all tools can change and showed in detail how to start automation on projects.
@@zamboz01 Hmmm... When I go to the description of the video (and click the "more" link), the link to the Gist is in the "Additional Info" section (the first one).
Not sure why it's not visible in your case or what to do to make it visible :(
Thanks Viktor
Great explanation
3:17 made me laugh hard
Thanks for great video :-)
Assuming it would be a big app with many microservices for the frontend, would you only build the images that have changed in the PR and use the existing ones for others?
I think it could be quite hard to do.
Further, how would you expose the endpoint with an ingress / https certs?
It would be a great solution to E2E all of it, but I see it as a challenge for large systems.
Cool thank you
Sounds great , but I think PR environment fit in stuff like frontend application , but if I have a stateless app that connect to db and there is a migration changes related to this PR so you can not use one database for PRs because they will conflicting each others and in that case you might use temporary database that will be removed after PR merge that will end up with a very complex situation, in addition to install stuff like traefik or nginx in the virtual cluster every PR and so on , what do you think ?
Yeah. The second part of the story is how to connect apps running in preview envs to those running on permanent ones like staging or reproduction. I'm working on two videos on that subject. They will be released in the upcoming weeks.
creating a load balancer as part of the workflow probably isn't a good idea because what'll happen is if the destruction of that load balancer is dependent on that PR being merged or closed in any way is that load balancer costs will sky rocket; they are quite expensive. what would be better, not to state the obvious, but just in case anyone was thinking through this, would be to use (preferably) an API gateway in which the PR adds, effectively, a route, or similarly, a service mesh (not a fan really), this way you're not wasting a ton of money on limbo lb's. my two cents.
Ingress controller is creating a load balancer. Adding a route to ingress (ingress resource) does not create an LB.
Great video Viktor. I have a question: do I have to build a new image when I merge the PR branch to production or can I use the image built from the PR branch, because the same image would be built on the same once it is merged?
You don't have to build a new image when merging a PR into the mainline. As a matter of fact, it should NOT be a new image. That way you know that what you're deploying to production is what was tested in previous environments.
In that case, I would create a new tag based on the PR image mostly so that it's properly versioned (e.g,. semantic versioning).
What is the process you're using for the lifecycle of your applications? Which tools do you prefer?
IMPORTANT: For reasons I do not comprehend (and Google support could not figure out), UA-cam tends to delete comments that contain links. Please exclude from the links to ensure that your comments are not removed.
I have some theoretical questions and consideration about dynamic environments and testing based on pull request.
1. Do you prefer PR/branch per user story (which delivers complete feature) or PR per task(which deliver only part of feature/story)? In latter case there is no sense creating environment for manual/uat testing per pull request (because desired feature which you want to test isn’t complete), but only for automated regression testing . Former approach, in my opinion breaks continuous integration rule of short leaved feature branches - potentially the branch is shared by many developers, and can live more than one, two days, which can cause merged conflicts. Do you agree?
2. When you merge your PR to main line, it is tested on open PR, but code which was merged from another PR, in meanwile, could breake something . So you should run all runtime tests (functional, performance, tests , etc on production like env.) also after merge. So you duplicate tests. So why not only run unit tests (which are faster) on creating PR, and runtime tests after merge? In that scenario, you really not need on demand environments. Or maybe there is another way to avoid conflicts between pull rquest?
1. I prefer a feature branch that results in a PR for a minimum deliverable to production. That that is called a story, a task, or something else is not that important. What does matter is that a branch is eventually merged to the mainline and delivered to production. The less time is spent between creating a branch and merging it to the mainline, the better.
2. A PR should always contain the latest code from the mainline. That means that you should merge the mainline into the PR whenever the mainline changes. Assuming that any change to the PR executes a pipeline that runs the tests and whichever other verifications you might have, it should not matter the order in which PRs are merged to the mainline. That also means that you do not have to run tests after merging to the mainline since you run them after merging the mainline to the PR. As for on-demand environments... If you do not have environments for PRs, you cannot run anything but static tests which would make merging them to the mainline, not a good idea, assuming that merge to the mainline is the signal that the process ends with deployment to prod should start.
In other words, PRs are where most of the action is happening and when we merge them to the mainline, we're doing that because we are confident that it's ready for production. We can never be 100%, but we should get close it.
I'm assuming the "production cluster" you have on here is on some cloud environment that allows github actions runners to connect to it publicly?
There is no need for GitHub Actions or any other tool to connect to the production cluster when using Argo CD or Flux. Those are pull-based tools running inside your clusters and the only thing you need in GitHubActions (or any other pipeline tool) is to make changes to the desired state stored in a Git repo.
I meant more about the pr-open action using vcluster. That command uses the kubeconfig you set as a secret, but it still needs to access the api somehow. Is your cluster just public in this case?
@@jdu1111 Yes. In that specific case, the cluster is accessible from GitHub actions. In a real-life scenario, that would not be the production cluster though. Also, I'm using GitHub Actions only as an example. You could accomplish the same result with, let's say, Tekton or Argo Workflows running inside the cluster used for preview environments. In that case, you would not need to enable access to it since the source (Tekton/Argo) is the same as the destination (it's running in the same cluster).
@@DevOpsToolkit I figured! Thanks for all the videos!
Hi Viktor, thanks for great video! But I’m interesting in what if I want to use my mainline branch to deploy changes to staging first and promote changes to production only via tag? I’m using Argocd and keep all manifests including app argocd custom resource inside the app repo, and I believe to deploy changes to staging it’s enough to set target revision on Argo app to HEAD, but what I should do when I set tag and my production argo app lives in another cluster and should point only to specific tag? Since argo can monitor only specific tag, I should tell him to what new tag he should sync changes. I see several approaches, one of them get direct access to production argo server and manually set target tag in my CD, but this approach violates gitops principles and I should provide access to cluster to my pipeline. The second way is create another repository to hold Argo app manifest there and new Argo app which will sync changes from separate repository and push new commit from my app repo to change target tag in Argo app. What do you think? Both solutions are not good enough from my perspective, but perhaps you have another recipe how to adopt trunkbased with argocd and releasing from tag? Thanks!
I always use specific (container) tags in Argo CD apps (together with env. specific variables). That way, I know exactly what are env. specific properties associated with each app in each environment. I'm not sure why you think that's not "good enough".
@@DevOpsToolkit not good enough, I mean I don’t like an idea with accessing cluster directly and I don’t like having extra repositories.
If you use a tag for Argo target revision then how do you update that tag? If I just change a tag in my app manifest and push to the git, Argo cd will not know about this change cuz it still syncing to the previous tag. We should have some way to tell Argo app to use another target revision and I’m interesting in how exactly this can be done?
@@kostyazgara3481I think we missunderstood each other. I am not suggesting anyone ever accessing any cluster (except in break-glass scenarios). I was referring to image tags defined in Argo CD App manifests stored in Git.
As for "extra" repositories... That's up to you. You can have many repos, one repo, or anything in between. When you define an Argo CD App, you have to specify a repo and a directory so you can organize "stuff" within directories as well.
> If you use a tag for Argo target revision then how do you update that tag?
You can update the manifest using pipelines (e.g., Jenkins, GitHub Actions, Argo Workflows) or you can use more specialized tools like Akuity Kargo. In any case, the point is that, one way or another, you modify the desired state in Git.
> If I just change a tag in my app manifest and push to the git, Argo cd will not know about this change cuz it still syncing to the previous tag.
I'm not sure why you would use Git tags for that (you have image tags that matter and the mainline should represent production). Nevertheless, if you do use Git tag, you need to reference those tag in App manifests and that means that you modify the manifest and push it back to Git. Remember, GitOps is all about defining the desired state. If you don't want to have the desired state in Git, GitOps is not a good option.
All in all, change Argo CD manifests and push them to Git. Do that through pipelines that you use for other types of tasks or through specialized tools that monitor contianer image registries. The end result must be the desired state stored in Git.
@@DevOpsToolkit I’d like to set target revision to git tag, cuz not all changes are present in new image tag. Let’s say I have added new lib which requires new env variable. To inject new env var obviously I need to modify deployment manifest, but this change is not present in new image, right? And this change should be applied by argo. But okay, seems I got you, the only true gitops way is somehow change argo app manifest. And this “somehow” becomes a tricky point to me, but hope I will find a best solution that will fit my needs. Thanks again!
Hi Viktor, are you planning on an ArgoCD autopilot video?
I am :)
It's alreadyt on my TODO list but, seeing that there is interest for it, I'll bump it up so that it's closer to the top.
What would you do when applications are not to be tested in isolation? e.g. the changed service in the Pull Request has a dependency with many other services. Would you deploy them all in the preview environment?
You can always configure your app (or apps) in preview envs to talk to other apps in other environments (e..g staging, production, etc.).
Have you tried the Kubernetes platform Deckhouse? Would like to hear your opinion
I haven't tried it (yet). Adding it to my TODO list...
What kind of fighting games do you play on that dual fightstick?
I rarely play fighting games, mostly because there's no one in my family to play them with :(
The last one I played is Shadow Dancer (en.wikipedia.org/wiki/Shadow_Dancer) and before that Wonder Boy (en.wikipedia.org/wiki/Wonder_Boy).
Below those stricks is an arcade machine loaded with 3k arcade games from the 80s and 90s.
Very nice setup! Do you have any hints on how it would work in a microservice architecture so that every PR not only deploys the single application but also the rest of the microservices it uses? Also, instead adding commits programmatically for deploying on production you can extract the kustomize part into a second repo containing all infrastructure part and just udpate that on each prod release. This way your main repo stays clean, you still have all versions in git and pull this infra repo from ArgoCD!
Unless you have only a few small apps, I do not think it's a good idea to deploy everything for every PR. That would cost too much (use too much compute). Nevertheless, if that's what you need to do, I would create a virtual cluster (e.g., vCluster) or a Namespace, fetch manifests that describe production, modify the tag of the app in question, and deploy them all, directly or by pushing changes to Git.
In my case, even though for simplicity reasons that's not how I organized the demo, production env repo contains only the references to manifests and env-specific overwrites. The base manifests are in repos of applications. I think that's similar, if not the same, as what you're describing when you said that the
main repo should stay clean.
@@DevOpsToolkit how else do you recommend creating a staging environment for a microservices architecture where you need to test the changes for 1 service?
Deploy that one microservice and connect it to other apps running in production or staging.
@@DevOpsToolkit for that case, wouldn't you have to ensure that the other staging apps are not running some modified code? I mean for testing my service, I should be sure that the other services are running the latest release branch and not some random branch which is being developed.
Maybe one would be using different namespaces for this, right? So a namespace with all the apps mimicking production apps and another namespace where we deploy testing apps. Then testing apps interact with the services in "mimicking" namespace
if i use Argo Workflows instead of GitHub Actions, how would be the integration of Argo Workflows with git repository(say bitbucket) for PR approval process ?
Normally, you'd use Argo Events to notify Argo Workflows (or something else) whenever specific events occur in Git (or anywhere else). In that context, think of Argo Events as the entity that a) creates Git webhooks and b) figures out where to redirect the requests coming from those webhooks. Argo Events is much more than that but, within the context of your question, that's what it does.
Is that what you meant or I misunderstood your question?
Is DevSpace still your recommendation for Local Kubernetes development?
Among those I used, yes, it is.
the cutaways to scary adults and children are terrifying fyi.
Thanks!
Thanks a ton, Joseph.
Contributions like that one are what keep the channel going and the expenses tolerable.