Plot twist: If you send your secret to a "secret manager" or an ESO, the secret is no longer "secret". Just because something works, it does not mean it works securely...
Thanks for another great video! I'd like to explore databases as a service with your videos. About secrets management with ESO, I use it all the time and I love it. I still find copying secret directly between kubernetes clusters a useful option, especially when you have a control plane cluster managing other clusters and secrets are consumed in kubernetes. This way you don't need to install (and secure) multiple ESO instances. I think it would be also useful to see an example of secret rotation in action (meaning not waiting for the secret to update in kubernetes but rotating it transparently while applications are using it).
thanks, love this advanced use cases!! I use reflector, to clone secret between namespaces within the same cluster and or eso with kubernetes provider to share secrets between clusters
Hey Viktor, Thanks for the information! your videos help me get the bigger picture of how things are managed in a mature k8s environment at scale. I'm a beginner starting out with Kubernetes and related tools in general, although I understand these approaches oftentimes there's confusion on how and whether are these applicable in my practice environment. (on prem cluster). Would be great if you could recommend a somewhat advanced project in order to get better at Kubernetes and the related tools. Thank you for your time reading this long paragraph.
We use it in my job the ESO, but the big problem in crossplane is reading secrets in composition without using a status for retaining this information.
Hi Victor, firstly thank you for sharing your wisdom :) I have one small question - looking on the final diagram. (besides other) you have created "external secret" in order to pull "root password" from aws secrets manager. How that "root password" was created? How did you put the password there? Was it generated? Or does the aws rds have even the capability to create secrets manager secret after the creation of rds? thank you
I generated that password with s simple aws command. It can be random if needed. I don't generate secrets like that one through manifests since that might end up being accidentally pushed to git.
Not directly related but doesn't have Cross plane a depends on tree for delaying execution loops until requirements are met? Been using vault webhook mutate and seems better for apps since injects secrets into running scope but it's harder to integrate
There is no need (most of the time) for a dependency loop since each resource has its own control loop. Those that cannot be created right away are created when the data they need (often available in other resources) becomes available.
Is it generally a bad practice for a namespaced claim (and consequently the composition) to refer (via k8 secret object observing) to a secret created by another claim's composition in a different namespace? Or should the latter's claim composition push to a common cluster secret store and then the former can pull it using ESO? Not thinking about cluster resources here, just something simpler that requires secret knowledge across namespaced resource claims.
@@DevOpsToolkit Totally agree for cross-cluster. For crossplane-cluster, it felt too much energy or friction for the reward. I was contemplating using reflector but I think crossplane can observe a secret in another namespace, is that correct?
Is ESO now the preferred method of integrating secrets with an external store (thinking Azure KeyVault in my case) considering it supports both push and pull now? Also, does this mean Crossplane's ESS (External Secret Stores) alpha feature will be dropped, or do they serve slightly different purposes?
I prefer ESO (not sure about others). Crossplane ESS is limited to Vault (at least now) while using ESO directly allows quite a few other providers. Not sure what is on the roadmap with Crossplane ESS...
@michals.2604 are you referring to ascp combined with SSCSID? If that's the case, the answer is yes. SSCSID is more secured since it does not use kubernetes secrets but volumes so data is not stored in etcd. That being said, whether storing something in etcd is secure or not depends on many factors like whether etcd is encrypted (it is in eks), whether it is accessible directly (it isn't in eks), etc. A more important issue is that you won't be able to use it with most third party apps since they are typically designed to expect secrets and not volume mounts.
Isn't password rotation an issue here? There would clearly be an outage when the password is changed depending on how often the secretsmanager contents are synced. The proper way to roll a password over without downtime would be to allow the old and new password during a certain time window but it seems to leak the entire "magic" of this setup when done
If you do need both the new and the old password (which i don't think you should need), you should use separate entries in your secrets manager. In any case, unless you plan to create secrets manually, the only reasonable path forward is with some operator.
@@DevOpsToolkit yeah I just mean it would be nice if the external secret would also abstract password rotation away. It's very difficult because it depends on the specifics of the actual systems using and providing the password I've seen many apps go down because some team goes full YOLO and changes the password in the password manager without actually checking where it gets synced in the cluster and then everything is just crashlooping, so clearly a drawback to the "magic" of just having a secret show up in k8s. It wouldn't be a problem if the abstraction of ExternalSecret took into account what kind of workflow is needed to "change" the value (both pull and push aspect of it). It's just a building block so something more is needed if you don't want downtime, otherwise "eventually" it will be alright some companies still have dumb policies like rotating every password at least once a year etc
Password rotation must follow the same rules as, let's say, changes to APIs or deployments of new releases. Changes need to be backwards compatible, at least until all start using it. Simply changing something does not work, no matter whether that's a change to secrets, APIs, releases, or anything else.
Have you seen by any chance ESO constantly updating K8s secrets, even though the value hasn't changed? Currently when looking at Events, some "externalsecret" objects are sitting at 27k+ Count..
@@DevOpsToolkit The reason is updated without the secret being updated.. You can see the count yourself │ Normal Updated 2m19s (x2487 over 41h) external-secrets Updated Secret │
I'm confused about this external secrets operator in general; seems like we alreay need a secret to connect to the "external source of secrets". Doesn't that basically defeat the purpose? I mean the secret will be there if a bad actor gets access to the cluster, and they can use it to get the actual secrets, no?
It's more for convenience and less for security, but the bad actor would need to get access to the control plane so it is for sure more secure than giving the app your aws root accesskey :P
@CarmineEspositoIT I wanted to make it more general so that it can apply to almost anything. Still, you're completely correct. If using AWS, it should be IRSA.
Plot twist: If you send your secret to a "secret manager" or an ESO, the secret is no longer "secret". Just because something works, it does not mean it works securely...
Thanks for another great video! I'd like to explore databases as a service with your videos.
About secrets management with ESO, I use it all the time and I love it.
I still find copying secret directly between kubernetes clusters a useful option, especially when you have a control plane cluster managing other clusters and secrets are consumed in kubernetes.
This way you don't need to install (and secure) multiple ESO instances.
I think it would be also useful to see an example of secret rotation in action (meaning not waiting for the secret to update in kubernetes but rotating it transparently while applications are using it).
thanks, love this advanced use cases!! I use reflector, to clone secret between namespaces within the same cluster and or eso with kubernetes provider to share secrets between clusters
Hey Viktor, Thanks for the information! your videos help me get the bigger picture of how things are managed in a mature k8s environment at scale. I'm a beginner starting out with Kubernetes and related tools in general, although I understand these approaches oftentimes there's confusion on how and whether are these applicable in my practice environment. (on prem cluster). Would be great if you could recommend a somewhat advanced project in order to get better at Kubernetes and the related tools. Thank you for your time reading this long paragraph.
Read the title singing Lando's song... "Secrets up and down side to side like a roller coaster..." 😂
We use it in my job the ESO, but the big problem in crossplane is reading secrets in composition without using a status for retaining this information.
Hi Victor, firstly thank you for sharing your wisdom :) I have one small question - looking on the final diagram. (besides other) you have created "external secret" in order to pull "root password" from aws secrets manager. How that "root password" was created? How did you put the password there? Was it generated? Or does the aws rds have even the capability to create secrets manager secret after the creation of rds? thank you
I generated that password with s simple aws command. It can be random if needed.
I don't generate secrets like that one through manifests since that might end up being accidentally pushed to git.
Not directly related but doesn't have Cross plane a depends on tree for delaying execution loops until requirements are met?
Been using vault webhook mutate and seems better for apps since injects secrets into running scope but it's harder to integrate
There is no need (most of the time) for a dependency loop since each resource has its own control loop. Those that cannot be created right away are created when the data they need (often available in other resources) becomes available.
Is it generally a bad practice for a namespaced claim (and consequently the composition) to refer (via k8 secret object observing) to a secret created by another claim's composition in a different namespace? Or should the latter's claim composition push to a common cluster secret store and then the former can pull it using ESO? Not thinking about cluster resources here, just something simpler that requires secret knowledge across namespaced resource claims.
I prefer using ESO to push and pull if multiple clusters are involved. In a single Cluster that might be too much of an overhead.
@@DevOpsToolkit Totally agree for cross-cluster. For crossplane-cluster, it felt too much energy or friction for the reward. I was contemplating using reflector but I think crossplane can observe a secret in another namespace, is that correct?
Yes. Managed resources are cluster-scoped and can observe any resource in any namespace as long as the service account allows it.
@@DevOpsToolkit Thanks for that. Your videos are fountains of knowledge in the devops space for me so really appreciate your answers
Is ESO now the preferred method of integrating secrets with an external store (thinking Azure KeyVault in my case) considering it supports both push and pull now? Also, does this mean Crossplane's ESS (External Secret Stores) alpha feature will be dropped, or do they serve slightly different purposes?
I prefer ESO (not sure about others).
Crossplane ESS is limited to Vault (at least now) while using ESO directly allows quite a few other providers.
Not sure what is on the roadmap with Crossplane ESS...
@@DevOpsToolkit Is not ESO significantly less secure compared to aws ascp and similar?
@michals.2604 are you referring to ascp combined with SSCSID? If that's the case, the answer is yes. SSCSID is more secured since it does not use kubernetes secrets but volumes so data is not stored in etcd. That being said, whether storing something in etcd is secure or not depends on many factors like whether etcd is encrypted (it is in eks), whether it is accessible directly (it isn't in eks), etc. A more important issue is that you won't be able to use it with most third party apps since they are typically designed to expect secrets and not volume mounts.
Isn't password rotation an issue here? There would clearly be an outage when the password is changed depending on how often the secretsmanager contents are synced.
The proper way to roll a password over without downtime would be to allow the old and new password during a certain time window but it seems to leak the entire "magic" of this setup when done
If you do need both the new and the old password (which i don't think you should need), you should use separate entries in your secrets manager. In any case, unless you plan to create secrets manually, the only reasonable path forward is with some operator.
@@DevOpsToolkit yeah I just mean it would be nice if the external secret would also abstract password rotation away. It's very difficult because it depends on the specifics of the actual systems using and providing the password
I've seen many apps go down because some team goes full YOLO and changes the password in the password manager without actually checking where it gets synced in the cluster and then everything is just crashlooping, so clearly a drawback to the "magic" of just having a secret show up in k8s. It wouldn't be a problem if the abstraction of ExternalSecret took into account what kind of workflow is needed to "change" the value (both pull and push aspect of it). It's just a building block so something more is needed if you don't want downtime, otherwise "eventually" it will be alright
some companies still have dumb policies like rotating every password at least once a year etc
Password rotation must follow the same rules as, let's say, changes to APIs or deployments of new releases. Changes need to be backwards compatible, at least until all start using it. Simply changing something does not work, no matter whether that's a change to secrets, APIs, releases, or anything else.
Have you seen by any chance ESO constantly updating K8s secrets, even though the value hasn't changed? Currently when looking at Events, some "externalsecret" objects are sitting at 27k+ Count..
I haven't had that problem. What's in events when you describe that external secret?
@@DevOpsToolkit The reason is updated without the secret being updated.. You can see the count yourself
│ Normal Updated 2m19s (x2487 over 41h) external-secrets Updated Secret │
I'm confused about this external secrets operator in general; seems like we alreay need a secret to connect to the "external source of secrets". Doesn't that basically defeat the purpose? I mean the secret will be there if a bad actor gets access to the cluster, and they can use it to get the actual secrets, no?
maybe you might look into spiffe / spire. It solves the bottom turtle problem...
It's more for convenience and less for security, but the bad actor would need to get access to the control plane so it is for sure more secure than giving the app your aws root accesskey :P
Good practice would be to use IRSA to connect to the external source of secrets instead of credentials stored in a secret
Sorry, what's IRSA?
@CarmineEspositoIT I wanted to make it more general so that it can apply to almost anything. Still, you're completely correct. If using AWS, it should be IRSA.