Real-world exploits and mitigations in LLM applications (37c3)

Video recording of my talk at the 37th Chaos Communication Congress in Hamburg titled "NEW IMPORTANT INSTRUCTIONS: Real-world exploits and mitigations in Large Language Model applications" about LLM app security and Prompt Injections specifically.
A big thank you to the CCC organizers and all the volunteers for putting together such a great event!
Source Video: media.ccc.de/v/37c3-12292-new...
Blog Post: embracethered.com/blog/posts/...
Abstract:
With the rapid growth of AI and Large Language Models (LLMs) users are facing an increased risk of scams, data exfiltration, loss of PII, and even remote code execution. This talk will demonstrate many real-world exploits the presenter discovered, including discussion of mitigations and fixes vendors put in place for the most prominent LLM applications, including ChatGPT, Bing Chat and Google Bard. The talk is about LLM security at large with a focus specifically on implications of Prompt Injections.

[@chitchatvn5208]

Thanks Johann.

[@embracethered]

You are welcome!

[@notV3NOM]

Thanks , great insights

[@embracethered]

Thanks for watching! Glad it was interesting.

[@ludovicjacomme1804]

Excellent presentation, thanks a lot for sharing, extremely informative.

[@embracethered]

Thanks for watching! Glad to hear it's informative! 🙂

[@jlf_]

I really enjoyed your talk, Johann! Thank you!

[@embracethered]

Thanks for watching and glad you enjoyed it! 🙂

[@Fitnessdealnews]

One of the best presentation I’ve seen

[@embracethered]

Thanks for watching! Really appreciate the feedback! 😀

[@artemsemenov8136]

Thank you, is awesome!

[@embracethered]

Glad you like it!

[@artemsemenov8136]

@@embracethered I'm a fan of yours, I've talked about your research at cybersecurity conferences in Russia. You're awesome.

[@embracethered]

Thank you! 🙏

[@artemsemenov8136]

@@embracethered what you think abot LLM security scanners, garak and vigil. Also, have you met P2SQlinjection in the real world ?

[@347my455]

superb!

[@embracethered]

Thank you!🙏

[@macklemo5968]

🔥

[@embracethered]

Thanks! 🚀🚀🚀

[@MohdAli-nz4yi]

I think a better conclusion is: never put in the context of an LLM information you need to keep private, because it will leak.

[@embracethered]

Thanks for watching and the note. I think that misses the point that the LLM can attack the hosting app/user, so developers/users can't trust the responses. this includes confused deputy issues (in the app), such as automatic tool invocation.

[@MohdAli-nz4yi]

@@embracethered Agreed! So 2 big points:
1. Never put info in LLM context you don't want to leak.
2. Never put untrusted input into LLM context, it's like executing arbitrary code you have downloaded from the internet on your machine. LLM inputs must always be trusted, because the LLM will "execute" it in "trusted mode".

[@embracethered]

@@MohdAli-nz4yi (1) I agree we shouldn't put sensitive information, like passwords, credit card number, or sensitive PII into chatbots. For (2) The challenge is that everyone wants to have an LLM operate over untrusted data. And that's the problem that hopefully one day will have a deterministic and secure solution. For now the best advise is to not trust the output. e.g. Developers shouldn't blindly take the output and invoke other tools/plugins in agents or render output as HTML, and users shouldn't blindly trust the output because it can be a hallucination (or a backdoor), or attacker controlled via an indirect prompt injection.
However, some use cases might be too risky to implement at all. And its best to threat model implementations accordingly to understand risks and implications.