Insecure output handling
In the previous examples, we learned about a few various security risks, threats, and exploits, especially against generative AI and LLMs.
One last (but not least) security risk we would like to cover in this book is the concept of insecure output handling. As the name applies, this risk is about the output of an LLM, specifically a flaw created when an application accepts LLM output without any additional analysis or scrutiny, thus making this insecure. In this risk, the completion is accepted as-is, regardless of if this came from a trusted LLM or not.
As a safeguard, always confirm the completion or output before taking any action based on the blindly accepted output. Some of the risks might include a potential breach of sensitive data and potential privileged access or possibly any remote code execution as well.
For example, many LLMs can handle or generate code. Let’s say an application blindly trusts an LLM-generated SQL query based on your input and then runs this against your database. Do you know what that SQL query is doing? Could it copy data to another table or location? Can it delete some fields, columns, transactions, or, worse, an entire database?
Important note
As you can see from just this single example, not managing insecure output handling tasks can be detrimental to your organization.
To mitigate this security risk, a review or audit of the outputs is critical. We do see emerging LLMs that can help with a security review; however, this discipline is still quite new and evolving.
Additionally, just as we covered in the prompt injection section before, using mature security tools and guidance, such as the OWASP ASVS (Application Security Verification Standard) guidelines, can ensure that you are putting the appropriate safeguards in place to protect against insecure output handling security risks.
The emergence of generative AI and LLMs has been quite exciting, as we have seen in the many exciting topics in this book. However, companies, organizations, governments, or any entities building applications and services that create or use LLMs need to handle this with caution and tread lightly, in the same way, they would if they were using a product or technology service that is still in beta or in its very early release. We always recommend verifying every component of your generative AI cloud solution or service, from the LLM itself to any relevant dataset or plugins used in the overall solution. Verifying and confirming each and every component against security risks may seem like a long, arduous task upfront, but the benefits of a safe, secure, generative AI cloud solution environment will serve you and your organization in the long term.
While we did cover some of the best practices and techniques to ensure a more secure generative AI enterprise service, let’s go into more detail on the “hows” of securing your cloud-based ChatGPT or other generative AI LLM solution in the next section.
