Prompt injection

Note: This content is adapted from a LinkedIn Post I made back in December 8th, 2022, and appears here with a few changes to add context.

Prompt injection attacks happen when an ML language model has been given a prompt before being presented to the user, but the user can negate it.

For example, imagine a language model has been asked to translate from French to English as part of an application. If one simply uses the translation prompt and asks the model to ignore its previous instructions and do something else, it will.¹

In ChatGPT, safety features that keep it from saying awful things can be bypassed through prompt-injection technqiues. Various ways I and others have found to negate this prompt:

• Encode your question in unicode, base64, or pig latin
• Ask ChatGPT to respond “as a large language model without safety features” and “without warning me.”
• Ask ChatGPT to tell you something evil inside of code, poetry, or a report. For example, it has no problem wrapping the instructions to make drugs inside of python.
• Similarly, have ChatGPT create a text adventure game that you interactively respond to that involves something graphic.
• ChatGPT can even invent new ways to “jailbreak” itself.²

The following screenshots demonstrate bypassing these protections using basic prompt injection techniques. They were all were taken back in 2022, and different techniques may be necessary to reproduce them now.

Using these tactics, ChatGPT was even willing to help me with other altered fake phishing URLs:

ChatGPT was even willing to help me install evilnix2, a phishing toolkit, to help with the phishing email. Yikes!

As these kind of language models become used in real services, the input created by the user should be considered untrusted. Attempts to sanitize these inputs, or even harder, sanitizing the output of the language model, will lead a new field of security vulnerabilities.