Defending against Prompt Injection with Structured Queries (StruQ) and Preference Optimization (SecAlign)

<!-- twitter --> <p>Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications. However, as LLMs have improved, so have the attacks against them. <a href="https://www.ibm.com/topics/prompt-injection">Prompt injection attack</a> is listed as the <a href="https://owasp.org/www-project-top-10-for-large-language-model-applications">#1 threat by OWASP</a> to LLM-integrated applications, where an LLM input contains a trusted prompt (instruction) and an untr