garak

garak (Generative AI Red-teaming and Assessment Kit) checks whether a large language model or dialogue system can be made to fail in undesirable ways. It probes for hallucination, data leakage, prompt injection, misinformation, toxic content generation, jailbreaks and many other weaknesses.

It works in a similar way to network scanners such as nmap or the Metasploit Framework, but for LLMs: probes send static, dynamic and adaptive attack prompts to the model under test, and detectors analyse the responses to identify failures. Results are summarised in a report, and garak supports many model providers and local models.

garak is developed by NVIDIA and is free and open source under the Apache 2.0 licence.