Skip to content

Releases: Azure/PyRIT

v0.9.0

09 May 01:42
@nina-msft nina-msft
v0.9.0
f942562
Compare
Choose a tag to compare
v0.9.0 Latest
Latest

What's Changed

Targets

  • HTTPTarget Improvements that properly parse the HTTP version, automatically calculate the content-length, and make headers case insensitive.
  • FIX: Fixed IndexError with RealtimeTarget to handle responses properly

Datasets

  • Social Engineering (Persuasion and Deception) Scenarios: See datasets/orchestrators/red_teaming/persuasion_deception and datasets/orchestrators/role_play/persuasion_script.yaml
  • Multilingual Vulnerability dataset from "A Framework to Assess Multilingual Vulnerabilities of LLMs"

Converters

  • Enhancements to the AsciiSmugglerConverter by adding support for two methods for encoding hidden data (embedding directly in a Unicode character (default: 😊) and appending hidden data to visible text).
  • ZalgoConverter: Adds Unicode characters to text to make it appear "glitchy"
  • ToxicSentenceGeneratorConverter: Generate toxic sentence starters based on seed prompts
  • FIX: Remove JSON Instructions for TranslationConverter to address intermittent failures due to JSON parsing issues and non-consistent responses from endpoints.

Orchestrators

  • [BREAKING] Rename MultiTurnAttackResult to OrchestratorResult as part of a bigger refactor to tack objectives and results.
  • FIX: Keep Conversation ID in PromptSendingOrchestrator if it is provided
  • FIX: Remove Harm-Specific Prevention from CrescendoOrchestrator

Scorers

  • Generic Scorer with Flexible Inputs: SelfAskGeneralScorer in pyrit/score/general_scorer.py. It can be configured to use different scoring types (e.g. True/False, float) and can format the prompt using a system prompt and a format string.
  • Criteria-Based Scorer (used with SelfAskScaleScorer): Provides evaluation criteria that is specific to a given objective.
  • CompositeScorer: Combines multiple True/False Results into a single True/False Result

Dependencies

  • Moves jupyter and ipykernel from required into an optional [dev] dependency. If you need to use Jupyter notebooks with PyRIT, you'll need to install using methods outlined here.
  • Moves azure-cognitiveservices-speech from required into an optional [speech] dependency.

Other

  • Added custom file name support to allows for saving data (image, audio, video, etc.) to storage under a custom name.
  • Custom Retry Decorator: pyrit_custom_result_retry to retry a function if a certain condition is true. This augments existing retry decorators which retry functions based on exception criteria.
  • Optimizations and various bug fixes to .devcontainer

Full list of changes

New Contributors

Full Changelog: v0.8.1...v0.9.0

Contributors

afogel, bashirpartovi, and 15 other contributors
Assets 2
Loading

v0.8.1

04 Apr 13:17
@romanlutz romanlutz
v0.8.1
f7e30cd
This commit was signed with the committer’s verified signature.
romanlutz Roman Lutz
GPG key ID: 30FB8FD707A41611
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.8.1

What's Changed

  • We have a new cookbook on Precomputing turns for orchestrators
  • OpenAIChatTargets now have an argument is_json_supported to allow specifying if the response_format request header should be set. This is supported by OpenAI, but not by several other providers that otherwise follow the OpenAI API.
  • There is now a Docker image for PyRIT users! Check out the steps outlined in the docker/README to try it out and feel free to provide feedback in GitHub issues or on Discord.
  • The Tom-and-Jerry jailbreak template was added!
  • When using AAD/Entra auth with OpenAITarget, the target auto-refreshes the auth token periodically now. This addresses a bug where the token would get stale after a period of time.
  • We also addressed bugs that resulted in exceptions from triggered content filters and empty exception which should lead to a smoother experience.

Full list of changes

  • MAINT post-v0.8.0 release update by @romanlutz in #837
  • MAINT: Making JSON support configurable with OpenAIChatTargets by @rlundeen2 in #833
  • FEAT: Add Dockerized PyRIT with Jupyter Notebook Support by @ErdemOzgen in #784
  • FEAT: add Tom-and-Jerry jailbreak by @hagsmand in #838
  • DOC: Adding cookbook around prepending turns by @rlundeen2 in #840
  • FIX: Small fix in cookbook by @jsong468 in #849
  • FIX catch content_filter with 200s instead of 500s by @romanlutz in #850
  • FIX: Amended dockerfile and requirements.txt to unblock ADO pipelines by @jsong468 in #853
  • FIX add zero width and insert punctuation converters to init.py file by @AnnaRevutsky in #848
  • FIX: AAD Auth refresh bug with OpenAITargets by @rlundeen2 in #855
  • FIX handle empty exception message in validation by @romanlutz in #859

New Contributors

Full Changelog: v0.8.0...v0.8.1

Contributors

romanlutz, ErdemOzgen, and 4 other contributors
Loading

v0.8.0

27 Mar 07:12
@romanlutz romanlutz
v0.8.0
0ec40d4
This commit was signed with the committer’s verified signature.
romanlutz Roman Lutz
GPG key ID: 30FB8FD707A41611
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.8.0

What's Changed

Targets:

  • HTTPTarget now supports rate limiting
  • Some users encountered errors in Azure OpenAI when hitting content filter errors using error code 500. PyRIT now catches content filter responses with both error codes 400 (as before) and 500 (new) and returns a clean response record.

Datasets:

  • fetch_babelscape_alert_dataset had a bug causing it to be limited to a single category even when users specified both. This is now fixed!
  • added fetch_red_team_social_bias_dataset
  • added fetch_darkbench_dataset
  • added fetch_mlcommons_ailuminate_demo_dataset

Converters:

  • added UnicodeReplacementConverter
  • added sneaky_bits option to AsciiSmugglerConverter in the encoding_mode argument. The unicode_tags argument is now removed and replaced by more options in encoding_mode (i.e., unicode_tags, unicode_tags_control, and sneaky_bits).

Scanner: A basic version was introduced in v0.7.0 that supported only sending single-turn prompts. v0.8.0 expands on this with support for most multi-turn orchestrators (incl. adversarial chat targets and scorers) and memory. This feature is still considered experimental and may change considerably in the following versions.

Other:

  • support for Python 3.13 in addition to 3.10-3.12.
  • For single-piece responses, we now have a convenient get_value() method.
  • PyRIT used to print warnings that torch isn't installed (unless the corresponding extra was installed). This was caused by transformers and is now turned off as it doesn't serve any purpose.
  • In previous versions, PyRIT started supporting .env.local as an override to the .env file for endpoint secrets. However, when using this outside of the normal repository structure (e.g., when running PyRIT without cloning this repo) the code failed to discover .env.local in the current working directory. This is now fixed.

Full list of changes

  • [DevContainer] Provide a uniform development environment by @bashirpartovi in #787
  • FEAT: Add Rate Limit Support for HTTP Target by @nina-msft in #786
  • DOC Updating contribution docs by @bashirpartovi in #788
  • MAINT support python 3.13 by @AdrGav941 in #779
  • FIX: fixed dev container permission issue by @bashirpartovi in #789
  • FEAT: simplify extraction of converted values from responses by @paulinek13 in #783
  • MAINT: improve organization of dataset fetch functions (refactoring) by @paulinek13 in #785
  • FEAT: Added cross-platform compatibility and needed language support for toml and docker by @bashirpartovi in #797
  • MAINT: Update release version to 0.7.1.dev0 by @jsong468 in #800
  • FIX: prevent data overwrite in fetch_babelscape_alert_dataset by @paulinek13 in #799
  • DOC contributor guide flowchart, small text updates, and add Roakey to README by @romanlutz in #798
  • DOC: clarify OpenAITarget targets httpx_client_kwargs timeout settings by @clod81 in #801
  • FIX: Add exception on response parsing when call to Openrouter.ai by @hagsmand in #796
  • FIX make sure conversation IDs are not sent out as UUIDs to the database by @ayeganov in #723
  • FEAT support adversarial_chat and scoring in scanner to enable automated multi-turn-orchestrators by @romanlutz in #706
  • FIX move misplaced test file to tests/unit/converter by @romanlutz in #794
  • FEAT: Added Red Team Social Bias dataset by @MoolmanM in #714
  • DOC improve API reference for auth, cli, common, chat_message_normalizer by @romanlutz in #793
  • FEAT: UnicodeReplacementConverter by @nina-msft in #803
  • FIX: Updating pre-commit to fix build issues by @rlundeen2 in #810
  • MAINT: Making test_connect more resilient by @rlundeen2 in #806
  • [FIX] fix bad domain by @mgstate in #815
  • [FIX] Integration test fixes: add hugging face token in notebook and fix test_fetch_datasets by @jsong468 in #819
  • FEAT: Added memory config to scanner by @bashirpartovi in #808
  • FEAT: add DarkBench dataset by @paulinek13 in #821
  • MAINT: improving build/test time by @bashirpartovi in #820
  • FIX handle Azure OpenAI content_filter errors with HTTP status code 500 by @romanlutz in #825
  • FIX turn off transformers warning by @romanlutz in #829
  • TEST: Adding integration test for content filters by @rlundeen2 in #830
  • MAINT: Separating integration test local .env by @rlundeen2 in #817
  • FEAT: add MLCommons AILuminate v1.0 DEMO Prompt Set by @paulinek13 in #828
  • FIX find .env.local in current working directory by @romanlutz in #832
  • BREAKING FEAT: Sneaky Bits - Advanced Data Smuggling Techniques by @KutalVolkan in #827
  • FEAT add ps-fuzz prompts by @ryanjieh in #823

New Contributors

Full Changelog: v0.7.0...v0.8.0

Contributors

clod81, ayeganov, and 12 other contributors
Loading

v0.7.0

17 Mar 16:40
@jsong468 jsong468
v0.7.0
c2af232
Compare
Choose a tag to compare
v0.7.0

What's Changed

Targets:

  • [BREAKING] OpenAIChatTarget has become more generalized to more broadly support OpenAI-compatible models. See the blog describing the changes here!
  • If api_version is set to None when instantiating OpenAITarget objects, it will not be added as a query parameter to requests.
  • Added Google Gemini example environment variables to .env_example and added integration tests for Gemini/OpenAIChatTargets

Converters:

  • [New] AddImageVideoConverter: PyRIT's first video converter! it allows users to add an image to a video in at a specified position. More video converters to come!
  • [New] InsertPunctuationConverter: Inserts various punctuation into a prompt to test model robustness to perturbations.

Orchestrators:

  • [New] ManyShotJailbreakOrchestrator: Prepend a faux dialogue between a human and an AI assistant within a single prompt for the target.
  • [New] [BREAKING] ContextComplianceOrchestrator: Update the context to prime an objective_chat_target to answer. The context is set using instructions defined in context_description_instructions_path, along with an adversarial_chat to generate the first turns to send.
  • [BREAKING] RolePlayOrchestrator improvements: Refactored for greater code re-use
  • FlipAttackOrchestrator improvement: Allow for additional converters applied after the flip attack

Memory:

  • Multimodal Seed Prompts Encoding Metadata: Adding non-text seed prompts to the database will automatically have metadata populated, including format (png, wav, etc.) and things like bitrate and duration for audio and video seed prompts.
  • SeedPrompt Duplicates: Duplicate seed prompts within the same dataset (identical dataset_name) will no longer be uploaded to memory.
  • Using Configured Paths for Multimodal Seed Prompts: Multimodal SeedPrompt file paths within .yaml files no longer use relative paths that break based on where the .yaml files are accessed. Instead, configured paths (located in paths.py) are used.
  • [BREAKING] Removed calls to disposing memory engines in Orchestrator and Prompt Target objects and replaces it with the atexit and weakref methods of cleanup in the Memory interface to ensure cleanup on process exit. Orchestrators and targets no longer support the context manager protocol.
  • Added get_values() method to the SeedPromptDataset class to simplify prompt values extraction from datasets. Optional filtering to retrieve the first and/or last N values has also been implemented.

Scorers:

  • [New] HumanInTheLoopScorerGradio: Create scores from manual human input by running the Gradio interface in a separate process and adds the scores to the database. For now, the possible scores that users can give are "safe" and "unsafe."

Datasets:

  • [New] Added new fetch function for Aya Red-Teaming Dataset
  • [New] Added Pliny's prompts from the l1b3rt4s repo as templates
  • [New] Added the Babelscape ALERT dataset
  • Added support for filtering based on harm categories for PKU-SafeRLHF and AdvBench datasets

Misc:

  • Other changes include various maintenance improvements and bug fixes, addition of integration tests, website enhancements, dependency updates, and doc improvements.

Full list of changes

  • FIX unblock test pipelines by skipping certain tests on Ubuntu and adding Windows additionally by @romanlutz in #727
  • MAINT: Update release version to 0.6.1.dev0 by @nina-msft in #731
  • MAINT: Upgrading DuckDB by @jbolor21 in #712
  • [FEAT][MAINT][4019] Make multi-modal easier to configure in seedprompt files by @shivenchawla in #696
  • FEAT: set favicon for the website by @paulinek13 in #717
  • FEAT: simplify extracting prompt values by @paulinek13 in #718
  • FEAT: add a fetch function for Aya Red-teaming Dataset by @paulinek13 in #713
  • MAINT update Roakey image to have transparent background by @romanlutz in #735
  • FEAT Moonshot Attack Module: Insert Punctuation Attack by @u7780339 in #475
  • FEAT: include scored_prompt_id in orchestrator_identifier of the system prompt by @NicolePell in #725
  • FEAT: Create many shot jailbreak orchestrator by @AdrGav941 in #709
  • MAINT pre-commit hook to remove notebook header from notebooks by @jbolor21 in #737
  • FEAT Add Encoding Data to Multimodal Seed Prompts by @jsong468 in #740
  • FEAT added Pliny's prompts from the l1b3rt4s repo as templates by @joaodunas in #710
  • FEAT Adding babelscape dataset by @Jarro01X in #738
  • FIX: Upgrading Packages by @rlundeen2 in #741
  • FIX: Increasing pipeline timout by @rlundeen2 in #743
  • FEAT PyRIT to not upload duplicate seed-prompts by @shivenchawla in #742
  • MAINT: Azure SQL Integration Test Misc. Updates by @nina-msft in #745
  • FIX Small bug fixes (renaming file, editing MANIFEST) by @jsong468 in #746
  • [BREAKING] FEAT: OpenAI Generalization Improvements by @rlundeen2 in #747
  • FEAT: Add example_count field to ManyShotJailbreakOrchestrator by @nina-msft in #748
  • DOC: Blog: A More Generalized OpenAIChatTarget by @rlundeen2 in #751
  • DOC: Updating git docs by @rlundeen2 in #753
  • FIX: Fixing integration tests broken with OpenAIChatTarget Update by @rlundeen2 in #755
  • FEAT Video Converter: Adding Images to Videos by @jbolor21 in #702
  • FIX: Adding back static js by @rlundeen2 in #761
  • [BREAKING] FEAT: RolePlayOrchestrator Improvements by @rlundeen2 in #758
  • [BREAKING] FIX: Dispose Memory in Memory vs Class Objects by @nina-msft in #752
  • MAINT clean up dependencies by @romanlutz in #757
  • FEAT Adding converter support to many shot jailbreak orchestrator by @AdrGav941 in #760
  • FIX: Default API Version for TTS Target by @jbolor21 in #749
  • [BREAKING] FEAT: Adding Context Compliance Orchestrator by @rlundeen2 in #763
  • DOC: Add Instructions for Tagging Breaking Changes in PR Template by @nina-msft in #765
  • FEAT: support filtering based on harm categories for PKU-SafeRLHF dataset by @paulinek13 in #756
  • DOC Update CCA Documentation for Clarity by @eugeniavkim in #773
  • DOC: Update OpenAI Environment Variable Names in Documentation by @nina-msft in #776
  • FEAT: add harm categories to AdvBench Dataset by @paulinek13 in #732
  • FIX: Allow api_version to be set to None when instantiating OpenAITarget objects by @LeoVrana in #764
  • MAINT standardize Hugging Face token environment variable, add integration tests for Google Gemini and Open AI by @romanlutz in #778
  • FEAT: Gradio HiTL Scorer by @mart123p in #722
  • DOC: clarify OpenAIChatTarget usage with Ollama by @jsdlm in #777
  • FIX: small edits to make integration tests pass by @jsong468 in #780
  • MAINT add notice generation to component governance by @romanlutz in #781
  • MAINT update NOTICE file by @romanlutz in #782

New Contributors

Full Changelog: releases/v0.6.0...releases/v0.7.0

Contributors

NicolePell, mart123p, and 14 other contributors
Loading

v0.6.0

22 Feb 01:37
@nina-msft nina-msft
v0.6.0
ca068ba
Compare
Choose a tag to compare
v0.6.0

What's Changed

  • Cookbooks are live, and replace our How To Guide! Cookbooks try to tackle a problem and use the components that work best, instead of our typical documentation which illustrates that many pieces of PyRITs are swappable.

Cookbooks:

Targets:

  • OllamaChatTarget: Implement ability to forward custom parameters directly to the HTTP client
  • HuggingFaceChatTarget: Adds optional keywords device_map, torch_dtype and attn_implementation
  • [New] PlaywrightTarget: Interact with web applications using Playwright. This is particularly useful for testing interactions with web interfaces like chatbots.
  • [New] RealtimeTarget: Send and receive audio with the Realtime API.
  • [New] GroqChatTarget: Interact with Groq's OpenAI-compatible API.

Converters:

  • [New] ANSI Escape Code Converter: AnsiAttackConverter
  • [New] BinaryConverter: Convert input text into binary with configurable bits per character
  • PDFConverter: Updates to support templated and non-templated PDF generation & enabling text injection into existing PDFs
  • [New] TextToHexConverter: Convert text to hexadecimal encoded utf-8 string
  • Add easier querying for converter-supported input/output types

Orchestrators:

  • RedTeamingOrchestrator & CrescendoOrchestrator now support prepended conversations. You can set a system prompt on the objective target using this feature, or provide conversation history as context to continue execution from a specific point.
  • ScoringOrchestrator: Add ability to score responses using filters.
  • PromptSendingOrchestrator: Set Skip Criteria to specify which prompts to skip being sent to the target with this orchestrator.
  • [New] RolePlayingOrchestrator: Single-turn orchestrator which prepends some prompts which describe fictional scenarios to attempt and elicit harmful responses
  • XPIAOrchestrator: Fix to BlobNotFound exception
     
    Memory:
  • [BREAKING] All notebooks must explicitly initialize Central Memory through a new initialize_pyrit() function: #616. This puts ownership into the hands of the user to set where your prompts will be stored. Read more here: Memory
  • Ability to add memory labels on a per-prompt level, specifically useful in Multimodal scenarios
  • Conversation Scores now available when exporting Prompt Data
  • Filter Data by various queries (e.g. prompt ID, orchestrator ID, labels, etc) using get_prompt_request_pieces()
  • Consolidated method to Export Conversations using Filters: export_conversations()
  • SeedPrompts: Support for Multimodal Seed Prompts
  • [BREAKING] NormalizerRequestPieces replaced with SeedPrompts: #648

Scorers:

  • Add tasks by default to scorers to improve scorer accuracy

Misc:

  • Other changes include various maintenance improvements and bug fixes, addition of integration tests, new blog posts, and doc improvements.

Full list of changes

Read more

Contributors

ayeganov, AlexRRR, and 14 other contributors
Loading

v0.5.2

03 Dec 23:19
@rdheekonda rdheekonda
v0.5.2
1a2838a
Compare
Choose a tag to compare
v0.5.2

What's Changed

  • Pinned the httpx version to 0.27.2 and refactored the codebase to ensure compatibility.
  • Fixed AzureSQLMemory authentication issues by adding token refresh, pool recycling, and pre-ping mechanisms.
  • Redesigned PAIR attack technique to function as a specialized instance of TAP orchestrator, streamlining architecture.
  • Added support for local Hugging Face model checkpoints.

Full list of changes

  • [DOC] Updating README by @rlundeen2 in #579
  • Fix Azure SQL Authentication Errors: Add Token Refresh, Pool Recycling, and Pre-Ping by @rdheekonda in #576
  • FEAT: add support for local model checkpoints and trust_remote_code in HuggingFaceChatTarget by @KutalVolkan in #574
  • FEAT: Refactor PAIR to be a special instance of TAP by @rlundeen2 in #580
  • FIX: httpx proxy arg fix, pinned httpx version by @jsong468 in #589
  • FIX: Not raising exceptions on None responses by @rlundeen2 in #590
  • Fix Test Prompt Response Error Values by @rdheekonda in #591

Full Changelog: v0.5.0...v0.5.2

Contributors

rdheekonda, KutalVolkan, and 2 other contributors
Loading

v0.5.0

27 Nov 00:28
@romanlutz romanlutz
v0.5.0
c3a1a48
This commit was signed with the committer’s verified signature.
romanlutz Roman Lutz
GPG key ID: 30FB8FD707A41611
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.5.0

What's Changed

  • PyRIT now has a website

  • We've been working on standardizing orchestrators in terms of naming and functionality:

    • The endpoint (of type PromptTarget) that PyRIT attacks will be referred to as objective_target.
    • The endpoint (of type PromptChatTarget) that helps us craft attacks will be referred to as adversarial_chat.
    • Beyond that, we've settled on a common interface for multi-turn orchestrators with a shared result object.
    • Instead of an attack_strategy arg we require a file path called adversarial_chat_system_prompt_path to make the connection to the adversarial_chat target clearer. Some orchestrators have a default for this, of course.
    • The initial prompt to the adversarial_chat is now called adversarial_chat_seed_prompt to also help with clarity and connection to adversarial_chat
    • Sometimes we use multiple scorers. For that reason, objective_scorer will be the scorer that decides if the objective has been achieved. Other scorers have similarly specific names, e.g., on_topic_scorer in the CrescendoOrchestrator
    • The new standard name for all orchestrators to execute an attack is run_attack_async.

    The standardization is not fully completed yet but will continue in future releases. So far, CrescendoOrchestrator, TreeOfAttacksWithPruningOrchestrator, and RedTeamingOrchestrator have been adjusted.

  • Support for a centralized database using Azure SQL as an optional alternative to a local DuckDB database.

  • Introduced (multi-modal) SeedPrompts and SeedPromptDatasets as a starting point for red teaming ops with integration to our databases.

  • New orchestrators and auxiliary attacks:

    • FuzzerOrchestrator with 5 template converters
    • GCG support via Azure ML pipelines to optimize adversarial suffixes
    • FlipAttackOrchestrator

  • New targets:

    • HuggingFaceChatTarget
    • HTTPTarget
    • Open AI and Azure Open AI targets were refactored to simplify the logic. They now share a common interface OpenAITarget and you can decide between Azure vs. Open AI using is_azure_target=True or False.

  • New datasets:

    • HarmBench
    • PKU-SafeRLHF
    • wmdp-bio, wmdp-chem, and wmdp-cyber (now fetchable from the original data source)
    • AdvBench
    • Decoding Trust Stereotypes
    • LLM-LAT/harmful-dataset
    • tdc23 red teaming dataset
    • TrustAIRLab/forbidden_question_set
    • LibrAI 'Do Not Answer' Dataset

  • New converters:

    • QRCodeConverter
    • AzureSpeechAudioToTextConverter
    • URLConverter
    • HumanInTheLoopConverter
    • ColloquialWordswapConverter
    • UnicodeConfusableConverter (updated with new functionality)
    • CharSwapGenerator
    • MaliciousQuestionGeneratorConverter
    • AsciiSmugglerConverter
    • MathPromptConverter
    • AudioFrequencyConverter
    • ZeroWidthConverter
    • DiacriticConverter

  • New scorers:

    • SelfAskRefusalScorer
    • HumanInTheLoopScorer
    • InsecureCodeScorer

  • We generally use a .env file to configure details of endpoints that PyRIT needs to execute. A new .env.local override file allow for further customization.

  • Finally, PyRIT now comes with several extras that you can install using pip install pyrit[<extra>]

    • dev includes developer dependencies that you shouldn't need unless you plan on contributing to the project.
    • torch includes just pytorch which is needed for some targets (e.g. Hugging Face) or auxiliary attacks (e.g., GCG) but not core functionality. This allows you to choose whether you want to install it.
    • gcg includes extra dependencies that are only needed for running GCG. Since this requires dedicated compute (ideally with GPU) you can choose whether it is required for you.
    • all includes all of the above.

Full list of changes

Read more

Contributors

elgertam, AlexRRR, and 27 other contributors
Loading

v0.4.0

23 Aug 01:36
@rdheekonda rdheekonda
v0.4.0
b25caa8
Compare
Choose a tag to compare
v0.4.0

What's Changed

  1. New Advanced Attack Techniques: Expanded orchestrators with advanced attack techniques, including PAIR, tree of attacks, and crescendo strategies.
  2. New Targets: Crucible target, Prompt Shield Target, Azure OpenAI GPT-4o target
  3. New Converters: Added Tense, Emoji, image to text, and Character Space converters.
  4. New Scorers: Scale Scorer, Prompt Shield, and True/False Inverter Scorer
  5. Automatic Scoring & Memory Labels: Introduced automatic scoring in the PromptSendingOrchestrator. Added support for scoring with user-provided memory labels.
  6. Delegation SAS Authentication: Supported delegation SAS authentication for secure interactions with Azure Blob Storage targets.
  7. Improved Resiliency: Enhanced the resiliency of targets, converters, and orchestrators with robust error handling mechanisms.
  8. Bug Fixes & Performance: Various bug fixes, added support for Python 3.12, speedup unit tests
  9. Fetch functionality: Introduced functionality to fetch adversarial datasets, such as SecLists, XStest etc.,
  10. Updated Demo Codes: Replaced demo code examples with the GPT-4o target.

Full List of Changes

New Contributors

Full Changelog: v0.3.0...v0.4.0

Contributors

elgertam, dlmgary, and 11 other contributors
Loading

v0.3.0

28 Jun 21:19
@jbolor21 jbolor21
v0.3.0
d68caed
Compare
Choose a tag to compare
v0.3.0

What's Changed

  • New and improved scorers! Many new scorers have been added, and scorers can now be swapped out and made generic.
  • Many new attack techniques and variations have been introduced. These include skeleton key, most of GPTFuzz, adding text to images, repeated token attack, cipherchat, shorten/expand, tone, CodeChameleon, and more. A total of 13 new converters have been added!
  • Framework improvements:
    • Ability to duplicate conversations for reuse (this makes implementation easier for attacks like PAIR/TAP/crescendo).
    • Converters can be added to LLM responses.
    • All framework calls are now async and parallelizable.
    • Error handling and intelligent automatic retries in targets (e.g., for network errors) and converters/scorers (e.g., for JSON deserialization).

Full list of Changes

New Contributors

Full Changelog: v0.2.1...v0.3.0

Contributors

elgertam, cseifert1, and 11 other contributors
Loading

v0.2.1

01 May 22:53
@romanlutz romanlutz
v0.2.1
9e852f1
This commit was signed with the committer’s verified signature.
romanlutz Roman Lutz
GPG key ID: 30FB8FD707A41611
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.1

What's Changed

  • added user authentication support for AOAI Chat Targets
  • request validation in targets
  • support for exporting conversations from the memory

Full list of changes

Full Changelog: v0.2.0...v0.2.1

Contributors

rdheekonda, rlundeen2, and nina-msft
Loading