Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create an example using Granite Guardian + the reference stack #33

Open
deanwampler opened this issue Jan 15, 2025 · 4 comments
Open

Create an example using Granite Guardian + the reference stack #33

deanwampler opened this issue Jan 15, 2025 · 4 comments
Labels
evaluators Implementations of evaluations, including benchmarks and datasets Examples Tickets for building user-facing examples. reference stack All tools for the reference stack.
Milestone
2025-01-31

Comments

@deanwampler
Copy link
Contributor

No description provided.

@deanwampler deanwampler added evaluators Implementations of evaluations, including benchmarks and datasets Examples Tickets for building user-facing examples. reference stack All tools for the reference stack. labels Jan 15, 2025
@deanwampler deanwampler moved this to Planning in FA2: TSEI Tasks Jan 15, 2025
@deanwampler deanwampler added this to the 2025-01-31 milestone Jan 15, 2025
@deanwampler
Copy link
Contributor Author

Look at the Granite Cookbook example...

@deanwampler
Copy link
Contributor Author

@bnayahu there are several example notebooks for Granite Guardian in the IBM Granite Community "snack" repo: https://github.com/ibm-granite-community/granite-snack-cookbook/tree/main/recipes/Granite_Guardian. Do you know if there is an integration with unitxt and/or lm-evaluation-harness for Granite Guardian?

@bnayahu
Copy link
Contributor

bnayahu commented Jan 29, 2025

There is, but it's still evolving. The current implementation in Unitxt is centered on RAG metrics and harmfulness assessment is not there yet.

@deanwampler
Copy link
Contributor Author

Looks like it starts here: https://github.com/IBM/unitxt/blob/main/prepare/metrics/rag_granite_guardian.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
evaluators Implementations of evaluations, including benchmarks and datasets Examples Tickets for building user-facing examples. reference stack All tools for the reference stack.
Projects
FA2: TSEI Tasks
Status: Planning
Milestone
2025-01-31
Development

No branches or pull requests
2 participants
@deanwampler @bnayahu