Skip to content

docs: add LLM-as-Judge examples and repo guidance - #2

Merged
dpaluy merged 3 commits into
masterfrom
chore/examples-pr
Apr 30, 2026
Merged

docs: add LLM-as-Judge examples and repo guidance#2
dpaluy merged 3 commits into
masterfrom
chore/examples-pr

Conversation

@dpaluy

@dpaluy dpaluy commented Apr 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • Update the README quick-start judge model to gpt-5.5 and add a pointer to the standalone examples guide.
  • Add repository-local AGENTS.md guidance for future agent work in this gem.
  • Add standalone LLM-as-Judge examples for RAG scoring, batch pass/fail output, model comparison, custom metrics, and Minitest assertions.
  • Bump the gem version to 0.1.2, add changelog notes, and exclude examples from packaged gem files.

Why

Users asked for practical LLM-as-Judge examples that are easy to demo and adapt. This PR keeps those examples separate from the main README while still linking to them, and it includes the repo-local guidance and ignore rule needed for future agent work.

Changes

docs(examples): make LLM-as-judge demos self-contained

  • What: Added live Ruby examples under examples/ plus a detailed examples README covering setup, run commands, expected output, and interpretation.
  • What: Added batch output that prints per-sample PASS/FAIL decisions and metric-score breakdowns.
  • What: Bumped RubricLLM::VERSION to 0.1.2 and added CHANGELOG.md release notes.
  • What: Updated rubric_llm.gemspec to exclude examples/ from packaged gem files.
  • Why: Keep adoption examples separate from the root README while preserving a lean runtime gem package.

docs: add repository guidance and examples pointer

  • What: Added AGENTS.md with repo structure, test, style, and security guidance.
  • What: Updated the root README quick-start model to gpt-5.5 and linked to examples/README.md.
  • Why: Keep local workflow state out of git and make the examples discoverable from the main README.

Testing

  • bundle exec rake
  • package check confirmed version=0.1.2, examples_in_package=[], and agents_in_package=[]
  • gem build rubric_llm.gemspec --output /tmp/rubric_llm-0.1.2.gem
  • git diff --check
  • Live provider example scripts were not run because they require OpenAI API calls.

Risk & Rollout

  • Risk: Low. Most changes are documentation/examples, and examples plus AGENTS.md are excluded from packaged gem files.
  • Rollback: Revert the PR commits to remove examples, repo guidance, README edits, and release/package metadata.
  • Monitoring: Confirm release packaging still excludes examples before publishing 0.1.2.

Add standalone live examples that show RAG scoring, batch decisions, model comparison, custom metrics, and Minitest assertions without expanding the root README. Bump the gem version for the example release while keeping the examples excluded from packaged gem files.

Constraint: Keep root README unchanged; examples live separately
Constraint: Examples should remain outside packaged gem files
Rejected: Package examples in the gem | demo scripts increase installed payload without runtime value
Confidence: high
Scope-risk: narrow
Directive: Keep live provider-call examples outside the root README and outside packaged gem files unless release policy changes
Tested: bundle exec rake
Tested: ruby -e spec package check reported version=0.1.2 and examples_in_package=[]
Tested: gem build rubric_llm.gemspec --output /tmp/rubric_llm-0.1.2.gem
Tested: git diff --cached --check
Not-tested: Live provider example scripts; they require OpenAI API calls
@dpaluy dpaluy closed this Apr 30, 2026
@dpaluy
dpaluy deleted the chore/examples-pr branch April 30, 2026 03:52
@dpaluy
dpaluy restored the chore/examples-pr branch April 30, 2026 03:52
@dpaluy dpaluy reopened this Apr 30, 2026
@dpaluy dpaluy changed the title docs(examples): make LLM-as-judge demos self-contained docs: add LLM-as-judge examples and repo guidance Apr 30, 2026
@dpaluy dpaluy changed the title docs: add LLM-as-judge examples and repo guidance docs: add LLM-as-Judge examples and repo guidance Apr 30, 2026
@dpaluy
dpaluy force-pushed the chore/examples-pr branch from 437bf39 to d6217ed Compare April 30, 2026 04:05
Bring the examples PR back to the intended full scope: include repository agent guidance, ignore local OMX workflow state, update the README quick-start judge model to gpt-5.5, and point readers to the standalone examples.

Constraint: Keep a single PR containing README, .gitignore, AGENTS.md, and examples changes
Constraint: Do not commit .omx workflow state
Confidence: high
Scope-risk: narrow
Directive: Keep .omx ignored as local workflow/session state
Tested: bundle exec rake
Tested: ruby -e package check reported version=0.1.2, examples_in_package=[], agents_in_package=[]
Tested: git diff --check
Not-tested: Live provider example scripts; they require OpenAI API calls
@dpaluy
dpaluy force-pushed the chore/examples-pr branch from d6217ed to ecc73f0 Compare April 30, 2026 04:08
Add a compact outline to the standalone examples guide so readers can jump directly to setup, each runnable example, and the batch-output explanation.

Constraint: Keep the examples guide detailed but easier to scan
Confidence: high
Scope-risk: narrow
Tested: git diff --check
Not-tested: Runtime examples; docs-only change
@dpaluy
dpaluy merged commit 8f5b5f1 into master Apr 30, 2026
2 checks passed
@dpaluy
dpaluy deleted the chore/examples-pr branch April 30, 2026 04:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant