Skip to content

Add “Selecting a Model Based on Stripe Conversion” notebook #1873

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jun 1, 2025

Conversation

joshbickett
Copy link
Contributor

Summary

Adds a new notebook “Selecting a Model Based on Stripe Conversion: A Practical Eval for Startups.”
The post walks through HyperWrite’s real-world A/B-test workflow for choosing an LLM using Stripe payment conversion as the success metric.

Motivation

Benchmark scores may fail to predict whether users will pay.
This guide shows startups how to evaluate and pick a cheaper or higher-performing model based on revenue-linked conversion data, not offline metrics. It fills a gap in the cookbook by providing:

  • A step-by-step, production-tested A/B methodology
  • Ready-to-run stats code (one-tailed Z-test)
  • Concrete cost-savings case study (GPT-4.1 vs. Claude 3.5 Sonnet)

The pattern is directly applicable to any OpenAI-powered app that charges through Stripe.

For new content

When contributing new content, read through our contribution guidelines, and mark the following action items as completed:

  • I have added a new entry in registry.yaml (and, optionally, in authors.yaml) so that my content renders on the cookbook website.
  • I have conducted a self-review of my content based on the contribution guidelines:
    • Relevance: This content is related to building with OpenAI technologies and is useful to others.
    • Uniqueness: I have searched for related examples in the OpenAI Cookbook, and verified that my content offers new insights or unique information compared to existing documentation.
    • Spelling and Grammar: I have checked for spelling or grammatical mistakes.
    • Clarity: I have done a final read-through and verified that my submission is well-organized and easy to understand.
    • Correctness: The information I include is correct and all of my code executes successfully.
    • Completeness: I have explained everything fully, including all necessary references and citations.

@joshbickett
Copy link
Contributor Author

@shyamal-anadkat @Lupie here is the PR for your review!

@shyamal-anadkat shyamal-anadkat requested a review from Copilot May 31, 2025 01:47
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds a new notebook demonstrating how to choose an LLM based on real Stripe payment conversion, plus configuration entries to surface it in the cookbook site.

  • Registers the new notebook in registry.yaml
  • Introduces the notebook file under examples/stripe_model_eval/
  • Adds authors metadata in authors.yaml

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
registry.yaml New entry for “Selecting a Model Based on Stripe Conversion”
examples/stripe_model_eval/selecting_a_model_based_on_stripe_conversion.ipynb Full notebook content for the new guide
authors.yaml Added YAML entries for two authors (third still missing)
Comments suppressed due to low confidence (3)

registry.yaml:2111

  • The author shyamal-anadkat is listed here but not defined in authors.yaml. Add a matching entry in authors.yaml to avoid broken author links.
-    - shyamal-anadkat

examples/stripe_model_eval/selecting_a_model_based_on_stripe_conversion.ipynb:152

  • The notebook references diagram.svg but that file isn’t included in this PR. Please add the SVG or update the path so the diagram displays correctly.
+        "<img src=\"diagram.svg\" width=\"600\">"

registry.yaml:2105

  • The indentation for the new list entry doesn’t match the surrounding items (which use leading spaces before the dash). Align this entry to the same indent level so it’s recognized correctly.
+- title: Selecting a Model Based on Stripe Conversion – A Practical Eval for Startups

Copy link
Collaborator

@shyamal-anadkat shyamal-anadkat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! thanks

@shyamal-anadkat shyamal-anadkat merged commit 2fc6dbb into openai:main Jun 1, 2025
@joshbickett
Copy link
Contributor Author

Great, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants