Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(ai): ai-proxy and ai-proxy-multi #12030

Merged
merged 4 commits into from
Mar 11, 2025
Merged

refactor(ai): ai-proxy and ai-proxy-multi #12030

merged 4 commits into from
Mar 11, 2025

Conversation

shreemaan-abhishek
Copy link
Contributor

@shreemaan-abhishek shreemaan-abhishek commented Mar 10, 2025 •
edited
Loading

Description

This PR accomplishes the following:

  1. Adjust priorities of ai plugins in order to incorporate upcoming ai-rate-limiting plugin.
  2. Refactors ai-proxy and ai-proxy-multi, integrating their proxy logic with APISIX's phase concept to work in conjunction with the ai-rate-limiting plugin under development
  3. Handles AI requests and responses accordingly, also obtaining usage data for AI provider

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

@shreemaan-abhishek shreemaan-abhishek marked this pull request as ready for review March 11, 2025 07:00
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. plugin tech debt labels Mar 11, 2025
membphis
membphis reviewed Mar 11, 2025
View reviewed changes
apisix/plugins/ai-proxy/base.lua
request_table.model = get_model_name_func(conf)
function _M.before_proxy(conf, ctx)
local ai_instance = ctx.picked_ai_instance
local ai_driver = require("apisix.plugins.ai-drivers." .. ai_instance.provider)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may failed, we need to check the return value or use pcall

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this line guarantees safety: https://github.com/apache/apisix/pull/12030/files#diff-6df73662ffc3d9815275bcf80c1ed0cebc86ce2dea275087cd299c6dea3b465aR73

@shreemaan-abhishek shreemaan-abhishek merged commit 53e5b02 into apache:master Mar 11, 2025
36 of 37 checks passed
@shreemaan-abhishek shreemaan-abhishek deleted the refactor-ai-proxy branch March 11, 2025 10:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@membphis membphis membphis approved these changes

@nic-6443 nic-6443 nic-6443 approved these changes

@AlinsRan AlinsRan AlinsRan approved these changes

Assignees
No one assigned
Labels
plugin size:XL This PR changes 500-999 lines, ignoring generated files. tech debt
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@shreemaan-abhishek @membphis @nic-6443 @AlinsRan