-
Notifications
You must be signed in to change notification settings - Fork 565
Add initial AI api-review configuration #2489
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Hello @JoelSpeed! Some important instructions when contributing to openshift/api: |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just some general questions for learning purposes.
Nothing worth blocking this on IMO, especially if you are finding it useful.
**Explanation:** [Why this change is needed] | ||
|
||
|
||
I'll run a comprehensive API review for the OpenShift API changes in the specified GitHub PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like something an LLM would spit out when you ask for a review - is it necessary to include this in the instructions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, this whole section looks like it is something an LLM spit out as an execution plan. Should something like this be hand-rolled with explicit instructions on how to conduct the review and important considerations?
I guess my curiosity here is if we made an explicit guidelines type document that humans could follow, an LLM should be able to follow along relatively easily and we can potentially enforce more nuanced guardrails.
Not worth blocking this on - more so asking questions for myself as I've not worked with LLMs in this capacity before.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an artefact of how these documents are built. Using Claude locally, I've given it text based prompts, and it has converted that into instructions that it can read. So 95% of this document is it translating my instructions and feedback into rules that it can later apply.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm. That is interesting. So this file is automatically updated with more detailed instructions by Claude as the AGENTS.md file is updated?
If you update this file by hand to "improve it", what happens?
|
||
|
||
I'll run a comprehensive API review for the OpenShift API changes in the specified GitHub PR. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to create a separate H1 heading to explain to the LLM that this is the steps for how to actually conduct the review?
### Testing | ||
```bash | ||
make test-unit # Run unit tests | ||
make integration # Run integration tests (in tests/ directory) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we teach it how to run the integration tests with more focused arguments?
That way running the integration tests don't take longer than necessary when reviewing a change that only impacts a subset of the APIs/tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that's a good idea, I'll have a go at that
@JoelSpeed: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This adds an initial AGENTS.md configuration for how to API review via an AI agent such as claude.
It also implements a
/api-review
command that can be used locally to review PRs for anyone who has claude installed.I hope we can get folks using this to self help, but my long term goal is to integrate this into coderabbit or some other review tool that can post the comments directly on the PR.
As an example of the output, see #2488 (comment)
Currently highlights of its instructions: