tests: add LLM based test generation tool #59626
Conversation
ti-chi-bot
bot
added
release-note-none
YangKeao
force-pushed
the
add-llm-test
branch
3 times, most recently
from
b167ea9 to
edd17d4
Compare
|
/retest |
YangKeao
force-pushed
the
add-llm-test
branch
2 times, most recently
from
cca5df7 to
f20e3ba
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #59626 +/- ##
================================================
+ Coverage 73.0024% 73.3852% +0.3827%
================================================
Files 1697 1697
Lines 468869 468909 +40
================================================
+ Hits 342286 344110 +1824
+ Misses 105529 103740 -1789
- Partials 21054 21059 +5
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
|
||
| func createGenerateCmd() *cobra.Command { | ||
| var ( | ||
| openaiToken string |
There was a problem hiding this comment.
Can I use the other LLM such as deepseek?
There was a problem hiding this comment.
Sure. The API of deepseek is compatible with OpenAI. Actually, I personally used deepseek to run this tool on my machine.
I've ran this tool with OpenAI gpt-4o, claude sonnet and deepseek-r1. Deepseek R1 gave the best result. Nearly every SQLs generated by it are valid, while the SQLs generated by other models will have some grammar errors 🤦 and cannot handle the escape character well. (But I haven't tested with gpt-o1 or newer models).
YangKeao
force-pushed
the
add-llm-test
branch
from
f20e3ba to
20a72c8
Compare
| @@ -0,0 +1,218 @@ | |||
| { | |||
| "delete": [ | |||
There was a problem hiding this comment.
Actually I'm not sure whether it's good to submit all generated test cases to the repo 🤔 .
|
|
||
| func createGenerateCmd() *cobra.Command { | ||
| var ( | ||
| openaiToken string |
There was a problem hiding this comment.
Sure. The API of deepseek is compatible with OpenAI. Actually, I personally used deepseek to run this tool on my machine.
I've ran this tool with OpenAI gpt-4o, claude sonnet and deepseek-r1. Deepseek R1 gave the best result. Nearly every SQLs generated by it are valid, while the SQLs generated by other models will have some grammar errors 🤦 and cannot handle the escape character well. (But I haven't tested with gpt-o1 or newer models).
YangKeao
force-pushed
the
add-llm-test
branch
3 times, most recently
from
a03170b to
0d0045e
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bb7133 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ti-chi-bot
bot
added
approved
needs-1-more-lgtm
[LGTM Timeline notifier]Timeline:
|
|
/retest |
1 similar comment
|
/retest |
YangKeao
force-pushed
the
add-llm-test
branch
2 times, most recently
from
26a1b2c to
9d6cce0
Compare
|
/retest |
Signed-off-by: Yang Keao <[email protected]>
YangKeao
force-pushed
the
add-llm-test
branch
from
9d6cce0 to
f7349e7
Compare
There was a problem hiding this comment.
can we generate this file on the fly, if so there's no need to upload to repo, quite large😓
There was a problem hiding this comment.
The output is not stable, so it'll give different queries each time.
If someone want to generate queries which haven't been tested (and have no duplicated issues) yet, these history will be helpful (we could tell LLM to avoid existing queries).
What problem does this PR solve?
Issue Number: close #59625
Problem Summary:
What changed and how does it work?
expression,dml, andctefeature. However, it only found bugs forexpressionfor now (it's kind of expected from my perspective. TiDB is good!).Reference to https://github.com/YangKeao/tidb/blob/add-llm-test/tests/llmtest/README.md if you want to know how to use it.
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.