feat(improve-api-endpoints): Added Datasets and Annotation APIs #12237

jasonfish568 · 2024-12-30T17:13:23Z

Added AnnotationReplyActionApi, AnnotationListApi, AnnotationUpdateDeleteApi
Added GET and PATCH endpoints to DatasetApi
Applied current_user in validate_app_token just like what it is done in validate_dataset_token
Added available model API

Summary

Basically copied the functions from console to the service api. Also added clear_partial_member_list() to Dataset Delete endpoint as what has been added in the console api.

I will keep working on this feature until it fulfills my needs. More endpoints will be added.

Checklist

Important

Please review the checklist below before submitting your pull request.

This change requires a documentation update, included: Dify Document
I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
I've updated the documentation accordingly.
I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

- Added AnnotationReplyActionApi, AnnotationListApi, AnnotationUpdateDeleteApi - Added GET and PATCH endpoints to DatasetApi - Applied current_user in validate_app_token just like what it is done in validate_dataset_token Related to: langgenius#11727

This API endpoint shares the same API token with Dataset API. It is essential for patching the knowledge base in terms of embedding model. Related to: langgenius#11727

Related to: PR#12237(langgenius/dify#12237)

Because app token is assigned to a specific app, the validate_app_token decorator has already validated the app_id. The returned app_model is the app targeted. Therefore there is no need to pass app_id separately. Simply use app_model.id instead. So app_id is removed from the api routes. Related to: langgenius#11727

crazywoola · 2025-01-02T01:59:05Z

Please fix the errors in CI.

jasonfish568 · 2025-01-02T04:30:58Z

Please fix the errors in CI.

Errors fixed.

crazywoola · 2025-01-03T01:57:52Z

Please fix the errors in CI.

Errors fixed.

Please run dev/reformat to resolve the errors in the CI.

jasonfish568 · 2025-01-03T03:07:18Z

Please fix the errors in CI.

Errors fixed.

Please run dev/reformat to resolve the errors in the CI.

Sorry for the trouble. Code has now been reformated.

crazywoola · 2025-01-03T08:41:48Z

The code part LGTM, however you should add according docs to the API Access as well in CN, EN & JA.

jasonfish568 · 2025-01-03T09:58:04Z

The code part LGTM, however you should add according docs to the API Access as well in CN, EN & JA.

langgenius/dify-docs#424

The relevant docs have been added and amended outdated part.

I have included both Chinese and English versions. However as I do not speak Japanese, I wouldn't be able to verify the translated version therefore I would wait for a Japanese contributor get it translated in a later time.

jasonfish568 · 2025-01-09T06:17:03Z

The code part LGTM, however you should add according docs to the API Access as well in CN, EN & JA.

Could you let me know when this PR will be merged? How does dify team manage PRs? When do PRs ususally get merged?

crazywoola · 2025-01-13T09:54:30Z

The code part LGTM, however you should add according docs to the API Access as well in CN, EN & JA.

Could you let me know when this PR will be merged? How does dify team manage PRs? When do PRs ususally get merged?

Hello, for newly added apis, you need to update the docs in mdx files as well.

There is a conflict file.
Here are the docs I mentioned.

It is added to Chat App due to Annotation is under Apps, and there is no individual Annotation API doc page. It seems correct to have Annotation APIs put here.

jasonfish568 · 2025-01-16T04:47:00Z

The code part LGTM, however you should add according docs to the API Access as well in CN, EN & JA.

Could you let me know when this PR will be merged? How does dify team manage PRs? When do PRs ususally get merged?

Hello, for newly added apis, you need to update the docs in mdx files as well.

There is a conflict file.

Here are the docs I mentioned.

Outcome

Conflict solved
Docs added to the mdx files

Suggestion:

I have always been wondering why there are docs.dify.ai and these API docs in the portal. It does not only confuse developers but also becomes a nightmare for contributors like me. I personally don't know mdx and my VSCode doesn't provide preview for mdx.

It would be a lot better if we can consolidate all docs to the same place without having to prepare duplicate docs all over the places. docs.dify.ai is already a built up platform hosting our docs. using markdown is also a standard way for the community to prepare docs.

Anyway, I'm just a newbie contributor. Dify team might have been preparing a new doc platform. So it's your call.

jasonfish568 · 2025-01-31T06:00:25Z

@JohnJyong Happy New Year. Just a follow up on this PR. Please kindly process at your earliest convenience.

jasonfish568 · 2025-02-06T04:56:56Z

The code part LGTM, however you should add according docs to the API Access as well in CN, EN & JA.

Could you let me know when this PR will be merged? How does dify team manage PRs? When do PRs ususally get merged?

Hello, for newly added apis, you need to update the docs in mdx files as well.

There is a conflict file.

Here are the docs I mentioned.

Hi,

I was wondering if you could speed up the merge of this PR? I'm desparately willing to use these APIs for my project.

Thanks

Update to lastest version

Update

jasonfishblu · 2025-03-19T09:32:37Z

@crazywoola Could you please clarify that version 1.0 has got these APIs implemented? The Dify Doc PR got denied due to not planned.

If this is never gonna be a feature in the official package, then I won't be bothered to contribute. My pay rate is too high to be wasted. Very much disappointed that when other contributors spent quite some time but received no response.

crazywoola · 2025-03-21T11:46:13Z

@crazywoola Could you please clarify that version 1.0 has got these APIs implemented? The Dify Doc PR got denied due to not planned.

If this is never gonna be a feature in the official package, then I won't be bothered to contribute. My pay rate is too high to be wasted. Very much disappointed that when other contributors spent quite some time but received no response.

Sorry for the late response, this is a great feature that we would like to have.

I know that the explanation is rather insipid. The fact is, we had a major infra update this week.

See Chore: frontend infrastructure upgrade #16420

So most of the PRs can not get merged during this time.

Can you resolve the conflicts, I will merged it asap. :)

@jasonfishblu

jasonfishblu · 2025-03-25T00:58:30Z

Thanks for the reply, I will arrange a time to get the conflicts sorted.

crazywoola · 2025-03-26T03:33:20Z

Thanks for the reply, I will arrange a time to get the conflicts sorted.

Cool Thanks, sorry for the inconvience.

jasonfish568 · 2025-04-05T06:00:38Z

Thanks for the reply, I will arrange a time to get the conflicts sorted.

Cool Thanks, sorry for the inconvience.

Hi Crazywoola,

I have managed to get the conflicts resolved. New doc added by other contributors did get honoured so no info missed.

Please help get this PR merged to prevent further conflicts at your earliest convenience.

Very much appreciated!

Jason

Copilot

Copilot reviewed 5 out of 11 changed files in this pull request and generated 1 comment.

Files not reviewed (6)

web/app/(commonLayout)/datasets/template/template.en.mdx: Language not supported
web/app/(commonLayout)/datasets/template/template.zh.mdx: Language not supported
web/app/components/develop/template/template.zh.mdx: Language not supported
web/app/components/develop/template/template_advanced_chat.en.mdx: Language not supported
web/app/components/develop/template/template_advanced_chat.zh.mdx: Language not supported
web/app/components/develop/template/template_chat.en.mdx: Language not supported

Comments suppressed due to low confidence (1)

api/controllers/service_api/dataset/dataset.py:156

The retrieval and update for 'partial_member_list' is performed twice in the GET method which appears redundant. Consider removing one of the duplicate calls to streamline the response assembly.

if data.get("permission") == "partial_members":

api/controllers/service_api/app/annotation.py

…genius#12237)

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. 💪 enhancement New feature or request labels Dec 30, 2024

feat(improve-api-endpoints): Added available model API

d7854a5

This API endpoint shares the same API token with Dataset API. It is essential for patching the knowledge base in terms of embedding model. Related to: langgenius#11727

jasonfish568 marked this pull request as draft December 31, 2024 05:43

jasonfish568 added a commit to jasonfish568/dify-docs that referenced this pull request Dec 31, 2024

docs: Add Annotation API documentation

85f8276

Related to: PR#12237(langgenius/dify#12237)

jasonfish568 marked this pull request as ready for review December 31, 2024 07:11

dosubot bot added the 📚 documentation Improvements or additions to documentation label Dec 31, 2024

jasonfish568 mentioned this pull request Dec 31, 2024

docs: Add Dataset API and Annotation API documentation langgenius/dify-docs-archived#424

Open

jasonfish568 added 2 commits January 2, 2025 12:15

style: fixed ruff errors

a7d85c2

style: Fix ruff error

d589f0b

style: Fix lint errors

b489355

crazywoola requested a review from JohnJyong January 13, 2025 09:54

jasonfish568 added 4 commits January 14, 2025 16:40

Merge branch 'main' into main

f00a65e

docs: Added docs in mdx files for the Dataset APIs implemented

8c8c87e

Merge branch 'main' of github.com:jasonfish568/dify

65e96df

docs: Added annotation API documentation to Chat App documentation.

6a8c603

It is added to Chat App due to Annotation is under Apps, and there is no individual Annotation API doc page. It seems correct to have Annotation APIs put here.

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jan 16, 2025

jasonfish568 added 2 commits February 10, 2025 13:47

Merge pull request #1 from langgenius/main

ec8290d

Update to lastest version

Merge pull request #2 from langgenius/main

55c6476

Update

crazywoola self-requested a review March 21, 2025 11:47

Merge branch 'main' into main

edd9755

crazywoola approved these changes Apr 7, 2025

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Apr 7, 2025

crazywoola merged commit fd44394 into langgenius:main Apr 7, 2025
7 checks passed

crazywoola requested a review from Copilot April 7, 2025 03:10

Copilot AI reviewed Apr 7, 2025

View reviewed changes

api/controllers/service_api/app/annotation.py Show resolved Hide resolved

chen622 mentioned this pull request Apr 17, 2025

fix: change the method of update_dataset api in document #18197

Merged

5 tasks

jsincorporated pushed a commit to jsincorporated/asaAi that referenced this pull request Jul 8, 2025

feat(improve-api-endpoints): Added Datasets and Annotation APIs (lang…

64a130d

…genius#12237)

feat(improve-api-endpoints): Added Datasets and Annotation APIs #12237

feat(improve-api-endpoints): Added Datasets and Annotation APIs #12237

Uh oh!

Conversation

jasonfish568 commented Dec 30, 2024 • edited by crazywoola Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

crazywoola commented Jan 2, 2025

Uh oh!

jasonfish568 commented Jan 2, 2025

Uh oh!

crazywoola commented Jan 3, 2025

Uh oh!

jasonfish568 commented Jan 3, 2025

Uh oh!

crazywoola commented Jan 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasonfish568 commented Jan 3, 2025

Uh oh!

jasonfish568 commented Jan 9, 2025

Uh oh!

crazywoola commented Jan 13, 2025

Uh oh!

jasonfish568 commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Outcome

Suggestion:

Uh oh!

jasonfish568 commented Jan 31, 2025

Uh oh!

jasonfish568 commented Feb 6, 2025

Uh oh!

jasonfishblu commented Mar 19, 2025

Uh oh!

crazywoola commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasonfishblu commented Mar 25, 2025

Uh oh!

crazywoola commented Mar 26, 2025

Uh oh!

jasonfish568 commented Apr 5, 2025

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jasonfish568 commented Dec 30, 2024 •

edited by crazywoola

Loading

crazywoola commented Jan 3, 2025 •

edited

Loading

jasonfish568 commented Jan 16, 2025 •

edited

Loading

crazywoola commented Mar 21, 2025 •

edited

Loading