Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] github的gpt4o识别图片报错 #4013

Open
BiFangKNT opened this issue Sep 18, 2024 · 37 comments
Open

[Bug] github的gpt4o识别图片报错 #4013

BiFangKNT opened this issue Sep 18, 2024 · 37 comments
Labels
🐛 Bug Something isn't working | 缺陷 vision

Comments

@BiFangKNT
Copy link
Contributor

BiFangKNT commented Sep 18, 2024

📦 部署环境

Docker

📌 软件版本

lobehub/lobe-chat-database:1.19.0

💻 系统环境

Other Linux

🌐 浏览器

Chrome

🐛 问题描述

github的模型
image

客户端请求开启或关闭,对话都可以正常进行的
LobeChat_随便聊聊_2024-09-19 (3)

客户端请求开启状态下,发送图片会报431错误(以下域名皆为占位符,实际是正确域名,在DNSPOD上正常解析)
LobeChat_随便聊聊_2024-09-19

{
  "error": {
    "stack": "Error: 431 status code (no body)\n    at ty.generate (https://lobechat.example.com/_next/static/chunks/68541-eb29d6606b3544d2.js:25:21597)\n    at rY.makeStatusError (https://lobechat.example.com/_next/static/chunks/68541-eb29d6606b3544d2.js:25:13162)\n    at rY.makeRequest (https://lobechat.example.com/_next/static/chunks/68541-eb29d6606b3544d2.js:25:14085)\n    at async Object.chat (https://lobechat.example.com/_next/static/chunks/7029-e4524a0dde512cb2.js:1:14880)\n    at async n (https://lobechat.example.com/_next/static/chunks/7029-e4524a0dde512cb2.js:1:54593)\n    at async https://lobechat.example.com/_next/static/chunks/7029-e4524a0dde512cb2.js:1:49329",
    "status": 431,
    "headers": {
      "content-length": "0",
      "x-request-time": "0.369"
    }
  },
  "endpoint": "https://models.inference.ai.azure.com",
  "provider": "github"
}

客户端请求关闭状态下,发送图片会报400错误
LobeChat_随便聊聊_2024-09-19 (1)

{
  "error": {
    "code": "BadRequest",
    "type": null,
    "param": null,
    "message": "The provided image url is not accessible."
  },
  "endpoint": "https://models.inference.ai.azure.com",
  "provider": "github"
}

而使用OpenRouter上的gemini-pro-1.5-exp发送图片是能正常识别的
LobeChat_随便聊聊_2024-09-19 (2)

”视觉模型不能正常使用视觉功能“的这个问题在智谱的glm-4v-plus及glm-4v上亦有,很难不怀疑是否有共通的bug #3909

📷 复现步骤

数据库版lobe,使用github的gpt4o,发送图片。

🚦 期望结果

视觉功能正常,包括后面提及的智谱的视觉模型。

📝 补充信息

这里使用的OpenRouter上的gemini-pro-1.5-exp,是通过one-api整合到openai接口的key上的

@BiFangKNT BiFangKNT added the 🐛 Bug Something isn't working | 缺陷 label Sep 18, 2024
@lobehubbot
Copy link
Member

👀 @BiFangKNT

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事,并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

@dosubot dosubot bot added the vision label Sep 18, 2024
Copy link

dosubot bot commented Sep 18, 2024

I found some similar issues and discussions that might be relevant to your problem:

  1. [Bug] 4o not support image upload (Open) - This issue discusses a similar problem with image uploads and provides some troubleshooting steps, such as checking network and cross-origin settings, verifying file type support, and ensuring correct Azure-specific configurations [1].

  2. [Bug] Cannot upload images using the LLaVA 34B model (Closed) - This issue was resolved by configuring the model with the correct environment variables or model panel settings [2].

  3. [Bug] Fail to send prompt with image with Azure GPT-4 2024-04-09 (Closed) - This issue was resolved by a pull request that fixed the Azure OpenAI Vision models issue [3].

Suggested Solutions:

  1. Network and Cross-Origin Settings: Ensure that your network is stable and that the cross-origin settings for your file storage service are correctly configured.

  2. File Type Support: Verify that the image file type is supported by the GitHub GPT-4.0 model. Only certain image file types may be supported for upload.

  3. Azure-Specific Configurations: Ensure that the Azure-specific configurations such as apiVersion, apikey, and endpoint are correctly set.

  4. Logging: Check the server logs for more detailed error messages. The code logs errors on the server side using console.error when an exception occurs during the image upload process.

  5. Model Configuration: Make sure the model is correctly configured with the necessary environment variables or through the model panel settings.

If the issue persists, you may need to review the specific configurations for GitHub's GPT-4.0 model in your codebase to ensure they are correctly defined and implemented.

To continue talking to Dosu, mention @dosu.

@BiFangKNT
Copy link
Contributor Author

应该不是跨域问题,如果是,那么下面的gemini-pro-1.5-exp就不应该能识别图片成功

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


It should not be a cross-domain problem. If it is, then the following gemini-pro-1.5-exp should not be able to successfully identify the image.

@SpeedupMaster
Copy link
Contributor

image
我这边用着没问题呀,可以正常识别图片

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


image
I have no problem using it here. I can recognize pictures normally.

@BiFangKNT
Copy link
Contributor Author

image 我这边用着没问题呀,可以正常识别图片

你这是github的模型吗?能否看看模型的tag?

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


![image](https://private-user-images.githubusercontent.com/130642629/368799983-0c18a395-e162-414f-979b-98c1ff850e4a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc 3MiOiJna NDI2MjkvMzY4Nzk5OTgzLTBjMThhMzk1LWUxNjItNDE0Zi05NzliLTk4YzFmZjg1MGU0YS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1 BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTE5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkxOVQxNDMyNDhaJlgtQW16LUV4cGlyZXM9MzAw JlgtQW16LVNpZ25hdHVyZT1lYzc4MzM3N2JlNzkxYmIwMzM0NDRkNTM3NjEzYTkwZDdmYT U5MDlkOWY2NTc5ZDVmNGI0NTI5Y2FiOGNjNmY1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zd CJ9.4IuKt9A4rrCqxmDkWfI62f4IiRGIwwTPcHz5YTo-4E8) My side There is no problem in using it. I can recognize pictures normally.

Is this your github model? Can I see the tag of the model?

@SAnBlog
Copy link

SAnBlog commented Sep 20, 2024

GitHub的models审核多久通过的?

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


How long does it take for GitHub's models to be approved?

@BiFangKNT
Copy link
Contributor Author

GitHub的models审核多久通过的?

不知道。它说通过了有邮件提醒,但是我没收到。这天我听闻lobe更新了github模型,就去试了试,才发现能够使用。

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


How long does it take for GitHub’s models to be approved?

have no idea. It said there would be an email reminder after it was approved, but I didn't receive it. That day I heard that lobe had updated the github model, so I tried it and found out that it could be used.

@SAnBlog
Copy link

SAnBlog commented Sep 20, 2024

GitHub的models审核多久通过的?

不知道。它说通过了有邮件提醒,但是我没收到。这天我听闻lobe更新了github模型,就去试了试,才发现能够使用。

好吧,那你是多久之前申请的?

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


How long does it take for GitHub’s models to be approved?

Don't know. It said there would be an email reminder after it was approved, but I didn't receive it. That day I heard that lobe had updated the github model, so I tried it and found out that it could be used.

Okay, so how long ago did you apply?

@SpeedupMaster
Copy link
Contributor

image 我这边用着没问题呀,可以正常识别图片

你这是github的模型吗?能否看看模型的tag?

LobeChat默认的,没做任何修改

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


![image](https://private-user-images.githubusercontent.com/130642629/368799983-0c18a395-e162-414f-979b-98c1ff850e4a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ pc3MiOiJna 2NDI2MjkvMzY4Nzk5OTgzLTBjMThhMzk1LWUxNjItNDE0Zi05NzliLTk4YzFmZjg1MGU0YS 5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD 1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTE5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkxOVQxNDMyNDhaJlgtQW16LUV4cGlyZXM9MzA wJlgtQW16LVNpZ25hdHVyZT1lYzc4MzM3N2JlNzkxYmIwMzM0NDRkNTM3NjEzYTkwZDdmY TU5MDlkOWY2NTc5ZDVmNGI0NTI5Y2FiOGNjNmY1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9z dCJ9.4IuKt9A4rrCqxmDkWfI62f4IiRGIwwTPcHz5YTo-4E8) Mine It’s no problem if you use it on the side. You can recognize pictures normally.

Is this a github model? Can I see the tag of the model?

LobeChat is the default without any modification.

@SAnBlog
Copy link

SAnBlog commented Sep 21, 2024

image 我这边用着没问题呀,可以正常识别图片

你这是github的模型吗?能否看看模型的tag?

LobeChat默认的,没做任何修改

你的GitHub模型是申请多久审批通过的?

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


![image](https://private-user-images.githubusercontent.com/130642629/368799983-0c18a395-e162-414f-979b-98c1ff850e4a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.ey Jpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjY3NTY2NjgsIm5iZiI6MTcyNjc1NjM2OCwicGF0aCI6Ii8xMz A2NDI2MjkvMzY4Nzk5OTgzLTBjMThhMzk1LWUxNjItNDE0Zi05NzliLTk4YzFmZjg1MGU0YS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhb D1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTE5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkxOVQxNDMyNDhaJlgtQW16LUV4cGlyZXM9Mz AwJlgtQW16LVNpZ25hdHVyZT1lYzc4MzM3N2JlNzkxYmIwMzM0NDRkNTM3NjEzYTkwZDdm YTU5MDlkOWY2NTc5ZDVmNGI0NTI5Y2FiOGNjNmY1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9 zdCJ9.4IuKt9A4rrCqxmDkWfI62f4IiRGIwwTPcHz5YTo-4E8) I There is no problem in using it here. Pictures can be recognized normally.

Is this a github model? Can I see the tag of the model?

LobeChat is the default without any modification.

How long did it take for your GitHub model to be approved?

@SpeedupMaster
Copy link
Contributor

image 我这边用着没问题呀,可以正常识别图片

你这是github的模型吗?能否看看模型的tag?

LobeChat默认的,没做任何修改

你的GitHub模型是申请多久审批通过的?

申请第二天就通过了

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


![image](https://private-user-images.githubusercontent.com/130642629/368799983-0c18a395-e162-414f-979b-98c1ff850e4a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9. eyJpc3 MiOiJna zA2NDI2MjkvMzY4Nzk5OTgzLTBjMThhMzk1LWUxNjItNDE0Zi05NzliLTk4YzFmZjg1MGU0YS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlh bD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTE5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkxOVQxNDMyNDhaJlgtQW16LUV4cGlyZXM9M zAwJlgtQW16LVNpZ25hdHVyZT1lYzc4MzM3N2JlNzkxYmIwMzM0NDRkNTM3NjEzYTkwZDd mYTU5MDlkOWY2NTc5ZDVmNGI0NTI5Y2FiOGNjNmY1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG 9zdCJ9.4IuKt9A4rrCqxmDkWfI62f4IiRGIwwTPcHz5YTo-4E8) I have no problem using it here. I can recognize pictures normally.

Is this a github model? Can I see the tag of the model?

LobeChat is the default without any modification.

How long did it take for your GitHub model to be approved?

The application was approved the next day

@BiFangKNT
Copy link
Contributor Author

GitHub的models审核多久通过的?

不知道。它说通过了有邮件提醒,但是我没收到。这天我听闻lobe更新了github模型,就去试了试,才发现能够使用。

好吧,那你是多久之前申请的?

一个星期吧

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


How long does it take for GitHub's models to pass review?

Don't know. It said there would be an email reminder after it was approved, but I didn't receive it. That day I heard that lobe had updated the github model, so I tried it and found out that it could be used.

Okay, how long ago did you apply?

A week

@BiFangKNT
Copy link
Contributor Author

image 我这边用着没问题呀,可以正常识别图片

请问你的lobe是部署在哪里的?是数据库版本吗?s3是minio吗?

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


![image](https://private-user-images.githubusercontent.com/130642629/368799983-0c18a395-e162-414f-979b-98c1ff850e4a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc 3MiOiJna DI2MjkvMzY4Nzk5OTgzLTBjMThhMzk1LWUxNjItNDE0Zi05NzliLTk4YzFmZjg1MGU0YS5wb mc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS 0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTIxJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkyMVQyMjU4MTZaJlgtQW16LUV4cGlyZXM9MzA wJlgtQW16LVNpZ25hdHVyZT03NmM1ZjA3ZTA5YjdiOTY1YTFiZGFjZmJmNDI0ZmIwOGRkY2 NmZDY2OTJjZjc4ZTUzODJhZDQ5ODQ1MWEyYmY1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdC J9.NIl_sTpikRXQoE5N0Bl38ET_OrAjKMcLbq7tzbp5lHM) I use it here No problem, pictures can be recognized normally

Where is your lobe deployed? Is it the database version? Is s3 minio?

@BiFangKNT
Copy link
Contributor Author

现在我通过开启容器ipv6,解决了许多问题,详见 #4058
于是我再试了一次gpt4o,关闭客户端请求状态下,报错改变了:

Route: [github] ProviderBizError: {
  headers: {
  apim-request-id: '26426ceb-ee10-45b3-b324-c4633ae4c02b',
  connection: 'keep-alive',
  content-length: '0',
  date: 'Sat, 21 Sep 2024 23:08:25 GMT',
  request-context: 'appId=',
  strict-transport-security: 'max-age=31536000; includeSubDomains; preload',
  x-aml-cluster: 'vienna-eastus2-02',
  x-content-type-options: 'nosniff',
  x-envoy-upstream-service-time: '2',
  x-ms-client-request-id: 'Not-Set',
  x-ms-region: 'East US 2',
  x-ratelimit-remaining-requests: '199999',
  x-ratelimit-remaining-tokens: '19993346',
  x-request-time: '0.292'
},
  stack: 'Error: 431 status code (no body)\n    at tl.generate (/app/.next/server/edge-chunks/702.js:4:20377)\n    at sW.makeStatusError (/app/.next/server/edge-chunks/702.js:4:13235)\n    at sW.makeRequest (/app/.next/server/edge-chunks/702.js:4:14179)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.chat (/app/.next/server/edge-chunks/210.js:9:3181)\n    at async /app/.next/server/app/api/chat/[provider]/route.js:1:1429\n    at async /app/.next/server/edge-chunks/490.js:1:90730\n    at async $.execute (/app/.next/server/edge-chunks/490.js:1:88440)\n    at async $.handle (/app/.next/server/edge-chunks/490.js:1:91997)\n    at async ex.handler (/app/.next/server/edge-chunks/490.js:2:32454)',
  status: 431
}

而开启客户端请求状态下,没有变化

而gpt4o-mini在开启客户端请求状态下,返回也是431错误,在关闭下,则是上面说到过的400错误

可以看出这个问题应该主要是431错误引起的,也就是请求头字段太大。希望官方排查一下,优化一下github api逻辑 @arvinxx

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


Now I have solved many problems by enabling container ipv6. For details, see #4058
So I tried gpt4o again. When the client request was turned off, the error message changed:

Route: [github] ProviderBizError: {
  headers: {
  apim-request-id: '26426ceb-ee10-45b3-b324-c4633ae4c02b',
  connection: 'keep-alive',
  content-length: '0',
  date: 'Sat, 21 Sep 2024 23:08:25 GMT',
  request-context: 'appId=',
  strict-transport-security: 'max-age=31536000; includeSubDomains; preload',
  x-aml-cluster: 'vienna-eastus2-02',
  x-content-type-options: 'nosniff',
  x-envoy-upstream-service-time: '2',
  x-ms-client-request-id: 'Not-Set',
  x-ms-region: 'East US 2',
  x-ratelimit-remaining-requests: '199999',
  x-ratelimit-remaining-tokens: '19993346',
  x-request-time: '0.292'
},
  stack: 'Error: 431 status code (no body)\n at tl.generate (/app/.next/server/edge-chunks/702.js:4:20377)\n at sW.makeStatusError (/app/. next/server/edge-chunks/702.js:4:13235)\n at sW.makeRequest (/app/.next/server/edge-chunks/702.js:4:14179)\n at process.processTicksAndRejections ( node:internal/process/task_queues:95:5)\n at async Object.chat (/app/.next/server/edge-chunks/210.js:9:3181)\n at async /app/.next/ server/app/api/chat/[provider]/route.js:1:1429\n at async /app/.next/server/edge-chunks/490.js:1:90730\n at async $.execute ( /app/.next/server/edge-chunks/490.js:1:88440)\n at async $.handle (/app/.next/server/edge-chunks/490.js:1:91997)\n at async ex.handler (/app/.next/server/edge-chunks/490.js:2:32454)',
  status: 431
}

When the client request state is turned on, there is no change.

When gpt4o-mini is turned on for client requests, it returns a 431 error. When it is turned off, it returns the 400 error mentioned above.

It can be seen that this problem should be mainly caused by the 431 error, that is, the request header field is too large. I hope the official will investigate and optimize the github api logic @arvinxx

@hominsu
Copy link

hominsu commented Sep 25, 2024

The same issue occurred when using GitHub's gpt4-o.

@SAnBlog
Copy link

SAnBlog commented Sep 26, 2024

现在我通过开启容器ipv6,解决了许多问题,详见 #4058 于是我再试了一次gpt4o,关闭客户端请求状态下,报错改变了:

Route: [github] ProviderBizError: {
  headers: {
  apim-request-id: '26426ceb-ee10-45b3-b324-c4633ae4c02b',
  connection: 'keep-alive',
  content-length: '0',
  date: 'Sat, 21 Sep 2024 23:08:25 GMT',
  request-context: 'appId=',
  strict-transport-security: 'max-age=31536000; includeSubDomains; preload',
  x-aml-cluster: 'vienna-eastus2-02',
  x-content-type-options: 'nosniff',
  x-envoy-upstream-service-time: '2',
  x-ms-client-request-id: 'Not-Set',
  x-ms-region: 'East US 2',
  x-ratelimit-remaining-requests: '199999',
  x-ratelimit-remaining-tokens: '19993346',
  x-request-time: '0.292'
},
  stack: 'Error: 431 status code (no body)\n    at tl.generate (/app/.next/server/edge-chunks/702.js:4:20377)\n    at sW.makeStatusError (/app/.next/server/edge-chunks/702.js:4:13235)\n    at sW.makeRequest (/app/.next/server/edge-chunks/702.js:4:14179)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.chat (/app/.next/server/edge-chunks/210.js:9:3181)\n    at async /app/.next/server/app/api/chat/[provider]/route.js:1:1429\n    at async /app/.next/server/edge-chunks/490.js:1:90730\n    at async $.execute (/app/.next/server/edge-chunks/490.js:1:88440)\n    at async $.handle (/app/.next/server/edge-chunks/490.js:1:91997)\n    at async ex.handler (/app/.next/server/edge-chunks/490.js:2:32454)',
  status: 431
}

而开启客户端请求状态下,没有变化

而gpt4o-mini在开启客户端请求状态下,返回也是431错误,在关闭下,则是上面说到过的400错误

可以看出这个问题应该主要是431错误引起的,也就是请求头字段太大。希望官方排查一下,优化一下github api逻辑 @arvinxx

存在一样的问题,431,发生在4o

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


Now I have solved many problems by enabling the container ipv6, see #4058 for details. So I tried gpt4o again, and when the client request was turned off, the error changed:

Route: [github] ProviderBizError: {
headers: {
apim-request-id: '26426ceb-ee10-45b3-b324-c4633ae4c02b',
connection: 'keep-alive',
content-length: '0',
date: 'Sat, 21 Sep 2024 23:08:25 GMT',
request-context: 'appId=',
strict-transport-security: 'max-age=31536000; includeSubDomains; preload',
x-aml-cluster: 'vienna-eastus2-02',
x-content-type-options: 'nosniff',
x-envoy-upstream-service-time: '2',
x-ms-client-request-id: 'Not-Set',
x-ms-region: 'East US 2',
x-ratelimit-remaining-requests: '199999',
x-ratelimit-remaining-tokens: '19993346',
x-request-time: '0.292'
},
stack: 'Error: 431 status code (no body)\n at tl.generate (/app/.next/server/edge-chunks/702.js:4:20377)\n at sW.makeStatusError (/app/ .next/server/edge-chunks/702.js:4:13235)\n at sW.makeRequest (/app/.next/server/edge-chunks/702.js:4:14179)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Object.chat (/app/.next/server/edge-chunks/210.js:9:3181)\n at async /app/.next /server/app/api/chat/[provider]/route.js:1:1429\n at async /app/.next/server/edge-chunks/490.js:1:90730\n at async $.execute (/app/.next/server/edge-chunks/490.js:1:88440)\n at async $.handle (/app/.next/server/edge-chunks/490.js:1:91997)\ n at async ex.handler (/app/.next/server/edge-chunks/490.js:2:32454)',
status: 431
}

When the client request is turned on, there is no change

When gpt4o-mini is turned on for client requests, it returns a 431 error. When it is turned off, it returns the 400 error mentioned above.

It can be seen that this problem should be mainly caused by the 431 error, that is, the request header field is too large. I hope the official will investigate and optimize the github api logic @arvinxx

The same problem exists, 431, occurred on 4o

@BiFangKNT
Copy link
Contributor Author

@arvinxx 大佬,应该是lobe这边的问题,我用github官方的测试代码测试了,minio里的图片能够正常识别

测试脚本如下:

import os
import base64
import requests
from openai import OpenAI

token = os.environ["GITHUB_TOKEN"]
endpoint = "https://models.inference.ai.azure.com"
model_name = "gpt-4o-mini"


def get_image_data_url(image_file: str, image_format: str) -> str:
    """
    Helper function to convert an image file (from local or URL) to a data URL string.

    Args:
        image_file (str): The path or URL to the image file.
        image_format (str): The format of the image file.

    Returns:
        str: The data URL of the image.
    """
    try:
        # Check if the image_file is a URL
        if image_file.startswith("http://") or image_file.startswith("https://"):
            response = requests.get(image_file)
            response.raise_for_status()  # Raise an exception for HTTP errors
            image_data = base64.b64encode(response.content).decode("utf-8")
        else:
            with open(image_file, "rb") as f:
                image_data = base64.b64encode(f.read()).decode("utf-8")
    except Exception as e:
        print(f"Error reading '{image_file}': {e}")
        exit()

    return f"data:image/{image_format};base64,{image_data}"


client = OpenAI(
    base_url=endpoint,
    api_key=token,
)

response = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that describes images in details.",
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "图片里有什么?",
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": get_image_data_url("https://minio.example.top/lobe/files/479414/9294edbb-e5ca-4a4d-84b1-52685fb738bc.png", "png"),
                        "detail": "low"
                    },
                },
            ],
        },
    ],
    model=model_name,
)

print(response.choices[0].message.content)

图片如下:
111

脚本返回如下:

这个图像包含一个标志和文字。标志的形状呈现为一个渐变的圆形,颜色由浅蓝色渐变到绿色。下方的文字“Logoipsum”使用简单、现代的无衬线字体,颜色为黑色。这种设计通常给人一种简洁、现代的感觉。

进程已结束,退出代码为 0

@BiFangKNT
Copy link
Contributor Author

@arvinxx 大佬,上面的脚本是基于OPENAI SDK的,我又尝试了Azure Al Inference SDK,其无法访问我的图片链接。

推测:后者不支持访问ipv6。
建议:使用OPENAI SDK,而不是Azure Al Inference SDK

基于Azure Al Inference SDK的测试脚本如下:

import os
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import (
    SystemMessage,
    UserMessage,
    TextContentItem,
    ImageContentItem,
    ImageUrl,
    ImageDetailLevel,
)
from azure.core.credentials import AzureKeyCredential

# 获取 GitHub Token
token = os.environ["GITHUB_TOKEN"]
endpoint = "https://models.inference.ai.azure.com"
model_name = "gpt-4o-mini"

# 创建聊天补全客户端
client = ChatCompletionsClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(token),
)

# 发送请求并加载在线图片链接
response = client.complete(
    messages=[
        # 系统消息,用于设定 AI 助手的角色
        SystemMessage(
            content="You are a helpful assistant that describes images in details."
        ),
        # 用户消息,包含文本和图片的在线 URL
        UserMessage(
            content=[
                TextContentItem(text="图片里有什么?"),
                ImageContentItem(
                    image_url=ImageUrl(
                        url="https://minio.example.top/lobe/files/479414/9294edbb-e5ca-4a4d-84b1-52685fb738bc.png",  # 替换为在线图片的 URL
                        detail=ImageDetailLevel.LOW  # 设置图像详细级别
                    ),
                ),
            ],
        ),
    ],
    model=model_name,
)

# 输出助手的回复
print(response.choices[0].message.content)

其返回如下:

Traceback (most recent call last):
  File "C:\Users\13204\PycharmProjects\atest\venv\github测试azure.py", line 25, in <module>
    response = client.complete(
  File "C:\Users\13204\PycharmProjects\atest\venv\lib\site-packages\azure\core\tracing\decorator.py", line 94, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "C:\Users\13204\PycharmProjects\atest\venv\lib\site-packages\azure\ai\inference\_patch.py", line 643, in complete
    raise HttpResponseError(response=response)
azure.core.exceptions.HttpResponseError: (BadRequest) The provided image url is not accessible.
Code: BadRequest
Message: The provided image url is not accessible.

其中图片链接改为支持ipv4的链接,脚本就能正常返回:

这幅图展示了一个身穿蓝色和白色服装的角色,背景则充满了梦幻的紫色和黑色元素。角色的发色为浅色,可能是银色或白色,周围环绕着多个光线和碎片,营造出一种动态和神秘的氛围。背景中有旋转的齿轮和其他抽象的图形,增加了画面的深度和复杂性。整体感觉充满了幻想和未来感。

进程已结束,退出代码为 0

@BiFangKNT
Copy link
Contributor Author

BiFangKNT commented Sep 28, 2024

The same issue occurred when using GitHub's gpt4-o.

@hominsu You can refer to the case I mentioned above.

@BiFangKNT
Copy link
Contributor Author

现在我通过开启容器ipv6,解决了许多问题,详见 #4058 于是我再试了一次gpt4o,关闭客户端请求状态下,报错改变了:

Route: [github] ProviderBizError: {
  headers: {
  apim-request-id: '26426ceb-ee10-45b3-b324-c4633ae4c02b',
  connection: 'keep-alive',
  content-length: '0',
  date: 'Sat, 21 Sep 2024 23:08:25 GMT',
  request-context: 'appId=',
  strict-transport-security: 'max-age=31536000; includeSubDomains; preload',
  x-aml-cluster: 'vienna-eastus2-02',
  x-content-type-options: 'nosniff',
  x-envoy-upstream-service-time: '2',
  x-ms-client-request-id: 'Not-Set',
  x-ms-region: 'East US 2',
  x-ratelimit-remaining-requests: '199999',
  x-ratelimit-remaining-tokens: '19993346',
  x-request-time: '0.292'
},
  stack: 'Error: 431 status code (no body)\n    at tl.generate (/app/.next/server/edge-chunks/702.js:4:20377)\n    at sW.makeStatusError (/app/.next/server/edge-chunks/702.js:4:13235)\n    at sW.makeRequest (/app/.next/server/edge-chunks/702.js:4:14179)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.chat (/app/.next/server/edge-chunks/210.js:9:3181)\n    at async /app/.next/server/app/api/chat/[provider]/route.js:1:1429\n    at async /app/.next/server/edge-chunks/490.js:1:90730\n    at async $.execute (/app/.next/server/edge-chunks/490.js:1:88440)\n    at async $.handle (/app/.next/server/edge-chunks/490.js:1:91997)\n    at async ex.handler (/app/.next/server/edge-chunks/490.js:2:32454)',
  status: 431
}

而开启客户端请求状态下,没有变化
而gpt4o-mini在开启客户端请求状态下,返回也是431错误,在关闭下,则是上面说到过的400错误
可以看出这个问题应该主要是431错误引起的,也就是请求头字段太大。希望官方排查一下,优化一下github api逻辑 @arvinxx

存在一样的问题,431,发生在4o

@SAnBlog 可以参考一下我上面的案例。

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


Now I have solved many problems by enabling ipv6 in the container. See #4058 for details. So I tried gpt4o again. When the client request was turned off, the error message changed:

Route: [github] ProviderBizError: {
headers: {
apim-request-id: '26426ceb-ee10-45b3-b324-c4633ae4c02b',
connection: 'keep-alive',
content-length: '0',
date: 'Sat, 21 Sep 2024 23:08:25 GMT',
request-context: 'appId=',
strict-transport-security: 'max-age=31536000; includeSubDomains; preload',
x-aml-cluster: 'vienna-eastus2-02',
x-content-type-options: 'nosniff',
x-envoy-upstream-service-time: '2',
x-ms-client-request-id: 'Not-Set',
x-ms-region: 'East US 2',
x-ratelimit-remaining-requests: '199999',
x-ratelimit-remaining-tokens: '19993346',
x-request-time: '0.292'
},
stack: 'Error: 431 status code (no body)\n at tl.generate (/app/.next/server/edge-chunks/702.js:4:20377)\n at sW.makeStatusError (/app /.next/server/edge-chunks/702.js:4:13235)\n at sW.makeRequest (/app/.next/server/edge-chunks/702.js:4:14179)\n at process. processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Object.chat (/app/.next/server/edge-chunks/210.js:9:3181)\n at async /app/. next/server/app/api/chat/[provider]/route.js:1:1429\n at async /app/.next/server/edge-chunks/490.js:1:90730\n at async $. execute (/app/.next/server/edge-chunks/490.js:1:88440)\n at async $.handle (/app/.next/server/edge-chunks/490.js:1:91997) \n at async ex.handler (/app/.next/server/edge-chunks/490.js:2:32454)',
status: 431
}

When the client request is turned on, there is no change.
When gpt4o-mini is turned on for client requests, it returns a 431 error. When it is turned off, it returns the 400 error mentioned above.
It can be seen that this problem should be mainly caused by the 431 error, that is, the request header field is too large. I hope the official will investigate and optimize the github api logic @arvinxx

The same problem exists, 431, occurred on 4o

@SAnBlog You can refer to my case above.

@BiFangKNT
Copy link
Contributor Author

@arvinxx 大佬,我发现使用Azure Al Inference SDK也行,不过需要多一步对图像的处理。如果把图片转为base64,再用SDK的ImageUrl方法包装成ImageUrl对象的话,模型就能正常识别了。

测试脚本如下:

import os
import base64
import requests
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import (
    SystemMessage,
    UserMessage,
    TextContentItem,
    ImageContentItem,
    ImageUrl,  # 导入 ImageUrl 类型
)
from azure.core.credentials import AzureKeyCredential

def fetch_image_as_base64(url):
    """
    Fetches an image from the given URL and returns it as a base64-encoded string.
    """
    try:
        response = requests.get(url)
        response.raise_for_status()
        image_data = base64.b64encode(response.content).decode("utf-8")
        return f"data:image/png;base64,{image_data}"
    except Exception as e:
        print(f"Error fetching image from {url}: {e}")
        return None

# 使用你的环境变量和接入点配置
token = os.environ["GITHUB_TOKEN"]
endpoint = "https://models.inference.ai.azure.com"
model_name = "gpt-4o-mini"

# 创建 Azure 客户端
client = ChatCompletionsClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(token),
)

# 使用自定义函数从 IPv6 URL 获取图片数据
image_url = "https://minio.example.top/lobe/files/479414/9294edbb-e5ca-4a4d-84b1-52685fb738bc.png"
base64_image = fetch_image_as_base64(image_url)

# 确保获取到图片数据后进行请求
if base64_image:
    # 创建 ImageUrl 对象并传递 base64 URL
    image_url_obj = ImageUrl(url=base64_image, detail="low")  # 符合 SDK 的构造要求

    response = client.complete(
        messages=[
            SystemMessage(
                content="You are a helpful assistant that describes images in details."
            ),
            UserMessage(
                content=[
                    TextContentItem(text="图片里有什么?"),
                    ImageContentItem(
                        image_url=image_url_obj  # 传递 ImageUrl 对象
                    ),
                ],
            ),
        ],
        model=model_name,
    )

    # 输出模型识别结果
    print(response.choices[0].message.content)
else:
    print("Failed to retrieve and encode image.")

脚本返回如下:

这是一幅现代风格的图像,包含一个渐变色的图形元素,呈现为一个半圆形,颜色从蓝色逐渐过渡到绿色。图形的下方有一段文字,使用了黑色的无衬线字体,上面写着“Logoipsum”。整体设计简洁,体现了现代设计的趋势。

进程已结束,退出代码为 0

@arvinxx
Copy link
Contributor

arvinxx commented Sep 29, 2024

我昨天试了直接发请求是正常能跑通的,但是不知道为啥换到应用里发就会报错… 估计得 debug 下

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


Yesterday I tried sending a request directly and it worked normally, but I don’t know why I got an error when I changed it to the application... I guess I need to debug it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 Bug Something isn't working | 缺陷 vision
Projects
Status: Roadmap - Chat 1.x
Development

No branches or pull requests

6 participants