Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: apisix may return http 504 when use dubbo-proxy #10429

Open
huntkalio opened this issue Oct 31, 2023 · 0 comments
Open

bug: apisix may return http 504 when use dubbo-proxy #10429

huntkalio opened this issue Oct 31, 2023 · 0 comments
Assignees
@Revolyssup
Labels
bug Something isn't working

Comments

@huntkalio
Copy link

huntkalio commented Oct 31, 2023 •
edited
Loading

Current Behavior

When I use wrk test apisix dubbo-proxy performance，wrk may receive http status 504 when performance test Just started。After some time it will always receive 200 OK。
It may due to apisix new connection Establish with dubbo-backend

wrk result may look like:

  2 threads and 100 connections
 Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.66ms    9.53ms 309.91ms   99.07%
    Req/Sec     4.98k   387.98     5.89k    71.55%
  172430 requests in 17.41s, 30.92MB read
  Non-2xx or 3xx responses: 28
Requests/sec:   9905.90
Transfer/sec:      1.78MB

Non-2xx or 3xx responses: 28

Expected Behavior

no 504 when use dubbo-proxy

Error Logs

10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.000 "-" "-" 10.55.250.95:20880 504 0.000 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.004 "-" "-" 10.55.250.95:20880 504 0.004 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.000 "-" "-" 10.55.250.95:20880 504 0.000 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.000 "-" "-" 10.55.250.95:20880 504 0.000 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.000 "-" "-" 10.55.250.95:20880 504 0.000 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.000 "-" "-" 10.55.250.95:20880 504 0.000 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.002 "-" "-" 10.55.250.95:20880 504 0.002 "http://10.55.250.95:9082"
10.55.250.66 - - [31/Oct/2023:08:55:12 +0000] 10.55.250.95:9082 "GET /hello HTTP/1.1" 504 239 0.000 "-" "-" 10.55.250.95:20880 504 0.000 "http://10.55.250.95:9082"



023/10/31 08:55:12 [error] 33#33: *997 multi: connect reuse unfinished 00007F7C991015D8
2023/10/31 08:55:12 [error] 33#33: *996 multi: connect reuse unfinished 00007F7C991014E0
2023/10/31 08:55:12 [error] 33#33: *997 multi: connect reuse unfinished 00007F7C991015D8
2023/10/31 08:55:12 [error] 33#33: *996 multi: connect reuse unfinished 00007F7C991014E0
2023/10/31 08:55:12 [warn] 33#33: *997 multi: multi connection detach not empty 00007F7C991015D8
2023/10/31 08:55:12 [error] 33#33: *998 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *990 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *993 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *994 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, subrequest: "/hello", upstream: "dubbo://10.55.250.95:20880"
2023/10/31 08:55:12 [error] 33#33: *1001 multi: connect reuse unfinished 00007F7C991018C0
2023/10/31 08:55:12 [error] 33#33: *1001 multi: connect reuse unfinished 00007F7C991018C0
2023/10/31 08:55:12 [error] 33#33: *1001 multi: connect reuse unfinished 00007F7C991018C0
2023/10/31 08:55:12 [error] 33#33: *1001 multi: connect reuse unfinished 00007F7C991018C0
2023/10/31 08:55:12 [warn] 33#33: *1001 multi: multi connection detach not empty 00007F7C991018C0
2023/10/31 08:55:12 [error] 33#33: *992 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *1000 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *993 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *990 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *998 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *999 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, subrequest: "/hello", upstream: "dubbo://10.55.250.95:20880"
2023/10/31 08:55:12 [error] 33#33: *1004 multi: connect reuse unfinished 00007F7C99101AB0
2023/10/31 08:55:12 [error] 33#33: *1004 multi: connect reuse unfinished 00007F7C99101AB0
2023/10/31 08:55:12 [error] 33#33: *1004 multi: connect reuse unfinished 00007F7C99101AB0
2023/10/31 08:55:12 [error] 33#33: *1004 multi: connect reuse unfinished 00007F7C99101AB0
2023/10/31 08:55:12 [error] 33#33: *1004 multi: connect reuse unfinished 00007F7C99101AB0
2023/10/31 08:55:12 [error] 33#33: *1004 multi: connect reuse unfinished 00007F7C99101AB0
2023/10/31 08:55:12 [warn] 33#33: *1004 multi: multi connection detach not empty 00007F7C99101AB0
2023/10/31 08:55:12 [error] 33#33: *994 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *999 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *998 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *990 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *993 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *992 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *1000 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *1002 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, subrequest: "/hello", upstream: "dubbo://10.55.250.95:20880"
2023/10/31 08:55:12 [error] 33#33: *1007 multi: connect reuse unfinished 00007F7C99101CA0
2023/10/31 08:55:12 [error] 33#33: *1007 multi: connect reuse unfinished 00007F7C99101CA0
2023/10/31 08:55:12 [error] 33#33: *1007 multi: connect reuse unfinished 00007F7C99101CA0
2023/10/31 08:55:12 [error] 33#33: *1007 multi: connect reuse unfinished 00007F7C99101CA0
2023/10/31 08:55:12 [error] 33#33: *1007 multi: connect reuse unfinished 00007F7C99101CA0
2023/10/31 08:55:12 [error] 33#33: *1007 multi: connect reuse unfinished 00007F7C99101CA0
2023/10/31 08:55:12 [error] 33#33: *1007 multi: connect reuse unfinished 00007F7C99101CA0
2023/10/31 08:55:12 [error] 33#33: *1007 multi: connect reuse unfinished 00007F7C99101CA0
2023/10/31 08:55:12 [warn] 33#33: *1007 multi: multi connection detach not empty 00007F7C99101CA0
2023/10/31 08:55:12 [error] 33#33: *995 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *1002 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *1000 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *992 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *993 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *990 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *998 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *999 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *994 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, request: "GET /hello HTTP/1.1", upstream: "dubbo://10.55.250.95:20880", host: "10.55.250.95:9082"
2023/10/31 08:55:12 [error] 33#33: *1005 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.55.250.66, server: _, subrequest: "/hello", upstream: "dubbo://10.55.250.95:20880"
2023/10/31 08:55:12 [error] 33#33: *1010 multi: connect reuse unfinished 00007F7C99101E90
2023/10/31 08:55:12 [error] 33#33: *1010 multi: connect reuse unfinished 00007F7C99101E90
2023/10/31 08:55:12 [error] 33#33: *1010 multi: connect reuse unfinished 00007F7C99101E90
2023/10/31 08:55:12 [error] 33#33: *1010 multi: connect reuse unfinished 00007F7C99101E90
2023/10/31 08:55:12 [error] 33#33: *1010 multi: connect reuse unfinished 00007F7C99101E90
2023/10/31 08:55:12 [error] 33#33: *1010 multi: connect reuse unfinished 00007F7C99101E90
2023/10/31 08:55:12 [error] 33#33: *1010 multi: connect reuse unfinished 00007F7C99101E90
2023/10/31 08:55:12 [error] 33#33: *1010 multi: connect reuse unfinished 00007F7C99101E90
2023/10/31 08:55:12 [error] 33#33: *1010 multi: connect reuse unfinished 00007F7C99101E90
2023/10/31 08:55:12 [error] 33#33: *1010 multi: connect reuse unfinished 00007F7C99101E90

Steps to Reproduce

curl http://127.0.0.1:9180/apisix/admin/upstreams/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "nodes": {
        "10.55.250.95:20880": 1
    },
    "type": "roundrobin"
}'

curl http://127.0.0.1:9180/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uris": [
        "/hello"
    ],
    "plugins": {
        "dubbo-proxy": {
            "service_name": "org.apache.dubbo.samples.tengine.DemoService",
            "service_version": "0.0.0",
            "method": "tengineDubbo"
        }
    },
    "upstream_id": 1
}'

wrk  http://127.0.0.1:9082/hello      -c 100  -d 300 -t 2

Environment

I use the apache/apisix image。below is config

deployment:
  #role: data_plane
  #role_data_plane:
  #  config_provider: yaml
  admin_key_required: false
  admin:
    admin_key:
      - name: admin
        key: edd1c9f034335f136f87ad84b625c8f1  # using fixed API token has security risk, please update it when you deploy to production environment
        role: admin
    allow_admin:                  # Limit Admin API access by IP addresses.
      - 0.0.0.0/0              # If not set, any IP address is allowed.
      # - "::/64"
    admin_listen:                 # Set the Admin API listening addresses.
      ip: 0.0.0.0                 # Set listening IP.
      port: 9180                  # Set listening port. Beware of port conflict with node_listen.
  etcd:
    host:                         # Set etcd address(es) in the same etcd cluster.
      - "http://xxxxxx:2381"   # If TLS is enabled for etcd, use https://127.0.0.1:2379.

apisix:
  control:
   ip: 0.0.0.0
   port: 9090

plugins:
  - dubbo-proxy

plugin_attr:
  dubbo-proxy:
    upstream_multiplex_count: 2


nginx_config:
  worker_processes: 1
  • APISIX version (run apisix version): 3.6.0
  • Operating system (run uname -a): Linux 5878cfa3eb6a 3.10.0-1160.25.1.el7.x86_64 change: added doc of how to load plugin. #1 SMP Wed Apr 28 21:49:45 UTC 2021 x86_64 GNU/Linux
  • OpenResty / Nginx version (run openresty -V or nginx -V): /1.21.4.2
  • etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
  • APISIX Dashboard version, if relevant:
  • Plugin runner version, for issues related to plugin runners:
  • LuaRocks version, for installation issues (run luarocks --version):

dubbo backend:
java -Djava.net.preferIPv4Stack=true -jar dubbo-demo-provider.one-jar.jar

https://github.com/apache/dubbo-samples/tree/master/2-advanced/dubbo-samples-tengine
@Revolyssup Revolyssup moved this to 📋 Backlog in Apache APISIX backlog Nov 1, 2023
@Revolyssup Revolyssup self-assigned this Nov 1, 2023
@shreemaan-abhishek shreemaan-abhishek added the bug Something isn't working label Jan 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Revolyssup Revolyssup

Labels
bug Something isn't working
Projects
Apache APISIX backlog
Status: 📋 Backlog
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Revolyssup @huntkalio @shreemaan-abhishek