bug in Sniffing #3289

Alireza78na · 2024-04-19T09:29:32Z

There is an issue with the Sniffing section that causes software like Spotify or some Google APIs to not route properly. For example, I have configured Xray to route geosite:Spotify traffic through Warp, but still a large portion of the APIs related to Spotify are not routing correctly.

I'm confident that the geosite file is complete and correct because enabling fake DNS resolves all issues, but unfortunately using fake DNS for all domains causes other problems.

Note that these issues have arisen after updating to the latest version and did not exist in previous versions.

Please look into this issue.

Fangliding · 2024-04-19T10:58:23Z

provid config, log
and the last version that works properly

Alireza78na · 2024-04-19T12:47:22Z

log file:
error.log

server Config:

{
  "api": {
    "services": [
      "HandlerService",
      "LoggerService",
      "StatsService"
    ],
    "tag": "api"
  },
  "inbounds": [
    {
      "listen": "127.0.0.1",
      "port": 62789,
      "protocol": "dokodemo-door",
      "settings": {
        "address": "127.0.0.1"
      },
      "tag": "api"
    }
  ],
  "log": {
    "access": "none",
    "dnsLog": false,
    "error": "none",
    "loglevel": "warning"
  },
  "outbounds": [
    {
      "protocol": "freedom",
      "settings": {},
      "tag": "direct"
    },
    {
      "protocol": "blackhole",
      "settings": {},
      "tag": "blocked"
    },
    {
      "tag": "warp",
      "protocol": "wireguard",
      "settings": {
        "mtu": 1420,
        "secretKey": "***********************************",
        "address": [
          "********"
        ],
        "workers": 2,
        "peers": [
          {
            "publicKey": "*******************************",
            "allowedIPs": [
              "0.0.0.0/0",
              "::/0"
            ],
            "endpoint": "************************",
            "keepAlive": 20
          }
        ],
        "kernelMode": false
      }
    },
    {
      "tag": "IPv4",
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "UseIPv4"
      }
    }
  ],
  "policy": {
    "levels": {
      "0": {
        "statsUserDownlink": true,
        "statsUserUplink": true
      }
    },
    "system": {
      "statsInboundDownlink": true,
      "statsInboundUplink": true,
      "statsOutboundDownlink": true,
      "statsOutboundUplink": true
    }
  },
  "routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [
      {
        "inboundTag": [
          "api"
        ],
        "outboundTag": "api",
        "type": "field"
      },
      {
        "ip": [
          "geoip:private"
        ],
        "outboundTag": "blocked",
        "type": "field"
      },
      {
        "outboundTag": "blocked",
        "protocol": [
          "bittorrent"
        ],
        "type": "field"
      },
      {
        "type": "field",
        "outboundTag": "blocked",
        "domain": [
          "geosite:speedtest"
        ]
      },
      {
        "type": "field",
        "domain": [
          "geosite:netflix"
        ],
        "outboundTag": "IPv4"
      },
      {
        "type": "field",
        "outboundTag": "warp",
        "domain": [
          "geosite:spotify"
        ]
      }
    ]
  },
  "stats": {},
  "dns": {
    "servers": [
      "94.140.14.14",
      "94.140.15.15",
      "2a10:50c0::ad1:ff",
      "2a10:50c0::ad2:ff"
    ],
    "queryStrategy": "UseIP",
    "tag": "dns_inbound"
  },
  "fakedns": null
}

last working version: 1.8.9

i should say that in the web browsing we usually dont have a problem for spotify etc.
the problem appears when we use its Apps in Android , IOS , Windows etc.
i use VMESS WS

Fangliding · 2024-04-19T12:51:45Z

I didn't see any valid inbound in your config

Alireza78na · 2024-04-19T12:58:30Z

I'm sorry, but what do you mean by config? I sent the server config. Do you mean the client config for connecting? Please explain a bit so I don't send the wrong files. Thank you.

Fangliding · 2024-04-19T12:59:43Z

Threre is only one inbound in your config and It's an api-in

mmmray · 2024-04-19T13:24:05Z

can you try blocking udp port 443? spotify uses quic, and I think the sniffer does not work on quic at all (see #2616)

I am not sure why/how fakedns works, but I suppose since it connects dns lookups with udp/tcp traffic, it can do better sniffing.

Alireza78na · 2024-04-19T13:25:10Z

I think its because of using 3-xui panel.
I don't use xray manually and I use MHSanaei panel. Maybe this is the reason why this is happening.
In any case, I am ready to build a server with a similar configuration if necessary and provide it to you for further investigation.

Alireza78na · 2024-04-19T13:28:35Z

can you try blocking udp port 443? spotify uses quic, and I think the sniffer does not work on quic at all (see #2616)

I am not sure why/how fakedns works, but I suppose since it connects dns lookups with udp/tcp traffic, it can do better sniffing.

Thanks, i will try it and let you know the result here

mmmray · 2024-04-19T13:28:59Z

ok, i just saw that the config you posted doesn't have a sniffing config at all...

if it's not possible to extract the generated xray config from 3x-ui (including a complete inbound section) then it is impossible to have a productive conversation in the xray-core issue tracker. i suggest to investigate how to extract a full server config from 3x-ui, or ask on their support channels

Alireza78na · 2024-04-19T13:47:54Z

can you try blocking udp port 443? spotify uses quic, and I think the sniffer does not work on quic at all (see #2616)

I am not sure why/how fakedns works, but I suppose since it connects dns lookups with udp/tcp traffic, it can do better sniffing.

I tried this , but nothing solved for me.
I blocked 443/udp in firewall for inbound and even outbound , but still problem not solved.

Also i will try to get full config from 3x-ui and send it here

Alireza78na · 2024-04-19T13:55:51Z

I should add this that many people also have my problem and We have repeatedly requested this matter from the panel creators, but a sufficient solution was not provided, and ultimately, we were referred to you. This issue is important because services like Spotify regularly block server IPs, and we cannot constantly change server IPs. The only solution is to use Warp
When all traffic passes through Warp and we do not use sniffing, the issue is resolved. However, as you know, using it in this way significantly reduces the connection quality.

mmmray · 2024-04-19T14:04:41Z

We have repeatedly requested this matter from the panel creators, but a sufficient solution was not provided, and ultimately, we were referred to you.

the separation of responsibilities is important, and what exactly you are asking of them is important.

3x-ui is not responsible for supporting sniffing, but they are responsible IMO for providing a way to debug xray-core issue

if they don't have a simple way to get the full server config, xray-core developers will have to understand 3x-ui/marzban and debug it

Fangliding · 2024-04-19T14:05:10Z

If the problem only exists in 1.8.10, this may indeed be a bug
To be honest, it's difficult to find valid information in the logs you provided
It's best for you to write a config file, restart xray, and access Spotify once to reduce invalid information in the log
Just a sentence, "I can't use Spotify", we can hardly help you

us254 · 2024-04-21T04:57:45Z

It could be due to the "IPIfNonMatch" domain strategy, where if the IP is already known (from cache or previous resolution), it takes precedence over the domain rule, See if changing the domain strategy to "AsIs" makes a difference in the routing decisions.

Alireza78na · 2024-04-21T08:03:07Z

It could be due to the "IPIfNonMatch" domain strategy, where if the IP is already known (from cache or previous resolution), it takes precedence over the domain rule, See if changing the domain strategy to "AsIs" makes a difference in the routing decisions.

Thank you for your guidance, but this method didn't work. In fact, the settings were like this from the beginning.

Alireza78na · 2024-04-21T08:09:45Z

If the problem only exists in 1.8.10, this may indeed be a bug To be honest, it's difficult to find valid information in the logs you provided It's best for you to write a config file, restart xray, and access Spotify once to reduce invalid information in the log Just a sentence, "I can't use Spotify", we can hardly help you

Thank you for your patience and guidance. I created a separate server for testing and also prepared the debug log and config.json files.

I'm sending them here. If needed, I can give you server access to personally review them.

I should also mention that on this server, since the IP hasn't been blocked by Spotify yet, everything works correctly. However, I don't think it will affect the Sniffing investigation process.

If I mistakenly sent something wrong again, please let me know.

Log:
error.log

Config:
config.json

RPRX · 2024-04-21T20:08:29Z

Note that these issues have arisen after updating to the latest version and did not exist in previous versions.

v1.8.10 似乎没有修改 sniffer 相关代码

Alireza78na · 2024-04-22T14:17:22Z

Note that these issues have arisen after updating to the latest version and did not exist in previous versions.

v1.8.10 似乎没有修改 sniffer 相关代码

so maybe we have some issue in DNS, because my problem solved with this solution: #1019issue

please check it

chise0713 · 2024-04-24T19:31:20Z

It could be due to the "IPIfNonMatch" domain strategy, where if the IP is already known (from cache or previous resolution), it takes precedence over the domain rule, See if changing the domain strategy to "AsIs" makes a difference in the routing decisions.

I've been wondering, is your understanding of the logic of IPIfNonMatch correct?

Because the document and the source code seems shows that IPIfNonMatch is not work as what you tought.
There is no "from cache or previous resolution", it is always use domain to match first, if it doesn't match any of domain rules, it will resovle domain to IP address then match rules again.

us254 · 2024-04-24T21:31:27Z

It could be due to the "IPIfNonMatch" domain strategy, where if the IP is already known (from cache or previous resolution), it takes precedence over the domain rule, See if changing the domain strategy to "AsIs" makes a difference in the routing decisions.

I've been wondering, is your understanding of the logic of IPIfNonMatch correct?

Because the document and the source code seems shows that IPIfNonMatch is not work as what you tought. There is no "from cache or previous resolution", it is always use domain to match first, if it doesn't match any of domain rules, it will resovle domain to IP address then match rules again.

you are correct. Thank you for bringing this to my attention

RPRX · 2024-04-26T03:35:18Z

Note that these issues have arisen after updating to the latest version and did not exist in previous versions.

v1.8.10 似乎没有修改 sniffer 相关代码

so maybe we have some issue in DNS, because my problem solved with this solution: #1019issue

please check it

v1.8.10 似乎也没有修改 dns 相关代码，请逐个测试近期的 commits，看一下哪个 commit 出现了你的问题

@Fangliding 等会儿在 issue 模板中加一下吧，如果新版有 bug 需先测出是哪个 commit 的问题再开 issue

pulsarice · 2024-04-26T18:09:26Z

I did some testing on spotify app on android.
Spotify has blocked my server's IP address, so I have to route spotify.com to WARP
(Quic protocol is disabled in both cases.)

First without fakeDNS (Spotify doesn't work)

accepted tcp:graph.facebook.com:443 [inbound -> warp] 
accepted tcp:graph.facebook.com:443 [inbound -> warp] 
accepted tcp:clienttoken.spotify.com:443 [inbound -> warp] 
accepted tcp:apresolve.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-dealer.spotify.com:443 [inbound -> warp] 
accepted tcp:relaycdn.anchor.fm:443 [inbound >> direct] 
accepted tcp:34.158.0.131:4070 [inbound >> direct] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-dealer.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp]

Now with fakeDNS enabled (Spotify works)

accepted tcp:graph.facebook.com:443 [inbound -> warp] 
accepted tcp:graph.facebook.com:443 [inbound -> warp] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:apresolve.spotify.com:443 [inbound -> warp] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:apresolve.spotify.com:443 [inbound -> warp] 
accepted tcp:relaycdn.anchor.fm:443 [inbound >> direct] 
accepted tcp:login5.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-dealer.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:ap-gew4.spotify.com:4070 [inbound -> warp] 
accepted tcp:gew4-dealer.spotify.com:443 [inbound -> warp] 
accepted tcp:image-cdn-ak.spotifycdn.com:443 [inbound >> direct]

The difference is that tcp:34.158.0.131:4070 is now tcp:ap-gew4.spotify.com:4070
It seems that sniffing didn't catch that. Is it because of a possible non-standard port/protocol used by spotify app?

mmmray · 2024-04-28T23:39:12Z

I just took a look with wireshark. Yes, from what I can tell port 4070 is an entirely proprietary protocol, and xray simply does not have support for sniffing a domain name from it.

i don't think there's a bug, only perhaps the absence of a feature. Correct me if I'm wrong, but xray's "geosite" support requires domain name sniffing to apply, and figuring out the domain name from a tcp connection is either a per-protocol thing to support (http, tls, ...), or something to bypass entirely with the big hammer of fakedns.

I think you can fix it with fakedns as you did here or write a rule that simply routes the entire IP 34.158.0.131 to warp. The IP belongs to Google, but since it accepts a custom protocol on a custom port, I find it very unlikely that spotify shares it with any other google cloud customer.

pulsarice · 2024-04-29T08:17:31Z

write a rule that simply routes the entire IP 34.158.0.131 to warp

Or blindly route all traffic with destination port of 4070 to warp :)) Is any other known service using that port number?
Thank you for investigating the matter.

yuhan6665 · 2024-04-29T14:54:22Z

Thanks for the great investigation @pulsarice @mmmray

Fangliding · 2024-04-29T14:56:38Z

Wait, why these did not happen in 1.8.9

mmmray · 2024-04-29T14:58:29Z

Is any other known service using that port number?

No idea, but it seems much more likely, statistically. I suggest to add the IP to geosite repo 🤷‍♂️

Wait, why these did not happen in 1.8.9

No idea. Did anybody actually try downgrading? I wonder if spotify simply updated around the time xray 1.8.10 was released.

Fangliding · 2024-04-29T15:01:57Z

@mmmray GEOSITE only contains domain, no IP
and maybe you can try to use 1.8.9 and see how this “protocol 4070” be routed

yuhan6665 · 2024-04-29T15:02:41Z

Wait, why these did not happen in 1.8.9

There is no change on our side, most likely user don't know/bother to test each version (I think I cannot guarantee to use binary search to find the problematic version for an issue as well)
This is the essence of any large product support, you are diving in mud water to find thing ;)
On the bright side, it make me twice as happy to see people dig the gold (root cause) from mud

mmmray · 2024-04-29T15:05:47Z

GEOSITE only contains domain, no IP

I wonder, can it be changed? I think supporting IPs in geosite file format would help a lot with proprietary protocols.

I do not have a working xray routing setup at all at the moment, I only looked at xray docs and spotify traffic, if nobody else does it i can do it next week ~~or i forget about it entirely~~

yuhan6665 · 2024-04-29T15:10:32Z

There is a separate project for IP https://github.com/Loyalsoldier/geoip
Although I understand the goal of this project is to sort based on geolocation. It might not be realistic to have more categories

pulsarice · 2024-04-29T15:11:59Z

Wait, why these did not happen in 1.8.9

It started happening to me many months ago, after a specific version update of spotify app. I actually downgraded my spotify app back then and can confirm that older version worked.

Fangliding · 2024-04-29T15:12:26Z

GEOSITE only contains domain, no IP

I wonder, can it be changed? I think supporting IPs in geosite file format would help a lot with proprietary protocols.

I do not have a working xray routing setup at all at the moment, I only looked at xray docs and spotify traffic, if nobody else does it i can do it next week ~~or i forget about it entirely~~

Some special IP ranges may be contained by some third-party geoip
However, the IP is different from the domain, and due to the existence of services such as CDN and balancers, services on one IP are likely to be frequently replaced

Fangliding · 2024-04-29T15:15:30Z

Wait, why these did not happen in 1.8.9

It started happening to me many month ago, after a specific version update of spotify app. I actually downgraded my spotify app back then and can confirm that older version worked.

So nobody sure if it's working correct on 1.8.9.............
Damn I need add this to template....
Perhaps one day it will become as complex as aviation safety regulations.
Ordinary people think they are outrageous, but there are even more outrageous things behind them

pulsarice · 2024-04-29T15:17:48Z

So you're not sure if it's working correct on 1.8.9

v1.8.9 wasn't even conceived when this started happening to me.

Fangliding · 2024-04-29T15:19:09Z

So you're not sure if it's working correct on 1.8.9

v1.8.9 wasn't even conceived when this started happening to me.

I mean the author of this issue, I read the wrong person
Edited

Kiya6955 · 2024-07-04T13:20:26Z

I just took a look with wireshark. Yes, from what I can tell port 4070 is an entirely proprietary protocol, and xray simply does not have support for sniffing a domain name from it.

i don't think there's a bug, only perhaps the absence of a feature. Correct me if I'm wrong, but xray's "geosite" support requires domain name sniffing to apply, and figuring out the domain name from a tcp connection is either a per-protocol thing to support (http, tls, ...), or something to bypass entirely with the big hammer of fakedns.

I think you can fix it with fakedns as you did here or write a rule that simply routes the entire IP 34.158.0.131 to warp. The IP belongs to Google, but since it accepts a custom protocol on a custom port, I find it very unlikely that spotify shares it with any other google cloud customer.

This ip was working for me till past week.
This week i had problem again and i found new ip address 35.186.224.28.
Dst ports was different so i decided to route whole traffic of this ip behind Warp.
Now im using

      {
        "outboundTag": "warp",
        "ip": [
          "35.186.224.28"
        ],
        "type": "field"
      },
      {
        "outboundTag": "warp",
        "domain": [
          "geosite:spotify"
        ],
        "type": "field"
      }

pulsarice · 2024-07-04T17:50:34Z

This ip was working for me till past week. This week i had problem again and i found new ip address 35.186.224.28. Dst ports was different so i decided to route whole traffic of this ip behind Warp. Now im using
      {
        "outboundTag": "warp",
        "ip": [
          "35.186.224.28"
        ],
        "type": "field"
      },
      {
        "outboundTag": "warp",
        "domain": [
          "geosite:spotify"
        ],
        "type": "field"
      }

I simply routed port 4070 and spotify.com domain to warp and haven't had any problems since:

  {
    "type": "field",
    "domain": [
      "domain:spotify.com"
    ],
    "outboundTag": "warp"
  },
  {
    "type": "field",
    "port": "4070",
    "outboundTag": "warp"
  }

yuhan6665 closed this as completed Apr 29, 2024

mmrabbani mentioned this issue Oct 21, 2024

Warp routing for Spotify is not working well alireza0/x-ui#1492

Closed

bug in Sniffing #3289

bug in Sniffing #3289

Comments

Alireza78na commented Apr 19, 2024

Fangliding commented Apr 19, 2024

Alireza78na commented Apr 19, 2024 • edited Loading

Fangliding commented Apr 19, 2024

Alireza78na commented Apr 19, 2024

Fangliding commented Apr 19, 2024

mmmray commented Apr 19, 2024

Alireza78na commented Apr 19, 2024

Alireza78na commented Apr 19, 2024

mmmray commented Apr 19, 2024

Alireza78na commented Apr 19, 2024

Alireza78na commented Apr 19, 2024

mmmray commented Apr 19, 2024

Fangliding commented Apr 19, 2024 • edited Loading

us254 commented Apr 21, 2024 • edited Loading

Alireza78na commented Apr 21, 2024

Alireza78na commented Apr 21, 2024

RPRX commented Apr 21, 2024

Alireza78na commented Apr 22, 2024

chise0713 commented Apr 24, 2024

us254 commented Apr 24, 2024

RPRX commented Apr 26, 2024

pulsarice commented Apr 26, 2024

mmmray commented Apr 28, 2024 • edited Loading

pulsarice commented Apr 29, 2024

yuhan6665 commented Apr 29, 2024

Fangliding commented Apr 29, 2024

mmmray commented Apr 29, 2024

Fangliding commented Apr 29, 2024

yuhan6665 commented Apr 29, 2024

mmmray commented Apr 29, 2024

yuhan6665 commented Apr 29, 2024

pulsarice commented Apr 29, 2024 • edited Loading

Fangliding commented Apr 29, 2024

Fangliding commented Apr 29, 2024 • edited Loading

pulsarice commented Apr 29, 2024

Fangliding commented Apr 29, 2024

Kiya6955 commented Jul 4, 2024 • edited Loading

pulsarice commented Jul 4, 2024

Alireza78na commented Apr 19, 2024 •

edited

Loading

Fangliding commented Apr 19, 2024 •

edited

Loading

us254 commented Apr 21, 2024 •

edited

Loading

mmmray commented Apr 28, 2024 •

edited

Loading

pulsarice commented Apr 29, 2024 •

edited

Loading

Fangliding commented Apr 29, 2024 •

edited

Loading

Kiya6955 commented Jul 4, 2024 •

edited

Loading