[Bug?]: No candidates found in corporate environment

[ds1371dani]

Self-service

• I'd be willing to implement a fix

Describe the bug

this issue is the same as #5989.

we have a npm registry in our corporate gitlab instance (https://gitlab.****/api/v4/projects/375/packages/npm/)
when i publish a new package to the repository and then try to install the new version in my project i get the following error:

yarn add @******/[email protected]
➤ YN0000: · Yarn 4.4.1
➤ YN0000: ┌ Resolution step
➤ YN0082: │ @******/utils@npm:7.4.4-alpha.14: No candidates found
➤ YN0000: └ Completed in 0s 516ms
➤ YN0000: · Failed with errors in 0s 527ms

this command yarn npm info @******/utils gives:

{
  name: '@******/utils',
  versions: [
    '7.4.4-alpha.10',
    '7.4.4-alpha.11',
    '7.4.4-alpha.12',
    '7.4.4-alpha.13',
    '7.4.4-alpha.14',
    '7.4.4',
    '7.4.5-alpha.11'
  ],
  'dist-tags': {
    latest: '7.4.4-alpha.14'
  },
  version: '7.4.4-alpha.14',
  devDependencies: {
    '@******/api': '^3.1.17',
    '@******/eslint-config-*******': '^2.4.2',
    '@******/prettier-config-*******': '^1.3.2',
    '@******/redux': '^2.16.11',
    '@rollup/plugin-commonjs': '^22.0.2',
    '@rollup/plugin-json': '^4.1.0',
    '@types/history': '^4.7.11',
    '@types/jest': '^28.1.8',
    '@types/nanoevents': '^1.0.0',
    '@types/react': '^18.0.17',
    '@types/react-native': '^0.69.5',
    '@types/react-redux': '^7.1.24',
    '@types/redux': '^3.6.0',
    '@types/shelljs': '^0.8.11',
    '@types/shortid': '^0.0.29',
    eslint: '^8.23.0',
    history: '4.10.1',
    husky: '4.3.8',
    jest: '^29.4.0',
    'lint-staged': '^13.0.3',
    prettier: '^2.7.1',
    'query-string': '^7.1.1',
    react: '^18.2.0',
    'react-redux': '^8.0.2',
    'react-router-dom': '^6.3.0',
    'react-use': '^17.4.0',
    redux: '^4.2.0',
    rollup: '^2.78.1',
    'rollup-plugin-copy': '^3.4.0',
    'rollup-plugin-terser': '^7.0.2',
    'rollup-plugin-typescript2': '^0.33.0',
    'timezone-mock': '^1.3.6',
    'ts-jest': '^29.0.5',
    tslib: '^2.4.0',
    typescript: '^4.8.2',
    yarn: '^1.22.19'
  },
  dependencies: {
    'date-fns': '^2.29.3',
    'eslint-plugin-you-dont-need-momentjs': '^1.6.0',
    i18next: '^23.8.2',
    'jalali-moment': '^3.3.11',
    nanoevents: '^2.0.0',
    'qrcode-generator': '^1.4.4',
    'react-i18next': '^14.0.2',
    shelljs: '^0.8.5',
    shortid: '^2.2.16'
  },
  peerDependencies: {
    '@******/api': '>=3.0.7',
    '@******/redux': '>=2.8.9',
    history: '>=4.10.1 <5.0.0',
    react: '>=18.0.0',
    'react-native': '>=0.63.34 <0.69.0',
    'react-redux': '>=7.2.2',
    'react-router-dom': '>=5.3.0',
    'react-use': '>=17.2.1',
    redux: '>=4.0.5'
  }
}

the content of ~/.yarn/berry/metadata/npm/3fb1ad/gitlab.********/@*****-utils.json:

{
  "metadata": {
    "dist-tags": {
      "latest": "7.4.4-alpha.13"
    },
    "versions": {
      "7.4.4": ...,
      "7.4.4-alpha.10": ...,
      "7.4.4-alpha.11": ...,
      "7.4.5-alpha.11": ...,
      "7.4.4-alpha.12": ...,
      "7.4.4-alpha.13": ...
    }
  },
  "etag": "W/\"12ae32cb1ec02d01eda3581b127c1fee\"",
  "lastModified": "Wed, 28 Aug 2024 13:36:24 GMT"
}

To reproduce

I cannot provide, as this only occurs in my company repo

Environment

System:
  OS: Linux 6.8 Ubuntu 24.04 LTS 24.04 LTS (Noble Numbat)
  CPU: (6) x64 Intel(R) Core(TM) i5-9600K CPU @ 3.70GHz
Binaries:
  Node: 21.7.0 - /tmp/xfs-226a6d5a/node
  Yarn: 4.4.1 - /tmp/xfs-226a6d5a/yarn
  npm: 10.5.0 - ~/.nvm/versions/node/v21.7.0/bin/npm

Additional context

No response

[derHodrig]

I can confirm the issue. It oocurs on self hostet gitlab instances.
@ds1371dani maybe we should test it with a private repository on official gitlab. I am currently on vacation, i could test this next week.

It seems the metadata crawls only the official npm registries but not others, so maybe it also wont work on public repository packages?

Yarn install should also look for a .yarnrc.yml and include the custom npm registry, which is not yet the case as it seems.

[paul-soporan]

Given the fact that the issue only reproduces on GitLab registries, which makes it quite hard for us to recreate the environment, would it be possible for any of you to debug it further using the unminified binary?

If not, would it be possible for you to at least create a test GitLab registry that we could install a package from so that we can debug it ourselves?

Details for debugging (Click to expand)

• You can run yarn set version from sources --no-minify to use an unminified binary from the latest commit.
• After that, you can search for async function getPackageMetadata, which is the function that handles npm registry metadata caching (in the source code it is located in npmHttpUtils).
• In there, you can log various variables that can help us discover what the issue is:
  • You can put it inside an if (structUtils_exports.stringifyIdent(ident2) === '@******/utils') to only have it log calls for that specific package's metadata.
  • You can log variables such as registry, version (maybe even headers).
  • You can also go inside loadPackageMetadataInfoFromDisk and loadPackageMetadataInfoFromNetwork to debug it further.

Basically, Yarn first loads the metadata from disk and puts it inside an in-memory cache (PACKAGE_DISK_METADATA_CACHE).
It then uses it directly if a specific version is requested (7.4.4-alpha.14) and it is found in the cached metadata. Otherwise, it makes a network call (cached in-memory inside PACKAGE_NETWORK_METADATA_CACHE) with the If-None-Match & If-Modified-Since headers.
If the response is 304, it uses the cached metadata. Otherwise, it uses the fresh metadata, adds it to the in-memory on-disk cache (PACKAGE_DISK_METADATA_CACHE), and writes it to disk too.

For the metadata to be stale, something has to go wrong somewhere in those functions.

[arcanis]

I wonder if perhaps the gitlab self-hosted registry returns invalid caching headers. But as Paul said, we either need a reproduction we can run ourselves, or that you dig into it to find out where we miss something (you can run yarn set version from sources --no-minify to install in your project an unminified yarn.js where you can add some console.log statements).

[ds1371dani]

@paul-soporan I tried debugging as you said, inside loadPackageMetadataInfoFromNetwork > wrapNetworkRequest the response return statusCode = 304 with an empty response.body so it returns the cached values on the system instead of server. it seems this should be fixed in gitlab instead of yarn, any suggestions on where to create an issue?

is it possible to force yarn to receive the metadata from server every time for the time being?