linkcheck inconsistent behavior from client

[fmigneault]

Describe the bug

Running on this project: https://github.com/crim-ca/weaver/

Local Ubuntu 20.04.6 LTS instance, Python 3.10.13 produces :

(         changes: line 1633) broken    https://github.com/geopython/pywps/commit/343d82539576b1e73eee3102654749c3d3137cff - 406 Client Error: Not Acceptable for url: https://github.com/geopython/pywps/commit/343d82539576b1e73eee3102654749c3d3137cff
(autoapi/weaver/base/index: line   72) redirect  https://stackoverflow.com/a/5191224 - with Found to https://stackoverflow.com/questions/5189699/how-to-make-a-class-property/5191224#5191224

On GitHub CI, Python 3.10.13 produces (https://github.com/crim-ca/weaver/actions/runs/7962895777/job/22037962351?pr=604) :

(         changes: line 1633) ok        https://github.com/geopython/pywps/commit/343d82539576b1e73eee3102654749c3d3137cff
(autoapi/weaver/base/index: line   72) broken    https://stackoverflow.com/a/5191224 - 403 Client Error: Forbidden for url: https://stackoverflow.com/a/5191224

Both locations run Sphinx 7.2.6

The errors for the same URLs are replicated over multiple runs, but somehow do not align on different machines.

How to Reproduce

Use the listed URLs in some RST document, and see if they get flagged by linkcheck.
The command make check-links can be used from https://github.com/crim-ca/weaver/ to test exactly the repo where the problem occurs.

Environment Information

Platform:              linux; (Linux-5.15.0-97-generic-x86_64-with-glibc2.31)
Python version:        3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0])
Python implementation: CPython
Sphinx version:        7.2.6
Docutils version:      0.18.1
Jinja2 version:        3.1.2
Pygments version:      2.17.1

> pip freeze
alabaster==0.7.13
alembic==1.12.1
amqp==5.2.0
anyascii==0.3.2
argcomplete==3.1.6
asciitree==0.3.3
astor==0.8.1
astroid==2.15.8
attrs==23.1.0
Babel==2.13.1
bandit==1.7.5
Beaker==1.12.1
beautifulsoup4==4.12.2
billiard==4.2.0
boto3==1.29.4
boto3-stubs==1.29.4
botocore==1.32.4
botocore-stubs==1.32.4
bump2version==1.0.1
CacheControl==0.13.1
celery==5.3.5
certifi==2023.11.17
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
cligj==0.7.2
codacy-coverage==1.3.11
colander==1.8.3
coloredlogs==15.0.1
contextlib2==21.6.0
cornice==6.0.1
cornice-swagger @ git+https://github.com/fmigneault/cornice.ext.swagger.git@b66bd4d85f5f5550f39d046742ae9799aee1f153
coverage==7.3.2
cryptography==41.0.5
cwl-upgrader==1.2.10
cwl-utils==0.31
cwltool==3.1.20230906142556
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.7
dnspython==2.4.2
doc8==0.11.2
docformatter==1.5.0
docker==6.1.3
docutils==0.18.1
dparse==0.6.3
duration==1.1.1
esgf-compute-api @ git+https://github.com/ESGF/esgf-compute-api.git@8faf829abb3ba0434715c4db44440cfb9c71b55f
execnet==2.0.2
fasteners==0.19
filelock==3.13.1
fiona==1.9.5
flake8==5.0.4
flynt==1.0.1
future==0.18.3
geotiff==0.2.10
gitdb==4.0.11
GitPython==3.1.40
greenlet==3.0.1
gunicorn==21.2.0
humanfriendly==10.0
humanize==4.8.0
hupper==1.12
idna==3.4
imagesize==1.4.1
importlib-resources==6.1.1
iniconfig==2.0.0
iso8601==2.1.0
isodate==0.6.1
isort==5.12.0
Jinja2==3.1.2
jmespath==1.0.1
json2xml==4.1.0
jsonschema==4.20.0
jsonschema-specifications==2023.11.1
kombu==5.3.4
lazy-object-proxy==1.9.0
lxml==4.9.3
Mako==1.3.0
markdown-it-py==3.0.0
MarkupSafe==2.1.3
mccabe==0.7.0
mdurl==0.1.2
mistune==2.0.5
mock==3.0.5
moto==4.2.14
msgpack==1.0.7
mypy==1.7.0
mypy-boto3-s3==1.29.3
mypy-extensions==1.0.0
networkx==3.2.1
numcodecs==0.12.1
numpy==1.26.2
oauthlib==3.2.2
OWSLib==0.29.3
packaging==21.3
parameterized==0.9.0
Paste==3.7.1
PasteDeploy==3.1.0
path==16.7.1
path.py==12.5.0
pbr==6.0.0
Pint==0.22
plaster==1.1.2
plaster-pastedeploy==1.0.1
platformdirs==4.0.0
pluggy==1.3.0
prompt-toolkit==3.0.41
prov==1.5.1
psutil==5.9.6
py==1.11.0
pycodestyle==2.9.1
pycparser==2.21
pydocstyle==6.3.0
pydot==1.4.2
pyflakes==2.5.0
Pygments==2.17.1
pylint==2.17.7
pylint-per-file-ignores==1.3.2
pylint-quotes==0.2.3
pymongo==4.6.0
pyparsing==3.1.1
pyproj==3.6.1
pyramid==2.0.2
pyramid-beaker==0.8
pyramid-celery @ git+https://github.com/crim-ca/pyramid_celery.git@3d4c5096661176b176cab4b7d31b282317c70301
pyramid-mako==1.1.0
pyramid-rewrite==0.2
pyramid-storage==1.3.0
pytest==6.2.5
pytest-fixture-config==1.7.0
pytest-httpserver==1.0.8
pytest-rerunfailures==12.0
pytest-server-fixtures==1.7.0
pytest-shutil==1.7.0
python-dateutil==2.8.2
pytz==2023.3.post1
pywps==4.6.0
PyYAML==6.0.1
rdflib==7.0.0
referencing==0.31.0
requests==2.31.0
requests-file==1.5.1
requests-magpie @ git+https://github.com/Ouranosinc/requests-magpie@b26ed3c2cee4395c1bd745c7e102a442ba2199ae
responses==0.24.1
restructuredtext-lint==1.4.0
retry==0.9.2
rich==13.7.0
rpds-py==0.13.1
ruamel.yaml==0.17.40
ruamel.yaml.clib==0.2.8
s3transfer==0.7.0
safety==2.3.5
schema-salad==8.4.20231117150958
shapely==2.0.2
shellescape==3.8.1
simplejson==3.19.2
six==1.16.0
smmap==5.0.1
snowballstemmer==2.2.0
soupsieve==2.5
Sphinx==7.2.6
sphinx-argparse==0.4.0
sphinx-autoapi==3.0.0
sphinx-autodoc-typehints==1.25.2
sphinx-paramlinks==0.6.0
sphinx-rtd-theme==1.3.0
sphinxcontrib-applehelp==1.0.7
sphinxcontrib-devhelp==1.0.5
sphinxcontrib-htmlhelp==2.0.4
sphinxcontrib-jquery==4.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.6
sphinxcontrib-redoc==1.6.0
sphinxcontrib-serializinghtml==1.1.9
SQLAlchemy==2.0.23
stevedore==5.1.0
stopit==1.1.2
termcolor==2.3.0
tifffile==2023.9.26
toml==0.10.2
tomli==2.0.1
tomlkit==0.12.3
translationstring==1.4
types-awscrt==0.19.13
types-s3transfer==0.7.0
typing_extensions==4.8.0
tzdata==2023.3
untokenize==0.1.1
urllib3==2.0.7
urlmatch==1.0.1
venusian==3.1.0
vine==5.1.0
waitress==2.1.2
wcwidth==0.2.11
-e git+ssh://git@github-crim/crim-ca/weaver.git@781dac1bb5429beba99580a4f1118224ef46b6a1#egg=weaver
WebOb==1.8.7
websocket-client==1.6.4
WebTest==3.0.0
Werkzeug==3.0.1
wrapt==1.16.0
WSGIProxy==0.2.2
WSGIProxy2==0.5.1
xmltodict==0.13.0
zarr==2.12.0
zope.deprecation==5.0
zope.interface==6.1

Sphinx extensions

https://github.com/crim-ca/weaver/blob/d911c161721c2e25ee4663d34a49d2b218261710/docs/source/conf.py#L54-L68

Additional context

Error seen on crim-ca/weaver#604 (but unrelated changes)
Last run that did not raise any issue by linkcheck as on crim-ca/weaver@d234e11 (ie: https://github.com/crim-ca/weaver/actions/runs/7890099969/job/21545963203)
No apparent change to documentation links between them (crim-ca/weaver@d234e11...cmd-err-handling).

[jayaddison]

Hi @fmigneault - unfortunately I think that this is most likely to be due to stackoverflow choosing to respond differently to HTTP requests depending on the origin. If that's so, then in the short-term may have to adapt by using linkcheck configuration options.

I'm going to convert this issue into a discussion topic since it isn't immediately clear that there is a bug or enhancement request for the Sphinx codebase here - but perhaps we'll discover one as a result.