[SPARK-51980][PYTHON][TESTS][3.5] Enable --use-pep517 in dev/run-pip-tests

### What changes were proposed in this pull request?
This pr enable `--use-pep517` option in `dev/run-pip-tests` refer to:

```
25l  DEPRECATION: Legacy editable install of pyspark==4.1.0.dev0 from file:///__w/spark/spark/python/packaging/classic (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at pypa/pip#11457
```

### Why are the changes needed?
To restore Github Action test failed:

- https://github.com/apache/spark/actions/runs/14778042174/job/41496865818

```
Installing collected packages: py4j, pyspark
25l  DEPRECATION: Legacy editable install of pyspark==4.1.0.dev0 from file:///__w/spark/spark/python/packaging/classic (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at pypa/pip#11457

  Running setup.py develop for pyspark

   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]    error: subprocess-exited-with-error

    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [74 lines of output]
        /tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
        !!

                ********************************************************************************
                Please consider removing the following classifiers in favor of a SPDX license expression:

                License :: OSI Approved :: Apache Software License

                See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
                ********************************************************************************

        !!
          self._finalize_license_expression()
        running develop
        /tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:90: DevelopDeprecationWarning: develop command is deprecated.
        !!

                ********************************************************************************
                Please avoid running ``setup.py`` and ``develop``.
                Instead, use standards-based tools like pip or uv.

                By 2025-Oct-31, you need to update your project and remove deprecated calls
                or your builds will no longer be supported.

                See pypa/setuptools#917 for details.
                ********************************************************************************

        !!
          self.initialize_options()
        Obtaining file:///__w/spark/spark/python
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
          Checking if build backend supports build_editable: started
          Checking if build backend supports build_editable: finished with status 'done'
          Getting requirements to build editable: started
          Getting requirements to build editable: finished with status 'error'
          error: subprocess-exited-with-error

          × Getting requirements to build editable did not run successfully.
          │ exit code: 255
          ╰─> [1 lines of output]
              Temp path for symlink to parent already exists deps
              [end of output]

          note: This error originates from a subprocess, and is likely not a problem with pip.
        error: subprocess-exited-with-error

        × Getting requirements to build editable did not run successfully.
        │ exit code: 255
        ╰─> See above for output.

        note: This error originates from a subprocess, and is likely not a problem with pip.
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 35, in <module>
          File "/__w/spark/spark/python/packaging/classic/setup.py", line 254, in <module>
            setup(
          File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
            return distutils.core.setup(**attrs)
          File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 186, in setup
            return run_commands(dist)
          File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
            dist.run_commands()
          File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
            self.run_command(cmd)
          File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/dist.py", line 1104, in run_command
            super().run_command(command)
          File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
            cmd_obj.run()
          File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/command/develop.py", line 39, in run
            subprocess.check_call(cmd)
          File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/subprocess.py", line 373, in check_call
            raise CalledProcessError(retcode, cmd)
        subprocess.CalledProcessError: Command '['/tmp/tmp.zGt6QETKsu/3.9/bin/python3.9', '-m', 'pip', 'install', '-e', '.', '--use-pep517', '--no-deps']' returned non-zero exit status 1.
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [74 lines of output]
    /tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
    !!

            ********************************************************************************
            Please consider removing the following classifiers in favor of a SPDX license expression:

            License :: OSI Approved :: Apache Software License

            See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
            ********************************************************************************

    !!
      self._finalize_license_expression()
    running develop
    /tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:90: DevelopDeprecationWarning: develop command is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` and ``develop``.
            Instead, use standards-based tools like pip or uv.

            By 2025-Oct-31, you need to update your project and remove deprecated calls
            or your builds will no longer be supported.

            See pypa/setuptools#917 for details.
            ********************************************************************************

    !!
      self.initialize_options()
    Obtaining file:///__w/spark/spark/python
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Checking if build backend supports build_editable: started
      Checking if build backend supports build_editable: finished with status 'done'
      Getting requirements to build editable: started
      Getting requirements to build editable: finished with status 'error'
      error: subprocess-exited-with-error

      × Getting requirements to build editable did not run successfully.
      │ exit code: 255
      ╰─> [1 lines of output]
          Temp path for symlink to parent already exists deps
          [end of output]

      note: This error originates from a subprocess, and is likely not a problem with pip.
    error: subprocess-exited-with-error

    × Getting requirements to build editable did not run successfully.
    │ exit code: 255
    ╰─> See above for output.

    note: This error originates from a subprocess, and is likely not a problem with pip.
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 35, in <module>
      File "/__w/spark/spark/python/packaging/classic/setup.py", line 254, in <module>
        setup(
      File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
        return distutils.core.setup(**attrs)
      File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 186, in setup
        return run_commands(dist)
      File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
        dist.run_commands()
      File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
        self.run_command(cmd)
      File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/dist.py", line 1104, in run_command
        super().run_command(command)
      File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
        cmd_obj.run()
      File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/command/develop.py", line 39, in run
        subprocess.check_call(cmd)
      File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/subprocess.py", line 373, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/tmp/tmp.zGt6QETKsu/3.9/bin/python3.9', '-m', 'pip', 'install', '-e', '.', '--use-pep517', '--no-deps']' returned non-zero exit status 1.
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
   ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
25h
Cleaning up temporary directory - /tmp/tmp.zGt6QETKsu
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #50861 from LuciferYang/SPARK-51980-3.5.

Authored-by: yangjie01 <[email protected]>
Signed-off-by: yangjie01 <[email protected]>

Author: LuciferYang

dev/run-pip-tests

@@ -70,7 +70,7 @@ fi
 PYSPARK_VERSION=$(python3 -c "exec(open('python/pyspark/version.py').read());print(__version__)")
 PYSPARK_DIST="$FWDIR/python/dist/pyspark-$PYSPARK_VERSION.tar.gz"
 # The pip install options we use for all the pip commands
-PIP_OPTIONS="--upgrade --no-cache-dir --force-reinstall"
+PIP_OPTIONS="--upgrade --no-cache-dir --force-reinstall --use-pep517"
 # Test both regular user and edit/dev install modes.
 PIP_COMMANDS=("pip install $PIP_OPTIONS $PYSPARK_DIST"
 	      "pip install $PIP_OPTIONS -e python/")