[Fix-18166] Restore file parameter transfer feature

[loftiest]

Was this PR generated or assisted by AI?

NO

Purpose of the pull request

Fix #18166 – Restore the file parameter transfer feature

Brief change log

• Add TaskFilesTransferUtils to transfer file paramter.
• Integrate upstream file download in PhysicalTaskExecutor#initializeTaskContext.
• Add success hook in PhysicalTaskExecutor#trackTaskExecutorState to trigger output file upload only on SUCCEEDED.
• Update CuringParamsServiceImpl to map input file parameter values as keys, so they can be matched against the workflow varPool output file parameters and resolved from prepareParamsMap.
• Add unit tests for TaskFilesTransferUtils and PhysicalTaskExecutor.

Verify this pull request

This change added tests and can be verified as follows:

• Added unit tests covering download/upload with FILE parameters, including edge cases.
• Added integration test verifying that file transfer and varPool can coexist and work correctly together.
• Manually tested with Shell tasks on a local cluster, confirming downstream task receives the file.

Pull Request Notice

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

[boring-cyborg]

Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)

[sonarqubecloud]

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 60%)

See analysis details on SonarQube Cloud

[ruanwenjun]

We do not recommend using localparams to store file paths, as this can easily lead to issues such as file renaming and may be affected by varpool. Instead, we should add a new switch at the task level to indicate whether the current task’s output should be uploaded, and include a parameter to specify whether to use the output from a particular upstream task.

[loftiest]

We do not recommend using localparams to store file paths, as this can easily lead to issues such as file renaming and may be affected by varpool. Instead, we should add a new switch at the task level to indicate whether the current task’s output should be uploaded, and include a parameter to specify whether to use the output from a particular upstream task.

Thanks for the suggestion! I agree that storing file paths in localParams and coupling with varPool can be fragile, especially with renaming. A dedicated task-level switch would indeed be a more robust long-term solution.

However, the goal of this PR is to restore the file parameter transfer feature, rather than redesign it. The existing FILE parameter mechanism has been used and documented for several versions. Changing it now would be a breaking change and should go through a separate improvement proposal.

I think your idea is great as a new feature/improvement. Could we track that in a new issue? I'd be happy to help design and implement it.

Thanks!