Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion dolphinscheduler-master/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,43 @@ Exposed interfaces:

## Tests

`src/test/java` — unit + integration. Integration tests extend `AbstractMasterIntegrationTestCase`; they simulate distributed scenarios including failover.
`src/test/java` — unit + integration. Integration tests extend `AbstractMasterIntegrationTestCase` (a `@SpringBootTest`) and live under `server/master/integration/cases/`; they simulate distributed scenarios including failover.

### Running the suite

From the repo root:

```bash
# Whole module (unit + integration). Use `clean` to avoid the JaCoCo
# "Cannot process instrumented class" failure when classes from a previous
# build are still on disk.
./mvnw -pl dolphinscheduler-master -am clean test

# Single test class. -Dsurefire.failIfNoSpecifiedTests=false is required
# whenever -am pulls in upstream modules: the -Dtest filter applies to
# every reactor module, and surefire fails the build on any module that
# has zero matches without this flag.
./mvnw -pl dolphinscheduler-master -am clean test \
-Dtest=WorkerGroupDispatcherTest \
-Dsurefire.failIfNoSpecifiedTests=false

# Single method
./mvnw -pl dolphinscheduler-master -am clean test \
-Dtest=WorkerGroupDispatcherTest#dispatch \
-Dsurefire.failIfNoSpecifiedTests=false
```

### Local environment

- **No Docker required.** Integration tests boot Spring Boot against an in-memory H2 (`spring-it-application.yaml`), with a fake registry — Testcontainers is not used here. The repo-wide `-Dm1_chip=true` flag only applies to `dolphinscheduler-api-test` / `dolphinscheduler-e2e`; ignore it when running this module.
- **Surefire forks 4 JVMs in parallel** (`forkCount=4`, `reuseForks=true`, see root `pom.xml`). Tests must therefore be parallel-safe — never share static mutable state across cases. Per-fork JaCoCo files land at `target/jacoco-${forkNumber}.exec`.
- The `Surefire is going to kill self fork JVM. The exit has elapsed 30 seconds after System.exit(0).` line at the end is **a harmless warning** — `BUILD SUCCESS` on the same run is the real signal. It just means a fork held a non-daemon thread past `System.exit`; do not chase it unless the build itself fails.

### When tests fail

1. Re-run with `clean` first — the JaCoCo error above and stale generated sources both vanish after a clean rebuild.
2. For integration tests, check `target/surefire-reports/*-output.txt` per fork — exceptions from the embedded master/H2 are logged there, not on stdout.
3. New lifecycle events / failover paths must be exercised against `AbstractMasterIntegrationTestCase` (see the gotcha above) — extend an existing case under `integration/cases/` rather than writing a bare unit test.

## Related modules

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@
import org.apache.dolphinscheduler.eventbus.AbstractDelayEventBus;
import org.apache.dolphinscheduler.server.master.engine.task.dispatcher.event.TaskDispatchableEvent;

import lombok.SneakyThrows;

public class TaskDispatchableEventBus<V extends TaskDispatchableEvent<T>, T extends Comparable<T>>
extends
AbstractDelayEventBus<V> {
Expand All @@ -30,8 +28,8 @@
super.publish(v);
}

@SneakyThrows
public V take() {
@Override
public V take() throws InterruptedException {

Check warning on line 32 in dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/task/dispatcher/TaskDispatchableEventBus.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Remove this method to simply inherit it.

See more on https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&issues=AZ4NUlmpAFXVgHJS620C&open=AZ4NUlmpAFXVgHJS620C&pullRequest=18240
return super.take();
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,14 @@ public synchronized void start() {
@Override
public void run() {
while (runningFlag.get()) {
TaskDispatchableEvent<ITaskExecution> taskEntry = workerGroupEventBus.take();
final TaskDispatchableEvent<ITaskExecution> taskEntry;
try {
taskEntry = workerGroupEventBus.take();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
log.info("{} interrupted, exiting run loop", this.getName());
return;
}
ITaskExecution taskExecution = taskEntry.getData();
try (
TaskExecutorMDCUtils.MDCAutoClosable ignore =
Expand Down Expand Up @@ -123,9 +130,9 @@ private void doDispatchTask(ITaskExecution taskExecution) {

// If dispatch failed, will put the task back to the queue
// The task will be dispatched after waiting time.
// the waiting time will increase multiple of times, but will not exceed 60 seconds
long waitingTimeMillis = Math.min(
taskExecution.getTaskExecutionContext().increaseDispatchFailTimes() * 1_000L, 60_000L);
// the waiting time grows with the fail count, bounded between 1 and 60 seconds
long waitingTimeMillis = Math.max(1_000L, Math.min(
taskExecution.getTaskExecutionContext().increaseDispatchFailTimes() * 1_000L, 60_000L));
dispatchTask(taskExecution, waitingTimeMillis);
log.warn("Dispatch Task: {} failed will retry after: {}/ms", taskInstanceId,
waitingTimeMillis, ex);
Expand Down
Loading