Skip to content

FLY-2591: add Codecov coverage reporting - #207

Open
Freshenext wants to merge 2 commits into
mainfrom
feature/FLY-2591
Open

FLY-2591: add Codecov coverage reporting#207
Freshenext wants to merge 2 commits into
mainfrom
feature/FLY-2591

Conversation

@Freshenext

@Freshenext Freshenext commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What

  • Add codecov.yml at the repo root with the same thresholds liquidity-provider-server uses: project target auto with a 1% allowed drop, patch target 80% with a 5% allowed shortfall.
  • CI now runs npm run test:coverage instead of npm run test, then uploads coverage/lcov.info with codecov/codecov-action (SHA-pinned to v5.5.5, matching how this repo pins every action).
  • The upload is gated on a preceding step that checks whether CODECOV_TOKEN is set. Without a token the upload is skipped and the run emits a warning annotation plus a job-summary line, instead of failing the whole job.
  • Add a Codecov badge to the README next to the CI and CodeQL badges.

No change to jest config was needed. jest.config.js already lists lcov in coverageReporters and writes to coverage/, and coverage/ is already in .gitignore.

Why

Integrators consume this SDK directly, so an untested regression reaches third parties instead of stopping inside the squad. npm run test:coverage already existed but nothing read its output. With this change each PR gets a coverage delta comment and the README shows the current number, the same setup LPS has today.

Why the upload is gated

A step-level if: cannot read the secrets context, so the token is surfaced through env on a Check for Codecov token step that writes configured=true|false to $GITHUB_OUTPUT; the upload reads that output.

Two cases need it. The repository is not yet onboarded to Codecov and has no CODECOV_TOKEN secret, so without the gate fail_ci_if_error: true fails every CI run. And pull requests from forks never receive Actions secrets, so that case does not go away once onboarding lands.

fail_ci_if_error: true is kept deliberately. The gate only skips the step when there is no token at all. Once the secret exists, a genuine upload failure still fails the job.

A skipped upload is not silent: it shows as a Coverage not uploaded warning on the checks tab and in the run summary.

How to verify

  • npm ci && npm run test:coverage writes coverage/lcov.info. Verified locally: 39 suites, 357 tests pass, and the file lands at that path.
  • Both YAML files parse (checked with js-yaml).
  • The ignore: list in codecov.yml covers the generated and non-source paths of this repo: src/api/bindings/**, src/blockchain/bindings/**, src/index.ts, **/*.test.ts, integration-test/**, lib/**, scripts/**. It mirrors the existing coveragePathIgnorePatterns in jest.config.js.

QA

Four checks. The first two are the regression this PR fixes; the last two prove the gate did not simply make CI unconditionally green.

1. Confirm the old behaviour failed

Open the CI run for commit 290cdec (the previous head of this branch). Code integrity validation fails, and the failing step is Upload coverage to Codecov, not a test or lint step.

2. Confirm the new behaviour passes, with the skip visible

Open the CI run for the current head. Expect all of:

  • Code integrity validation passes.
  • The step Upload coverage to Codecov is marked skipped, not green. A green upload would mean a token appeared and the gate was never exercised.
  • The run page shows a warning annotation titled Coverage not uploaded.
  • The job summary shows the same message.

3. Exercise the gate logic locally, both branches

git clone git@github.com:rsksmart/flyover-sdk.git
cd flyover-sdk
git checkout feature/FLY-2591
npm ci && npm run build
npm run test:coverage
ls -l coverage/lcov.info    # the file the upload step would send

Save the gate script exactly as CI runs it:

cat > /tmp/gate.sh <<'SCRIPT'
if [ -n "$CODECOV_TOKEN" ]; then
  echo "configured=true" >> "$GITHUB_OUTPUT"
else
  echo "configured=false" >> "$GITHUB_OUTPUT"
  echo "::warning title=Coverage not uploaded::CODECOV_TOKEN is not available, so the Codecov upload was skipped."
  echo ":warning: **Coverage not uploaded**" >> "$GITHUB_STEP_SUMMARY"
fi
SCRIPT

Run it with no token:

export GITHUB_OUTPUT=$(mktemp) GITHUB_STEP_SUMMARY=$(mktemp)
unset CODECOV_TOKEN
bash /tmp/gate.sh
cat "$GITHUB_OUTPUT"          # expect: configured=false
cat "$GITHUB_STEP_SUMMARY"    # expect: the warning line

Run it with a token:

export GITHUB_OUTPUT=$(mktemp) GITHUB_STEP_SUMMARY=$(mktemp)
export CODECOV_TOKEN=dummy-value
bash /tmp/gate.sh
cat "$GITHUB_OUTPUT"          # expect: configured=true
cat "$GITHUB_STEP_SUMMARY"    # expect: empty

configured=true is what makes the upload step run, so this is the path that will be taken once the real secret is added.

4. Confirm CI still fails when it should

The gate must not mask real failures. On a scratch branch, break something and push:

git checkout -b qa/FLY-2591-still-fails
# in any src/**/*.test.ts, change one expected value so the assertion fails
npm run test:coverage         # expect a non-zero exit locally
git commit -am "QA: deliberately failing test, do not merge"
git push origin qa/FLY-2591-still-fails

Open a draft PR from that branch. Code integrity validation must fail at Run unit tests with coverage, before the gate is ever reached. Delete the branch afterwards.

Follow-up, not a blocker

Codecov onboarding for this repository and the CODECOV_TOKEN secret are tracked separately. Until that lands, coverage is computed on every run but not published, and the README badge will read unknown. Adding the secret is the only step needed to turn uploads on; no further change to this workflow is required.

Out of scope, per the ticket: raising the coverage number, and making the Codecov check required for merge.

Related issues

  • Jira: FLY-2591
  • Follow-up: FLY-2600 (Codecov onboarding + CODECOV_TOKEN secret)

Copilot AI lite review requested due to automatic review settings August 11, 2026 03:22
@Freshenext
Freshenext requested a review from a team as a code owner August 11, 2026 03:22
@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

PackageVersionScoreDetails
actions/codecov/codecov-action 0fb7174895f61a3b6b78fc075e0cd60383518dac 🟢 8.2
Details
CheckScoreReason
Code-Review🟢 10all changesets reviewed
Dependency-Update-Tool🟢 10update tool detected
Maintained🟢 810 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 8
Packaging⚠️ -1packaging workflow not detected
Binary-Artifacts🟢 10no binaries found in the repo
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Token-Permissions🟢 9detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies⚠️ 1dependency not pinned by hash detected -- score normalized to 1
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Vulnerabilities🟢 100 existing vulnerabilities detected
Branch-Protection⚠️ -1internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration
Signed-Releases⚠️ -1no releases found
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Security-Policy🟢 10security policy file detected
SAST🟢 9SAST tool detected but not run on all commits
CI-Tests🟢 929 out of 30 merged PRs checked by a CI test -- score normalized to 9
Contributors🟢 10project has 14 contributing companies or organizations

Scanned Files

  • .github/workflows/ci.yml

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Codecov coverage reporting configuration and CI integration so this SDK can publish and track coverage deltas in PRs (and display coverage status in the README), aligning with the setup used in liquidity-provider-server.

Changes:

  • Add codecov.yml with project/patch coverage thresholds and ignore patterns.
  • Update CI to run npm run test:coverage and upload coverage/lcov.info via codecov/codecov-action (SHA-pinned).
  • Add a Codecov badge to README.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
README.md Adds Codecov badge alongside existing CI/CodeQL badges.
codecov.yml Defines Codecov status checks, PR comment behavior, and ignored paths.
.github/workflows/ci.yml Switches CI to run coverage and attempts to upload to Codecov.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/ci.yml
Luisfc68
Luisfc68 previously approved these changes Aug 13, 2026
Comment thread codecov.yml
The upload step ran unconditionally with fail_ci_if_error, so the whole
CI job failed whenever CODECOV_TOKEN was absent. That is the case today,
and it is permanently the case for pull requests from forks, where
Actions secrets are not exposed to the workflow.

Gate the upload on a preceding step that checks the token. When it is
missing the run emits a warning annotation and a job-summary line, so a
skipped upload stays visible instead of passing silently. Keep
fail_ci_if_error, so a real upload failure still fails the job once the
token exists.
Copilot AI review requested due to automatic review settings August 14, 2026 16:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

.github/workflows/ci.yml:45

  • The PR description's “Known blocker” says the Codecov upload step will fail CI until onboarding/secret provisioning, but the workflow now explicitly skips the upload when CODECOV_TOKEN is missing (and only runs the action when configured). Please update the PR description accordingly so reviewers/maintainers aren't misled about merge readiness.
      - name: Check for Codecov token
        id: codecov_token
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
        run: |
          if [ -n "$CODECOV_TOKEN" ]; then
            echo "configured=true" >> "$GITHUB_OUTPUT"
          else
            echo "configured=false" >> "$GITHUB_OUTPUT"
            echo "::warning title=Coverage not uploaded::CODECOV_TOKEN is not available, so the Codecov upload was skipped. Expected on pull requests from forks, where Actions secrets are not exposed; otherwise the repository still needs to be onboarded to Codecov and the secret added."
            echo ":warning: **Coverage not uploaded** — \`CODECOV_TOKEN\` is not available, so the Codecov upload was skipped." >> "$GITHUB_STEP_SUMMARY"
          fi
      - name: Upload coverage to Codecov
        if: steps.codecov_token.outputs.configured == 'true'
        uses: codecov/codecov-action@0fb7174895f61a3b6b78fc075e0cd60383518dac # v5.5.5

.github/workflows/ci.yml:30

  • npm run test:coverage runs jest --coverage --collectCoverageFrom="./src/**" (package.json) and this repo contains many src/**/*.test.ts files, so CI will include test files in the generated lcov report. Since Codecov is configured to ignore **/*.test.ts, the coverage numbers shown by Jest vs. Codecov may diverge, and instrumentation work is wasted on test files. Consider excluding **/*.test.ts (and any other test patterns) in Jest collection (e.g., via jest config or the test:coverage script) so the report and Codecov match.
      - name: Run unit tests with coverage
        run: npm run test:coverage

@Freshenext
Freshenext requested a review from Luisfc68 August 14, 2026 16:40
@Freshenext

Copy link
Copy Markdown
Contributor Author

@Luisfc68 sorry, your approval was auto-dismissed — the main ruleset has dismiss_stale_reviews_on_push, and this branch needed one more commit. Re-requesting.

The only change since you approved is 0494da2, which gates the Codecov upload on whether CODECOV_TOKEN is set. Everything you already reviewed in codecov.yml and the README is untouched, and the 80% patch target stays as agreed with @Freshenext.

Why it was needed: the upload ran unconditionally with fail_ci_if_error: true, so CI failed on every run while the repo has no token, and would keep failing on pull requests from forks even after Codecov onboarding lands, since forks never receive Actions secrets. A step-level if: cannot read the secrets context, hence the extra step that surfaces the token through env and writes an output.

CI is green now, and the gate is verified rather than assumed. On the latest run the Upload coverage to Codecov step reports skipped (not green, so it really was gated), and the run carries a Coverage not uploaded warning annotation — a missing token stays visible instead of looking like a clean pass. fail_ci_if_error: true is kept, so a genuine upload failure still fails the job once the secret exists.

Both review threads are resolved, and the PR description now carries step-by-step QA instructions, including a check that a deliberately failing test still fails CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants