jd-client vardiff runaway: share counter increments before validation

[warioishere]

Bug Description

JD-Client's vardiff continuously increases difficulty even when miners fail to submit valid shares due to difficulty being too high. This creates a runaway loop where difficulty spirals upward.

Root Cause

In miner-apps/jd-client/src/lib/channel_manager/downstream_message_handler.rs at line 972, the vardiff share counter is incremented before share validation:

vardiff.increment_shares_since_last_update();  // Line 972 - BEFORE validation
let res = standard_channel.validate_share(msg.clone());  // Line 973
match res {
    Ok(ShareValidationResult::Valid(share_hash)) => { ... }
    Ok(ShareValidationResult::Invalid(e)) => { ... }
    Err(e) => { ... }
}

This means:

1. Miner submits a share (even if difficulty is too low to meet the target)
2. Vardiff counter increments unconditionally
3. Share validation happens
4. Share is rejected (doesn't meet target)
5. Vardiff thinks the share was successful
6. On next vardiff cycle (60s), it calculates: high share rate → hashrate must be high → increase difficulty
7. Difficulty increases further, making the problem worse

Expected Behavior

The share counter should only increment for valid shares that pass validation. The increment should move inside the Ok(ShareValidationResult::Valid(...)) block (line 976).

Observed Behavior

• Vardiff continuously increases difficulty
• Miner fails to submit shares (difficulty too high)
• Failed shares still count toward vardiff share rate
• Difficulty increases further
• Loop continues until difficulty is impossibly high

Reproduction

1. Connect a miner to jd-client
2. Wait for vardiff to increase difficulty beyond miner's capability
3. Observe miner failing to submit valid shares
4. Watch vardiff continue to increase difficulty instead of reducing it

Affected Code

File: miner-apps/jd-client/src/lib/channel_manager/downstream_message_handler.rs
Lines: 972 (standard channel), 1188 (extended channel - same issue)

Vardiff logic: stratum/sv2/channels-sv2/src/vardiff/classic.rs
Line 170: When realized_share_per_min == 0.0, it reduces hashrate estimate (easier difficulty), but this never triggers because failed shares still increment the counter.

Suggested Fix

Move vardiff.increment_shares_since_last_update() to after validation succeeds:

let res = standard_channel.validate_share(msg.clone());
let mut is_downstream_share_valid = false;
match res {
    Ok(ShareValidationResult::Valid(share_hash)) => {
        vardiff.increment_shares_since_last_update();  // Only count valid shares
        // ... rest of success handling
    }
    // ... error cases
}

Same fix needed for extended channels at line 1188.

Impact

This bug makes jd-client's vardiff unusable for miners with variable hashrate or when difficulty temporarily exceeds miner capability. Once difficulty gets too high, it never corrects downward.

[warioishere]

Bitaxe Already at 13K, if I let it go for a few hours, I already saw diff over 1M

[warioishere]

thats weird, now I have it ah 1

maybe thatswhy it later jumps to M or more.

[GitGab19]

Vardiff mechanism is used to reach and maintain the configured shares submission rate (defined as shares_per_minute parameter in config files) from the downstream clients.

So, it doesn't matter if the client sends valid or invalid shares, in any case the counter used by vardiff has to be incremented whenever we receive a share.

That said, can you please share more information about your testing session? Which values are you using in configs?

[warioishere]

Config: shares_per_minute = 15, bitaxe ~1 TH/s.

I am using my draft sv2 integration for the ESP-Miner bitaxeorg/ESP-Miner#1553

I am trying to bring sv2 to solominers too.

We're seeing very erratic difficulty swings — sometimes it drops to 1, sometimes it climbs past 10K, and if left running for hours it has gone over 1M. It sometimes stabilizes then around a normal value of diff of 1k for the target of 15 shares per minute,, but then we see these jumps to too high rates for the miner to ever deliver a valid share. Since I am counting what jd-client submits to upstream, I have a invalid share counting on our sv2 server (own typescript sv2 implementation as I cannot code rust)

One alternative approach worth considering which almost all solo-pools like ck, publicpool and etc uses: instead of counting all submissions (valid + invalid), only count accepted shares — and track the actual difficulty of each share in a sliding window rather than just a raw count divided by elapsed time.

The formula would be:

• Keep the last N accepted shares (e.g. 30, max 300s window)
• difficulty_per_second = sum(share_difficulties) / window_seconds
• target_difficulty = difficulty_per_second * (60 / target_shares_per_minute)
• Only adjust if current difficulty is more than 2x off from target

Why this would fix the swings:

• If difficulty goes too high and the miner stops finding valid shares, the cache empties and difficulty immediately drops. With the current approach, invalid/stale retries still bump the counter, keeping the realized rate non-zero — which either prevents the drop or pushes difficulty even higher.
• The 2x hysteresis threshold prevents small fluctuations from triggering constant re-adjustments, which is what causes the oscillation between extremes.

[GitGab19]

I am using my draft sv2 integration for the ESP-Miner bitaxeorg/ESP-Miner#1553

I am trying to bring sv2 to solominers too.

This is great to hear, we are also working in that direction at the Pool level (see #135), so it seems a perfect timing for it!

I have a invalid share counting on our sv2 server (own typescript sv2 implementation as I cannot code rust)

Why did you chose to implement again the Sv2 server? Which challenges did you face? We also have pre-compiled binary and docker image for our Pool if you don't want to compile it from source.

---

That said, why are you using JDC if you're testing the simpler Sv2 implementation on the ESPMiner? I would connect it directly to the Sv2 Pool without any JDC for initial testing.

[warioishere]

Ah nice, I didnt see that you work on a Solo Mining Mode, I already modified my sv2 server to parses address from user_identifier → calls getCoinbaseOutputsForToken(address) → builds a P2WPKH/P2PKH scriptPubKey output → sends it back in AllocateMiningJobTokenSuccess.coinbase_outputs.

I know you have precompiled binaries, but I heavily forked benjamins wilsons solo pool in typescript, and implemented the sv2 protocol from scratch, because I build a ton of statistic services around the original build and wanted to keep everything like that.

Connecting the ESP Miner directly to the sv2-pool is working already fine via standard channels, but currently, I am testing connecting the miner to a jd-client with a tp provider connecting to bitcoincore to have full template control. I have done some setup and forget fullstack deployment with everything on from this repo. But I am currently testing if everything works perfectly:

https://github.com/warioishere/sv2-apps

[GitGab19]

About what you're doing on https://github.com/warioishere/sv2-apps, there are many things which we're currently building as well: UI for JDC (and tProxy), single docker compose which runs every necessary component (UI included), with containers orchestration, etc.

It might make sense to join the efforts on that, since we're currently working on it right now, and we're planning to ship it in our next release: https://github.com/stratum-mining/sv2-ui

But I think we are going a bit out of scope here, so maybe it's better to discuss these things over Discord, or create a dedicated github discussion.

---

Going back to the original issue:

Connecting the ESP Miner directly to the sv2-pool is working already fine via standard channels

Do you mean our Pool, or your forked public-pool?

[warioishere]

I also didnt see that, shit, so much effort I already put on this haha, do you plan to connect the jd-client to core directly or via Sjors TP?

I ment my pool, can you send me your pooladress so I can verify it connects also correctly to your reference implementation?

[GitGab19]

I also didnt see that, shit, so much effort I already put on this haha, do you plan to connect the jd-client to core directly or via Sjors TP?

JDC uses the bitcoin_core_sv2 crate we shipped a couple of months ago, and it's able to directly connect to Bitcoin Core, without the need for the TP in between.

I ment my pool, can you send me your pooladress so I can verify it connects also correctly to your reference implementation?

You can find all the relevant information about our hosted Pool here:

sv2-apps/miner-apps/jd-client/config-examples/mainnet/jdc-config-bitcoin-core-ipc-hosted-infra-example.toml

Lines 65 to 68 in 534b67c

	[[upstreams]]
	authority_pubkey = "9auqWEzQDVyd2oe1JVGFLMLHZtCo2FFqZwtKA5gd9xbuEu7PH72"
	pool_address = "75.119.150.111"
	pool_port = 3333

[warioishere]

thanks for clarification, jst back to the vardiff method and the fluctuation of the diff which jd clients sends.

my suggestion would be

// Line 106–133
public getSuggestedDifficulty(clientDifficulty: number): number | null {

    // < 5 valid shares and > 60s passed → cold-start fallback
    if (this.submissionCache.length < 5) {
        const elapsedSeconds =
            (Date.now() - this.submissionCacheStart.getTime()) / 1000;

        if (elapsedSeconds > 60) {
            return this.nearestDifficultyStep(
                clientDifficulty / this.targetSharesPerMinute
            );
        }

        return null;
    }

    // Sum difficulties of all cached valid shares
    const sum = this.submissionCache.reduce(
        (prev, curr) => prev + curr.difficulty,
        0
    );

    const diffSeconds =
        (this.submissionCache[last].time -
            this.submissionCache[0].time) / 1000;

    const difficultyPerSecond = sum / diffSeconds;
    const targetDifficulty =
        difficultyPerSecond * this.targetSubmissionPerSecond;

    // Only adjust if more than 2x off — hard hysteresis
    if (
        clientDifficulty * 2 < targetDifficulty ||
        clientDifficulty / 2 > targetDifficulty
    ) {
        return this.nearestDifficultyStep(targetDifficulty);
    }

    return null; // no adjustment needed
}


// updateHashRate() — only called for valid shares
// Line 47 — feeds the submissionCache
public updateHashRate(targetDifficulty: number) {
    this.updateSubmissionCache(date, targetDifficulty); // only valid shares reach here
    ...
}


// updateSubmissionCache()
// 300s window, max 30 entries, keyed by { time, difficulty }

That's the entire mechanism in typescript. Three methods, ~60 lines total.

[GitGab19]

Can you try to run your Sv2 ESPMiner against our Pool and see what happens wrt difficulty?

Because there's a possibility that the problems you are facing with our JDC are not related to our vardiff mechanism per se, but it might be dependent on other issues we're already tracking on JDC, like this one.

[warioishere]

okay running now, but maybe useful info: this didn't happen against my own sv2 pool, no diff jumps, but I will also report back on your pool.

[plebhash]

I also didnt see that, shit, so much effort I already put on this haha, do you plan to connect the jd-client to core directly or via Sjors TP?

JDC uses the bitcoin_core_sv2 crate we shipped a couple of months ago, and it's able to directly connect to Bitcoin Core, without the need for the TP in between.

just to further clarify on this: both options are available... you can choose to connect via IPC directly to Bitcoin Core, or alternatively deploy sv2-tp (which connects to Bitcoin Core via IPC), and then connect your JDC to it via Template Distribution Protocol over a TCP connection

[warioishere]

can be closed, maybe jst some randowm weirdness from my side