runtime: add runtime.Yield()

[dt]

This change is 

[petermattis]

I'm good with experimenting with this Go runtime enhancement. I definitely want to see experimental evidence of the benefit.

PS Should probably update the print in schedtrace to include sched.bgqsize.

[sumeerbhola]

Given that we are getting closer to merging this, I think we need a plan for how we will maintain this. Specifically the couple of lines of code scattered over various parts in proc.go e.g. we need a list of what cases we need to integrate with the changes, and how to go about finding those cases in the scheduler code.

[dt]

the couple of lines of code scattered over various parts in proc.go

Per the discussion thread above and in the google doc, I've cut these "lines scattered around" (assuming we're referring to the same ones) since my current approach is to just do searches of runqs when npidle is zero. This seems to perform about the same, and better handled added a check of netpoll which I realized I wanted anyway, so the diff is now much closer to a pure, more self contained addition: we need two additions to findRunnable, though they don't depend on anything other than the new yieldq, then two new fields: the yieldq in schedt and the yieldchecks counter/timestamp in G. Other than that it's just the pure addition of the Yield() function and its helpers now.

[dt]

I definitely want to see experimental evidence of the benefit.

Here's without/with making all Pacer.Pace() calls include a call to runtime.Yield() (even those where Pacer is nil/elastic AC is off) on a 5x8vcpu cluster running a TPCC 5k IMPORT while serving kv95 20k QPS in the foreground (note the units changed in the first one). CPU utilization is smoother and higher with Yields.

Without a foreground workload, IMPORT throughput is higher with Yields than without, somewhat surprisingly (by 5-10%) while it is slightly lower when there is a workload to yield to. CPU profiling indicate that while maintaining 98% CPU utilization, the Yield() calls account for about 0.8-0.9% of IMPORT's CPU usage and when under <90% utilization more like 0.4-0.6%.

[petermattis]

I like that this iteration is less intrusive than the previous one. Mostly comments about needing more comments.

[petermattis]

Is the expense of runqempty problematic? Seems like you're only doing this infrequently.

[dt]

Not that expensive (unless you have a silly number of cores) at least in relative terms in this branch where we have a whole syscall to netpoll. But I think it makes sense to skip it above when checking the local runq and if we're okay skipping it there, we should be okay skipping it here too (they can all be stolen from concurrently all the same).

[petermattis]

The results are very compelling.

[sumeerbhola]

@sumeerbhola reviewed 1 of 2 files at r3, all commit messages.
Reviewable status: 1 of 5 files reviewed, 25 unresolved discussions (waiting on @dt, @petermattis, and @tbg)

---

src/runtime/runtime2.go line 515 at r3 (raw file):

	runningnanos  int64 // wall time spent in the running state

	yieldchecks uint32 // a packed approx time and count of maybeYield checks.

This needs a longer code comment elaborating on what exactly this represents and what the packing scheme is.

---

src/runtime/proc.go line 420 at r3 (raw file):

		// To avoid thrashing between yields, set yieldchecks to 1: if we yield
		// right back and see this sentinel we'll park instead to break the cycle.
		gp.yieldchecks = 1

So sometimes yieldchecks is a packed field and sometimes 1? This needs code comments.

---

src/runtime/proc.go line 7251 at r3 (raw file):

}

// yield_put is the gopark unlock function for Yield. It enqueues the goroutine

I'm confused by the "unlock function" terminology, given there are callers of gopark that pass nil. And I want to make sure we document in a code comment why this is correct from the perspective of the following comment in gopark:

// unlockf must not access this G's stack, as it may be moved between
// the call to gopark and the call to unlockf.
//
// Note that because unlockf is called after putting the G into a waiting
// state, the G may have already been readied by the time unlockf is called
// unless there is external synchronization preventing the G from being
// readied. If unlockf returns false, it must guarantee that the G cannot be
// externally readied.

We don't access the stack, so fine. Is "readied" means transitioning back to runnable state? I suppose that can't happen either because we haven't even put it in the yieldq yet, so no one can discover it and make it runnable. Anything else I am missing?

[dt]

Reviewable status: 0 of 6 files reviewed, 25 unresolved discussions (waiting on @petermattis, @sumeerbhola, and @tbg)

---

src/runtime/proc.go line 7251 at r3 (raw file):

Previously, sumeerbhola wrote…

I'm confused by the "unlock function" terminology, given there are callers of gopark that pass nil. And I want to make sure we document in a code comment why this is correct from the perspective of the following comment in gopark:

// unlockf must not access this G's stack, as it may be moved between
// the call to gopark and the call to unlockf.
//
// Note that because unlockf is called after putting the G into a waiting
// state, the G may have already been readied by the time unlockf is called
// unless there is external synchronization preventing the G from being
// readied. If unlockf returns false, it must guarantee that the G cannot be
// externally readied.

We don't access the stack, so fine. Is "readied" means transitioning back to runnable state? I suppose that can't happen either because we haven't even put it in the yieldq yet, so no one can discover it and make it runnable. Anything else I am missing?

Yeah, that's pretty much it: nothing is going to ready this G until findRunnable pull it from the yieldq so it is our to pu there, so I think we can guarantee it isn't externally readied.

[dt]

Only one data point but I just rebased this from go 1.23 to go 1.25 since CRDB just switched today and the rebase was clean with no conflicts on the scaled back touch points of just findRunnable+two new fields. I think the previous version with the extra pending work atomic did have master/1.25 -> 1.23 conflicts when I went the other way, so maybe some evidence that being more contained is indeed helpful here.

[petermattis]

This approach looks good to me. I'll defer to @sumeerbhola for final approval. Really excited about the benefits you're seeing in test scenarios.

[petermattis]

Nit: anything parked?

[sumeerbhola]

@sumeerbhola reviewed 2 of 3 files at r5.
Reviewable status: 2 of 6 files reviewed, 39 unresolved discussions (waiting on @dt, @petermattis, and @tbg)

---

src/runtime/proc.go line 463 at r5 (raw file):

which it will be when they're zero if we don't yield,

what is "they" in "they're"?
I am still not very clear on the yieldchecks states and state transitions. Please spell out the state machine and transitions somewhat precisely in a code comment, ideally where yieldchecks is declared.

---

src/runtime/proc.go line 470 at r5 (raw file):
Thinking out aloud. There are 3 assignments to yieldchecks in this file
yieldchecks = 1
yieldchecks = now
yieldchecks = prev | (yieldCountMask / 2)

Say one of the latter two ran and it parked itself in the global yieldq. At some point it starts running again. It will call yield and the first assignment runs and it puts itself in the back of the local runq. For it not to park itself in the yieldq on the next yield, the local runq must be empty, yes?

Say it is empty, so it falls through to the code below. Where we increment it to 2, and so count = 2 and count +1 = 3, and we don't immediately check the clock. Then the next yield sets sees the transition since count = 3 and count + 1 = 4. But the clock value is 0 since we haven't initialized it, so now != prev is true and we will check all the other queues. Seems ok, but this transition between the two modes of using yieldchecks needs a precise explanation and preferably a statements of invariants. I know the earlier comment is trying to do that, but IMHO it doesn't spell it out in detail. Without that, the reader has to do the hard work of fully figuring it out.

// We can clobber yieldchecks here since we're
// actively yielding -- we don't need the counter to decide to do so. And
// our sentinel will in turn be clobbered the very next time the time is put
// in the upper bits, which it will be when they're zero if we don't yield, ...

---

src/runtime/proc.go line 523 at r5 (raw file):
A longer comment would help. Something like:

// count & (count + 1) will be 0 on transitions from 2^k-1 to 2^k for every value of k, so k=0, 1, ..., which means we read the clock with exponential backoff. When k=11, we reach the maximum value of the counter, and we will also sample on the transition from 2^11-1 to 2^11, after which k will become 0 and we will resume faster sampling.

---

after which k will become 0 and we will resume faster sampling.

I see now that we don't do that in that we do gp.yieldchecks = prev | (yieldCountMask / 2) in that we are going back to 2^10-1. But we are clobbering the time bits, which seems wrong.

---

src/runtime/proc.go line 456 at r5 (raw file):

		// yieldq and potentially switching Ps. While that's our preferred choice,
		// we want to avoid thrashing back and forth between multiple Yield-calling
		// goroutines: in such a case it is better to just park one so the other

... just park one on the global yieldq ...

---

src/runtime/proc.go line 457 at r5 (raw file):

		// we want to avoid thrashing back and forth between multiple Yield-calling
		// goroutines: in such a case it is better to just park one so the other
		// stops seeing it in the queue and yielding to it. To detect and break this

... stops seeing it in the local P's runq and yielding ...

---

src/runtime/proc.go line 510 at r5 (raw file):

	// uint32 on G: its 11 lower bits store a counter while the remaining 21 bits
	// store nanos quantized to 0.25ms "epochs" by discarding the lower 18 bits.
	// of a int64 nanotime() value. For counter values after increment of 2^k-1,

what is k?

---

src/runtime/proc.go line 515 at r5 (raw file):

	//
	// Choosing 11 bits for a counter allows backing off to a rate of checking the
	// clock once every 1k calls if called extremely frequently; it seems unlikely

Isn't 10 bits enough to represent 1023, so why 11 bits?

---

src/runtime/proc.go line 519 at r5 (raw file):

	// higher backoff. The 21 remaining bits allows ~9mins between rollover of
	// the epoch: the slim chance of a false negative is quite acceptable as if we
	// hit it, we just delay one check of the runqs by a quarter millisecond.

This false negative comment is not very clear to me.

IIUC, the risk is that when (count & (count + 1)) == 0, 9 minutes (with fidelity of 0.25ms) have elapsed and we won't check the other runqs. Since we vary the count interval between sampling the clock, due to the exponential backoff on the count interval, we will not keep hitting this pathological case of 9min having elapsed.

The above understanding is not consistent with the comment "we just delay one check of the runqs by a quarter millisecond", since if 9min have elapsed and we exponentially backoff, then 18min will have elapsed. I am probably missing something.

Could you add a longer code comment.

---

src/runtime/proc.go line 520 at r5 (raw file):

	// the epoch: the slim chance of a false negative is quite acceptable as if we
	// hit it, we just delay one check of the runqs by a quarter millisecond.
	const yieldCountBits, yieldCountMask = 11, (1 << 11) - 1

so yieldCountMask is 2047, and not 1023, yes?

---

src/runtime/proc.go line 521 at r5 (raw file):

	// hit it, we just delay one check of the runqs by a quarter millisecond.
	const yieldCountBits, yieldCountMask = 11, (1 << 11) - 1
	const yieldEpochShift = 18 - yieldCountBits // only need to shift by the differnce, then mask.

18 - 11? Why?
18 is the quantization of the time, which corresponds to 1 epoch.
I would expect we would right shift time by 18. Then we would clear out everything except for the lowest 11 bits.
But we are doing

now := uint32(nanotime()>>yieldEpochShift) &^ yieldCountMask

So we are right shifting by 7 and then clearing the lowest 11 bits. Confused.
Can this yieldchecks be moved into a struct with methods and a simple unit test. Hopefully all the methods will inline. It would also help with my earlier state machine comment.

---

Hmm, we do gp.yieldchecks = now, so I can see we want the quantized value and not use the lowest 11 bits. So that means taking the clock value, quantizing by clearing the lowest 18 bits, then right shifting by 18 bits and then left shifting by 11 bits. I think what is happening here is doing this in one step by right shifting by 7 and clearing the lowest 11. This is very straightforward code, if there were code comments explaining what it is doing, but without those code comments at least for me this is not decipherable without spending far too much time on it.

---

src/runtime/proc.go line 522 at r5 (raw file):

	const yieldCountBits, yieldCountMask = 11, (1 << 11) - 1
	const yieldEpochShift = 18 - yieldCountBits // only need to shift by the differnce, then mask.
	gp.yieldchecks++

When this goroutine comes back from

---

src/runtime/proc.go line 568 at r5 (raw file):

// park to Yield is considered "waiting" rather than "runnable" as it is blocked
// in this state until there is strictly spare execution capacity available to
// resume it, unlike runnable goroutines which generally take runs running at

take turns?