Out-of-band ConPTY, CsWin32 interop, and the concurrency harness that found it all

[JohnCampionJr]

Answers #33 — thanks for the go-ahead. Everything here came out of driving this library hard from a desktop app (terminal panes and long-running children, many at once), so each piece is a measurement rather than a preference.

Rebased onto main now that #7, #8 and #9 are merged, so those three fixes are no longer in this diff. CI is green on all three platforms.

---

Out-of-band ConPTY

Microsoft ships ConPTY out of band as Microsoft.Windows.Console.ConPTY — conpty.dll + OpenConsole.exe, the same implementation Windows Terminal carries. Both paths are wired up behind PORTAPTY_CONPTY so the choice could be measured rather than argued. Out-of-band is the default, with an automatic fallback to in-box when conpty.dll isn't beside the assembly — so a consumer who hasn't referenced the package can't break.

It appeared to cost ~3.0 seconds per pseudoconsole. It doesn't. ConPTY asks the terminal what it is (Primary Device Attributes) and blocks three seconds waiting for a reply a read-only consumer never sends. Answering it up front:

	first output (ms)
before	3016, 3012, 3011, 3013, 3019
after	15, 9, 9, 8, 8

Out-of-band then measures 9ms per pseudoconsole against in-box's 13ms — and pins behaviour to one implementation rather than to whatever Windows build the user happens to run.

Two things made that A/B lie before it told the truth, both written up in docs/conpty-out-of-band.md:

• Recent Windows 11 ships its own System32\conpty.dll, so an unqualified DllImport resolves the OS copy and the "out-of-band" arm is quietly in-box. DefaultDllImportSearchPaths(AssemblyDirectory) is load-bearing, and leaving it off is silent rather than fatal.
• The only direct evidence of which implementation is live is a process census — in-box spawns a conhost.exe per pseudoconsole, out-of-band an OpenConsole.exe. Three earlier A/B runs produced plausible, near-identical tables because both arms were conhost.

CsWin32 replaces Vanara

The point is the runtime dependency: Vanara ships an assembly every consumer then carries, for about twenty entry points. CsWin32 is a build-time generator with PrivateAssets=all — nothing at runtime, nothing in a consumer's graph — and the generated interop is trimming/AOT friendly. The idea is Sylinko's, from their fork; this is an independent implementation of the same move.

Windows job-object race

CreateProcessW wasn't given CREATE_SUSPENDED, so a child could run and exit before AssignProcessToJobObject reached it — surfacing as "Failed to assign process to job object". Now created suspended, assigned, then resumed.

net10.0, MSTest on MTP, and the harness

MTP reports crashes and hangs instead of absorbing them, which matters here because almost everything interesting is threads and process lifetime — a swallowed hang looks exactly like a pass.

ConcurrentSpawnTests is where the rest of this came from: 24 concurrent spawns, 20 samples per cell, minimal and realistic shells. It's what showed the reader strategy dominating everything else — 137ms to first output on a dedicated thread against 7546ms pooled — and what surfaced forkpty's thread-unsafety on macOS as a reproducible 5/24 rather than an occasional mystery.

---

Verified: 23 tests, 0 failed on macOS (3 Windows-only skipped). Your existing CI should stay green — it already builds and stages the native shim, and global.json gives dotnet test the MTP runner. I deliberately left your build-*.yml and PublishNuget.yml alone.

Verified on Windows, which is where most of this had to be: the ConPTY latency numbers above, the in-box vs out-of-band process census, and the job-object race were all found and measured on real Windows machines — the race in particular only exists there. macOS covers the forkpty and controller-fd work. Linux is the one platform I have exercised least, and your matrix covers it.

Packaging, and a consumer that can see it

The tests reference the library by project, which bypasses the .nupkg entirely — no runtimes/ resolution, no buildTransitive/, no native asset staging — so no packaging defect is visible from them. Two were sitting there, both silent:

• A package consumer got conpty.dll with no OpenConsole.exe. ConPTY ships its host-staging logic under build/, which imports for a direct reference only. conpty.dll with no host doesn't error; it falls back to in-box conhost. buildTransitive/Porta.Pty.targets forwards it, so a consumer needs no metadata.
• The POSIX shim wasn't staged for project-reference consumers, which is why the workflows hand-copied it — to a path that stopped existing once the test output went RID-specific. Now Content-flowed from the library, so the copy steps are gone and a guard asserts it landed.

samples/Porta.Pty.Demo is a real consumer — spawns a pty, echoes a random token, asserts it comes back — and the verify scripts pack the library and build it against a local feed, so what runs is what a consumer gets:

scripts/verify-consumer.sh                  Linux and macOS
scripts/Verify-ConPtyConsumerStaging.ps1    Windows: conpty.dll + both hosts, then the round trip (-Aot for native AOT)
scripts/Verify-ConPtyHost.ps1               Windows: which host actually launched, from the process tree

All three run in CI. The Windows leg covers win-x64 as well as the runner's own architecture, since an x64 process runs on ARM64 Windows under emulation. Also verified on a physical Windows ARM64 machine: both RIDs build clean and pass the round trip with both hosts staged.

[tomlm]

Hey, this looks good. Can you comment on the extra metadata that it looks like a consumer needs to use?

[tomlm]

I think this change is significant enough to warrant V2.0.0

[tomlm]

why were keywords and symbols removed?

[JohnCampionJr]

glitch? this came from a previously private fork. will fix

[tomlm]

same, this warrents v2.0

[tomlm]

do people who use Porta.Pty need to do this?

[Copilot]

🟡 Changes recommended

Linux/macOS CI currently fails, and packaged consumers do not reliably receive the out-of-band ConPTY host.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds out-of-band ConPTY support, replaces Vanara with CsWin32, fixes PTY lifecycle races, and introduces concurrency-focused testing.

Changes:

• Adds selectable ConPTY implementations and safer Windows process startup.
• Fixes Unix PTY allocation and descriptor cleanup.
• Migrates to .NET 10, MSTest/MTP, and expanded concurrency tests.

File summaries

File	Description
.gitignore	Ignores native build outputs.
docs/conpty-out-of-band.md	Documents ConPTY behavior and measurements.
global.json	Selects Microsoft Testing Platform.
src/Porta.Pty.Native/build.sh	Builds native binaries by platform and architecture.
src/Porta.Pty.Native/porta_pty.c	Serializes PTY allocation.
src/Porta.Pty.Tests/ConcurrentSpawnTests.cs	Adds concurrency and latency harnesses.
src/Porta.Pty.Tests/Porta.Pty.Tests.csproj	Migrates tests to .NET 10 and MSTest.
src/Porta.Pty.Tests/PtyTests.cs	Converts existing tests to MSTest assertions.
src/Porta.Pty/Linux/PtyConnection.cs	Closes Linux controller descriptors.
src/Porta.Pty/Linux/PtyProvider.cs	Improves spawn diagnostics.
src/Porta.Pty/Mac/PtyConnection.cs	Closes macOS controller descriptors.
src/Porta.Pty/Mac/PtyProvider.cs	Improves spawn diagnostics.
src/Porta.Pty/NativeMethods.txt	Defines CsWin32 generation inputs.
src/Porta.Pty/PlatformServices.cs	Simplifies platform provider selection.
src/Porta.Pty/Porta.Pty.csproj	Targets .NET 10 and replaces Vanara.
src/Porta.Pty/PtyProvider.cs	Exposes ConPTY selection diagnostics.
src/Porta.Pty/Unix/PtyConnection.cs	Adds controller cleanup.
src/Porta.Pty/Windows/JobObject.cs	Ports job-object interop to CsWin32.
src/Porta.Pty/Windows/NativeMethods.cs	Ports attribute-list interop.
src/Porta.Pty/Windows/PseudoConsole.cs	Implements dual ConPTY selection.
src/Porta.Pty/Windows/PseudoConsoleConnection.cs	Uses new handle abstractions.
src/Porta.Pty/Windows/PtyProvider.cs	Fixes process assignment race and handshake.

Review details

• Files reviewed: 20/22 changed files
• Comments generated: 6
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JohnCampionJr]

Hey, this looks good. Can you comment on the extra metadata that it looks like a consumer needs to use?

on it :)

[tomlm]

Hey do you think this fixes issue #6 ?

[JohnCampionJr]

Hey do you think this fixes issue #6 ?

I bet it does, but I haven't personally tested it with AOT. Let me see what I can do.

[JohnCampionJr]

Both of your questions turned out to have the same answer underneath, and chasing the first one found a real defect. Rebased onto main (so #7, #8 and #9 are out of the diff) and CI is green on all three platforms now.

"the extra metadata that it looks like a consumer needs to use"

None — but you were right to ask, because until a few commits ago there was some, and it was undocumented and silent.

Microsoft.Windows.Console.ConPTY splits its payload across two locations, and only one half travels transitively:

path in the ConPTY package	reaches a transitive consumer?
runtimes/win-<arch>/native/conpty.dll	yes — ordinary native asset resolution
build/native/runtimes/<arch>/OpenConsole.exe	no — build/ imports for a direct PackageReference only, and ConPTY ships no buildTransitive/

Porta.Pty references ConPTY directly, so our build staged the host and our tests passed. A consumer of Porta.Pty got conpty.dll and nothing else — and conpty.dll with no host to launch doesn't fail. It falls back to in-box conhost silently: no error, no warning, exactly the behaviour the out-of-band package was taken to avoid.

I found it by packing the library and building a win-x64 consumer against it: the output directory held conpty.dll alone.

buildTransitive/Porta.Pty.targets now forwards the host, so a consumer needs no metadata at all. Two details in there are load-bearing:

• The ConPTY package directory is derived from the resolved conpty.dll, not composed from NuGetPackageRoot plus a version literal. The consumer's graph decides which ConPTY version wins, and a hard-coded version resolves to a path that quietly doesn't exist.
• DestinationSubDirectory, not TargetPath. _CopyFilesMarkedCopyLocal builds $(OutDir)%(DestinationSubDirectory)%(Filename)%(Extension) and never reads TargetPath — with TargetPath both hosts copy flat and the second overwrites the first, leaving a win-x64 consumer holding the ARM64 host.

Setting ConptyRequiresx64Host from a shipped .props can't fix this, incidentally — the targets that read those flags are precisely the ones that never import.

The only knob is an opt-out: PortaPtyStageConPtyHost=false, for a consumer staging the host itself.

Why the tests couldn't have caught that

They reference the library by project, which bypasses the .nupkg entirely — no runtimes/ resolution, no buildTransitive/, no native asset staging. Every packaging defect is invisible from there.

So there's now samples/Porta.Pty.Demo, a real consumer that spawns a pty, echoes a random token and asserts it comes back. The verify scripts pack the library and build the sample against a local feed, so what runs is what a consumer gets:

scripts/verify-consumer.sh                  Linux and macOS
scripts/Verify-ConPtyConsumerStaging.ps1    Windows: conpty.dll + both hosts, then the round trip
scripts/Verify-ConPtyHost.ps1               Windows: which host actually launched, from the process tree

All three run in CI. The Windows leg does win-x64 as well as the runner's own architecture, since an x64 process runs on ARM64 Windows under emulation.

Verified on a physical Windows ARM64 machine: both win-arm64 and win-x64 build clean and pass the round trip, with conpty.dll and both OpenConsole.exe hosts staged.

The Linux/macOS CI failures

Mine, and worth stating plainly: the test project is RID-specific now (so the ConPTY natives land beside the test assembly on Windows), which moved its output from bin/Debug/<tfm>/ to bin/Debug/<tfm>/<rid>/. Your staging step copied the shim to a fixed bin/Debug/net6.0/ under a || true, so it silently did nothing and the run died 20 lines later as DllNotFoundException naming nothing.

Rather than patch the path, the library now stages its own shim into a project-reference consumer's output via Content, so the copy steps are gone entirely and a guard asserts the shim landed.

That fix had a trap in it worth flagging, because it packs cleanly while being badly wrong: a second item pointing at the same file with Pack="false" is reconciled by NuGet, Pack="false" wins, and the package silently loses that RID. Since the RID is the build host's, the package ends up always missing exactly the platform it was built on, with every other platform present — so the listing looks plausible. Measured: packing on osx-arm64 produced a package containing osx-x64 alone. Each RID is now declared exactly once, the host's as Content.

Does this fix #6?

Testing it rather than guessing — -Aot on the consumer script publishes the demo with PublishAot=true and runs it, and I'll report back with the result.

The reason to expect it does: "Unable to convert object to its binary format" is a serialization path failing with no JIT behind it. CsWin32 source-generates its P/Invoke, so there's nothing for the trimmer to fail to see; the Vanara wrappers it replaces are mostly reflection-free but not uniformly, and Vanara also shipped a runtime assembly every consumer carried for roughly twenty entry points. That's the main reason for the swap — AOT was the bonus, and it's the part I want measured before claiming it.

[JohnCampionJr]

@tomlm it does indeed fix AOT. But the larger issue this all revealed is that this also breaks the portability of the earlier version. currently trying to find a way to have it all.

[JohnCampionJr]

Whew, at long last. It all works perfectly now. It is fully portable, and AOT is confirmed, closes #6

Claude Below:

Bumped to 2.0.0 as you asked — the netstandard2.0 → net10.0 move is a breaking change for any .NET Framework or netstandard consumer, and Vanara leaving the dependency graph is a second one. The API itself is unchanged, so there's now an "Upgrading to 2.0" note in the README saying exactly that.

While in there I also restored four properties this branch had dropped: Keyword, ContentTargetFolders, IncludeSymbols, SymbolPackageFormat. That was the private-fork glitch, not a decision — and losing IncludeSymbols would have meant shipping a release with no .snupkg.

Correcting my earlier answer on consumer metadata

I said "none" a few comments up. That was right about properties and wrong by omission, and rather than caveat it I'd rather fix it: a consumer now needs nothing at all, including no RuntimeIdentifier.

The gap was real. Native assets are only flattened into the app root for a RID-specific build; a portable build keeps the runtimes/ tree, so conpty.dll sat under runtimes/win-x64/native/ with no OpenConsole.exe anywhere near it, and the out-of-band path silently became conhost. My first instinct was to document the requirement and warn about it at build time. That's the wrong shape for a library — forcing every consumer to pin a RID isn't a reasonable thing to ask — so instead:

• Each host is staged next to its own conpty.dll, with the destination derived from the DLL's DestinationSubDirectory rather than assuming the flattened case:

  build	conpty.dll	host
  RID-specific	app root	x64/, arm64/
  portable	runtimes/win-<arch>/native/	runtimes/win-<arch>/native/<arch>/

• An import resolver finds it in either layout. The imports are pinned to DllImportSearchPath.AssemblyDirectory deliberately — Windows 11 ships a conhost-backed System32\conpty.dll that an unpinned DllImport binds happily — and that pin can't see runtimes/. Ordinary P/Invoke would resolve it through deps.json, but that's the resolution we're refusing. The resolver loads by absolute path, so it keeps the protection and extends the reach.

Verified on Windows ARM64 by process census: portable build, OpenConsole.exe in the process tree by default, conhost.exe with PORTAPTY_CONPTY=inbox as the control. The Linux/macOS equivalent runs in CI on every push.

This rests on conpty.dll launching OpenConsole.exe from an <arch>/ subdirectory of its own directory. Nothing in the ConPTY package documents that — no readme, and conpty.h says nothing — so it's confirmed by census rather than by contract, and scripts/Verify-ConPtyHost.ps1 -NoRid stays as a regression guard in case it changes.

"do people who use Porta.Pty need to do this?"

No. That was on ConptyRequiresx64Host in the test project, and the answer splits in two:

• A package consumer needs none of it. The library carries the ConPTY reference and ships buildTransitive/Porta.Pty.targets. samples/Porta.Pty.Demo sets no Conpty* property and no ConPTY PackageReference, and gets conpty.dll plus its host.
• The test project needs it because it references the library by project. Package build assets — build/ and buildTransitive/ alike — are a NuGet mechanism that doesn't apply across a ProjectReference, and MSBuild properties don't flow across one either.

That second point is also why the packaging bugs hid for so long: a ProjectReference can't exercise any of it, so the test suite was structurally incapable of noticing. Hence samples/Porta.Pty.Demo, which the verify scripts build from a real .nupkg — it spawns a pty, echoes a random token and asserts it comes back. That's what found the ConPTY host gap Copilot flagged, and two more besides.

The Copilot review

All six addressed:

finding
Transitive consumers don't get the ConPTY host	Fixed — buildTransitive/Porta.Pty.targets
Linux/macOS CI failing	Fixed — see below
Probe used AppContext.BaseDirectory, imports use AssemblyDirectory	Fixed, and made structural: one resolved path now serves both, so they can't disagree
PseudoConsoleImplementation reported the selected branch, not what runs	Fixed — reports oob-no-host separately
<see cref> to a method that doesn't exist	Fixed
Doc pointed at a ci.yml that doesn't exist	Fixed

The third and fourth were the good ones. On the fourth in particular — conpty.dll with no host falls back silently, so reporting oob on the strength of having selected it produces exactly the false A/B the docs warn about, both arms conhost and the numbers agreeing beautifully. Out-of-band is now only claimed when both halves are present; otherwise the in-box path is taken deliberately, so the two arms stay genuinely distinct.

The CI failures were mine

The test project is RID-specific now, so the ConPTY natives land beside the test assembly on Windows. That moved its output from bin/Debug/<tfm>/ to bin/Debug/<tfm>/<rid>/, and the POSIX staging steps copied the shim to a fixed bin/Debug/net6.0/ under a || true — so they silently did nothing and the run died 20 lines later as DllNotFoundException, naming nothing.

Rather than patch the path, the library now stages its own shim into a project-reference consumer's output, so those copy steps are gone and a guard asserts it landed instead.

Worth flagging one trap from that fix, since it packs cleanly while being badly wrong: a second item pointing at the same file with Pack="false" gets reconciled by NuGet, Pack="false" wins, and the package silently loses that RID — and since it's the build host's RID, the package ends up always missing exactly the platform it was built on while every other platform is present. Packing on osx-arm64 produced a package containing osx-x64 alone. Each RID is declared exactly once now.

Does it fix #6?

Yes — measured, not assumed. Native AOT publishes clean and spawns correctly on win-arm64, win-x64 under emulation, and macOS.

samples/Porta.Pty.AotDemo asserts RuntimeFeature.IsDynamicCodeSupported is false before it does anything, so a misconfigured publish that quietly produced an ordinary binary fails rather than passing. It runs on all three platforms in CI.

The reason it works: "Unable to convert object to its binary format" is a serialization path failing with no JIT behind it. CsWin32 source-generates its P/Invoke, so there's nothing for the trimmer to fail to see — the Vanara wrappers were mostly reflection-free but not uniformly. AOT was a bonus of the swap rather than its motivation; the motivation was that Vanara shipped a runtime assembly every consumer carried for about twenty entry points.

Where it stands

CI green on all three platforms. Beyond the existing suite, each push now also:

• builds a real package consumer and runs a pty round trip through it — RID-specific and portable
• asserts conpty.dll and both hosts are staged, on win-x64 as well as the runner's own architecture
• publishes and runs the Native AOT sample

Verified on physical Windows ARM64 as well: the suite, both RIDs, portable, and AOT.

Happy to split any of this out if the PR is too much in one piece — the packaging work and the ConPTY/CsWin32 work are separable, though the packaging fixes are only visible because the ConPTY work made a consumer care.