Managed Return Value debug info cannot encode multi-register floating-point return values

[tommcdon]

Issue opened in reference to #129321 (comment)

The managed return value (MRV) debug info protocol (VarLocType in cordebuginfo.h) has no encoding for values returned in two floating-point registers. This affects ICorDebugILFrame3::GetReturnValueForILOffset for types returned via multi-reg FP on Linux x64.

Type	Win x64	Unix x64	ARM64	x86
Vector128 (16B)	Ret buffer	Multi-reg (XMM0+XMM1) *	Single V0 reg	Ret buffer
Vector256 (32B)	Ret buffer	Ret buffer	Ret buffer	Ret buffer
Vector512 (64B)	Ret buffer	Ret buffer	Ret buffer	Ret buffer

Only the * case is affected. All other combinations either use the ret buffer path (stack-based encoding) or a single FP register (VLT_REG_FP).

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @tommcdon, @dotnet/dotnet-diag
See info in area-owners.md if you want to be subscribed.

[jakobbotsch]

We might want to change things so that registers are treated uniformly, as a single enum. SysV x64 can also return structs in mixed XMM/general purpose registers, so we would end up needing VLT_REG_FP_REG and VLT_REG_REG_FP too otherwise.
In fact we probably end up with a weird encoding right now for that case, so we'll probably want to fix that as part of enabling structs.