Add .NET Swift interop tooling components and layout

[kotlarmilos]

Description

This is the initial PR for the projection tooling which introduces the .NET Swift interop tooling components based on the Binding Tools for Swift and highlights the functionalities of the tool. Subsequent PRs will cover the process of projecting Swift types into .NET.

For importing Swift into .NET, the key components are presented, where they work together to allow the generation of C# bindings and Swift wrappers for interop. The process begins with the source Swift library, from which a module declaration is generated. Subsequently, the tool generates Swift wrappers and its module declaration for cases where direct P/Invoke is not possible. Finally, the tool generates C# bindings. For exporting .NET to Swift, the tool generates corresponding Swift and C# wrappers to enable type mapping.

Regarding distribution and validation, the tooling will be part of Xamarin publishing, shipped as NuGet package and accessible via .NET CLI. Validation will be achieved through supporting libraries and MAUI samples, as outlined in dotnet/runtime#95636.

[stephen-hawley]

It's worse than this. Value types in swift have specific semantics for what happens when an instance goes out of scope. This is, of course, completely different than the semantics of C#, but can be approximated by making the type IDisposable. Early on in BTfS, I tried to classify structs into two types: blittable (contains 0 or more fields that are blittable) and non-blittable so that we handle them differently. Blittable types should be mappable to C# structs directly. What I found in reality is that there were so many edge cases that trying to do something with a little more efficiency in some cases created nothing but problems. Over and over again in BTfS, I was schooled that wherever possible, a general solution is best. This is why structs and non-trivial enums are best implemented as a common class implementing IDisposable with a payload that is opaque.

[jkoritzinsky]

Yeah I think the best way to model structs is either with an IDisposable class or with a struct that contains an IDisposable field that exposes the opaque pointer. To handle lifetimes, I would recommend that we emit a finalizer to ensure that we release memory correctly.

[kotlarmilos]

Thanks, updated.

[agocke]

What is the purpose of the IDisposable implementation? Given that IDisposable in C# doesn't actually indicate de-allocation, I would not expect it to do things like decrease a refcount on the Swift side.

[AaronRobinsonMSFT]

It is for disposal or release of native resources. In this case that is the intent. The fact that it is a refcount on the native side seems like implementation detail of the taret platform we are interoping with.

[jkoritzinsky]

Yeah I think the best way to model structs is either with an IDisposable class or with a struct that contains an IDisposable field that exposes the opaque pointer. To handle lifetimes, I would recommend that we emit a finalizer to ensure that we release memory correctly.

[kotlarmilos]

I have updated this PR according to the received feedback. The scope has been narrowed to focus on the general mechanisms of projections. For comprehensive details please refer to dotnet/runtimelab#2512.

We've added validation steps for the projection tooling with required runtime support in dotnet/runtime#95636. Additionally, there are proposed stages for evolving the tooling in dotnet/runtime#95633.

Feel free to get involved and share your perspectives or feedback on any provided issue.

[kotlarmilos]

This PR should outline the general concepts of projection tooling. All details should be added to dotnet/runtimelab#2512.

Please let me know if there is anything else that you think should be added here.

[agocke]

Could there be a little more elaboration on the IDisposable piece? My expectation is that we would use IDisposable for managed resources, but all the things mentioned as being cleaned up sound like they are native resources. That sounds like a better fit for destructors. What makes IDisposable necessary here?

[kotlarmilos]

Could there be a little more elaboration on the IDisposable piece? My expectation is that we would use IDisposable for managed resources, but all the things mentioned as being cleaned up sound like they are native resources. That sounds like a better fit for destructors. What makes IDisposable necessary here?

Good point. I've updated the docs to reflect the decision.

The IDisposable provides an explicit mechanism for releasing unmanaged resources. Destructors are managed by the GC and offer a way to release unmanaged resources when an object is collected by the GC. While destructors abstract away memory management from the user, the Idisposable pattern provides deterministic control over when resources and can lead to better performance as it prevents the need for GC collection cycles. The IDisposable pattern is the typical .NET approach for dealing with unmanaged resources and thus is selected as default option at initial stage. If it is determined that the IDisposable pattern introduces unnecessary overhead for users, and that destructors can adequately manage the release of unmanaged resources, appropriate updates to the memory management approach will be made.

[agocke]

the Idisposable pattern provides deterministic control over when resources and can lead to better performance as it prevents the need for GC collection cycles

How is this relevant to interop? It sounds like it doesn’t make any semantic difference if the object is released early or late, it’s just a design difference. That is, it doesn’t sound like there are any non-memory resources being tracked. In that case I would expect us to stick with C# design, which is to use the GC for all memory resources. Is there something else being tracked here?

[AaronRobinsonMSFT]

Is there something else being tracked here?

Composition can be a concern here. For example, in COM Aggregation there are potentially two distinct memory models that are represented by a single object - WinRT/COM class is the base of some .NET class. In the COM scenario this is relatively simple because it is expected the user to handle circular references that strattle the interop boundary. For WinRT, the Jupiter runtime has a reference tracking mechansim that means more cooperation is needed to collect a single object that is managed by the two systems.

I don't know if our Swift interop scenarios will have need for that sort of mechanism, but from experience interop scenarios that are non-deterministic make tooling very complicated - memory tracking, code coverage, etc. If we expect existing Swift tooling to be usable when .NET is involved having an explicit option is beneficial - even if it is an opt-in like CreateObjectFlags.UniqueInstance.

[kotlarmilos]

The primary reason for considering destructors is to utilize the GC and the C# design pattern. The current tooling (BTfS) implements the IDisposable strategy. Since we are at an early stage, it is challenging to make a decision.

I don't know if our Swift interop scenarios will have need for that sort of mechanism, but from experience interop scenarios that are non-deterministic make tooling very complicated

Opting for IDisposable for the sake of simplicity seems like a reasonable argument. @agocke are you open to this approach (starting with IDisposable), or do you prefer starting with destructors initially and transitioning to IDisposable if required?

[agocke]

My concern is mainly that we overload the use of IDisposable a lot in .NET, and that makes it confusing for customers. In some cases, like for files, it's pretty important that you call IDisposable to prevent running out of handles. In this case, pretty much every Swift type would come with IDisposable. Without knowing details of the type, it seems impossible for me, the caller, to understand whether I should or shouldn't dispose it.

Moreover, I think the preferred behavior for "trivial" deinit calls would be to not call Dispose (or more accurately, not wrap the type in a using). Since adding a using in C# produces a try-finally and hurts codegen, it's probably better to avoid a pattern that causes an explosion of usings. Instead, I think it would be ideal if we only used IDisposable for custom deinit. In those cases, we can be relatively confident that something more than just memory is being freed.

So overall I'm very supportive of some way to explicitly call deinit/dispose on Swift type, but making everything IDisposable feels like an anti-pattern that will just cause a lot of user confusion.

[kotlarmilos]

Thanks for info. I've updated the section and the MVP accordingly.

[jkotas]

What are the compatibility rules for adding/removing custom deinit in Swift?

Are introduction or removal of a custom deinit considered to be a compatible change in Swift? If they are compatible changes, it would be problematic to map custom deinit to IDIsposable. Adding/removing IDisposable is a breaking change in .NET.

[agocke]

This is unfortunate -- I bet Swift is fine with removing a deinit since there's always an implicit deinit that it can fall back on.

I think the fundamental problem here is that Swift does not have a distinction between "deterministic" destruction and non-deterministic destruction. In contrast, C# does. And, importantly, that distinction is very important in C#. If you require deterministic destruction then timely destruction may be a requirement for your application, where delaying destruction may cause your application to fail. Conversely, if you don't require it, using deterministic destruction (IDisposable/using) for all resources is very expensive and will likely cause significant performance degradation. It's also a large programmer burden as disposability is transitive and any IDisposable fields require the parent type to also be IDisposable.

"Custom deinit" is a heuristic that attempts to map between C#, which differentiates the two concepts, and Swift, which doesn't. But since there is no way to determine statically whether a Swift type actually requires deterministic destruction, I don't think there's any heuristic that will be perfectly accurate.

That said, maybe we can do better than "custom deinit". Suggestions welcome.

[kotlarmilos]

Does this imply that we should have a solid reason for implementing the IDisposable pattern on a type as introducing or removing explicit memory management via IDisposable could lead to breaking changes?

This is the case where direct mapping is not possible, and introducing IDisposable only when absolutely necessary can help minimize breaking changes caused by switching to/from the explicit memory management.

[AaronRobinsonMSFT]

This is the case where direct mapping is not possible, and introducing IDisposable only when absolutely necessary

Another option here would be to follow the built-in COM mechanism and add a helper function like Marshal.FinalReleaseComObject(). This could be on Marshal or another more targeted type, and use any number of tricks to provide efficient deterministic release semantics. This has the added benefit of not forcing users to litter their code with type checks/casts.

Note that ComWrappers did use IDisposable for specific COM scenarios. In that case though the user is driving the experience by explicitly requesting the deterministic support - see this section of the ComWrappers tutorial.

[stephen-hawley]

I'll offer a datapoint here that there are at least 4 issues at play here:

• Swift has very different semantics than C# for value types. Value types can induce code to get executed when they go out of scope (for example, a value type contains a reference type).
• Swift maintains two different reference counts: a strong reference count and a weak reference count on heap-allocated types, not a simple reference count. The weak reference count is always used implicitly at allocation time (IIRC - it's been a while since I put this under a magnifying glass) and may be used explicitly by assigning to a member declared as weak
• At binding time, it is not always deterministic that a value type will always be blittable (for example, a non-frozen struct could get a member added in the future) and this will create a compatibility issue for us in the future

In BTfS, when we are presented with a Swift object, the C# binding implements IDisposable and takes a swift weak reference to the object via the swift runtime routine swift_unownedRetain and keep a weak GCHandle. Dispose (false) gets called by the finalizer.

Initially, I tried to draw a distinction between blittable and non-blittable value types, but things that make that challenging were all the special cases: value types that contain mutating methods or properties, inout parameters, frozen/not frozen, opaque layout of enums etc. It was an endless supply of bugs in the marshaling. My solution for this was to make all values type handling uniform.

Not being an expert on it, does COM interop exist on non-windows platforms?

I'm pinging @rolfbjarne here as well since he was responsible for the handling of ObjC mapping, which required similar work.

[kotlarmilos]

Let's review some examples of existing bindings to gain a deeper understanding of how the IDisposable pattern is utilized. I've done a review of the Xamarin Google APIs for iOS Components and Google Mobile Unity Ads, focusing specifically on the following applications: cloud messaging, mobile ads, analytics, and crashlytics. The goal of this review was to identify the use cases of IDisposable patterns. The collected data could help us move this discussion/decision forward.

In the existing Xamarin ObjC interop, the NSObject represents a base object and implements IDisposable.

Cloud Messaging

In the cloud messaging bindings, the following APIs require implementing the IDisposable pattern:

• MessageInfo
• MessagingDelegate
• Messaging
• MessagingExtensionHelper

In the sample application, these types are used in AppDelegate, but there are no explict disposing of objects.

There is another example where IDisposable pattern is used to remove notification handles

var sendSuccessToken = Messaging.Notifications.ObserveSendSuccess (HandleSendMessageSuccess);
var sendErrorToken = Messaging.Notifications.ObserveSendError (HandleSendMessageError);

// Call this lines to stop receiving notifications
sendSuccessToken.Dispose ();
sendErrorToken.Dispose ();

// Another option: Calling native code could explicitly release observers.

var sendSuccessToken = NSNotificationCenter.DefaultCenter.AddObserver (Messaging.SendSuccessNotification, HandleSendMessageSuccess);
var sendErrorToken = NSNotificationCenter.DefaultCenter.AddObserver (Messaging.SendSuccessNotification, HandleSendMessageError);

// Call this lines to stop receiving notifications
NSNotificationCenter.DefaultCenter.RemoveObserver (sendSuccessToken);
NSNotificationCenter.DefaultCenter.RemoveObserver (sendErrorToken);

Analytics

In the analytics bindings, the following APIs require implementing the IDisposable pattern:

• Analytics
• DictionaryBuilder
• Fields
• Logger
• TrackedViewController
• Tracker
• EcommerceFields
• EcommerceProduct
• EcommerceProductAction
• EcommercePromotion

In the samples, they are not explicitly released.

MobileAds

In the mobile ads bindings, there are 43 interfaces with base types implementing IDisposable section.

There are 6 instances of disposal in the sample application:

// You need to explicitly Dispose Interstitial when you dont need it anymore
// to avoid crashes if pending request are in progress
void RemoveAdFromTableView ()
{
	if (adViewTableView != null) {
		if (adOnTable) {
			dvcDialog.Root.RemoveAt (idx: 2, anim: UITableViewRowAnimation.Fade);
		}
		adOnTable = false;

		// You need to explicitly Dispose BannerView when you dont need it anymore
		// to avoid crashes if pending request are in progress
		adViewTableView.Dispose();
		adViewTableView = null;
	}
}

GoogleAds Mobile Unity

In the google mobile bindings, the following types implement IDisposable, but none of them are disposed explicitly.

• AdManagerBannerClient
• AppOpenAdClient
• BannerClient
• InterstitialClient
• NativeOverlayAdClient
• RewardedAdClient
• RewardedInterstitialAdClient

Crashlytics

In the crashlytics bindings, the following APIs require implementing IDisposable:

• Crashlytics
• ExceptionModel
• StackFrame

In the sample application, none of them are disposed explicitly.

Xamarin Designer

The Xamarin Designer (deprecated) for iOS is a visual designer that contain controls with IDisposable pattern. Samples from the mentioned bindings that implement Xamarin Designer templates contain a ReleaseDesignerOutlets function that explicitly releases all unmanaged references. Here is an example:

[Outlet]
UIKit.UILabel LblPreview { get; set; }

[Outlet]
UIKit.UILabel LblSubtitle { get; set; }

[Outlet]
UIKit.UILabel LblTitle { get; set; }

void ReleaseDesignerOutlets ()
{
	if (LblPreview != null) {
		LblPreview.Dispose ();
		LblPreview = null;
	}

	if (LblSubtitle != null) {
		LblSubtitle.Dispose ();
		LblSubtitle = null;
	}

	if (LblTitle != null) {
		LblTitle.Dispose ();
		LblTitle = null;
	}
}

[agocke]

Thanks @kotlarmilos, this is a lot of great work!

Based on real-world examples, it doesn't seem like people actually use Dispose in most cases. Based on that, I would recommend using a static method like Marshal.ReleaseComObject (or even a different interface if necessary) rather than adding a bunch of ceremony to the API, but that's just my opinion.

My main concern is accurately providing guidance to authors on whether or not manual disposal is expected, and it seems like it's not.

@jkoritzinsky @AaronRobinsonMSFT @jkotas thoughts?

[AaronRobinsonMSFT]

Thanks @kotlarmilos, this is a lot of great work!

+1. Thank you for this sort of due-diligence and investigation.

I would recommend using a static method like Marshal.ReleaseComObject (or even a different interface if necessary) rather than adding a bunch of ceremony to the API

Agree. I accept this is anecdotal, but it does paint a compelling argument for avoiding a ubiquitious IDisposable requirement. I think the onus is now on the other side of the argument to make a case for IDisposable everywhere.

[vitek-karas]

+1 on Andy's and Aaron's responses - and thanks @kotlarmilos for the analysis.

Just curious - did we consider adding an instance method onto the base type (assuming there will be a base type for all projected types)? I know this is sort of against .NET/C#, but I might go as far as adding a void Dispose() onto the base type without implementing IDisposable. Unfortunately C#'s using is not pattern based (unlike most other C# features) and needs the type to implement IDisposable. But it would make the disposing feel a bit more first class and more discoverable I think.

Additional notes:

• Implementing IDisposable on the projected types can trigger analyzer warnings- for example CA2000, or CA2213 - although these analyzers are not enabled by default currently (that said it seems that other tools do report similar diagnostics - Resharper and so on)
• I do agree that for certain projected types implementing IDisposable would be a good idea (like the above mentioned photos) - this means that eventually the tooling should be able to take a hint to implement the interface on a given projected type.
• Based on the above description it seems that the cases where Dispose is called are actually not about releasing resource pressure, but about actual functionality - like de-registering something. It just so happens that the semantics is tied to "deinit" because in the Swift that's the right design. So requiring it on all types feels weird...

[agocke]

Something like Dispose is fine but I might be careful about using Dispose itself because it could be confused with IDisposable and raise the same problems. Not implementing the interface is a kind of subtle shift that doesn't seem to present a lot of clarity.

Implementing IDisposable on the projected types can trigger analyzer warnings- for example CA2000, or CA2213 - although these analyzers are not enabled by default currently (that said it seems that other tools do report similar diagnostics - Resharper and so on)

This is indeed my main concern -- some IDisposable types require disposal for correct usage. The Swift types don't. It's really hard to tell general IDisposable implementations that do require Dispose from the ones that don't, so my general recommendation for people is to avoid IDisposable if it's not intended to be used in normal operation.

[kotlarmilos]

Closing this PR as it is obsolete. The feedback will be incorporated into the design of the tooling.