-
Notifications
You must be signed in to change notification settings - Fork 162
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add .NET Swift interop tooling components and layout #312
Changes from all commits
8cf069f
9c21864
76d1433
ea4d636
0bb0e3c
659e44d
722a72c
4ccb501
b9325c6
84b2fa4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -130,9 +130,15 @@ CoreCLR and NativeAOT currently block the `VectorX<T>` types from P/Invokes as t | |
|
||
##### Automatic Reference Counting and Lifetime Management | ||
|
||
Swift has a strongly-defined lifetime and ownership model. This model is specified in the Swift ABI and is similar to Objective-C's ARC (Automatic Reference Counting) system. When .NET calls into Swift, the .NET GC is responsible for managing all managed objects. Unmanaged objects from C# should either implement `IDisposable` or utilize a designated thin wrapper over the Swift memory allocator, currently accessible through the `NativeMemory` class, to explicitly release memory. It's important to ensure that when a Swift callee function allocates an "unsafe" or "raw" pointer types, such as UnsafeMutablePointer and UnsafeRawPointer, where explicit control over memory is needed, and the pointer is returned to .NET, the memory is not dereferenced after the call returns. Also, if a C# managed object is allocated in a callee function and returned to Swift, the .NET GC will eventually collect it, but Swift will keep track using ARC, which represents an invalid case and should be handled by projection tools. | ||
Swift has a strongly-defined lifetime and ownership model. This model is specified in the Swift ABI and is similar to Objective-C's ARC (Automatic Reference Counting) system. When .NET calls into Swift, the .NET GC is responsible for managing all managed objects. | ||
|
||
The Binding Tools for Swift tooling handles these explicit lifetime semantics with generated Swift code. In the new Swift/.NET interop, management of these lifetime semantics will be done by the Swift projection layer and not by the raw calling-convention support. If any GC interaction is required to handle the lifetime semantics correctly, we should take an approach more similar to the `ComWrappers` support (higher-level, less complex interop interface) rather than the Objective-C interop support (lower-level, basically only usable by the ObjCRuntime implementation). | ||
There are two strategies for managing native memory in .NET: destructors and `IDisposable`. Destructors abstract away memory management from the user and are managed by the GC. They provide a way to release unmanaged resources when an object is collected by the GC. `IDisposable` offers an explicit mechanism for releasing unmanaged resources with deterministic control over when resources are released. The preferred behavior for general cases would be to implement destructors. This approach aligns with the .NET pattern and offers codegen benefits by avoiding excessive `using` statements. Ideally, the tooling should only use `IDisposable` for custom deinit. | ||
|
||
Using `IDisposable` can lead to better overall performance as it prevents the need for GC collection cycles. However, implementing Swift types as `IDisposable` in .NET can be confusing for customers. In that case, every Swift type would come with `IDisposable`, and without knowing the details of the type, it may be challenging for the caller to determine whether to explicitly dispose it. | ||
|
||
Swift type projections in C# should implement destructors and utilize a designated thin wrapper over the Swift memory allocator, currently accessible through the NativeMemory class, to explicitly release memory. In custom scenarios where more than just memory is being freed, they will implement `IDisposable` pattern. It's important to ensure that when a Swift callee function allocates an "unsafe" or "raw" pointer types, such as UnsafeMutablePointer and UnsafeRawPointer, where explicit control over memory is needed, and the pointer is returned to .NET, the memory is not dereferenced after the call returns. Also, if a C# managed object is allocated in a callee function and returned to Swift, the .NET GC will eventually collect it, but Swift will keep track using ARC, which represents an invalid case and should be handled by projection tools. | ||
|
||
In the new Swift/.NET interop, management of these lifetime semantics will be done by the Swift projection layer and not by the raw calling-convention support. If any GC interaction is required to handle the lifetime semantics correctly, we should take an approach more similar to the `ComWrappers` support (higher-level, less complex interop interface) rather than the Objective-C interop support (lower-level, basically only usable by the ObjCRuntime implementation). | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's worse than this. Value types in swift have specific semantics for what happens when an instance goes out of scope. This is, of course, completely different than the semantics of C#, but can be approximated by making the type There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah I think the best way to model structs is either with an There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks, updated. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What is the purpose of the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is for disposal or release of native resources. In this case that is the intent. The fact that it is a refcount on the native side seems like implementation detail of the taret platform we are interoping with. |
||
### Projecting Swift into .NET | ||
|
||
|
@@ -142,7 +148,13 @@ All designs in this section should be designed such that they are trimming and A | |
|
||
#### Swift to .NET Language Feature Projections | ||
|
||
##### Structs/Value Types | ||
The following subheadings describe projections of Swift types into C#. This section illustrates general mechanisms and practices applied in the tooling projection. The complete documentation of the projection tooling is available at https://github.com/dotnet/runtimelab/tree/feature/swift-bindings/docs. | ||
|
||
##### Primitive types | ||
|
||
Swift primitive types are implemented as frozen structs that conform to Swift-specific lowering processes handled by the runtime. However, most of these types are below the size limit for being passed by reference and can fit within the underlying calling convention. | ||
|
||
##### Structs/Enums | ||
|
||
Unlike .NET, Swift's struct types have strong lifetime semantics more similar to C++ types than .NET structs. At the Swift ABI layer, there are broadly three types of structs/enums: "POD/Trivial" structs, "Bitwise Takable/Movable" structs, and non-bitwise movable structs. The [Swift documentation](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#layout-and-properties-of-types) covers these different kinds of structs. Let's look at how we could map each of these categories of structs into .NET. | ||
|
||
|
@@ -152,11 +164,23 @@ Unlike .NET, Swift's struct types have strong lifetime semantics more similar to | |
|
||
Structs that are non-bitwise-movable are more difficult. They cannot be moved by copying their bits; their copy constructors must be used in all copy scenarios. When mapping these structs to C#, we must take care that we do not copy the underlying memory and to call the deallocate function when the C# usage of the struct falls out of scope. These use cases best match up to C# class semantics, not struct semantics. | ||
|
||
We plan to interop with Swift's Library Evolution mode, which brings an additional wrinkle into the Swift struct story. Swift's Library Evolution mode abstracts away all type layout and semantic information unless a type is explicitly marked as `@frozen`. In the Library Evolution case, all structs have "opaque" layout, meaning that their exact layout and category cannot be determined until runtime. As a result, we need to treat all "opaque" layout structs as possibly non-bitwise-movable at compile time as we will not know until runtime what the exact layout is. Swift/C++ interop is not required to use the Library Evolution mode in all cases as it can statically link against Swift libraries, so it is not limited by opaque struct layouts in every case. The size and layout information of a struct is available in its [Value Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#value-witness-table), so we can look up this information at runtime for allocating struct instances and manipulating struct memory correctly. | ||
We plan to interop with Swift's Library Evolution mode, which brings an additional wrinkle into the Swift struct story. Swift's Library Evolution mode abstracts away all type layout and semantic information unless a type is explicitly marked as `@frozen`. In the Library Evolution case, all structs have "opaque" layout, meaning that their exact layout and category cannot be determined until runtime. The size and layout information of concrete types is available in its [Value Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#value-witness-table), so we can look up this information at runtime for allocating struct instances and manipulating struct memory correctly. As a result, we need to treat all "opaque" layout structs as possibly non-bitwise-movable at compile time as we will not know until runtime what the exact layout is. Swift/C++ interop is not required to use the Library Evolution mode in all cases as it can statically link against Swift libraries, so it is not limited by opaque struct layouts in every case. Every concrete type in Swift has a structure that provides information about how to manipulate values of that type. | ||
|
||
##### Tuples | ||
Swift structs and enums have richer semantics than in .NET and are projected as C# classes to streamline handling of both simple blittable and more complex scenarios. These C# classes have a single property that holds the data payload for the type. They typically include two constructors: one that corresponds to the init method in the Swift class, and another internal constructor used to create uninitialized types invoked by the marshaler in cases when it is a return value from a function. | ||
|
||
##### Classes/Protocols | ||
|
||
If possible, Swift tuples should be represented as `ValueTuple`s in .NET. If this is not possible, then they should be represented as types with a `Deconstruct` method similar to `ValueTuple` to allow a tuple-like experience in C#. | ||
A public Swift class is projected as a final C# class, while a virtual Swift class is projected as an internal C# class. A final class has a straightforward inheritance model, while a virtual class introduces more complexity, particularly related to subclassing and simulated vtable methods. Another important type in Swift is protocols. Swift allows any type to implement a protocol and supports retroactive modeling through extensions. Since the protocol's implementation can't be part of the object, Swift uses a [Protocol Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#protocol-witness-tables), which functions like a vtable for each conformance. A protocol type in Swift is represented using an [existential container](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#existential-metadata) that includes payload, type metadata pointer, and a protocol witness table pointer. | ||
|
||
Projections can utilize [`IUnmanagedVirtualMethodTableProvider` interface](https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.marshalling.iunmanagedvirtualmethodtableprovider?view=net-8.0) to retrieve vtable for a given target type from an object. | ||
|
||
This subheading will be updated with more details on projection once simpler types are reviewed, like structs and enums. | ||
|
||
##### Tuples/Closures | ||
|
||
A Swift tuple can be projected as `ValueTuple` type in C#. Swift uses two types of closures: escaping and non-escaping. Escaping closures can exist beyond their original context, while non-escaping closures should not persist beyond their declaration context and cannot reference external data. The Swift compiler has a mechanism to convert a non-escaping closure into an escaping one within the scope of another closure. | ||
|
||
This subheading will be updated with more details on projection once simpler types are reviewed, like structs and enums. | ||
|
||
##### SIMD types | ||
|
||
|
@@ -171,18 +195,9 @@ As mentioned in the calling-convention section above, none of the libraries we a | |
|
||
#### Projection Tooling Components | ||
|
||
The projection tooling should be split into these components: | ||
The projection tooling is based on the [Binding Tools for Swift](https://github.com/xamarin/binding-tools-for-swift). The tooling contain components that can consume a compiled Apple Swift library interface and generate C# source code bindings that allow it to be surfaced as a .NET library. The tool will not generate any Swift wrappers and it's users responsibility to provide Swift wrappers for cases where direct binding is not possible. The projection tooling will utilize the runtime core interop source-gen infrastructure to implement marshalling codegen. | ||
|
||
##### Importing Swift into .NET | ||
|
||
1. A tool that takes in a `.swiftinterface` file or Swift sources and produces C# code. | ||
2. A library that provides the basic support for Swift interop that the generated code builds on. | ||
3. User tooling to easily generate Swift projections for a given set of `.framework`s. | ||
- This tooling would build a higher-level interface on top of the tool in item 1 that is more user-friendly and project-system-integrated. | ||
4. (optional) A NuGet package, possibly referencable by `FrameworkReference` or automatically included when targeting macOS, Mac Catalyst, iOS, or tvOS platforms that exposes the platform APIs for each `.framework` that is exposed from Swift to .NET. | ||
- This would be required to provide a single source of truth for Swift types so they can be exposed across an assembly boundary. | ||
|
||
##### Exporting .NET to Swift | ||
##### Exporting .NET to Swift | ||
|
||
There are two components to exporting .NET to Swift: Implementing existing Swift types in .NET and passing instances of those types to Swift, and exposing novel types from .NET code to Swift code to be created from Swift. Exposing novel types from .NET code to Swift code is considered out of scope at this time. | ||
|
||
|
@@ -191,13 +206,17 @@ For implementing existing Swift types in .NET, we will require one of the follow | |
1. A Roslyn source generator to generate any supporting code needed to produce any required metadata, such as type metadata and witness tables, to pass instances of Swift-type-implementing .NET types defined in the current project to Swift. | ||
2. An IL-post-processing tool to generate the supporting code and metadata from the compiled assembly. | ||
|
||
If we were to use an IL-post-processing tool here, we would break Hot Reload in assemblies that implement Swift types, even for .NET-only code, due to introducing new tokens that the Hot Reload "client" (aka Roslyn) does not know about. As a result, we should prefer the Roslyn source generator approach. | ||
If we want to use an IL-post-processing tool here, we would break Hot Reload in assemblies that implement Swift types, even for .NET-only code, due to introducing new tokens that the Hot Reload "client" (aka Roslyn) does not know about. | ||
|
||
### Distribution | ||
|
||
The calling convention work will be implemented by the .NET runtimes in dotnet/runtime. | ||
The projection tooling will be implemented as a .NET CLI tool and integrated into the Xamarin publishing infrastructure. It will be included in the macios workload for Apple platforms and also available as a standalone package independent of MAUI framework. | ||
|
||
The calling convention work will be implemented in the [dotnet/runtime](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/Swift/SwiftTypes.cs) repository. | ||
|
||
### Validation | ||
|
||
The projection tooling will not ship as part of the runtime. It should be available as a separate NuGet package, possibly as a .NET CLI tool package. The projections should either be included automatically as part of the TPMs for macOS, iOS, and tvOS, or should be easily referenceable. | ||
The interop will be showcased through CryptoKit library in the runtime repository and MAUI libraries and samples: https://github.com/dotnet/runtime/issues/95636. | ||
|
||
## Q & A | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are the compatibility rules for adding/removing custom deinit in Swift?
Are introduction or removal of a custom deinit considered to be a compatible change in Swift? If they are compatible changes, it would be problematic to map custom deinit to IDIsposable. Adding/removing IDisposable is a breaking change in .NET.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is unfortunate -- I bet Swift is fine with removing a deinit since there's always an implicit deinit that it can fall back on.
I think the fundamental problem here is that Swift does not have a distinction between "deterministic" destruction and non-deterministic destruction. In contrast, C# does. And, importantly, that distinction is very important in C#. If you require deterministic destruction then timely destruction may be a requirement for your application, where delaying destruction may cause your application to fail. Conversely, if you don't require it, using deterministic destruction (
IDisposable
/using) for all resources is very expensive and will likely cause significant performance degradation. It's also a large programmer burden as disposability is transitive and any IDisposable fields require the parent type to also be IDisposable."Custom deinit" is a heuristic that attempts to map between C#, which differentiates the two concepts, and Swift, which doesn't. But since there is no way to determine statically whether a Swift type actually requires deterministic destruction, I don't think there's any heuristic that will be perfectly accurate.
That said, maybe we can do better than "custom deinit". Suggestions welcome.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this imply that we should have a solid reason for implementing the
IDisposable
pattern on a type as introducing or removing explicit memory management viaIDisposable
could lead to breaking changes?This is the case where direct mapping is not possible, and introducing
IDisposable
only when absolutely necessary can help minimize breaking changes caused by switching to/from the explicit memory management.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another option here would be to follow the built-in COM mechanism and add a helper function like
Marshal.FinalReleaseComObject()
. This could be onMarshal
or another more targeted type, and use any number of tricks to provide efficient deterministic release semantics. This has the added benefit of not forcing users to litter their code with type checks/casts.Note that
ComWrappers
did useIDisposable
for specific COM scenarios. In that case though the user is driving the experience by explicitly requesting the deterministic support - see this section of theComWrappers
tutorial.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll offer a datapoint here that there are at least 4 issues at play here:
weak
In BTfS, when we are presented with a Swift object, the C# binding implements
IDisposable
and takes a swift weak reference to the object via the swift runtime routineswift_unownedRetain
and keep a weak GCHandle.Dispose (false)
gets called by the finalizer.Initially, I tried to draw a distinction between blittable and non-blittable value types, but things that make that challenging were all the special cases: value types that contain mutating methods or properties,
inout
parameters, frozen/not frozen, opaque layout of enums etc. It was an endless supply of bugs in the marshaling. My solution for this was to make all values type handling uniform.Not being an expert on it, does COM interop exist on non-windows platforms?
I'm pinging @rolfbjarne here as well since he was responsible for the handling of ObjC mapping, which required similar work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's review some examples of existing bindings to gain a deeper understanding of how the
IDisposable
pattern is utilized. I've done a review of the Xamarin Google APIs for iOS Components and Google Mobile Unity Ads, focusing specifically on the following applications: cloud messaging, mobile ads, analytics, and crashlytics. The goal of this review was to identify the use cases of IDisposable patterns. The collected data could help us move this discussion/decision forward.In the existing Xamarin ObjC interop, the
NSObject
represents a base object and implements IDisposable.Cloud Messaging
In the cloud messaging bindings, the following APIs require implementing the IDisposable pattern:
MessageInfo
MessagingDelegate
Messaging
MessagingExtensionHelper
In the sample application, these types are used in AppDelegate, but there are no explict disposing of objects.
There is another example where IDisposable pattern is used to remove notification handles
Analytics
In the analytics bindings, the following APIs require implementing the IDisposable pattern:
Analytics
DictionaryBuilder
Fields
Logger
TrackedViewController
Tracker
EcommerceFields
EcommerceProduct
EcommerceProductAction
EcommercePromotion
In the samples, they are not explicitly released.
MobileAds
In the mobile ads bindings, there are 43 interfaces with base types implementing IDisposable section.
There are 6 instances of disposal in the sample application:
GoogleAds Mobile Unity
In the google mobile bindings, the following types implement IDisposable, but none of them are disposed explicitly.
AdManagerBannerClient
AppOpenAdClient
BannerClient
InterstitialClient
NativeOverlayAdClient
RewardedAdClient
RewardedInterstitialAdClient
Crashlytics
In the crashlytics bindings, the following APIs require implementing IDisposable:
Crashlytics
ExceptionModel
StackFrame
In the sample application, none of them are disposed explicitly.
Xamarin Designer
The Xamarin Designer (deprecated) for iOS is a visual designer that contain controls with IDisposable pattern. Samples from the mentioned bindings that implement Xamarin Designer templates contain a ReleaseDesignerOutlets function that explicitly releases all unmanaged references. Here is an example:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @kotlarmilos, this is a lot of great work!
Based on real-world examples, it doesn't seem like people actually use Dispose in most cases. Based on that, I would recommend using a static method like Marshal.ReleaseComObject (or even a different interface if necessary) rather than adding a bunch of ceremony to the API, but that's just my opinion.
My main concern is accurately providing guidance to authors on whether or not manual disposal is expected, and it seems like it's not.
@jkoritzinsky @AaronRobinsonMSFT @jkotas thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1. Thank you for this sort of due-diligence and investigation.
Agree. I accept this is anecdotal, but it does paint a compelling argument for avoiding a ubiquitious
IDisposable
requirement. I think the onus is now on the other side of the argument to make a case forIDisposable
everywhere.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 on Andy's and Aaron's responses - and thanks @kotlarmilos for the analysis.
Just curious - did we consider adding an instance method onto the base type (assuming there will be a base type for all projected types)? I know this is sort of against .NET/C#, but I might go as far as adding a
void Dispose()
onto the base type without implementingIDisposable
. Unfortunately C#'susing
is not pattern based (unlike most other C# features) and needs the type to implementIDisposable
. But it would make the disposing feel a bit more first class and more discoverable I think.Additional notes:
IDisposable
on the projected types can trigger analyzer warnings- for example CA2000, or CA2213 - although these analyzers are not enabled by default currently (that said it seems that other tools do report similar diagnostics - Resharper and so on)IDisposable
would be a good idea (like the above mentioned photos) - this means that eventually the tooling should be able to take a hint to implement the interface on a given projected type.Dispose
is called are actually not about releasing resource pressure, but about actual functionality - like de-registering something. It just so happens that the semantics is tied to "deinit" because in the Swift that's the right design. So requiring it on all types feels weird...There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Something like
Dispose
is fine but I might be careful about usingDispose
itself because it could be confused with IDisposable and raise the same problems. Not implementing the interface is a kind of subtle shift that doesn't seem to present a lot of clarity.This is indeed my main concern -- some
IDisposable
types require disposal for correct usage. The Swift types don't. It's really hard to tell generalIDisposable
implementations that do require Dispose from the ones that don't, so my general recommendation for people is to avoidIDisposable
if it's not intended to be used in normal operation.