Add 'optional' imports/exports (WIP)

Author: lukewagner

design/mvp/Binary.md

@@ -235,8 +235,10 @@ instancedecl  ::= 0x00 t:<core:type>                       => t
                 | 0x01 t:<type>                            => t
                 | 0x02 a:<alias>                           => a
                 | 0x04 ed:<exportdecl>                     => ed
-importdecl    ::= na:<nameattributes> et:<externtype>      => (import na et)
-exportdecl    ::= na:<nameattributes> et:<externtype>      => (export na et)
+importdecl    ::= na:<nameattributes> oet:<optexterntype>  => (import na oet)
+exportdecl    ::= na:<nameattributes> oet:<optexterntype>  => (export na oet)
+optexterntype ::= et:<externtype>                          => et
+                | 0x10 et:<externtype>                     => optional et
 externtype    ::= 0x00 0x11 i:<core:typeidx>               => (core module (type i))
                 | 0x01 i:<typeidx>                         => (func (type i))
                 | 0x02 b:<valuebound>                      => (value b) 🪙
@@ -308,6 +310,7 @@ canon    ::= 0x00 0x00 f:<core:funcidx> opts:<opts> ft:<typeidx> => (canon lift
            | 0x02 rt:<typeidx>                                   => (canon resource.new rt (core func))
            | 0x03 rt:<typeidx>                                   => (canon resource.drop rt (core func))
            | 0x04 rt:<typeidx>                                   => (canon resource.rep rt (core func))
+           | 0x07 si:<sortidx>                                   => (canon optional.present si (core global)) ❓
            | 0x24                                                => (canon backpressure.inc (core func)) 🔀
            | 0x25                                                => (canon backpressure.dec (core func)) 🔀
            | 0x09 rs:<resultlist> opts:<opts>                    => (canon task.return rs opts (core func)) 🔀
@@ -403,7 +406,7 @@ flags are set.
 (See [Import and Export Definitions](Explainer.md#import-and-export-definitions)
 in the explainer.)
 ```ebnf
-import         ::= na:<nameattributes> et:<externtype>                  => (import na et)
+import         ::= na:<nameattributes> oet:<optexterntype>              => (import na oet)
 export         ::= na:<nameattributes> si:<sortidx> et?:<externtype>?   => (export na si et?)
 nameattributes ::= 0x00 len:<u32> en:<externname>                       => "en"                 (if len = |en|)
                  | 0x01 len:<u32> en:<externname>                       => "en"                 (if len = |en|)
@@ -573,6 +576,7 @@ named once.
 * The two `list` type codes should be merged into one with an optional immediate
   and similarly for `func`.
 * The redundant `0x00` and `0x01` opcodes of `nameattributes` will be merged.
+* The `0x10` for `optional` in `optexterntype` overlapping with `externtype`
 * The vestigial `0x00` in `thread.*` and `waitable-set.*` built-ins may be
   removed.
 * Most built-ins should have a `<canonopt>*` immediate instead of an ad hoc

design/mvp/CanonicalABI.md

@@ -40,6 +40,7 @@ specified here.
   * [`canon resource.new`](#canon-resourcenew)
   * [`canon resource.drop`](#canon-resourcedrop)
   * [`canon resource.rep`](#canon-resourcerep)
+  * [`canon optional.present`](#-canon-optionalpresent) ❓
   * [`canon context.get`](#-canon-contextget) 🔀
   * [`canon context.set`](#-canon-contextset) 🔀
   * [`canon backpressure.{inc,dec}`](#-canon-backpressureincdec) 🔀
@@ -3910,6 +3911,21 @@ Note that the "locally-defined" requirement above ensures that only the
 component instance defining a resource can access its representation.
 
 
+### ❓ `canon optional.present`
+
+For a canonical definition:
+```wat
+(canon optional.present $sortidx (core global $present))
+```
+validation specifies:
+* `$sortidx` must refer to an `(optional <externtype'>)`
+* `$present` is given type `(global i32)`
+
+The new immutable global `$present` is appended to the core global index space
+and is either `1` or `0` depending on whether the `$sortidx` is present or not,
+resp.
+
+
 ### 🔀 `canon context.get`
 
 For a canonical definition:

design/mvp/Explainer.md

@@ -26,6 +26,7 @@ more user-focused explanation, take a look at the
     * [Canonical ABI](#canonical-abi)
     * [Canonical built-ins](#canonical-built-ins)
       * [Resource built-ins](#resource-built-ins)
+      * [Optionality built-ins](#-optionality-built-ins)
       * [Concurrency built-ins](#-concurrency-built-ins)
       * [Error Context built-ins](#-error-context-built-ins)
   * [Value definitions](#-value-definitions)
@@ -75,6 +76,7 @@ shipped as part of a future WASI Developer Preview release:
 * 🐘: [memory64]
 * 📡: getters and setters
 * ➡️: `stream.forward` and `future.forward` built-ins
+* ❓: `optional` imports and exports
 
 
 ## Grammar
@@ -643,8 +645,9 @@ instancedecl  ::= core-prefix(<core:type>)
                 | <type>
                 | <alias>
                 | <exportdecl>
-importdecl    ::= (import <externnamelit> <attributes> bind-id(<externtype>))
-exportdecl    ::= (export <externnamelit> <attributes> bind-id(<externtype>))
+importdecl    ::= (import <externnamelit> <attributes> <optexterntype>)
+exportdecl    ::= (export <externnamelit> <attributes> <optexterntype>)
+optexterntype ::= optional? bind-id(<externtype>)
 externtype    ::= (<sort> (type <idx>) )
                 | core-prefix(<core:moduletype>)
                 | <functype>
@@ -918,19 +921,20 @@ capabilities of the component model.
 
 #### Declarators
 
-The `importdecl` and `exportdecl` declarators correspond to component `import`
-and `export` definitions, respectively, allowing an identifier to be bound for
-use by subsequent declarators. The definitions of `labellit`, `externnamelit`
-and `attribute` are given in the [imports and exports](#import-and-export-definitions)
-section below. Following the precedent of [`core:typeuse`], the text format
+The `importdecl` and `exportdecl` rules describe component-level `import` and
+`export` definitions at the type level. Thus, *both* imports and exports have an
+`externtype`, not just imports. Both declarators allow an identifier to be bound
+for use by subsequent declarators, so that imported and exported types can
+be used to define imported or exported function types.
+
+The definitions of `labellit`, `externnamelit` and `attribute` are given in the
+[imports and exports](#import-and-export-definitions) section below.
+
+Following the precedent of [`core:typeuse`], the text format of `externtype`
 allows both references to out-of-line type definitions (via `(type <typeidx>)`)
 and inline type expressions that the text format desugars into out-of-line type
 definitions.
 
-🪙 The `value` case of `externtype` describes a runtime value that is imported or
-exported at instantiation time as described in the
-[value definitions](#value-definitions) section below.
-
 The `type` case of `externtype` describes an imported or exported type along
 with its "bound":
 
@@ -955,6 +959,38 @@ to some preceding type definition. This allows:
   could generate in C++ a `typedef std::vector<uint64_t> bytes` or in JS an
   exported field named `bytes` that aliases `Uint64Array`.
 
+🪙 The `value` case of `externtype` describes a runtime value that is imported or
+exported at instantiation time as described in the
+[value definitions](#value-definitions) section below.
+
+❓ The `optional` immediate indicates that the containing import or export may
+or may not be "present" at instantiation time. 
+export unconditionally adds an index to `T`'s index space so that index spaces
+have runtime-invariant contents. The meaning of "not present" depends on the
+`externtype'`:
+* For `func`, "not present" means "traps if called".
+* For `instance`, "not present" means "the contained exports are either
+  all-present or all not-present". Consequently, `alias export $i "e"`
+  produces an `optional` alias for `"e"` if `$i` is `optional`.
+* For `type`, "not present" means that any `func` or `instance` `externtype`
+  that transitively refers to a "not present" `type` also becomes "not present",
+  (even if that `func` or `instance` is not declared `optional`).
+* 🪙 For `value`, "not present" means "a `none` value will be passed into any
+  `start` function that takes this value as a parameter" (as described below).
+  Consequently, an import of `(optional (value T))` is given the value type
+  `(option T)` when validating a `start` call.
+
+To determine whether an `optional`-typed `sortidx` is present at runtime, Core
+WebAssembly code can import a `global i32` boolean for that `sortidx` using the
+ABI built-in `optional.present` (as described below).
+
+TODO: For example...
+```wat
+... dependent explicitly optional
+... dependent implicitly optional
+... maybe show `optional.present` for a few and describe the relationship
+```
+
 The `alias` declarator reuses the syntax for component-level `alias` definitions
 so that `export` and `outer` aliases can be used inside `component` and
 `instance` *types*. For example, `export` aliases are necessary to define the
@@ -1090,6 +1126,11 @@ in the future to better support API Evolution (being careful to understand how
 subtyping manifests itself in the wide variety of source languages so that
 subtype-compatible updates don't inadvertently break source-level clients).
 
+❓ TODO: mention `optional` in subtyping:
+* `(X ...) <: (X optional ...)`
+* `(instance (X optional ...)) <: (instance optional (X ...))`
+  * dependently optional on all the parts: present iff all parts are
+
 Module, component and instance subtyping allows a subtype to export more and
 import less than is declared by the supertype, ignoring the exact order of
 imports and exports and considering only names. For example, here, `$I1` is a
@@ -1335,6 +1376,15 @@ replaced by `$R` when validating the instantiations of `$c1` and `$c2`. These
 type-checking rules for instantiating type imports mirror the *elimination*
 rule of [universal types]  (∀T).
 
+❓ TODO:
+* for `optional` import:
+  * if provide nothing: substitute bottom type (can't name: no valid functions)
+    * e.g., but can still lower (to trap) and call `optional.present`
+  * else: substitute
+    * `optional` is part of what's substituted, so it goes away if provide
+      non-optional def to `with`
+      * e.g., can export with non-optional type
+
 Importantly, this type substitution performed by the parent is not visible to
 the child at validation- or run-time. In particular, there are no runtime
 casts that can "see through" to the original type parameter, avoiding
@@ -1558,6 +1608,7 @@ canon ::= ...
         | (canon resource.new <typeidx> (core func <id>?))
         | (canon resource.drop <typeidx> (core func <id>?))
         | (canon resource.rep <typeidx> (core func <id>?))
+        | (canon optional.present <sortidx> (core global <id>?)) ❓
         | (canon context.get <core:valtype> <u32> (core func <id>?)) 🔀
         | (canon context.set <core:valtype> <u32> (core func <id>?)) 🔀
         | (canon backpressure.inc (core func <id>?)) 🔀
@@ -1684,6 +1735,12 @@ transferring ownership of the newly-created resource to the export's caller.
 
 For details, see [`canon_resource_rep`] in the Canonical ABI explainer.
 
+##### ❓ Optionality built-ins
+
+###### ❓ `optional.present`
+
+TODO
+
 ##### 🔀🧵 Concurrency built-ins
 
 See the [concurrency explainer] for background.
@@ -2658,27 +2715,29 @@ A third difference is that not only import definitions, but also export
 definitions append a new element to the index space of the imported/exported
 `sort` which can be optionally bound to an identifier in the text format. In the
 case of imports, the identifier is bound just like Core WebAssembly, as part of
-the `externtype` (e.g., `(import "x" (func $x))` binds the identifier `$x`). In
-the case of exports, the `<id>?` right after the `export` is bound while the
+the `externtype` (e.g., `(import "x" (func $x))` binds the identifier `$x`).
+In the case of exports, the `<id>?` right after the `export` is bound while the
 `<id>` inside the `<externidx>` is a reference to the preceding definition being
 exported (e.g., `(export $x "x" (func $f))` binds a new identifier `$x`).
 
 Given this, the syntax of imports and exports are defined as follows:
 ```ebnf
-import        ::= (import <externnamelit> <attributes> bind-id(<externtype>))
-export        ::= (export <id>? <externnamelit> <attributes> <externidx> <externtype>?)
+import        ::= (import <externnamelit> <attributes> <optexterntype>)
+export        ::= (export <id>? <externnamelit> <attributes> <externidx> <optexterntype>?)
 attributes    ::= <implements>? <versionsuffix>? <externid>?
 implements    ::= (implements <interfacenamelit>) 🏷️
 versionsuffix ::= (version-suffix <semversuffixlit>) 🔗
 externid      ::= (external-id <name>) 🏷️
 ```
 Imports and exports both have a *name* (`externnamelit`) and a list of
-*attributes* (`attributes`). Imports always include a type (`externtype`).
+*attributes* (`attributes`). Imports always include a type (`optexterntype`).
 Exports always refer to a preceding definition (`externidx`) whose type is used
 by default as the type of the export. However, this "inferred" type can be
-overridden with an optional type ascription (`externtype`) which must be
+overridden with an optional type ascription (`optexterntype`) which must be
 compatible with the exported definition's type.
 
+TODO: `optional`
+
 Import and export names are used by hosts and [parent][Donut Wrapped] components
 to *identify* imports and exports. Names are expected to be bound to source
 language variable-, function- and type-names. Thus, names are required to be

design/mvp/WIT.md

@@ -308,6 +308,9 @@ world union-my-world {
 }
 ```
 
+TODO: add `optional` to some of the `include` examples above (unions *both* imports and exports)
+TODO: add a note that `include` does not produce subtype or supertype, but other ways to combine worlds (`meet` and `join`) could be added in the future
+
 ### De-duplication of interfaces
 
 If two worlds share an imported or exported [interface name], then the union of
@@ -339,6 +342,8 @@ world union-my-world-b {
 }
 ```
 
+TODO: mention merging of `optional` and non-`optional` ~~> non-`optional`
+
 ### Name Conflicts and `with`
 
 When two or more included Worlds have the same name for an import or export
@@ -436,12 +441,6 @@ world invalid-union-world {
 
 ```
 
-### A Note on Subtyping
-
-In the future, when `optional` export is supported, the world author may explicitly mark exports as optional to make a component targeting an included World a subtype of the union World.
-
-For now, we are not following the subtyping rules for the `include` statement. That is, the `include` statement does not imply any subtyping relationship between the included worlds and the union world.
-
 ## WIT Packages and `use`
 [use]: #wit-packages-and-use
 
@@ -563,6 +562,8 @@ interface my-interface {
 }
 ```
 
+TODO: `use` resolution when only used from `optional`y-imported interfaces --> `optional`
+
 #### Top-level `use`
 
 If a package being referred to has a version number, then using the above syntax
@@ -1075,6 +1076,7 @@ keyword ::= 'as'
           | 'list'
           | 'map'
           | 'option'
+          | 'optional'
           | 'own'
           | 'package'
           | 'record'
@@ -1441,10 +1443,10 @@ world-definition ::= export-item
                    | typedef-item
                    | include-item
 
-export-item ::= external-id? 'export' id ':' extern-type
-              | 'export' use-path ';'
-import-item ::= external-id? 'import' id ':' extern-type
-              | 'import' use-path ';'
+export-item ::= external-id? 'export' id ':' 'optional'? extern-type
+              | 'export' 'optional'? use-path ';'
+import-item ::= external-id? 'import' id ':' 'optional'? extern-type
+              | 'import' 'optional'? use-path ';'
 
 external-id ::= '@external-id' '(' string-literal ')' 🏷️
 
@@ -1508,6 +1510,8 @@ world my-world {
 }
 ```
 
+TODO: note `optional` and what it means
+
 [`componenttype`]: Explainer.md#type-definitions
 
 ## Item: `include`