Add 'optional' imports/exports (WIP)

Author: lukewagner

design/mvp/Binary.md

@@ -237,7 +237,9 @@ instancedecl  ::= 0x00 t:<core:type>                       => t
                 | 0x04 ed:<exportdecl>                     => ed
 importdecl    ::= na:<nameattributes> et:<externtype>      => (import na et)
 exportdecl    ::= na:<nameattributes> et:<externtype>      => (export na et)
-externtype    ::= 0x00 0x11 i:<core:typeidx>               => (core module (type i))
+externtype    ::= et:<externtype'>                         => et
+                | 0x10 et:<externtype'>                    => optional et ❓
+externtype'   ::= 0x00 0x11 i:<core:typeidx>               => (core module (type i))
                 | 0x01 i:<typeidx>                         => (func (type i))
                 | 0x02 b:<valuebound>                      => (value b) 🪙
                 | 0x03 b:<typebound>                       => (type b)
@@ -308,6 +310,7 @@ canon    ::= 0x00 0x00 f:<core:funcidx> opts:<opts> ft:<typeidx> => (canon lift
            | 0x02 rt:<typeidx>                                   => (canon resource.new rt (core func))
            | 0x03 rt:<typeidx>                                   => (canon resource.drop rt (core func))
            | 0x04 rt:<typeidx>                                   => (canon resource.rep rt (core func))
+           | 0x07 si:<sortidx>                                   => (canon optional.present si (core global)) ❓
            | 0x24                                                => (canon backpressure.inc (core func)) 🔀
            | 0x25                                                => (canon backpressure.dec (core func)) 🔀
            | 0x09 rs:<resultlist> opts:<opts>                    => (canon task.return rs opts (core func)) 🔀
@@ -573,6 +576,7 @@ named once.
 * The two `list` type codes should be merged into one with an optional immediate
   and similarly for `func`.
 * The redundant `0x00` and `0x01` opcodes of `nameattributes` will be merged.
+* The `0x10` for `optional` in `externtype` overlapping with `externtype'`
 * The vestigial `0x00` in `thread.*` and `waitable-set.*` built-ins may be
   removed.
 * Most built-ins should have a `<canonopt>*` immediate instead of an ad hoc

design/mvp/CanonicalABI.md

@@ -40,6 +40,7 @@ specified here.
   * [`canon resource.new`](#canon-resourcenew)
   * [`canon resource.drop`](#canon-resourcedrop)
   * [`canon resource.rep`](#canon-resourcerep)
+  * [`canon optional.present`](#-canon-optionalpresent) ❓
   * [`canon context.get`](#-canon-contextget) 🔀
   * [`canon context.set`](#-canon-contextset) 🔀
   * [`canon backpressure.{inc,dec}`](#-canon-backpressureincdec) 🔀
@@ -3931,6 +3932,21 @@ Note that the "locally-defined" requirement above ensures that only the
 component instance defining a resource can access its representation.
 
 
+### ❓ `canon optional.present`
+
+For a canonical definition:
+```wat
+(canon optional.present $sortidx (core global $present))
+```
+validation specifies:
+* the target of `$sortidx` is declared with `optional`
+* `$present` is given type `(global i32)`
+
+The new immutable global `$present` is appended to the core global index space
+and is either `1` or `0` depending on whether the target of `$sortidx` is
+present or not, resp.
+
+
 ### 🔀 `canon context.get`
 
 For a canonical definition:

design/mvp/Explainer.md

@@ -22,10 +22,12 @@ more user-focused explanation, take a look at the
     * [Definition types](#definition-types)
     * [Declarators](#declarators)
     * [Type checking](#type-checking)
+    * [Optionality](#-optionality)
   * [Canonical definitions](#canonical-definitions)
     * [Canonical ABI](#canonical-abi)
     * [Canonical built-ins](#canonical-built-ins)
       * [Resource built-ins](#resource-built-ins)
+      * [Optionality built-ins](#-optionality-built-ins)
       * [Concurrency built-ins](#-concurrency-built-ins)
       * [Error Context built-ins](#-error-context-built-ins)
   * [Value definitions](#-value-definitions)
@@ -75,6 +77,7 @@ shipped as part of a future WASI Developer Preview release:
 * 🐘: [memory64]
 * 📡: getters and setters
 * ➡️: `stream.forward` and `future.forward` built-ins
+* ❓: `optional` imports and exports
 
 
 ## Grammar
@@ -643,9 +646,11 @@ instancedecl  ::= core-prefix(<core:type>)
                 | <type>
                 | <alias>
                 | <exportdecl>
-importdecl    ::= (import <externnamelit> <attributes> bind-id(<externtype>))
-exportdecl    ::= (export <externnamelit> <attributes> bind-id(<externtype>))
-externtype    ::= (<sort> (type <idx>) )
+importdecl    ::= (import <externnamelit> <attributes> <externtype>)
+exportdecl    ::= (export <externnamelit> <attributes> <externtype>)
+externtype    ::= bind-id(<externtype'>)
+                | optional bind-id(<externtype'>) ❓
+externtype'   ::= (<sort> (type <idx>) )
                 | core-prefix(<core:moduletype>)
                 | <functype>
                 | <componenttype>
@@ -923,9 +928,10 @@ and `export` definitions, respectively, allowing an identifier to be bound for
 use by subsequent declarators. The definitions of `labellit`, `externnamelit`
 and `attribute` are given in the [imports and exports](#import-and-export-definitions)
 section below. Following the precedent of [`core:typeuse`], the text format
-allows both references to out-of-line type definitions (via `(type <typeidx>)`)
-and inline type expressions that the text format desugars into out-of-line type
-definitions.
+for `externtype` allows both references to out-of-line type definitions (via
+`(type <typeidx>)`) and inline type expressions that the text format desugars
+into out-of-line type definitions. The meaning of the `optional` immediate of
+`externtype` is given [below](#optionality).
 
 🪙 The `value` case of `externtype` describes a runtime value that is imported or
 exported at instantiation time as described in the
@@ -1359,6 +1365,71 @@ distinct. Thus, in a sense, the generativity of resource types *generalizes*
 traditional name-based nominal typing, providing a finer granularity of
 isolation than otherwise achievable with a shared global namespace.
 
+#### ❓ Optionality
+
+If the type of an import or export (`externtype`) includes the `optional`
+immediate, this indicates that the import or export may not be "present" at
+instantiation time (on an instance-by-instance basis). Regardless of what
+happens at instantiation time, at validation time, `optional` imports and
+exports unconditionally append an index to the sort's index space and include
+`optional` in the type of this index.
+
+The meaning of "not present" depends on the `externtype'`:
+* For `func`, "not present" means "traps if called".
+* 🪙 For `value`, "not present" means "a `none` value will be passed into any
+  `start` function that takes this value as a parameter". Consequently, an
+  import of type `optional (value $T)` is given the value type `(option $T)`
+  when validating a `start` call.
+* For `instance`, "not present" means "the contained exports are either
+  all-present or all-not-present". Consequently, `alias export $i "e"`
+  produces an `optional`-typed alias for `"e"` if `$i` is `optional` and,
+  conversely, `alias outer $C $d` introduces a *non*-`optional` alias for
+  `$d` even when `$d` is `optional`.
+* For `type`, "not present" means that any dependent `func`, `value` or
+  `instance` also becomes "not present". Consequently, any `externtype`
+  containing a `func`, `value`, `instance` that transitively refers to an
+  `optional` type must itself be `optional`.
+* For `component` and `core module`: these are temporarily rejected by
+  validation, but could be supported in the future by refining the rules for
+  `instantiate` to account for "not present".
+
+For example, the following component exercises these rules and is valid:
+```wat
+(component $C
+  (import "t" optional (type $t (sub resource)))
+  (import "foo" optional (func (result (own $t))))
+  (import "i" optional (instance $i
+    (alias outer $C $t (type $t')) ;; non-optional
+    (export "bar" (func (result (own $t'))))
+    (export "u" (type $u (sub resource)))
+    (export "baz" (func (result (own $u))))
+  ))
+  (alias export $i "u" (type $u')) ;; optional
+  (alias export $i "baz" (func $baz'))
+  (import "quux" optional (func (result (own $u'))))
+)
+```
+The `alias outer` introduces `$t'` as non-`optional` and thus `bar` is not
+required to be `optional`. The nested export of `u` is also non-`optional` and
+thus `baz` is also not required to be `optional`. However, outside the scope of
+`$i`, both `u` and `baz` become `optional` and thus `quux` must be declared
+`optional`.
+
+
+TODO: regarding subtyping:
+* `(X ...) <: (X optional ...)`
+* `(instance (X optional ...)) <: (instance optional (X ...))`
+  * dependently optional on all the parts: present iff all parts are
+
+
+TODO: regarding substitution
+* for `optional` import:
+  * if provide nothing: substitute bottom type (can't name: no valid functions)
+    * e.g., but can still lower (to trap) and call `optional.present`
+  * else: substitute
+    * `optional` is part of what's substituted, so it goes away if provide
+      non-optional def to `with`
+      * e.g., can export with non-optional type
 
 ### Canonical Definitions
 
@@ -1398,7 +1469,7 @@ two directions:
 Canonical definitions specify one of these two wrapping directions, the function
 to wrap and a list of configuration options:
 ```ebnf
-canon    ::= (canon lift core-prefix(<core:funcsortidx>) <canonopt>* bind-id(<externtype>))
+canon    ::= (canon lift core-prefix(<core:funcsortidx>) <canonopt>* bind-id(<functype>))
            | (canon lower <funcsortidx> <canonopt>* (core func <id>?))
 canonopt ::= string-encoding=utf8
            | string-encoding=utf16
@@ -1409,9 +1480,6 @@ canonopt ::= string-encoding=utf8
            | async 🔀
            | (callback core-prefix(<core:funcidx>)) 🔀
 ```
-While the production `externtype` accepts any `sort`, the validation rules
-for `canon lift` would only allow the `func` sort. In the future, other sorts
-may be added (viz., types), hence the explicit sort.
 
 The `string-encoding` option specifies the encoding the Canonical ABI will use
 for the `string` type. The `latin1+utf16` encoding captures a common string
@@ -1558,6 +1626,7 @@ canon ::= ...
         | (canon resource.new <typeidx> (core func <id>?))
         | (canon resource.drop <typeidx> (core func <id>?))
         | (canon resource.rep <typeidx> (core func <id>?))
+        | (canon optional.present <sortidx> (core global <id>?)) ❓
         | (canon context.get <core:valtype> <u32> (core func <id>?)) 🔀
         | (canon context.set <core:valtype> <u32> (core func <id>?)) 🔀
         | (canon backpressure.inc (core func <id>?)) 🔀
@@ -1684,6 +1753,12 @@ transferring ownership of the newly-created resource to the export's caller.
 
 For details, see [`canon_resource_rep`] in the Canonical ABI explainer.
 
+##### ❓ Optionality built-ins
+
+###### ❓ `optional.present`
+
+TODO
+
 ##### 🔀🧵 Concurrency built-ins
 
 See the [concurrency explainer] for background.
@@ -2665,7 +2740,7 @@ exported (e.g., `(export $x "x" (func $f))` binds a new identifier `$x`).
 
 Given this, the syntax of imports and exports are defined as follows:
 ```ebnf
-import        ::= (import <externnamelit> <attributes> bind-id(<externtype>))
+import        ::= (import <externnamelit> <attributes> <externtype>)
 export        ::= (export <id>? <externnamelit> <attributes> <externidx> <externtype>?)
 attributes    ::= <implements>? <versionsuffix>? <externid>?
 implements    ::= (implements <interfacenamelit>) 🏷️

design/mvp/WIT.md

@@ -308,6 +308,9 @@ world union-my-world {
 }
 ```
 
+TODO: add `optional` to some of the `include` examples above (unions *both* imports and exports)
+TODO: add a note that `include` does not produce subtype or supertype, but other ways to combine worlds (`meet` and `join`) could be added in the future
+
 ### De-duplication of interfaces
 
 If two worlds share an imported or exported [interface name], then the union of
@@ -339,6 +342,8 @@ world union-my-world-b {
 }
 ```
 
+TODO: mention merging of `optional` and non-`optional` ~~> non-`optional`
+
 ### Name Conflicts and `with`
 
 When two or more included Worlds have the same name for an import or export
@@ -436,12 +441,6 @@ world invalid-union-world {
 
 ```
 
-### A Note on Subtyping
-
-In the future, when `optional` export is supported, the world author may explicitly mark exports as optional to make a component targeting an included World a subtype of the union World.
-
-For now, we are not following the subtyping rules for the `include` statement. That is, the `include` statement does not imply any subtyping relationship between the included worlds and the union world.
-
 ## WIT Packages and `use`
 [use]: #wit-packages-and-use
 
@@ -563,6 +562,8 @@ interface my-interface {
 }
 ```
 
+TODO: `use` resolution when only used from `optional`y-imported interfaces --> `optional`
+
 #### Top-level `use`
 
 If a package being referred to has a version number, then using the above syntax
@@ -1075,6 +1076,7 @@ keyword ::= 'as'
           | 'list'
           | 'map'
           | 'option'
+          | 'optional'
           | 'own'
           | 'package'
           | 'record'
@@ -1441,10 +1443,10 @@ world-definition ::= export-item
                    | typedef-item
                    | include-item
 
-export-item ::= external-id? 'export' id ':' extern-type
-              | 'export' use-path ';'
-import-item ::= external-id? 'import' id ':' extern-type
-              | 'import' use-path ';'
+export-item ::= external-id? 'export' id ':' 'optional'? extern-type
+              | 'export' 'optional'? use-path ';'
+import-item ::= external-id? 'import' id ':' 'optional'? extern-type
+              | 'import' 'optional'? use-path ';'
 
 external-id ::= '@external-id' '(' string-literal ')' 🏷️
 
@@ -1508,6 +1510,8 @@ world my-world {
 }
 ```
 
+TODO: note `optional` and what it means
+
 [`componenttype`]: Explainer.md#type-definitions
 
 ## Item: `include`