Untold Editor — Component Extension System Proposal

[miogds]

I propose a lightweight, code-first Component Extension System for the Untold Editor. The editor will run macOS-only for now, while a separate visionOS Remote Display app will mirror rendering output and forward input to the macOS editor. Because visionOS (like other Apple App Stores) prohibits dynamic native code loading, the extension mechanism resides entirely on macOS. Extensions are authored as Swift/SwiftPM/Xcode modules, compiled and loaded by the macOS editor. The visionOS app renders frames streamed from macOS and relays input; it does not load third-party code.

At the core, we use Swift compiler plugins + macros to mark component properties for in-editor editing and hot binding without boilerplate (e.g., @EditorFloat, @EditorEnum). The compiler plugin emits machine-readable metadata that the editor ingests to auto-build UI controls and runtime bindings.

---

Goals & Non-Goals

Goals

• G1: macOS-only Editor (initially) with remote rendering to visionOS via a companion app that mirrors output and forwards input.
• G2: Bring back component extensions with a simple, declarative authoring model (attribute-driven).
• G3: Zero extra glue code: annotations generate metadata + bindings automatically.
• G4: Support loading project code (components) from Xcode/SPM builds directly into the editor on macOS.
• G5: Maintain parity with the pre-split component system for runtime semantics.

Non-Goals (Phase 1)

• N1: Native code extensions on visionOS.
• N2: Scripting-VM performance work (beyond basic console tooling).
• N3: On-device compilation for visionOS.
• N4: Network multi-user co-editing.

---

Platform Architecture Overview

macOS Editor (authoritative)

• Hosts the scene graph, asset pipeline, and component extension loader.
• Compiles/links project code (SPM/Xcode).
• Provides the Editor Service (IPC/WebSocket/QUIC) that the remote display uses.

visionOS Remote Display (companion app)

• No dynamic code loading.
• Receives streamed render frames (or GPU textures) from the macOS Editor.
• Forwards input (gaze/hand gestures/taps/keyboard) back to macOS Editor.
• Optional lightweight tooling overlays (HUDs) driven entirely by editor-pushed metadata.

Data Flow

1. Project opens in macOS Editor → builds components (debug profile) → emits Editor Metadata.
2. Editor loads compiled component bundles; binds UI based on metadata.
3. Editor renders via Metal → streams to visionOS app.
4. visionOS app forwards input → Editor updates state → re-renders.

---

Component Extension System Design

Authoring Model (Macros/Attributes)

Developers annotate Swift component properties with editor-aware macros:

final class SunLightComponent: Component {
    @EditorFloat(min: 0.0, max: 100_000.0, step: 10)
    var intensity: Float = 10_000

    @EditorColor
    var color: ColorRGB = .white

    @EditorEnum(options: ["Directional", "Point", "Spot"]) 
    var type: LightType = .directional

    @EditorBool
    var castsShadows: Bool = true
}

Supported macros (initial set):

• @EditorFloat(min:max:step:)
• @EditorInt(min:max:step:)
• @EditorBool
• @EditorColor
• @EditorEnum(options:default:)
• @EditorVector2/3/4
• @EditorQuat
• @EditorAsset(type:) (e.g., texture, mesh)
• @EditorGroup("Lighting") / @EditorCategory("Rendering")
• @EditorHidden (exposed in metadata but hidden in default UI)

Compiler Plugin Responsibilities

We use Swift Compiler Plugins + SwiftSyntax to:

• Validate macro usage and types.
• Generate a per-target <TargetName>.editor.json (or swift static registry) describing components, fields, types, attributes, constraints, and default values.
• (Optional) Emit a lightweight reflection table in Swift for runtime (for direct bindings without JSON parse on macOS).

Example metadata (excerpt):

{
  "components": [
    {
      "type": "SunLightComponent",
      "displayName": "Sun Light",
      "fields": [
        {
          "name": "intensity",
          "kind": "float",
          "min": 0.0,
          "max": 100000.0,
          "step": 10,
          "default": 10000
        },
        { "name": "color", "kind": "color", "default": [1,1,1] },
        { "name": "type", "kind": "enum",          "options": ["Directional","Point","Spot"], "default": "Directional" },
        { "name": "castsShadows", "kind": "bool", "default": true }
      ]
    }
  ]
}

Editor Runtime (Binding Layer)

• At project load, the Editor scans the build products for *.editor.json (or a compiled registry symbol), merges them, and auto-builds inspector UI.
• A BindingAdapter maps field edits to component instances via generated accessors or reflection (keypaths or function pointers).
• Changes propagate to the live scene and are serialized back to scene files.

UI Generation

• Inspector panels are created dynamically based on metadata.
• Grouping: @EditorGroup/@EditorCategory drive tabs/sections.
• Constraints (min/max/step) configure sliders/steppers.
• Enums render as segmented controls or dropdowns.
• Color uses native macOS color well + HDR support when available.

Backwards Compatibility

• Legacy components (pre-split) remain valid; if no macros are found, fields can be surfaced via a compat shim (manual registry or default reflection) to preserve prior behavior.

---

Build, Load, and Project Integration

Xcode/SPM Layout

• Components live in one or more SPM targets or Xcode subprojects within the main game/app workspace.
• The Editor Plugin is added as a build dependency for those targets to emit *.editor.json and optional registries.
• The Editor Host app watches the build folder (DerivedData or a configured Products/) for updated artifacts.

Loading Strategy (macOS)

• Compile components as .dylib or .bundle with stable ABI entry points (e.g., unbtols_component_init()).
• On editor start or project (re)build, the editor loads bundles dynamically.
• Code signing is required for hardened runtime; local development uses developer-signed artifacts.

Serialization

• Scene and prefab files store component types + field values.
• A schema version accompanies metadata; the editor migrates data if fields are renamed or types change.

---

Remote Rendering for visionOS

Constraints & Rationale

• visionOS App Store disallows dynamic native code loading; scripting is allowed but too slow for our use case.
• Therefore, all third-party/native extensions run on macOS.
• visionOS app is a remote viewer + input forwarder.

Link & Transport

• Preferred: QUIC or WebRTC for low-latency bi-directional transport (NAT-friendly).

Latency Targets

• Motion-to-photon: ≤ 40–60 ms (LAN), with frame pacing from the Editor.
• Adaptive bitrate and resolution scaling based on link quality.

Tooling Overlays

• HUD overlays are editor-driven, described via a tiny UI schema (buttons, sliders, pickers) pushed to the client; the client never executes extension code.

---

Security, Signing, and Stability

• macOS only dynamic loading; visionOS app contains no untrusted code.
• Enforce codesigning for bundles; optionally require a developer team ID allowlist.
• Sandboxed IPC with strict message schemas.
• Crash isolation: load extensions into a helper process with XPC; hot-reload by restarting helper without killing the editor.

---

Example: End-to-End Developer Flow

1. Create MyGameplayComponents SPM target.
2. Add dependency on UnbtolsEditorPlugin and import EditorMacros.
3. Annotate fields with @Editor* macros.
4. Build project in Xcode → plugin emits MyGameplayComponents.editor.json and registry.
5. Open project in Editor → dynamic load of MyGameplayComponents.bundle → inspector auto-populates.
6. Open visionOS Remote Display → see live scene, tweak properties in macOS editor; changes visible instantly in the headset.

---

Public API (Draft)

Macros

@propertyWrapper public struct EditorFloat { /* init(min:max:step:) */ }
@propertyWrapper public struct EditorInt { /* ... */ }
@propertyWrapper public struct EditorBool { }
@propertyWrapper public struct EditorColor { }
@propertyWrapper public struct EditorEnum<T: CaseIterable & Codable> { /* ... */ }
@propertyWrapper public struct EditorVector2 { }
@propertyWrapper public struct EditorVector3 { }
@propertyWrapper public struct EditorVector4 { }
@propertyWrapper public struct EditorQuat { }
@propertyWrapper public struct EditorAsset<T: AssetKind> { }

@attached(member) public macro EditorGroup(_ name: String) // grouping
@attached(member) public macro EditorCategory(_ name: String) // top-level tab
@attached(member) public macro EditorHidden() // hide in default UI

Registry & Hooks

public protocol Component: AnyObject { /* marker */ }
public protocol EditorRegistrableComponent: Component {
    static func editorTypeName() -> String
}

public struct EditorFieldMeta { /* name, kind, constraints, default */ }
public struct EditorComponentMeta { /* typeName, fields, groups, version */ }

public protocol EditorMetadataProvider {
    static func editorMetadata() -> EditorComponentMeta
}

// Optional runtime hook for value changed
public protocol EditorObservableField {
    mutating func willSetEditorValue(_ newValue: Any)
    mutating func didSetEditorValue(_ newValue: Any)
}

---

Migration from Pre-Split System

• Keep existing component registry reading logic.
• Introduce a dual-path loader: (a) legacy reflection/registry, (b) new metadata path.
• Provide a script to auto-annotate common scalar fields with sensible defaults.

---

Minimal Working Example

// Component
final class NewComponent: Component {
    @EditorFloat(min: 0, max: 1, step: 0.01)
    var sunlight: Float = 0.5
}

// Generated (conceptual)
struct __NewComponent_EditorMeta: EditorMetadataProvider {
    static func editorMetadata() -> EditorComponentMeta {
        .init(typeName: "NewComponent", fields: [
            .init(name: "sunlight", kind: .float, min: 0, max: 1, step: 0.01, default: 0.5)
        ], groups: [], version: 1)
    }
}

// Editor auto-UI (pseudo)
Inspector.register(meta: __NewComponent_EditorMeta.editorMetadata())