first commit

packages/@n8n/expression-runtime/docs/architecture-diagram.mmd

@@ -0,0 +1,91 @@
+%% Expression Runtime Architecture
+%% Three-layer design for environment-agnostic expression evaluation
+
+graph TB
+    subgraph "Host Process"
+        WF[Workflow Package]
+
+        subgraph "Layer 3: Evaluator"
+            EVAL[ExpressionEvaluator]
+            TOUR[Tournament]
+            CACHE[Code Cache]
+            OBS[Observability]
+        end
+
+        subgraph "Layer 2: Bridge"
+            BRIDGE_IF[RuntimeBridge Interface]
+            ISOVM[IsolatedVmBridge]
+            WEBW[WebWorkerBridge]
+            TASKR[TaskRunnerBridge]
+        end
+
+        DATASTORE[(Data Store)]
+    end
+
+    subgraph "Isolated Context (isolate/worker/subprocess)"
+        subgraph "Layer 1: Runtime"
+            RUNTIME[Runtime Entry]
+            PROXY[Lazy Proxy]
+            HELPERS[Helper Functions]
+            LODASH[lodash]
+            LUXON[Luxon]
+        end
+    end
+
+    WF -->|evaluate| EVAL
+    EVAL --> TOUR
+    EVAL --> CACHE
+    EVAL --> OBS
+    EVAL -->|execute| BRIDGE_IF
+
+    BRIDGE_IF -.->|implements| ISOVM
+    BRIDGE_IF -.->|implements| WEBW
+    BRIDGE_IF -.->|implements| TASKR
+
+    ISOVM -->|IPC/Reference| RUNTIME
+    WEBW -->|postMessage| RUNTIME
+    TASKR -->|IPC| RUNTIME
+
+    RUNTIME --> PROXY
+    RUNTIME --> HELPERS
+    RUNTIME --> LODASH
+    RUNTIME --> LUXON
+
+    PROXY -.->|getData request| ISOVM
+    ISOVM --> DATASTORE
+    DATASTORE -.->|value| ISOVM
+    ISOVM -.->|value| PROXY
+
+    style EVAL fill:#e1f5ff
+    style BRIDGE_IF fill:#fff4e1
+    style RUNTIME fill:#f0ffe1
+
+    style ISOVM fill:#fff4e1,stroke:#ff9800
+    style WEBW fill:#fff4e1,stroke:#9e9e9e,stroke-dasharray: 5 5
+    style TASKR fill:#fff4e1,stroke:#9e9e9e,stroke-dasharray: 5 5
+
+%% Data Flow Sequence
+
+sequenceDiagram
+    participant WF as Workflow
+    participant Eval as ExpressionEvaluator
+    participant Bridge as IsolatedVmBridge
+    participant Runtime as Runtime (Isolated)
+
+    WF->>Eval: evaluate(expr, data)
+    Eval->>Eval: Transform with Tournament
+    Eval->>Eval: Check code cache
+    Eval->>Bridge: execute(code, data)
+    Bridge->>Bridge: Register ivm.Reference callbacks with data
+    Bridge->>Runtime: evalSync("resetDataProxies()")
+    Runtime->>Runtime: Create lazy proxies for $json, $input, etc.
+    Bridge->>Runtime: Run compiled script
+
+    Runtime->>Runtime: Access $json.email
+    Runtime->>Bridge: __getValueAtPath(['$json','email']) [ivm.Reference]
+    Bridge->>Bridge: Navigate path in data
+    Bridge-->>Runtime: Value
+
+    Runtime-->>Bridge: Expression result
+    Bridge-->>Eval: Result
+    Eval-->>WF: Result

packages/@n8n/expression-runtime/docs/deep-lazy-proxy.md

@@ -0,0 +1,235 @@
+# Deep Lazy Proxy
+
+## Overview
+
+The Deep Lazy Proxy is a memory-efficient mechanism for providing workflow data to expression evaluation contexts. Instead of copying entire data structures upfront, it loads data on-demand as properties are accessed.
+
+## Key Features
+
+- **On-Demand Loading**: Only fetches data when accessed
+- **Metadata-Driven**: Returns object structure (keys, length) without values
+- **Caching**: Values are cached after first access to avoid redundant lookups
+- **Type Support**: Handles objects, arrays, functions, and primitives correctly
+- **Memory Efficient**: Large arrays and objects don't cause memory overhead
+
+## Architecture
+
+The deep lazy proxy is implemented in `src/runtime/lazy-proxy.ts`, which is bundled
+together with the other runtime modules into `dist/bundle/runtime.iife.js` and injected
+into the V8 isolate at startup.
+
+Key functions exposed on `globalThis` inside the isolate:
+
+- `createDeepLazyProxy(basePath)` — creates recursive object/array proxies
+- `resetDataProxies()` — called before each evaluation to reinitialise `$json`,
+  `$input`, `$node`, etc. as fresh lazy proxies backed by the three host callbacks
+- `__sanitize(key)` — runtime property-access guard that blocks `__proto__`,
+  `constructor`, `prototype`, etc.
+
+Host-side callbacks registered by `IsolatedVmBridge` as `ivm.Reference` objects
+(synchronous cross-isolate calls):
+
+- `__getValueAtPath(path[])` — returns a primitive, array metadata, or object metadata
+- `__getArrayElement(path[], index)` — returns a single array element (or its metadata)
+- `__callFunctionAtPath(path[], ...args)` — invokes a host-side function and returns the result
+
+## Usage
+
+The proxy system runs **inside the V8 isolate** and is not directly importable from
+host code. The host sets up the data context by calling `bridge.execute(code, data)`,
+which internally:
+
+1. Registers three `ivm.Reference` callbacks with the current `data` object
+2. Calls `resetDataProxies()` in the isolate to create fresh lazy proxies for
+   `$json`, `$binary`, `$input`, `$node`, `$parameter`, `$workflow`, `$prevNode`
+3. Runs the tournament-transformed expression code with `this === __data`
+
+From the expression's perspective it just sees normal objects:
+
+```typescript
+// Inside an expression (runs in isolate):
+$json.user.email        // triggers getValueAtPath(['$json','user','email'])
+$json.items[150].id     // triggers getArrayElement(['$json','items'], 150)
+$items()                // triggers callFunctionAtPath(['$items'])
+```
+
+### Array metadata
+
+Arrays are **never transferred in full** — only their length is returned. Elements
+are loaded individually on demand. Length can be determined from the host object
+in O(1), but serialization cost is proportional to the total byte size of all
+elements, which cannot be bounded from length alone.
+
+```typescript
+// __getValueAtPath returns:
+{ __isArray: true, __length: 1000 }            // always metadata only
+{ __isObject: true, __keys: ['name','email'] } // object — lazy
+42                                              // primitive
+```
+
+## How It Works
+
+### Metadata Pattern
+
+Instead of transferring entire objects/arrays, the proxy uses metadata:
+
+**Arrays** (all sizes):
+```typescript
+{
+  __isArray: true,
+  __length: 1000  // Only length; elements loaded on demand via __getArrayElement
+}
+```
+
+**Objects**:
+```typescript
+{
+  __isObject: true,
+  __keys: ['name', 'email', 'age']  // Only keys, not values
+}
+```
+
+### Caching
+
+Once a property is accessed, it's cached in the proxy's target object:
+
+```typescript
+proxy.$json.user.name  // First access: fetches via callback
+proxy.$json.user.name  // Second access: returns cached value
+```
+
+### Recursive Proxies
+
+When accessing nested objects or arrays, new proxies are created:
+
+```typescript
+proxy.$json.user  // Creates proxy for user object
+proxy.$json.items[50]  // Creates proxy for object at index 50
+```
+
+## Security
+
+### Function Handling
+
+- **Custom Functions**: Allowed and passed directly
+- **Native Functions**: Blocked for security (e.g., `Object.keys`)
+
+```typescript
+const customFn = (x: number) => x * 2;  // Allowed
+const nativeFn = Object.keys;  // Blocked (returns undefined)
+```
+
+Detection is done by checking if `fn.toString()` contains `'[native code]'`.
+
+### Symbol Properties
+
+Symbol properties return `undefined` to prevent security issues.
+
+## Performance
+
+### Memory Efficiency
+
+- **Arrays**: Always lazy-loaded — only length transferred, elements fetched on demand
+- **Objects**: Always lazy-loaded — only keys transferred, values fetched on demand
+
+### Access Patterns
+
+Best performance when:
+- Accessing few properties from large objects
+- Accessing specific array elements (not iterating entire array)
+- Accessing the same properties multiple times (caching means only the first access pays)
+
+Suboptimal performance when:
+- Iterating entire arrays (`.map()`, `.filter()`) — each element triggers a separate callback
+- Accessing most properties of large objects
+- No property reuse (no benefit from caching)
+
+## Known Limitations
+
+1. **Array Methods**: Methods like `.map()`, `.filter()` iterate all elements.
+   Each element triggers a separate `__getArrayElement` callback call, which is slow
+   for large arrays.
+   - **Workaround**: Avoid iterating large arrays in expressions; access specific indices instead
+
+2. **Circular References**: May cause infinite loops in the proxy handler.
+   - **Current**: No cycle detection; circular structures should be avoided in expression data
+
+## Testing
+
+### Integration Tests
+
+```bash
+cd packages/@n8n/expression-runtime
+pnpm test
+```
+
+Test coverage:
+- ✅ Basic property access
+- ✅ Nested properties
+- ✅ Array element access (lazy-loaded via `__getArrayElement`)
+- ✅ Object proxies
+- ✅ Function handling
+- ✅ Caching behavior
+- ✅ Edge cases (circular refs, symbols, "in" operator)
+
+## API Reference (inside the isolate bundle)
+
+These functions are available on `globalThis` within the V8 isolate after the
+runtime bundle (`dist/bundle/runtime.iife.js`) is loaded.
+
+### `resetDataProxies()`
+
+Called by the bridge before each expression evaluation. Reads `$json`, `$binary`,
+`$input`, `$node`, `$parameter`, `$workflow`, `$prevNode`, `$runIndex`, `$itemIndex`,
+and `$items` from `__data` (populated via host callbacks) and exposes them on both
+`globalThis` and `__data` so tournament-transformed code can access them via
+`this.$json`, `this.$input`, etc.
+
+### `createDeepLazyProxy(basePath)`
+
+Creates a recursive Proxy for a given property path. Intercepts property access and
+calls back to the host via `__getValueAtPath` to fetch structure metadata, then
+creates nested proxies for objects or arrays as needed.
+
+**Parameter:**
+- `basePath: string[]` — path from the root data object to the node this proxy represents
+
+## Examples
+
+### Accessing nested data (expression syntax)
+
+```
+{{ $json.order.customer.name }}    // lazy-loads order.customer.name
+{{ $json.order.items[1].product }} // lazy-loads array element at index 1
+{{ $json.items[0] }}               // fetches only the first element
+```
+
+### Array iteration is slow for large arrays
+
+```
+{{ $json.items.reduce((sum, x) => sum + x, 0) }}
+// items has 10 000 elements → length transferred, then 10 000 callback
+// calls to fetch each element. Prefer accessing specific indices.
+```
+
+Note: lodash (`_`) is not available in expressions — it is bundled internally for
+use by extension functions but not exposed on `globalThis`.
+
+## Contributing
+
+When modifying the proxy implementation:
+
+1. **Run tests**: `pnpm test proxy`
+2. **Type check**: `pnpm typecheck`
+3. **Build**: `pnpm build`
+4. **Add tests** for new features
+5. **Update this documentation**
+
+## Related Files
+
+- Proxy implementation: `packages/@n8n/expression-runtime/src/runtime/lazy-proxy.ts` — `createDeepLazyProxy`
+- Reset: `packages/@n8n/expression-runtime/src/runtime/reset.ts` — `resetDataProxies`
+- Security globals: `packages/@n8n/expression-runtime/src/runtime/safe-globals.ts` — `SafeObject`, `SafeError`, `__sanitize`
+- Runtime entry: `packages/@n8n/expression-runtime/src/runtime/index.ts` — wires all modules to `globalThis`
+- Bridge: `packages/@n8n/expression-runtime/src/bridge/isolated-vm-bridge.ts` — registers `ivm.Reference` callbacks, loads bundle, calls `resetDataProxies`
+- Build: `packages/@n8n/expression-runtime/esbuild.config.js` — bundles runtime to `dist/bundle/runtime.iife.js`

packages/@n8n/expression-runtime/docs/implementation-phases.md

@@ -0,0 +1,155 @@
+# Implementation Phases
+
+This document maps interfaces to implementation phases to help developers focus on what's needed when.
+
+## Phase 1.1: Core Runtime Package (MVP)
+
+**Goal**: Basic expression evaluation working in CLI/backend
+
+**Interfaces Needed**:
+- `RuntimeBridge` - Main bridge interface
+- `BridgeConfig` (without `debug` field)
+- `RuntimeHostInterface` - Runtime-to-host communication
+- `RuntimeGlobals` - Globals injected into runtime
+- `WorkflowDataProxy` - Data access helper
+- `IExpressionEvaluator` - Public API
+- `EvaluatorConfig` (without observability)
+- `WorkflowData` - Input data format
+- `EvaluateOptions` (basic)
+
+**Implementations Required**:
+- `IsolatedVmBridge` - For CLI/backend
+- `ExpressionEvaluator` - Main evaluator class
+- Runtime code (runs inside isolate, bundled via esbuild)
+- Lazy loading proxies
+- Expression code cache (per-evaluator, caches tournament-transformed code)
+
+**Can Skip**:
+- Observability (use `NoOpProvider` stub)
+- Debug mode
+- Specific error types (use generic `Error`)
+- Web Workers
+- Task runners
+
+## Phase 0.2: Observability Infrastructure (PARALLEL)
+
+**Goal**: Add metrics, traces, and logs
+
+**Interfaces Needed**:
+- `ObservabilityProvider`
+- `MetricsAPI`
+- `TracesAPI`
+- `LogsAPI`
+- `Span`
+
+**Implementations Required**:
+- `NoOpProvider` (zero overhead when disabled)
+- `OpenTelemetryProvider`
+- `PostHogProvider` (optional)
+- `CompositeProvider` (use multiple providers)
+
+**Integration**:
+- Add to `EvaluatorConfig.observability`
+- Emit metrics/traces from evaluator and bridge
+- Smart sampling implementation
+
+## Phase 1.2: Isolate Pooling
+
+**Goal**: Handle concurrent evaluations
+
+**New Interfaces**: None (uses existing `RuntimeBridge`)
+
+**Implementations Required**:
+- `IsolatePool` class
+- Pool configuration
+- Acquire/release mechanism
+- Disposal detection and replacement
+
+## Phase 1.3: Extension Framework
+
+**Goal**: 100% test compatibility
+
+**New Interfaces**: None
+
+**Implementation**: Extension functions in runtime
+
+## Phase 1.4: Error Handling
+
+**Goal**: Graceful error handling with clear messages
+
+**Interfaces Needed**:
+- `ExpressionError`
+- `MemoryLimitError`
+- `TimeoutError`
+- `SecurityViolationError`
+- `SyntaxError`
+
+**Implementation**: Error handling in all code paths
+
+## Phase 2+: Future Enhancements
+
+### Web Worker Support
+**Interfaces**: Already defined (same `RuntimeBridge`)
+
+**Implementations**:
+- `WebWorkerBridge`
+- Runtime bundled as ESM
+- Note: No lazy loading initially (pre-fetch data)
+
+### Chrome DevTools Debugging
+**Config**: `BridgeConfig.debug` field
+
+**Implementation**:
+- Inspector protocol integration
+- Debug mode in IsolatedVmBridge
+
+### Task Runner Integration (Architecture TBD)
+
+Task runners already have process-level isolation. Expression evaluation happens **inside the task runner** (no IPC to worker needed).
+
+**Option A**: Use `IsolatedVmBridge` locally within task runner
+- Adds another sandbox layer for extra security
+- Task runner creates local evaluator instance
+- No lazy loading needed (task runner has all data)
+
+**Option B**: Evaluate directly without extra sandbox
+- Reuse task runner's existing process isolation
+- Simpler, potentially faster
+- May be sufficient given process-level isolation
+
+**Decision pending** - will be made during Phase 2+ implementation.
+
+---
+
+## Quick Start Guides
+
+### For Frontend Developers (Web Worker Integration)
+
+**Phase 1**: Skip - Web Workers are Phase 2+
+
+**Phase 2**: Focus on:
+1. `RuntimeBridge` interface - Your bridge must implement this
+2. `BridgeConfig` - Configuration options
+3. `WorkflowDataProxy` - How to structure data
+4. Ignore: Observability interfaces (optional)
+
+**Key Difference**: Web Workers can't do lazy loading initially, so you'll need to pre-fetch all data before calling `execute()`.
+
+### For CLI/Backend Developers
+
+**Phase 1.1**: Focus on:
+1. `IsolatedVmBridge` implementation
+2. `ExpressionEvaluator` class
+3. Runtime code (runs inside isolate)
+4. Code cache implementation
+
+**Use**: `NoOpProvider` for observability initially
+
+**Phase 0.2**: Add real observability providers
+
+### For Testing
+
+Integration tests use `IsolatedVmBridge` directly (see `src/__tests__/integration.test.ts`).
+
+All interfaces in `src/types/` are stable enough to write against before the
+bridge implementation lands.