first commit

packages/testing/containers/HELM-TESTING.md

@@ -0,0 +1,175 @@
+# Helm Chart E2E Testing
+
+Test n8n Helm chart deployments using K3s (lightweight Kubernetes) inside Docker, powered by `@testcontainers/k3s`.
+
+## Prerequisites
+
+### Required
+
+- **Docker Desktop** (macOS/Windows) or **Docker Engine** (Linux)
+- **Privileged container support** — K3s runs as a privileged container
+- **helm** CLI — runs on your host machine ([install](https://helm.sh/docs/intro/install/))
+- **kubectl** CLI — runs on your host machine ([install](https://kubernetes.io/docs/tasks/tools/))
+- **n8n Docker image** — built locally (`pnpm build:docker`) or pulled from Docker Hub
+
+### Not Required
+
+- Kind, Minikube, or any other K8s distribution
+- Kubernetes cluster access
+- Any K8s tooling beyond helm + kubectl
+
+### Privileged Container Compatibility
+
+| Environment | Supported | Notes |
+|---|---|---|
+| Docker Desktop (macOS/Windows) | Yes | Privileged enabled by default |
+| Docker Engine (Linux) | Yes | Standard daemon supports it |
+| GitHub Actions (ubuntu runners) | Yes | Docker socket available |
+| Blacksmith runners | Yes | Standard Docker-capable VMs |
+| Rootless Docker | **No** | K3s requires privileged mode |
+| Docker-in-Docker | Depends | Outer container needs `--privileged` |
+| Podman | **No** | K3s requires Docker-compatible runtime |
+
+## How It Works
+
+```
+Host Machine (helm, kubectl)
+├── KUBECONFIG=/tmp/helm-kubeconfig-*.yaml
+└── Docker
+    └── K3s Container (privileged, NodePort 30080 → host random port)
+        ├── containerd (K3s runtime)
+        │   └── n8n image (preloaded from host Docker)
+        └── Kubernetes control plane
+            ├── n8n Pod (Helm-deployed)
+            └── Service (NodePort 30080 → Pod 5678)
+
+Playwright ──── http://localhost:<host-port> ──→ NodePort ──→ n8n Pod
+```
+
+1. `@testcontainers/k3s` starts K3s inside a Docker container with NodePort 30080 exposed
+2. The n8n Docker image is exported from host Docker and imported into K3s's containerd
+3. Host `helm` installs the n8n chart using a kubeconfig pointing at the K3s API
+4. The n8n service is patched to NodePort, routing traffic through K3s's exposed port
+5. Playwright tests connect via `N8N_BASE_URL=http://localhost:<host-port>`
+
+## Local Usage
+
+```bash
+# 1. Build the n8n Docker image (or use a Docker Hub image)
+pnpm build:docker
+
+# 2. Start the Helm stack (takes ~60-120s)
+cd packages/testing/containers
+pnpm stack:helm
+
+# 3. In another terminal, use the printed KUBECONFIG for debugging
+export KUBECONFIG=/tmp/helm-kubeconfig-*.yaml
+kubectl get pods
+kubectl logs -l app.kubernetes.io/name=n8n
+
+# 4. Run tests
+N8N_BASE_URL=http://localhost:<port> RESET_E2E_DB=true \
+  npx playwright test tests/e2e/building-blocks/ --workers=1
+
+# 5. Cleanup
+pnpm stack:helm:clean
+```
+
+### Test a Specific Version Matrix
+
+```bash
+# Test n8n 1.80.0 against chart version v1.2.0
+pnpm stack:helm --image n8nio/n8n:1.80.0 --chart-ref v1.2.0
+
+# Test latest n8n against a chart PR branch
+pnpm stack:helm --chart-ref fix/pvc-permissions
+
+# Test a GHCR image (e.g., from CI)
+pnpm stack:helm --image ghcr.io/n8n-io/n8n:ci-12345
+```
+
+### CLI Options
+
+```
+--mode <mode>         standalone (SQLite, default) or queue (PostgreSQL + Redis + workers)
+--image <image>       n8n Docker image (default: n8nio/n8n:local)
+--chart-ref <ref>     Git branch/tag for n8n-hosting (default: main)
+--chart-repo <url>    Git repo URL (default: https://github.com/n8n-io/n8n-hosting.git)
+--k3s-image <image>   K3s image (default: rancher/k3s:v1.32.2-k3s1)
+--url-file <path>     Write URL to file when ready (for CI automation)
+--help                Show help
+```
+
+## CI Usage
+
+The `test-e2e-helm.yml` workflow handles everything:
+
+- **Trigger:** Push to `helm-container-test` branch, or manual dispatch
+- **Build:** Creates n8n Docker image, pushes to GHCR
+- **Test:** Starts K3s, installs Helm chart, runs building-blocks E2E tests
+- **Cleanup:** Removes ephemeral GHCR image
+
+Manual dispatch also accepts `helm-chart-ref` to test a specific chart version.
+
+## Troubleshooting
+
+### K3s won't start
+
+Check that Docker supports privileged containers:
+```bash
+docker run --rm --privileged alpine echo "privileged works"
+```
+
+### Image not found in K3s
+
+Ensure the n8n Docker image exists locally:
+```bash
+docker images | grep n8nio/n8n
+```
+
+If empty, run `pnpm build:docker` first.
+
+### Helm install times out
+
+Use kubectl to inspect the cluster:
+```bash
+export KUBECONFIG=/tmp/helm-kubeconfig-*.yaml
+kubectl get pods -o wide
+kubectl describe pod -l app.kubernetes.io/name=n8n
+kubectl get events --sort-by=.lastTimestamp
+```
+
+### Port not accessible
+
+NodePort routing is stateless (kube-proxy), so connectivity issues typically indicate the pod is unhealthy. Check pod status:
+```bash
+export KUBECONFIG=/tmp/helm-kubeconfig-*.yaml
+kubectl get pods -o wide
+kubectl describe pod -l app.kubernetes.io/name=n8n
+kubectl version --client
+helm version
+```
+
+### Slow startup
+
+First run is slower due to K3s image pull. Typical timings:
+
+| Phase | First Run | Subsequent |
+|---|---|---|
+| K3s start | ~15-30s | ~5s (reuse) |
+| Image preload | ~10-20s | ~10-20s |
+| Helm install | ~5-10s | ~5-10s |
+| n8n boot | ~15-30s | ~15-30s |
+| **Total** | **~60-120s** | **~40-80s** |
+
+## Comparison: Testcontainers vs K3s + Helm
+
+| | Testcontainers (Stream 1) | K3s + Helm (Stream 2) |
+|---|---|---|
+| **Purpose** | Feature testing | Deployment validation |
+| **Speed** | 5-15s startup | 60-120s startup |
+| **K8s features** | None | PVC, RBAC, securityContext, NetworkPolicy |
+| **Custom services** | Kafka, Mailpit, OIDC, etc. | Only what the Helm chart defines |
+| **What it proves** | "n8n works with X" | "this chart config deploys correctly" |
+| **When to run** | Every PR | On demand, nightly, pre-release |
+| **Prerequisites** | Docker | Docker + helm + kubectl |

packages/testing/containers/README.md

@@ -0,0 +1,438 @@
+# n8n Test Containers
+
+A composable container stack for n8n testing. Describe what you need, it builds the environment.
+
+## Quick Start
+
+```bash
+#build the container
+pnpm build:docker
+```
+alternatively, you can set `N8N_DOCKER_IMAGE=n8nio/n8n:latest`
+
+
+```bash
+# Basic n8n (SQLite)
+pnpm stack
+
+# With PostgreSQL
+pnpm stack --postgres
+
+# Queue mode (Redis + PostgreSQL + worker)
+pnpm stack --queue
+
+# Multi-main cluster
+pnpm stack --mains 2 --workers 1
+
+# Cloud plan simulation
+pnpm stack --plan starter
+
+# Public tunnel for webhook testing
+pnpm stack --tunnel
+```
+
+When started, you'll see the URL: `http://localhost:[port]`
+
+## Using in Playwright Tests
+
+### Basic Test
+
+```typescript
+import { test, expect } from '../fixtures/base';
+
+test('my test', async ({ n8n }) => {
+  await n8n.page.goto('/workflow/new');
+  // ...
+});
+```
+
+### Enabling Services
+
+Use `test.use()` to request services:
+
+```typescript
+// Single service
+test.use({
+  capability: {
+    services: ['mailpit'],
+  },
+});
+
+// Multiple services
+test.use({
+  capability: {
+    services: ['mailpit', 'keycloak', 'victoriaLogs', 'victoriaMetrics', 'vector'],
+  },
+});
+
+// Queue mode with services
+test.use({
+  capability: {
+    mains: 2,
+    workers: 1,
+    services: ['victoriaLogs', 'victoriaMetrics', 'vector'],
+  },
+});
+```
+
+### Using Service Helpers
+
+Services provide type-safe helpers via `n8nContainer.services.*`:
+
+```typescript
+test('email test', async ({ n8nContainer }) => {
+  // Wait for email
+  const email = await n8nContainer.services.mailpit.waitForMessage({
+    to: 'test@example.com',
+  });
+  expect(email.subject).toBe('Welcome');
+});
+
+test('source control', async ({ n8nContainer }) => {
+  // Create git repo
+  const repo = await n8nContainer.services.gitea.createRepo('my-repo');
+  await repo.createBranch('develop');
+});
+
+test('metrics', async ({ n8nContainer }) => {
+  // Query Prometheus metrics
+  const result = await n8nContainer.services.observability.metrics.query('up');
+  expect(result[0].value).toBe(1);
+});
+```
+
+### Capability Shortcuts
+
+Common combinations have shortcuts in `fixtures/capabilities.ts`:
+
+```typescript
+// Instead of: { services: ['mailpit'] }
+test.use({ capability: 'email' });
+
+// Instead of: { services: ['keycloak'] }
+test.use({ capability: 'oidc' });
+
+// Instead of: { services: ['gitea'] }
+test.use({ capability: 'source-control' });
+```
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                        Test Code                             │
+│   n8nContainer.services.gitea.createRepo('my-repo')         │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│                         N8NStack                             │
+│   services: ServiceHelpers  ← Proxy with lazy instantiation │
+│   baseUrl, stop(), findContainers()                         │
+└─────────────────────────────────────────────────────────────┘
+                              │
+              ┌───────────────┼───────────────┐
+              ▼               ▼               ▼
+      ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
+      │ GiteaHelper │ │MailpitHelper│ │ Observability│
+      └─────────────┘ └─────────────┘ └─────────────┘
+              │               │               │
+              ▼               ▼               ▼
+      ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
+      │  Container  │ │  Container  │ │  Container  │
+      └─────────────┘ └─────────────┘ └─────────────┘
+```
+
+### Key Concepts
+
+| Concept | Description |
+|---------|-------------|
+| **Service** | Container definition with `start()`, optional `env()`, optional helper |
+| **Registry** | Central manifest of all services (`services/registry.ts`) |
+| **Stack** | Orchestrator that builds the environment from config |
+| **Helper** | Type-safe API for interacting with a service in tests |
+
+### Service Activation
+
+Services activate in two ways:
+
+| Mode | When | Example |
+|------|------|---------|
+| **Auto-start** | Service has `shouldStart()` returning true | Redis auto-starts in queue mode |
+| **User-enabled** | Listed in `services: []` array | `services: ['mailpit']` |
+
+## Adding a New Service
+
+### Do I Need a Helper?
+
+Helpers let tests interact with a service **outside of the n8n UI**. Ask yourself:
+
+> "Will tests need to arrange or assert data in this service directly?"
+
+| Scenario | Helper Needed? | Example |
+|----------|---------------|---------|
+| **Test arrangement** - Set up data before test | Yes | Create a git repo before testing source control sync |
+| **Test assertion** - Verify side effects | Yes | Check an email was sent after workflow execution |
+| **Infrastructure only** - n8n connects, tests don't | No | PostgreSQL, Redis - n8n uses them, tests don't touch them |
+| **Observability** - Query metrics/logs | Yes | Assert memory usage, check for error logs |
+
+**Examples:**
+
+```typescript
+// Mailpit helper - ARRANGE: no emails exist, ASSERT: email was sent
+const emails = await n8nContainer.services.mailpit.getMessages();
+expect(emails).toHaveLength(1);
+
+// Gitea helper - ARRANGE: create repo before test
+const repo = await n8nContainer.services.gitea.createRepo('test-repo');
+// Now test source control connection via UI
+
+// Observability helper - ASSERT: check metrics after load test
+const memory = await n8nContainer.services.observability.metrics.query('process_resident_memory_bytes');
+expect(memory[0].value).toBeLessThan(500_000_000);
+
+// Redis/Postgres - no helper needed, n8n connects automatically
+// Tests don't need to interact with these directly
+```
+
+**Rule of thumb:** If you'd otherwise need `docker exec` or raw HTTP calls in your test, you need a helper.
+
+### Minimal Service (No Helper)
+
+**1. Create `services/my-service.ts`:**
+
+```typescript
+import { GenericContainer, Wait } from 'testcontainers';
+import type { Service, ServiceResult } from './types';
+
+const HOSTNAME = 'myservice';
+const PORT = 8080;
+
+export interface MyServiceMeta {
+  host: string;
+  port: number;
+}
+
+export type MyServiceResult = ServiceResult<MyServiceMeta>;
+
+export const myService: Service<MyServiceResult> = {
+  description: 'My service description',
+
+  async start(network, projectName) {
+    const container = await new GenericContainer('myimage:latest')
+      .withNetwork(network)
+      .withNetworkAliases(HOSTNAME)
+      .withExposedPorts(PORT)
+      .withWaitStrategy(Wait.forListeningPorts())
+      .withLabels({
+        'com.docker.compose.project': projectName,
+        'com.docker.compose.service': HOSTNAME,
+      })
+      .withName(`${projectName}-${HOSTNAME}`)
+      .withReuse()
+      .start();
+
+    return {
+      container,
+      meta: { host: HOSTNAME, port: PORT },
+    };
+  },
+
+  // Optional: env vars for n8n
+  env(result) {
+    return {
+      MY_SERVICE_HOST: result.meta.host,
+      MY_SERVICE_PORT: String(result.meta.port),
+    };
+  },
+};
+```
+
+**2. Register in `services/types.ts` and `services/registry.ts`:**
+
+```typescript
+// types.ts - add to SERVICE_NAMES array
+export const SERVICE_NAMES = [
+  // ...existing
+  'myService',
+] as const;
+
+// registry.ts - add to services object
+import { myService } from './my-service';
+
+export const services: Record<ServiceName, Service<ServiceResult>> = {
+  // ...existing
+  myService,
+};
+```
+
+**Done.** Use with `services: ['myService']` in tests.
+
+> **Note:** The `ServiceName` type is derived from `SERVICE_NAMES`, and `Record<ServiceName, ...>` ensures the registry includes all services. TypeScript will error if they're out of sync.
+
+### Service With Helper
+
+Add a helper class and factory to the service file:
+
+```typescript
+// ... service definition from above ...
+
+// Helper class
+export class MyServiceHelper {
+  constructor(
+    private readonly container: StartedTestContainer,
+    private readonly meta: MyServiceMeta,
+  ) {}
+
+  async doSomething(): Promise<string> {
+    // Interact with the service
+    const response = await fetch(`http://${this.container.getHost()}:${this.container.getMappedPort(PORT)}/api`);
+    return response.text();
+  }
+}
+
+// Factory function
+export function createMyServiceHelper(ctx: HelperContext): MyServiceHelper {
+  const result = ctx.serviceResults.myService;
+  if (!result) {
+    throw new Error('MyService not running. Add services: ["myService"] to test.use()');
+  }
+  return new MyServiceHelper(result.container, result.meta as MyServiceMeta);
+}
+
+// Type registration (enables autocomplete)
+declare module './types' {
+  interface ServiceHelpers {
+    myService: MyServiceHelper;
+  }
+}
+```
+
+**Register in `services/types.ts` and `services/registry.ts`:**
+
+```typescript
+// types.ts - add to SERVICE_NAMES array
+export const SERVICE_NAMES = [
+  // ...existing
+  'myService',
+] as const;
+
+// registry.ts - add service and helper factory
+import { myService, createMyServiceHelper } from './my-service';
+
+export const services = { ...existing, myService };
+export const helperFactories = { ...existing, myService: createMyServiceHelper };
+```
+
+**Use in tests:**
+
+```typescript
+test('my test', async ({ n8nContainer }) => {
+  const result = await n8nContainer.services.myService.doSomething();
+});
+```
+
+### Optional: Add Capability Shortcut
+
+In `fixtures/capabilities.ts`:
+
+```typescript
+export const CAPABILITIES = {
+  // ...existing
+  'my-capability': { services: ['myService'] },
+};
+```
+
+Now usable as `test.use({ capability: 'my-capability' })`.
+
+## Available Services
+
+| Service | Helper | Description |
+|---------|--------|-------------|
+| `postgres` | - | PostgreSQL database |
+| `redis` | - | Redis for queue mode |
+| `mailpit` | ✓ | Email testing (SMTP + UI) |
+| `gitea` | ✓ | Git server for source control |
+| `keycloak` | ✓ | OIDC/SSO provider |
+| `victoriaLogs` | - | VictoriaLogs for log storage |
+| `victoriaMetrics` | - | VictoriaMetrics for metrics |
+| `vector` | - | Vector log collector (depends on victoriaLogs) |
+| `tracing` | ✓ | Jaeger for distributed tracing |
+| `kafka` | ✓ | Kafka broker for message queue testing |
+| `proxy` | - | HTTP proxy (MockServer) |
+| `taskRunner` | - | External task runner |
+| `loadBalancer` | - | Caddy for multi-main |
+| `cloudflared` | - | Cloudflare Tunnel for public webhook URLs |
+
+**Note:** For observability (logs + metrics), enable all three: `['victoriaLogs', 'victoriaMetrics', 'vector']`.
+The `observability` capability shortcut handles this automatically: `test.use({ capability: 'observability' })`.
+
+## CLI Options
+
+| Option | Description |
+|--------|-------------|
+| `--postgres` | Use PostgreSQL instead of SQLite |
+| `--queue` | Enable queue mode (adds Redis + PostgreSQL) |
+| `--mains <n>` | Number of main instances |
+| `--workers <n>` | Number of worker instances |
+| `--plan <name>` | Cloud plan preset (trial, starter, pro-1, pro-2, enterprise) |
+| `--name <name>` | Custom project name for parallel runs |
+| `--env KEY=VALUE` | Set environment variables |
+| `--observability` | Enable metrics/logs stack |
+| `--tracing` | Enable tracing stack (Jaeger) |
+| `--tunnel` | Enable Cloudflare Tunnel for public webhook URLs |
+| `--oidc` | Enable Keycloak |
+| `--source-control` | Enable Gitea |
+| `--mailpit` | Enable email testing (Mailpit) |
+
+## Telemetry
+
+Container stack telemetry tracks startup timing, configuration, and runner info. Useful for monitoring CI performance and debugging slow stacks.
+
+### Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `CONTAINER_TELEMETRY_WEBHOOK` | POST telemetry JSON to this URL |
+| `CONTAINER_TELEMETRY_VERBOSE` | Set to `1` for JSON output to console |
+
+### What's Collected
+
+```typescript
+{
+  timestamp: string;              // ISO timestamp
+  git: { sha, branch, pr? };      // Git context from CI env vars
+  ci: { runId, job, workflow };   // GitHub Actions context
+  runner: { provider, cpuCores, memoryGb };  // github | blacksmith | local
+  stack: { type, mains, workers, postgres, services };
+  timing: { total, network, n8nStartup, services: Record<string, number> };
+  containers: { total, services, n8n };
+  success: boolean;
+  errorMessage?: string;
+}
+```
+
+### Usage
+
+```bash
+# Verbose output locally
+CONTAINER_TELEMETRY_VERBOSE=1 pnpm stack
+
+# Send to webhook (CI)
+CONTAINER_TELEMETRY_WEBHOOK=https://n8n.example.com/webhook/telemetry
+```
+
+## Cleanup
+
+```bash
+# Remove all containers and networks
+pnpm stack:clean:all
+```
+
+## Tips
+
+- **Container Reuse**: Set `TESTCONTAINERS_REUSE_ENABLE=true` for faster restarts
+- **Parallel Testing**: Use `--name` to run multiple stacks without conflicts
+- **Custom Image**: Set `TEST_IMAGE_N8N=n8nio/n8n:dev` to use a different image
+- **Multi-Main**: Requires queue mode and license key in `N8N_LICENSE_ACTIVATION_KEY`
+- **Using podman**: This does not work with podman out of the box - you need to ensure testcontainers is set correctly [https://podman-desktop.io/tutorial/testcontainers-with-podman](https://podman-desktop.io/tutorial/testcontainers-with-podman)

packages/testing/containers/docker-image-not-found-error.ts

@@ -0,0 +1,24 @@
+import { TEST_CONTAINER_IMAGES } from './test-containers';
+
+// Custom error class for when the Docker image is not found locally/remotely
+// This can happen when using the "n8nio/n8n:local" image, which is not available on Docker Hub
+// This image is available after running `pnpm build:docker` at the root of the repository
+export class DockerImageNotFoundError extends Error {
+	constructor(containerName: string, originalError?: Error) {
+		const dockerImage = TEST_CONTAINER_IMAGES.n8n;
+
+		const message = `Failed to start container ${containerName}: Docker image '${dockerImage}' not found locally!
+
+This is likely because the image is not available locally.
+To fix this, you can either:
+  1. Build the image by running: pnpm build:docker at the root
+  2. Use a different image by setting: TEST_IMAGE_N8N=<image-tag>
+
+Example with different image:
+  TEST_IMAGE_N8N=n8nio/n8n:latest npm run stack`;
+
+		super(message);
+		this.name = 'DockerImageNotFoundError';
+		this.cause = originalError;
+	}
+}

packages/testing/containers/dockerfiles/kent/Dockerfile

@@ -0,0 +1,9 @@
+FROM python:3.12-slim
+
+RUN pip install --no-cache-dir kent flask-cors
+
+EXPOSE 8000
+
+COPY kent_cors.py /app/kent_cors.py
+
+CMD ["python", "/app/kent_cors.py"]

packages/testing/containers/dockerfiles/kent/kent_cors.py

@@ -0,0 +1,9 @@
+"""Kent server wrapper with CORS support for browser-based testing."""
+from flask_cors import CORS
+from kent.app import create_app
+
+app = create_app()
+CORS(app)  # Enable CORS for all origins
+
+if __name__ == "__main__":
+    app.run(host="0.0.0.0", port=8000)

packages/testing/containers/eslint.config.mjs

@@ -0,0 +1,19 @@
+import { defineConfig } from 'eslint/config';
+import { baseConfig } from '@n8n/eslint-config/base';
+
+export default defineConfig(baseConfig, {
+	rules: {
+		'@typescript-eslint/naming-convention': [
+			'error',
+			// Add exception for Docker Compose labels
+			{
+				selector: 'objectLiteralProperty',
+				format: null, // Allow any format
+				filter: {
+					regex: '^com\\.docker\\.',
+					match: true,
+				},
+			},
+		],
+	},
+});

packages/testing/containers/helm-stack.ts

@@ -0,0 +1,473 @@
+import { K3sContainer, type StartedK3sContainer } from '@testcontainers/k3s';
+import { execSync } from 'node:child_process';
+import {
+	copyFileSync,
+	existsSync,
+	mkdirSync,
+	mkdtempSync,
+	unlinkSync,
+	writeFileSync,
+} from 'node:fs';
+import { homedir, tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { setTimeout as wait } from 'node:timers/promises';
+
+import { TEST_CONTAINER_IMAGES } from './test-containers';
+
+const DEFAULT_K3S_IMAGE = 'rancher/k3s:v1.32.2-k3s1';
+const DEFAULT_CHART_REPO = 'https://github.com/n8n-io/n8n-hosting.git';
+const DEFAULT_CHART_REF = 'main';
+const N8N_NODE_PORT = 30080;
+const HEALTH_POLL_INTERVAL_MS = 2_000;
+const CONTAINERD_READY_TIMEOUT_MS = 30_000;
+const K3S_STARTUP_TIMEOUT_MS = 120_000;
+const COMMAND_TIMEOUT_MS = 600_000;
+
+export type HelmStackMode = 'standalone' | 'queue';
+
+export interface HelmStackConfig {
+	/** n8n Docker image to deploy (default: TEST_CONTAINER_IMAGES.n8n) */
+	n8nImage?: string;
+	/** K3s image (default: rancher/k3s:v1.32.2-k3s1) */
+	k3sImage?: string;
+	/** Git ref for the n8n-hosting repo (default: main) */
+	helmChartRef?: string;
+	/** Git repo URL for the Helm chart (default: n8n-io/n8n-hosting) */
+	helmChartRepo?: string;
+	/** Total startup timeout in ms (default: 300_000) */
+	startupTimeoutMs?: number;
+	/** Deployment mode: standalone (SQLite) or queue (PostgreSQL + Redis + workers) */
+	mode?: HelmStackMode;
+	/** Additional environment variables to inject into n8n pods via Helm set-flags (merge-last-wins over defaults) */
+	env?: Record<string, string>;
+}
+
+export interface HelmStack {
+	/** Base URL to access n8n running inside K3s */
+	baseUrl: string;
+	/** Stop the K3s container and clean up */
+	stop: () => Promise<void>;
+	/** Path to kubeconfig file (use with kubectl/helm from your terminal) */
+	kubeConfigPath: string;
+}
+
+// -- Logging ------------------------------------------------------------------
+
+function log(message: string) {
+	const timestamp = new Date().toISOString().slice(11, 19);
+	console.log(`[helm-stack ${timestamp}] ${message}`);
+}
+
+// -- Host command execution ---------------------------------------------------
+
+/** Execute a shell command on the host with the given environment. Returns stdout or throws with stderr. */
+function execOnHost(cmd: string, env: NodeJS.ProcessEnv, description: string): string {
+	try {
+		return execSync(cmd, { env, stdio: 'pipe', encoding: 'utf-8', timeout: COMMAND_TIMEOUT_MS });
+	} catch (error: unknown) {
+		const stderr = (error as { stderr?: string }).stderr ?? '';
+		const message = error instanceof Error ? error.message : String(error);
+		throw new Error(`${description} failed:\n${stderr || message}`);
+	}
+}
+
+// -- Image preloading (must run inside K3s containerd) ------------------------
+
+async function preloadImage(container: StartedK3sContainer, imageName: string): Promise<void> {
+	// Try crictl pull first (fast for public registry images like GHCR).
+	// Falls back to docker save + ctr import for local-only images (e.g. n8nio/n8n:local).
+	log(`Pulling ${imageName} inside K3s...`);
+	const pullResult = await container.exec(['crictl', 'pull', imageName]);
+	if (pullResult.exitCode !== 0) {
+		log('Registry pull failed, importing from local Docker...');
+		const tarPath = `/tmp/n8n-helm-${Date.now()}.tar`;
+		try {
+			execSync(`docker save ${imageName} -o ${tarPath}`, { stdio: 'pipe' });
+			execSync(`docker cp ${tarPath} ${container.getId()}:/tmp/n8n-image.tar`, {
+				stdio: 'pipe',
+			});
+
+			const importResult = await container.exec([
+				'ctr',
+				'--namespace',
+				'k8s.io',
+				'images',
+				'import',
+				'/tmp/n8n-image.tar',
+			]);
+			if (importResult.exitCode !== 0) {
+				throw new Error(`ctr import failed: ${importResult.output}`);
+			}
+			await container.exec(['rm', '-f', '/tmp/n8n-image.tar']);
+		} finally {
+			try {
+				unlinkSync(tarPath);
+			} catch (cleanupError: unknown) {
+				log(
+					`Warning: failed to clean up temp file ${tarPath}: ${cleanupError instanceof Error ? cleanupError.message : String(cleanupError)}`,
+				);
+			}
+		}
+	}
+
+	const { output } = await container.exec(['crictl', 'images']);
+	log(`Available images after preload:\n${output}`);
+}
+
+// -- Chart download -----------------------------------------------------------
+
+function cloneChartToHost(repo: string, ref: string): string {
+	log(`Downloading chart from ${repo} @ ${ref}...`);
+	const dir = mkdtempSync(join(tmpdir(), 'n8n-chart-'));
+	const repoPath = repo.replace('https://github.com/', '').replace('.git', '');
+	const tarUrl = `https://github.com/${repoPath}/archive/${ref}.tar.gz`;
+
+	execSync(`curl -fsSL "${tarUrl}" | tar xz -C "${dir}" --strip-components=1`, { stdio: 'pipe' });
+	log('Chart downloaded');
+	return dir;
+}
+
+// -- Example values file selection --------------------------------------------
+
+const EXAMPLE_VALUES_FILES: Record<HelmStackMode, string> = {
+	standalone: 'standalone.yaml',
+	queue: 'minimal.yaml',
+};
+
+function getExampleValuesFile(chartDir: string, mode: HelmStackMode): string {
+	return join(chartDir, 'charts', 'n8n', 'examples', EXAMPLE_VALUES_FILES[mode]);
+}
+
+// -- Helm install flags -------------------------------------------------------
+
+function parseImageName(imageName: string): { repository: string; tag: string } {
+	// Split on last colon to handle registry ports (e.g. localhost:5000/repo:tag)
+	const lastColon = imageName.lastIndexOf(':');
+	const hasTag = lastColon > 0 && !imageName.substring(lastColon).includes('/');
+	return hasTag
+		? { repository: imageName.substring(0, lastColon), tag: imageName.substring(lastColon + 1) }
+		: { repository: imageName, tag: 'latest' };
+}
+
+function buildHelmSetFlags(
+	imageName: string,
+	mode: HelmStackMode,
+	baseUrl: string,
+	envOverrides?: Record<string, string>,
+): string[] {
+	const { repository, tag } = parseImageName(imageName);
+
+	// Collect env vars as an array, then convert to indexed --set flags.
+	// This avoids fragile manual index tracking and makes it easy to add conditional entries.
+	const extraEnvs: Array<{ name: string; value: string }> = [
+		{ name: 'N8N_DIAGNOSTICS_ENABLED', value: 'false' },
+		{ name: 'N8N_DYNAMIC_BANNERS_ENABLED', value: 'false' },
+		// WEBHOOK_URL tells n8n its externally-accessible address (for invitation links, webhooks, etc.)
+		{ name: 'WEBHOOK_URL', value: baseUrl },
+	];
+
+	// License env vars from host (same pattern as testcontainers stack)
+	if (process.env.N8N_LICENSE_TENANT_ID) {
+		extraEnvs.push({ name: 'N8N_LICENSE_TENANT_ID', value: process.env.N8N_LICENSE_TENANT_ID });
+	}
+	if (process.env.N8N_LICENSE_ACTIVATION_KEY) {
+		extraEnvs.push({
+			name: 'N8N_LICENSE_ACTIVATION_KEY',
+			value: process.env.N8N_LICENSE_ACTIVATION_KEY,
+		});
+	}
+	if (process.env.N8N_LICENSE_CERT) {
+		extraEnvs.push({ name: 'N8N_LICENSE_CERT', value: process.env.N8N_LICENSE_CERT });
+	}
+
+	// Caller-provided env overrides (merge-last-wins)
+	if (envOverrides) {
+		for (const [name, value] of Object.entries(envOverrides)) {
+			extraEnvs.push({ name, value });
+		}
+	}
+
+	// Dynamic overrides on top of the example values file (-f).
+	// Mode-specific config (database type, queue settings, etc.) comes from the example file.
+	const flags = [
+		`--set image.repository=${repository}`,
+		`--set image.tag=${tag}`,
+		// --set-string because K8s env values must be strings
+		...extraEnvs.flatMap((env, i) => [
+			`--set config.extraEnv[${i}].name=${env.name}`,
+			`--set-string config.extraEnv[${i}].value=${env.value}`,
+		]),
+	];
+
+	if (mode === 'standalone') {
+		// Override persistence size (example uses 5Gi, we need less for tests)
+		flags.push('--set persistence.size=1Gi');
+	} else {
+		// Override placeholder hosts to point at our Bitnami services
+		flags.push(
+			'--set database.host=postgresql',
+			'--set redis.host=redis-master',
+			// Example file references n8n-core-secrets, we use n8n-secrets
+			'--set secretRefs.existingSecret=n8n-secrets',
+		);
+	}
+
+	return flags;
+}
+
+// -- K8s secrets --------------------------------------------------------------
+
+function createN8nSecret(env: NodeJS.ProcessEnv): void {
+	log('Creating n8n core secrets...');
+	execOnHost(
+		'kubectl create secret generic n8n-secrets --from-literal=N8N_ENCRYPTION_KEY=test-encryption-key-for-e2e-testing --from-literal=N8N_HOST=localhost --from-literal=N8N_PORT=5678 --from-literal=N8N_PROTOCOL=http',
+		env,
+		'Create n8n core secrets',
+	);
+}
+
+// -- Queue mode infrastructure ------------------------------------------------
+
+function deployQueueInfrastructure(env: NodeJS.ProcessEnv): void {
+	log('Adding Bitnami Helm repo...');
+	execOnHost(
+		'helm repo add bitnami https://charts.bitnami.com/bitnami && helm repo update',
+		env,
+		'Add Bitnami repo',
+	);
+
+	log('Deploying PostgreSQL...');
+	execOnHost(
+		'helm install postgresql bitnami/postgresql --set auth.username=n8n --set auth.password=n8n-test-password --set auth.database=n8n --set primary.resources.requests.cpu=100m --set primary.resources.requests.memory=256Mi --set primary.resources.limits.cpu=500m --set primary.resources.limits.memory=512Mi --wait --timeout 3m',
+		env,
+		'Deploy PostgreSQL',
+	);
+	log('PostgreSQL deployed');
+
+	log('Deploying Redis...');
+	execOnHost(
+		"helm install redis bitnami/redis --set architecture=standalone --set auth.enabled=false --set-json 'master.disableCommands=[]' --set master.resources.requests.cpu=100m --set master.resources.requests.memory=128Mi --set master.resources.limits.cpu=250m --set master.resources.limits.memory=256Mi --wait --timeout 3m",
+		env,
+		'Deploy Redis',
+	);
+	log('Redis deployed');
+
+	execOnHost(
+		'kubectl create secret generic n8n-db-secret --from-literal=password=n8n-test-password',
+		env,
+		'Create DB password secret',
+	);
+}
+
+// -- Health check -------------------------------------------------------------
+
+async function pollHealthEndpoint(baseUrl: string, timeoutMs: number): Promise<void> {
+	const url = `${baseUrl}/healthz/readiness`;
+	const startTime = Date.now();
+
+	while (Date.now() - startTime < timeoutMs) {
+		try {
+			const response = await fetch(url);
+			if (response.status === 200) {
+				return;
+			}
+		} catch {
+			// Retry
+		}
+		await wait(HEALTH_POLL_INTERVAL_MS);
+	}
+
+	throw new Error(`n8n health check at ${url} did not return 200 within ${timeoutMs / 1000}s`);
+}
+
+// -- Main entry point ---------------------------------------------------------
+
+export async function createHelmStack(config: HelmStackConfig = {}): Promise<HelmStack> {
+	const {
+		n8nImage = TEST_CONTAINER_IMAGES.n8n,
+		k3sImage = DEFAULT_K3S_IMAGE,
+		helmChartRef = DEFAULT_CHART_REF,
+		helmChartRepo = DEFAULT_CHART_REPO,
+		startupTimeoutMs = 300_000,
+		mode = 'standalone',
+		env: envOverrides,
+	} = config;
+
+	const containerName = `n8n-helm-${mode}-${Date.now().toString(36)}`;
+
+	log('Starting K3s + Helm stack');
+	log(`  Mode: ${mode}`);
+	log(`  Container: ${containerName}`);
+	log(`  n8n image: ${n8nImage}`);
+	log(`  K3s image: ${k3sImage}`);
+	log(`  Chart: ${helmChartRepo} @ ${helmChartRef}`);
+
+	// Step 1: Start K3s with NodePort exposed (bypasses flaky kubectl port-forward)
+	// Ryuk is disabled so the container survives process exit — clean up via stack:helm:clean.
+	log('Starting K3s container (privileged)...');
+	const k3s = await new K3sContainer(k3sImage)
+		.withName(containerName)
+		.withLabels({ 'n8n.helm': 'true', 'n8n.helm.mode': mode })
+		.withExposedPorts(N8N_NODE_PORT)
+		.withStartupTimeout(K3S_STARTUP_TIMEOUT_MS)
+		.start();
+	const hostPort = k3s.getMappedPort(N8N_NODE_PORT);
+	const baseUrl = `http://localhost:${hostPort}`;
+	log(`K3s started (NodePort ${N8N_NODE_PORT} -> host ${hostPort})`);
+
+	// Step 2: Write kubeconfig to a stable path and merge into ~/.kube/config
+	const kubeDir = join(homedir(), '.kube');
+	mkdirSync(kubeDir, { recursive: true });
+	const contextName = `n8n-helm-${mode}`;
+	const kubeConfigPath = join(kubeDir, `${contextName}.yaml`);
+
+	// K3s names everything 'default' — rename to avoid clashing with existing contexts
+	const rawKubeconfig = k3s.getKubeConfig();
+	const namedKubeconfig = rawKubeconfig
+		.replace(/\bname: default\b/g, `name: ${contextName}`)
+		.replace(/\bcurrent-context: default\b/, `current-context: ${contextName}`)
+		.replace(/\bcluster: default\b/g, `cluster: ${contextName}`)
+		.replace(/\buser: default\b/g, `user: ${contextName}`);
+	writeFileSync(kubeConfigPath, namedKubeconfig);
+
+	// Merge into ~/.kube/config so kubectl works without KUBECONFIG env var
+	const defaultKubeconfig = join(kubeDir, 'config');
+	if (existsSync(defaultKubeconfig)) {
+		copyFileSync(defaultKubeconfig, `${defaultKubeconfig}.bak`);
+	}
+	try {
+		const merged = execSync('kubectl config view --flatten', {
+			env: {
+				...process.env,
+				KUBECONFIG: existsSync(defaultKubeconfig)
+					? `${kubeConfigPath}:${defaultKubeconfig}`
+					: kubeConfigPath,
+			},
+			stdio: 'pipe',
+			encoding: 'utf-8',
+		});
+		writeFileSync(defaultKubeconfig, merged);
+		execSync(`kubectl config use-context ${contextName}`, { stdio: 'pipe' });
+		log(`Kubeconfig merged into ~/.kube/config (context: ${contextName})`);
+	} catch {
+		log(`Warning: could not merge kubeconfig — use: export KUBECONFIG=${kubeConfigPath}`);
+	}
+
+	const env: NodeJS.ProcessEnv = { ...process.env, KUBECONFIG: kubeConfigPath };
+
+	let chartDir = '';
+
+	try {
+		// Step 3: Wait for containerd readiness
+		log('Waiting for containerd...');
+		const deadline = Date.now() + CONTAINERD_READY_TIMEOUT_MS;
+		while (Date.now() < deadline) {
+			const result = await k3s.exec(['crictl', 'images']);
+			if (result.exitCode === 0) break;
+			await wait(HEALTH_POLL_INTERVAL_MS);
+		}
+
+		// Step 4: Preload n8n image into K3s containerd
+		await preloadImage(k3s, n8nImage);
+		log('Image preloaded');
+
+		// Step 5: Download chart to host
+		chartDir = cloneChartToHost(helmChartRepo, helmChartRef);
+
+		// Step 6: Create n8n core secrets (encryption key, host config)
+		createN8nSecret(env);
+
+		// Step 7: Deploy queue infrastructure if needed
+		if (mode === 'queue') {
+			deployQueueInfrastructure(env);
+		}
+
+		// Step 8: Install n8n chart using published example values file + dynamic overrides
+		log('Installing Helm chart (this may take a few minutes)...');
+		const valuesFile = getExampleValuesFile(chartDir, mode);
+		const setFlags = buildHelmSetFlags(n8nImage, mode, baseUrl, envOverrides).join(' ');
+		log(`Using values file: ${valuesFile}`);
+		const helmOutput = execOnHost(
+			`helm install n8n "${chartDir}/charts/n8n" -f "${valuesFile}" ${setFlags} --wait --timeout 5m`,
+			env,
+			'Helm install',
+		);
+		log(`Helm install complete:\n${helmOutput.trim()}`);
+
+		// Step 9: Patch service to NodePort so traffic goes through K3s's exposed port
+		// (bypasses kubectl port-forward which silently breaks after many connections)
+		log(`Patching n8n service to NodePort ${N8N_NODE_PORT}...`);
+		execOnHost(
+			`kubectl patch svc n8n-main --type merge -p '{"spec":{"type":"NodePort","ports":[{"port":5678,"targetPort":5678,"nodePort":${N8N_NODE_PORT}}]}}'`,
+			env,
+			'Patch service to NodePort',
+		);
+
+		// Step 10: Poll health endpoint
+		log(`Polling ${baseUrl}/healthz/readiness...`);
+		await pollHealthEndpoint(baseUrl, Math.min(startupTimeoutMs, 120_000));
+		log(`n8n is ready at ${baseUrl}`);
+
+		return {
+			baseUrl,
+			kubeConfigPath,
+			stop: async () => {
+				log('Shutting down...');
+				try {
+					unlinkSync(kubeConfigPath);
+				} catch {
+					/* ignore */
+				}
+				try {
+					execSync(`rm -rf "${chartDir}"`, { stdio: 'pipe' });
+				} catch {
+					/* ignore */
+				}
+				await k3s.stop();
+				log('K3s stopped');
+			},
+		};
+	} catch (error) {
+		// Dump debug info from host kubectl
+		try {
+			const podStatus = execOnHost('kubectl get pods -o wide 2>/dev/null || true', env, 'debug');
+			console.error('\n--- Pod Status ---');
+			console.error(podStatus);
+
+			const podLogs = execOnHost(
+				'kubectl logs -l app.kubernetes.io/name=n8n --tail=50 2>/dev/null || true',
+				env,
+				'debug',
+			);
+			if (podLogs.trim()) {
+				console.error('\n--- Pod Logs ---');
+				console.error(podLogs);
+			}
+
+			const events = execOnHost(
+				'kubectl get events --sort-by=.lastTimestamp 2>/dev/null || true',
+				env,
+				'debug',
+			);
+			console.error('\n--- Events ---');
+			console.error(events);
+			console.error('----------------\n');
+		} catch {
+			// Best-effort debugging output
+		}
+
+		try {
+			unlinkSync(kubeConfigPath);
+		} catch {
+			/* ignore */
+		}
+		if (chartDir)
+			try {
+				execSync(`rm -rf "${chartDir}"`, { stdio: 'pipe' });
+			} catch {
+				/* ignore */
+			}
+		await k3s.stop();
+		throw error;
+	}
+}

packages/testing/containers/helm-start-stack.ts

@@ -0,0 +1,153 @@
+#!/usr/bin/env tsx
+import { writeFileSync } from 'node:fs';
+import { parseArgs } from 'node:util';
+
+import { createHelmStack, type HelmStackMode } from './helm-stack';
+
+const colors = {
+	reset: '\x1b[0m',
+	bright: '\x1b[1m',
+	green: '\x1b[32m',
+	yellow: '\x1b[33m',
+	blue: '\x1b[34m',
+	red: '\x1b[31m',
+	cyan: '\x1b[36m',
+};
+
+const log = {
+	info: (msg: string) => console.log(`${colors.blue}ℹ${colors.reset} ${msg}`),
+	success: (msg: string) => console.log(`${colors.green}✓${colors.reset} ${msg}`),
+	error: (msg: string) => console.error(`${colors.red}✗${colors.reset} ${msg}`),
+	header: (msg: string) => console.log(`\n${colors.bright}${colors.cyan}${msg}${colors.reset}\n`),
+};
+
+function showHelp() {
+	console.log(`
+${colors.bright}n8n Helm Stack${colors.reset}
+
+Start n8n via Helm chart in a K3s (lightweight Kubernetes) container.
+
+${colors.yellow}Usage:${colors.reset}
+  pnpm stack:helm [options]
+
+${colors.yellow}Options:${colors.reset}
+  --mode <mode>         standalone (SQLite, default) or queue (PostgreSQL + Redis + workers)
+  --image <image>       n8n Docker image (default: n8nio/n8n:local)
+  --chart-ref <ref>     Git branch/tag for n8n-hosting repo (default: main)
+  --chart-repo <url>    Git repo URL (default: https://github.com/n8n-io/n8n-hosting.git)
+  --k3s-image <image>   K3s image (default: rancher/k3s:v1.32.2-k3s1)
+  --env <KEY=VALUE>     Set environment variable in n8n pods (repeatable)
+  --url-file <path>     Write URL to file when ready (for CI)
+  --help, -h            Show this help
+
+${colors.yellow}Examples:${colors.reset}
+  ${colors.bright}# Start with default local image${colors.reset}
+  pnpm stack:helm
+
+  ${colors.bright}# Test specific n8n version against specific chart${colors.reset}
+  pnpm stack:helm --image n8nio/n8n:1.80.0 --chart-ref v1.2.0
+
+  ${colors.bright}# Queue mode (PostgreSQL + Redis + workers)${colors.reset}
+  pnpm stack:helm --mode queue --image n8nio/n8n:latest
+
+  ${colors.bright}# E2E test mode (requires INCLUDE_TEST_CONTROLLER image)${colors.reset}
+  pnpm stack:helm --env E2E_TESTS=true --env NODE_ENV=development
+
+  ${colors.bright}# CI mode (writes URL to file)${colors.reset}
+  pnpm stack:helm --url-file /tmp/n8n-url.txt &
+
+${colors.yellow}Prerequisites:${colors.reset}
+  • Docker with privileged container support
+  • helm and kubectl CLIs installed locally
+  • n8n Docker image built locally (pnpm build:docker) or available on Docker Hub
+  • See HELM-TESTING.md for full requirements
+
+${colors.yellow}Notes:${colors.reset}
+  • helm and kubectl run on your host (not inside K3s)
+  • Startup takes ~60-120s (K3s boot + image load + Helm install)
+  • After startup, use kubectl with KUBECONFIG printed below
+  • Press Ctrl+C to stop
+`);
+}
+
+async function main() {
+	const { values } = parseArgs({
+		args: process.argv.slice(2),
+		options: {
+			help: { type: 'boolean', short: 'h' },
+			mode: { type: 'string' },
+			image: { type: 'string' },
+			'chart-ref': { type: 'string' },
+			'chart-repo': { type: 'string' },
+			'k3s-image': { type: 'string' },
+			env: { type: 'string', multiple: true },
+			'url-file': { type: 'string' },
+			'kubeconfig-file': { type: 'string' },
+		},
+		allowPositionals: false,
+	});
+
+	if (values.help) {
+		showHelp();
+		process.exit(0);
+	}
+
+	log.header('Starting n8n Helm Stack');
+
+	const mode = (values.mode as HelmStackMode) || undefined;
+
+	// Parse --env KEY=VALUE pairs into a record
+	const envOverrides: Record<string, string> = {};
+	for (const entry of values.env ?? []) {
+		const eqIndex = entry.indexOf('=');
+		if (eqIndex === -1) {
+			log.error(`Invalid --env format: "${entry}" (expected KEY=VALUE)`);
+			process.exit(1);
+		}
+		envOverrides[entry.slice(0, eqIndex)] = entry.slice(eqIndex + 1);
+	}
+
+	const stack = await createHelmStack({
+		n8nImage: values.image,
+		helmChartRef: values['chart-ref'],
+		helmChartRepo: values['chart-repo'],
+		k3sImage: values['k3s-image'],
+		mode,
+		env: Object.keys(envOverrides).length > 0 ? envOverrides : undefined,
+	});
+
+	log.header('Stack Ready');
+	log.success(`n8n URL: ${colors.bright}${colors.green}${stack.baseUrl}${colors.reset}`);
+	log.info(`Kubeconfig: ${colors.bright}${stack.kubeConfigPath}${colors.reset}`);
+
+	if (values['url-file']) {
+		writeFileSync(values['url-file'], stack.baseUrl);
+		log.info(`URL written to ${values['url-file']}`);
+	}
+
+	if (values['kubeconfig-file']) {
+		writeFileSync(values['kubeconfig-file'], stack.kubeConfigPath);
+		log.info(`Kubeconfig path written to ${values['kubeconfig-file']}`);
+	}
+
+	console.log('');
+	log.info('Debug with kubectl (context already active):');
+	log.info(`  ${colors.bright}kubectl get pods${colors.reset}`);
+	log.info(`  ${colors.bright}kubectl logs -l app.kubernetes.io/name=n8n${colors.reset}`);
+	console.log('');
+	log.info(`Cleanup: ${colors.bright}pnpm --filter n8n-containers stack:helm:clean${colors.reset}`);
+	console.log('');
+	if (envOverrides.E2E_TESTS === 'true') {
+		log.info('Run tests against this instance:');
+		log.info(
+			`  ${colors.bright}N8N_BASE_URL=${stack.baseUrl} RESET_E2E_DB=true npx playwright test tests/e2e/building-blocks/ --workers=1${colors.reset}`,
+		);
+	}
+
+	// Container stays alive in Docker (Ryuk disabled) — clean up via stack:helm:clean.
+}
+
+main().catch((error) => {
+	log.error(`Failed to start: ${error instanceof Error ? error.message : String(error)}`);
+	process.exit(1);
+});

packages/testing/containers/helpers/utils.ts

@@ -0,0 +1,80 @@
+import { setTimeout as wait } from 'node:timers/promises';
+import type { Readable } from 'stream';
+import type { StartedTestContainer } from 'testcontainers';
+
+/**
+ * Create a logger that prefixes messages with elapsed time since creation.
+ * Only outputs when CONTAINER_TELEMETRY_VERBOSE=1 is set.
+ */
+export function createElapsedLogger(prefix: string) {
+	const startTime = Date.now();
+	const isVerbose = process.env.CONTAINER_TELEMETRY_VERBOSE === '1';
+
+	return (message: string) => {
+		if (!isVerbose) return;
+		const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
+		console.log(`[${prefix} +${elapsed}s] ${message}`);
+	};
+}
+
+/**
+ * Create a log consumer that does not log to the console.
+ * Logs are collected in memory and can be output on error.
+ */
+export function createSilentLogConsumer() {
+	const logs: string[] = [];
+
+	const consumer = (stream: Readable) => {
+		stream.on('data', (chunk: Buffer | string) => {
+			logs.push(chunk.toString().trim());
+		});
+	};
+
+	const throwWithLogs = (error: unknown): never => {
+		if (logs.length > 0) {
+			console.error('\n--- Container Logs ---');
+			console.error(logs.join('\n'));
+			console.error('---------------------\n');
+		}
+		throw error;
+	};
+
+	return { consumer, throwWithLogs };
+}
+
+/**
+ * Polls a container's HTTP endpoint until it returns a 200 status.
+ * Logs a warning if the endpoint does not return 200 within the specified timeout.
+ *
+ * @param container The started container.
+ * @param endpoint The HTTP health check endpoint (e.g., '/healthz/readiness').
+ * @param timeoutMs Total timeout in milliseconds (default: 60,000ms).
+ */
+export async function pollContainerHttpEndpoint(
+	container: StartedTestContainer,
+	endpoint: string,
+	timeoutMs: number = 60000,
+): Promise<void> {
+	const startTime = Date.now();
+	const url = `http://${container.getHost()}:${container.getFirstMappedPort()}${endpoint}`;
+	const retryIntervalMs = 1000;
+
+	while (Date.now() - startTime < timeoutMs) {
+		try {
+			const response = await fetch(url);
+			if (response.status === 200) {
+				return;
+			}
+		} catch {
+			// Don't log errors, just retry
+		}
+
+		await wait(retryIntervalMs);
+	}
+
+	console.error(
+		`WARNING: HTTP endpoint at ${url} did not return 200 within ${
+			timeoutMs / 1000
+		} seconds. Proceeding with caution.`,
+	);
+}

packages/testing/containers/index.ts

@@ -0,0 +1,27 @@
+/**
+ * n8n Test Containers
+ *
+ * This package provides container management utilities for n8n testing.
+ * Services are accessed via n8nContainer.services.* in tests.
+ */
+
+// Stack orchestration - primary public API
+export { createN8NStack } from './stack';
+export type { N8NConfig, N8NStack } from './stack';
+
+// K3s + Helm chart stack - for Kubernetes deployment validation
+export { createHelmStack } from './helm-stack';
+export type { HelmStack, HelmStackConfig, HelmStackMode } from './helm-stack';
+
+// Service-only stack (no n8n containers) - for integration tests
+export { createServiceStack } from './service-stack';
+
+export type { StackTelemetryRecord } from './telemetry';
+
+// Performance plans (CLI-only)
+export * from './performance-plans';
+
+// Types used externally by tests
+export { type LogEntry, type MetricsHelper } from './services/observability';
+export { type GiteaHelper } from './services/gitea';
+export { KafkaHelper } from './services/kafka';

packages/testing/containers/n8n-image-pull-policy.ts

@@ -0,0 +1,19 @@
+import type { ImagePullPolicy } from 'testcontainers';
+import { PullPolicy } from 'testcontainers';
+
+/**
+ * Custom pull policy for n8n images:
+ * - Never try to pull the local image
+ * - Otherwise, use the default pull policy (pull only if not present)
+ */
+export class N8nImagePullPolicy implements ImagePullPolicy {
+	constructor(private readonly image: string) {}
+
+	shouldPull(): boolean {
+		if (this.image === 'n8nio/n8n:local') {
+			return false;
+		}
+
+		return PullPolicy.defaultPolicy().shouldPull();
+	}
+}

packages/testing/containers/n8n-start-stack.ts

@@ -0,0 +1,534 @@
+#!/usr/bin/env tsx
+import { writeFileSync } from 'node:fs';
+import { resolve } from 'node:path';
+import { parseArgs } from 'node:util';
+
+import { DockerImageNotFoundError } from './docker-image-not-found-error';
+import { BASE_PERFORMANCE_PLANS, isValidPerformancePlan } from './performance-plans';
+import { createServiceStack } from './service-stack';
+import type { CloudflaredResult } from './services/cloudflared';
+import type { KentResult } from './services/kent';
+import type { KeycloakResult } from './services/keycloak';
+import type { MailpitResult } from './services/mailpit';
+import type { NgrokResult } from './services/ngrok';
+import { services as SERVICE_REGISTRY } from './services/registry';
+import type { TracingResult } from './services/tracing';
+import type { ServiceName } from './services/types';
+import type { VictoriaLogsResult } from './services/victoria-logs';
+import type { VictoriaMetricsResult } from './services/victoria-metrics';
+import type { N8NConfig, N8NStack } from './stack';
+import { createN8NStack } from './stack';
+import { TEST_CONTAINER_IMAGES } from './test-containers';
+
+// ANSI colors for terminal output
+const colors = {
+	reset: '\x1b[0m',
+	bright: '\x1b[1m',
+	green: '\x1b[32m',
+	yellow: '\x1b[33m',
+	blue: '\x1b[34m',
+	red: '\x1b[31m',
+	cyan: '\x1b[36m',
+};
+
+const log = {
+	info: (msg: string) => console.log(`${colors.blue}ℹ${colors.reset} ${msg}`),
+	success: (msg: string) => console.log(`${colors.green}✓${colors.reset} ${msg}`),
+	error: (msg: string) => console.error(`${colors.red}✗${colors.reset} ${msg}`),
+	warn: (msg: string) => console.warn(`${colors.yellow}⚠${colors.reset} ${msg}`),
+	header: (msg: string) => console.log(`\n${colors.bright}${colors.cyan}${msg}${colors.reset}\n`),
+};
+
+function showHelp() {
+	console.log(`
+${colors.bright}n8n Stack Manager${colors.reset}
+
+Start n8n containers for development and testing.
+
+${colors.yellow}Usage:${colors.reset}
+  npm run stack [options]
+
+${colors.yellow}Options:${colors.reset}
+  --services-only   Start services only (no n8n containers), write .env for local dev
+  --services <list> Comma-separated services (e.g. postgres,redis,mailpit,proxy,kafka)
+  --postgres        Use PostgreSQL instead of SQLite
+  --queue           Enable queue mode (requires PostgreSQL)
+  --source-control  Enable source control (Git) container for testing
+  --oidc            Enable OIDC testing with Keycloak (requires PostgreSQL)
+  --observability   Enable observability stack (VictoriaLogs + VictoriaMetrics + Vector)
+  --tracing         Enable tracing stack (n8n-tracer + Jaeger) for workflow visualization
+  --kafka           Enable Kafka broker for message queue trigger testing
+  --tunnel          Enable Cloudflare Tunnel for public URL (via trycloudflare.com)
+  --ngrok           Enable ngrok tunnel for public URL (requires NGROK_AUTHTOKEN env var)
+  --mailpit         Enable Mailpit for email testing
+  --kent            Enable Kent (Sentry mock) for error tracking testing
+  --mains <n>       Number of main instances (default: 1)
+  --workers <n>     Number of worker instances (default: 1)
+  --name <name>     Project name for parallel runs
+  --env KEY=VALUE   Set environment variables
+  --plan <plan>     Use performance plan preset (${Object.keys(BASE_PERFORMANCE_PLANS).join(', ')})
+  --help, -h        Show this help
+
+${colors.yellow}Performance Plans:${colors.reset}
+${Object.entries(BASE_PERFORMANCE_PLANS)
+	.map(
+		([name, plan]) =>
+			`  ${name.padEnd(12)} ${plan.memory}GB RAM, ${plan.cpu} CPU cores - SQLite only`,
+	)
+	.join('\n')}
+
+${colors.yellow}Environment Variables:${colors.reset}
+  • TEST_IMAGE_N8N=<image>  Use a custom Docker image (default: n8nio/n8n:local)
+
+${colors.yellow}Examples:${colors.reset}
+  ${colors.bright}# Simple SQLite instance${colors.reset}
+  npm run stack
+
+  ${colors.bright}# PostgreSQL database${colors.reset}
+  npm run stack --postgres
+
+  ${colors.bright}# Queue mode (automatically uses PostgreSQL)${colors.reset}
+  npm run stack --queue
+
+  ${colors.bright}# With source control (Git) testing${colors.reset}
+  npm run stack --postgres --source-control
+
+  ${colors.bright}# With OIDC (Keycloak) for SSO testing${colors.reset}
+  npm run stack --postgres --oidc
+
+  ${colors.bright}# With observability stack (logs + metrics persist even after terminal closes)${colors.reset}
+  npm run stack --observability
+
+  ${colors.bright}# With tracing stack (Jaeger UI for workflow execution visualization)${colors.reset}
+  npm run stack --queue --tracing
+
+  ${colors.bright}# With public tunnel (webhooks accessible from internet)${colors.reset}
+  npm run stack --tunnel
+
+  ${colors.bright}# Custom scaling${colors.reset}
+  npm run stack --queue --mains 3 --workers 5
+
+  ${colors.bright}# With environment variables${colors.reset}
+  npm run stack --postgres --env N8N_LOG_LEVEL=info --env N8N_ENABLED_MODULES=insights
+
+  ${colors.bright}# Performance plan presets${colors.reset}
+${Object.keys(BASE_PERFORMANCE_PLANS)
+	.map((name) => `  npm run stack --plan ${name}`)
+	.join('\n')}
+
+  ${colors.bright}# Services only (local dev — writes .env for pnpm start)${colors.reset}
+  pnpm services --services postgres
+  pnpm services --services postgres,redis
+  pnpm services --services postgres,mailpit,proxy
+
+  ${colors.bright}# Parallel instances${colors.reset}
+  npm run stack --name test-1
+  npm run stack --name test-2
+
+${colors.yellow}Notes:${colors.reset}
+  • SQLite is the default database (no external dependencies)
+  • Task runner is always enabled (mirrors production)
+  • Queue mode requires PostgreSQL and enables horizontal scaling
+  • Use --name for running multiple instances in parallel
+  • Performance plans simulate cloud constraints (SQLite only, resource-limited)
+  • Press Ctrl+C to stop all containers
+`);
+}
+
+async function main() {
+	const { values } = parseArgs({
+		args: process.argv.slice(2),
+		options: {
+			help: { type: 'boolean', short: 'h' },
+			'services-only': { type: 'boolean' },
+			postgres: { type: 'boolean' },
+			queue: { type: 'boolean' },
+			services: { type: 'string' },
+			'source-control': { type: 'boolean' },
+			oidc: { type: 'boolean' },
+			observability: { type: 'boolean' },
+			tracing: { type: 'boolean' },
+			kafka: { type: 'boolean' },
+			tunnel: { type: 'boolean' },
+			ngrok: { type: 'boolean' },
+			mailpit: { type: 'boolean' },
+			kent: { type: 'boolean' },
+			mains: { type: 'string' },
+			workers: { type: 'string' },
+			name: { type: 'string' },
+			env: { type: 'string', multiple: true },
+			plan: { type: 'string' },
+		},
+		allowPositionals: false,
+	});
+
+	// Show help if requested
+	if (values.help) {
+		showHelp();
+		process.exit(0);
+	}
+
+	const servicesOnly = values['services-only'] ?? false;
+
+	// Build services array from CLI flags
+	const validServiceNames = new Set(Object.keys(SERVICE_REGISTRY));
+	const services: ServiceName[] = [];
+	if (values.services) {
+		for (const name of values.services.split(',').map((s) => s.trim())) {
+			if (!validServiceNames.has(name)) {
+				log.error(`Unknown service: '${name}'. Available: ${[...validServiceNames].join(', ')}`);
+				process.exit(1);
+			}
+			services.push(name as ServiceName);
+		}
+	}
+	if (values['source-control']) services.push('gitea');
+	if (values.oidc) services.push('keycloak');
+	if (values.observability) services.push('victoriaLogs', 'victoriaMetrics', 'vector');
+	if (values.tracing) services.push('tracing');
+	if (values.kafka) services.push('kafka');
+	if (values.tunnel) services.push('cloudflared');
+	if (values.ngrok) services.push('ngrok');
+	if (values.mailpit) services.push('mailpit');
+	if (values.kent) services.push('kent');
+
+	// Build configuration
+	const config: N8NConfig = {
+		postgres: values.postgres ?? false,
+		services,
+		projectName:
+			values.name ??
+			(servicesOnly
+				? `n8n-svc-${Math.random().toString(36).substring(7)}`
+				: `n8n-stack-${Math.random().toString(36).substring(7)}`),
+	};
+
+	// Handle queue mode (mains > 1 or workers > 0)
+	if (values.queue ?? values.mains ?? values.workers) {
+		const mains = parseInt(values.mains ?? '1', 10);
+		const workers = parseInt(values.workers ?? '1', 10);
+
+		if (isNaN(mains) || isNaN(workers) || mains < 1 || workers < 0) {
+			log.error('Invalid mains or workers count');
+			process.exit(1);
+		}
+
+		config.mains = mains;
+		config.workers = workers;
+	}
+
+	if (values.plan) {
+		const planName = values.plan;
+		if (!isValidPerformancePlan(planName)) {
+			log.error(`Invalid performance plan: ${values.plan}`);
+			log.error(`Available plans: ${Object.keys(BASE_PERFORMANCE_PLANS).join(', ')}`);
+			process.exit(1);
+		}
+
+		const plan = BASE_PERFORMANCE_PLANS[planName];
+
+		if (values.postgres) {
+			log.warn('Performance plans use SQLite only. PostgreSQL option ignored.');
+		}
+		if (values.queue || values.mains || values.workers) {
+			log.warn('Performance plans use SQLite only. Queue mode ignored.');
+		}
+
+		config.resourceQuota = plan;
+		config.postgres = false; // Force SQLite for performance plans
+		config.mains = 1; // Force single instance for performance plans
+		config.workers = 0;
+
+		log.info(
+			`Using ${planName} performance plan: ${plan.memory}GB RAM, ${plan.cpu} CPU cores (SQLite only)`,
+		);
+	}
+
+	// Parse environment variables
+	if (values.env && values.env.length > 0) {
+		config.env = {};
+
+		for (const envStr of values.env) {
+			const [key, ...valueParts] = envStr.split('=');
+			const value = valueParts.join('='); // Handle values with = in them
+
+			if (key && value) {
+				config.env[key] = value;
+			} else {
+				log.warn(`Invalid env format: ${envStr} (expected KEY=VALUE)`);
+			}
+		}
+	}
+
+	// Services-only mode: start containers, write .env, no n8n
+	if (servicesOnly) {
+		if (services.length === 0) {
+			log.error('No services specified. Use flags like --postgres, --redis, --mailpit, etc.');
+			process.exit(1);
+		}
+
+		log.header('Starting service containers');
+		log.info(`Project: ${config.projectName}`);
+		log.info(`Services: ${services.join(', ')}`);
+
+		try {
+			const stack = await createServiceStack({
+				services,
+				projectName: config.projectName,
+			});
+
+			// Collect host-compatible env vars from each service
+			const envVars: Record<string, string> = {};
+			for (const name of services) {
+				const result = stack.serviceResults[name];
+				if (!result) continue;
+
+				const service = SERVICE_REGISTRY[name];
+				Object.assign(
+					envVars,
+					service.env?.(result, true) ?? {},
+					service.extraEnv?.(result, true) ?? {},
+				);
+			}
+
+			// Write .env to packages/cli/bin/ because `pnpm start` runs os-normalize.mjs
+			// which does `cd packages/cli/bin` before launching n8n, and dotenv loads from cwd.
+			if (Object.keys(envVars).length > 0) {
+				const repoRoot = resolve(__dirname, '../../..');
+				const envPath = resolve(repoRoot, 'packages/cli/bin/.env');
+				const lines = [
+					'# Generated by pnpm services — do not edit',
+					`# Project: ${stack.projectName}`,
+					'# Stop with: pnpm --filter n8n-containers services:clean',
+					'',
+					...Object.entries(envVars).map(([key, value]) => `${key}=${value}`),
+					'',
+				];
+				writeFileSync(envPath, lines.join('\n'));
+				log.success(`Wrote ${Object.keys(envVars).length} env vars to packages/cli/bin/.env`);
+			}
+
+			// Print summary
+			log.header('Services running');
+			for (const name of services) {
+				const result = stack.serviceResults[name];
+				if (!result) continue;
+
+				const service = SERVICE_REGISTRY[name];
+				const vars = {
+					...(service.env?.(result, true) ?? {}),
+					...(service.extraEnv?.(result, true) ?? {}),
+				};
+				const varSummary = Object.entries(vars)
+					.map(([k, v]) => `${k}=${v}`)
+					.join(', ');
+				log.success(`${name}${varSummary ? `: ${varSummary}` : ''}`);
+			}
+
+			// Print mailpit UI URL if running
+			const mailpitResult = stack.serviceResults.mailpit as MailpitResult | undefined;
+			if (mailpitResult) {
+				console.log('');
+				log.info(`Mailpit UI: ${colors.cyan}${mailpitResult.meta.apiBaseUrl}${colors.reset}`);
+			}
+
+			console.log('');
+			log.info('Containers are running in the background');
+			log.info(`Run ${colors.bright}pnpm dev${colors.reset} in another terminal to start n8n`);
+			log.info(
+				`Cleanup: ${colors.bright}pnpm --filter n8n-containers services:clean${colors.reset}`,
+			);
+			console.log('');
+		} catch (error) {
+			log.error(
+				`Failed to start services: ${error instanceof Error ? error.message : String(error)}`,
+			);
+			process.exit(1);
+		}
+		return;
+	}
+
+	log.header('Starting n8n Stack');
+	log.info(`Project name: ${config.projectName}`);
+	displayConfig(config);
+
+	let stack: N8NStack;
+
+	try {
+		try {
+			stack = await createN8NStack(config);
+		} catch (error) {
+			if (error instanceof DockerImageNotFoundError) {
+				log.error(error.message);
+				process.exit(1);
+			}
+			throw error;
+		}
+
+		console.log('');
+		log.info(`n8n URL: ${colors.bright}${colors.green}${stack.baseUrl}${colors.reset}`);
+
+		// Display OIDC configuration if enabled
+		const keycloakResult = stack.serviceResults.keycloak as KeycloakResult | undefined;
+		if (keycloakResult) {
+			const { meta } = keycloakResult;
+			console.log('');
+			log.header('OIDC Configuration (Keycloak)');
+			log.info(`Discovery URL: ${colors.cyan}${meta.discoveryUrl}${colors.reset}`);
+			log.info(`Client ID: ${colors.cyan}${meta.clientId}${colors.reset}`);
+			log.info(`Client Secret: ${colors.cyan}${meta.clientSecret}${colors.reset}`);
+			console.log('');
+			log.header('Test User Credentials');
+			log.info(`Email: ${colors.cyan}${meta.testUser.email}${colors.reset}`);
+			log.info(`Password: ${colors.cyan}${meta.testUser.password}${colors.reset}`);
+		}
+
+		// Display observability configuration if enabled
+		const logsResult = stack.serviceResults.victoriaLogs as VictoriaLogsResult | undefined;
+		const metricsResult = stack.serviceResults.victoriaMetrics as VictoriaMetricsResult | undefined;
+		if (logsResult || metricsResult) {
+			console.log('');
+			log.header('Observability Stack (VictoriaObs)');
+			if (logsResult) {
+				log.info(
+					`VictoriaLogs UI: ${colors.cyan}${logsResult.meta.queryEndpoint}/select/vmui${colors.reset}`,
+				);
+			}
+			if (metricsResult) {
+				log.info(
+					`VictoriaMetrics UI: ${colors.cyan}${metricsResult.meta.queryEndpoint}/vmui${colors.reset}`,
+				);
+			}
+			// Vector is always started when observability is enabled in the new stack
+			log.success('Container logs collected by Vector (runs in background)');
+		}
+
+		const tracingResult = stack.serviceResults.tracing as TracingResult | undefined;
+		if (tracingResult) {
+			console.log('');
+			log.header('Tracing Stack (n8n-tracer + Jaeger)');
+			log.info(`Jaeger UI: ${colors.cyan}${tracingResult.meta.jaeger.uiUrl}${colors.reset}`);
+		}
+
+		const cloudflaredResult = stack.serviceResults.cloudflared as CloudflaredResult | undefined;
+		if (cloudflaredResult) {
+			console.log('');
+			log.header('Cloudflare Tunnel');
+			log.info(`Public URL: ${colors.cyan}${cloudflaredResult.meta.publicUrl}${colors.reset}`);
+			log.info('Webhooks are accessible from the internet via this URL');
+		}
+
+		const ngrokResult = stack.serviceResults.ngrok as NgrokResult | undefined;
+		if (ngrokResult) {
+			console.log('');
+			log.header('ngrok Tunnel');
+			log.info(`Public URL: ${colors.cyan}${ngrokResult.meta.publicUrl}${colors.reset}`);
+			log.info('Webhooks are accessible from the internet via this URL');
+		}
+
+		const mailpitResult = stack.serviceResults.mailpit as MailpitResult | undefined;
+		if (mailpitResult) {
+			console.log('');
+			log.header('Email Testing (Mailpit)');
+			log.info(`Mailpit UI: ${colors.cyan}${mailpitResult.meta.apiBaseUrl}${colors.reset}`);
+		}
+
+		const kentResult = stack.serviceResults.kent as KentResult | undefined;
+		if (kentResult) {
+			console.log('');
+			log.header('Sentry Mock (Kent)');
+			log.info(`Kent UI: ${colors.cyan}${kentResult.meta.apiUrl}${colors.reset}`);
+			log.info(`Backend DSN: ${colors.cyan}${kentResult.meta.sentryDsn}${colors.reset}`);
+			log.info(`Frontend DSN: ${colors.cyan}${kentResult.meta.frontendDsn}${colors.reset}`);
+		}
+
+		console.log('');
+		log.info('Containers are running in the background');
+		log.info(
+			'Cleanup with: pnpm --filter n8n-containers stack:clean:all (stops containers and removes networks)',
+		);
+		console.log('');
+	} catch (error) {
+		log.error(`Failed to start: ${error as string}`);
+		process.exit(1);
+	}
+}
+
+function displayConfig(config: N8NConfig) {
+	const dockerImage = TEST_CONTAINER_IMAGES.n8n;
+	const mains = config.mains ?? 1;
+	const workers = config.workers ?? 0;
+	const isQueueMode = mains > 1 || workers > 0;
+	const services = config.services ?? [];
+
+	// eslint-disable-next-line @typescript-eslint/prefer-nullish-coalescing
+	const usePostgres = config.postgres || isQueueMode || services.includes('keycloak');
+
+	let modeStr: string;
+	if (isQueueMode) {
+		const parts = [`${mains}M/${workers}W`, usePostgres ? 'PostgreSQL' : 'SQLite'];
+		if (mains > 1) parts.push('load-balanced');
+		modeStr = parts.join(', ');
+	} else {
+		modeStr = usePostgres ? 'single, PostgreSQL' : 'single, SQLite';
+	}
+
+	log.info(`Image: ${dockerImage}`);
+	log.info(`Mode: ${modeStr}`);
+
+	const enabledFeatures: string[] = [];
+	if (services.includes('gitea')) enabledFeatures.push('Source Control (Gitea)');
+	if (services.includes('keycloak')) enabledFeatures.push('OIDC (Keycloak)');
+	if (services.includes('victoriaLogs')) enabledFeatures.push('Observability');
+	if (services.includes('tracing')) enabledFeatures.push('Tracing (Jaeger)');
+	if (services.includes('mailpit')) enabledFeatures.push('Email (Mailpit)');
+	if (services.includes('kent')) enabledFeatures.push('Sentry Mock (Kent)');
+
+	if (enabledFeatures.length > 0) {
+		log.info(`Services: ${enabledFeatures.join(', ')}`);
+	}
+
+	// Display observability status
+	if (services.includes('victoriaLogs')) {
+		log.info('Observability: enabled (VictoriaLogs + VictoriaMetrics + Vector)');
+	} else {
+		log.info('Observability: disabled');
+	}
+
+	// Display tracing status
+	if (services.includes('tracing')) {
+		log.info('Tracing: enabled (n8n-tracer + Jaeger)');
+	} else {
+		log.info('Tracing: disabled');
+	}
+
+	// Display tunnel status
+	if (services.includes('cloudflared')) {
+		log.info('Tunnel: enabled (Cloudflare Quick Tunnel)');
+	} else if (services.includes('ngrok')) {
+		log.info('Tunnel: enabled (ngrok)');
+	} else {
+		log.info('Tunnel: disabled');
+	}
+	if (config.resourceQuota) {
+		log.info(`Resources: ${config.resourceQuota.memory}GB RAM, ${config.resourceQuota.cpu} CPU`);
+	}
+
+	if (config.env && Object.keys(config.env).length > 0) {
+		log.info(`Custom env: ${Object.keys(config.env).join(', ')}`);
+	}
+}
+
+// Run if executed directly
+if (require.main === module) {
+	// Keep the event loop alive while main() runs. Without this, the process
+	// can exit between async Docker API calls (e.g. after exposeHostPorts
+	// resolves but before Network.start() creates new I/O handles).
+	const keepAlive = setInterval(() => {}, 30_000);
+	main()
+		.catch((error) => {
+			log.error(`Unexpected error: ${error}`);
+			process.exit(1);
+		})
+		.finally(() => clearInterval(keepAlive));
+}

packages/testing/containers/network-stabilization.ts

@@ -0,0 +1,83 @@
+import { spawn } from 'child_process';
+import os from 'os';
+
+/**
+ * Wait for the Linux network stack to become quiet (no netlink events).
+ * This monitors actual kernel network events rather than guessing with fixed delays.
+ *
+ * Only runs in CI on Linux. On other platforms or local dev, resolves immediately.
+ *
+ * @param quietDurationMs - How long the network must be quiet before resolving (default: 2000ms)
+ * @param maxWaitMs - Maximum time to wait before giving up (default: 10000ms)
+ */
+export async function waitForNetworkQuiet(
+	quietDurationMs = 1000,
+	maxWaitMs = 10000,
+): Promise<void> {
+	// Only run in CI on Linux
+	if (!process.env.CI || os.platform() !== 'linux') {
+		return;
+	}
+
+	return await new Promise((resolve) => {
+		let lastEventTime = Date.now();
+		let checkInterval: NodeJS.Timeout | null = null;
+		let maxTimeout: NodeJS.Timeout | null = null;
+		let resolved = false;
+
+		const cleanup = () => {
+			if (resolved) return;
+			resolved = true;
+			if (checkInterval) clearInterval(checkInterval);
+			if (maxTimeout) clearTimeout(maxTimeout);
+			monitor.kill();
+		};
+
+		// Monitor network events using `ip monitor`
+		// Watches: link (interfaces), address (IP assignments), route (routing table)
+		const monitor = spawn('ip', ['monitor', 'link', 'address', 'route'], {
+			stdio: ['ignore', 'pipe', 'pipe'],
+		});
+
+		monitor.on('error', (error: Error) => {
+			// ip command not available - fall back to no-op
+			console.warn(`[network-stabilization] ip monitor not available: ${error.message}`);
+			cleanup();
+			resolve();
+		});
+
+		monitor.stdout.on('data', () => {
+			// Network change detected, reset timer
+			lastEventTime = Date.now();
+		});
+
+		monitor.stderr.on('data', (data) => {
+			console.warn(`[network-stabilization] ip monitor stderr: ${data}`);
+		});
+
+		// Check periodically if network has been quiet long enough
+		checkInterval = setInterval(() => {
+			const quietDuration = Date.now() - lastEventTime;
+			if (quietDuration >= quietDurationMs) {
+				console.log(`[network-stabilization] Network quiet for ${quietDuration}ms, proceeding`);
+				cleanup();
+				resolve();
+			}
+		}, 100);
+
+		// Maximum wait timeout
+		maxTimeout = setTimeout(() => {
+			console.warn(`[network-stabilization] Max wait (${maxWaitMs}ms) exceeded, proceeding anyway`);
+			cleanup();
+			resolve();
+		}, maxWaitMs);
+
+		// Handle process exit
+		monitor.on('close', () => {
+			if (!resolved) {
+				cleanup();
+				resolve();
+			}
+		});
+	});
+}

packages/testing/containers/package.json

@@ -0,0 +1,43 @@
+{
+  "name": "n8n-containers",
+  "private": true,
+  "version": "1.0.0",
+  "description": "",
+  "main": "index.ts",
+  "scripts": {
+    "stack": "tsx ./n8n-start-stack.ts",
+    "stack:help": "tsx ./n8n-start-stack.ts --help",
+    "stack:sqlite": "TESTCONTAINERS_REUSE_ENABLE=true npm run stack",
+    "stack:postgres": "TESTCONTAINERS_REUSE_ENABLE=true npm run stack -- --postgres",
+    "stack:queue": "TESTCONTAINERS_REUSE_ENABLE=true npm run stack -- --queue",
+    "stack:multi-main": "TESTCONTAINERS_REUSE_ENABLE=true npm run stack -- --mains 2 --workers 1",
+    "stack:starter": "TESTCONTAINERS_REUSE_ENABLE=true npm run stack -- --plan starter",
+    "stack:enterprise": "TESTCONTAINERS_REUSE_ENABLE=true npm run stack -- --plan enterprise",
+    "stack:observability": "TESTCONTAINERS_REUSE_ENABLE=true npm run stack -- --observability",
+    "stack:kafka": "TESTCONTAINERS_REUSE_ENABLE=true npm run stack -- --kafka",
+    "stack:clean:containers": "docker ps -aq --filter 'name=n8n-stack-*' | xargs -r docker rm -f 2>/dev/null",
+    "stack:clean:networks": "docker network ls --filter 'label=org.testcontainers=true' -q | xargs -r docker network rm 2>/dev/null",
+    "stack:clean:all": "pnpm run stack:clean:containers && pnpm run stack:clean:networks",
+    "stack:helm": "TESTCONTAINERS_RYUK_DISABLED=true tsx ./helm-start-stack.ts",
+    "stack:helm:clean": "docker ps -aq --filter 'label=n8n.helm=true' | xargs -r docker rm -f 2>/dev/null; rm -f ~/.kube/n8n-helm-*.yaml; kubectl config delete-context n8n-helm-standalone 2>/dev/null; kubectl config delete-context n8n-helm-queue 2>/dev/null; true",
+    "services": "tsx ./n8n-start-stack.ts --services-only",
+    "services:clean": "docker ps -aq --filter 'name=n8n-svc-*' | xargs docker rm -f 2>/dev/null; rm -f ../../../packages/cli/bin/.env",
+    "lint": "eslint . --quiet",
+    "lint:fix": "eslint . --fix"
+  },
+  "keywords": [],
+  "author": "",
+  "license": "ISC",
+  "devDependencies": {
+    "@aws-sdk/client-secrets-manager": "3.808.0",
+    "@testcontainers/k3s": "^11.11.0",
+    "@testcontainers/kafka": "^11.11.0",
+    "@testcontainers/mysql": "^11.11.0",
+    "@testcontainers/postgresql": "^11.0.3",
+    "@testcontainers/redis": "^11.0.3",
+    "get-port": "^7.1.0",
+    "mockserver-client": "^5.15.0",
+    "kafkajs": "catalog:",
+    "testcontainers": "^11.11.0"
+  }
+}

packages/testing/containers/performance-plans.ts

@@ -0,0 +1,28 @@
+/**
+ * Shared Performance Plan Types and Configurations
+ *
+ * This file provides the base performance plan definitions that can be used by:
+ * - CLI tools (n8n-start-stack.ts)
+ * - Playwright tests (cloud-only.ts)
+ *
+ */
+
+// Base performance plan configuration (resource constraints only)
+export interface BasePerformancePlan {
+	memory: number; // in GB
+	cpu: number; // in cores
+}
+
+export const BASE_PERFORMANCE_PLANS: Record<string, BasePerformancePlan> = {
+	trial: { memory: 0.75, cpu: 1 }, // 768MB RAM, 1000 millicore CPU
+	starter: { memory: 0.75, cpu: 1 }, // 768MB RAM, 1000 millicore CPU
+	pro1: { memory: 1.25, cpu: 1 }, // 1.25GB RAM, 1000 millicore CPU
+	pro2: { memory: 2.5, cpu: 1.5 }, // 2.5GB RAM, 1500 millicore CPU
+	enterprise: { memory: 8.0, cpu: 2.0 }, // 8GB RAM, 2.0 CPU core
+} as const;
+
+export type PerformancePlanName = keyof typeof BASE_PERFORMANCE_PLANS;
+
+export function isValidPerformancePlan(name: string): name is PerformancePlanName {
+	return name in BASE_PERFORMANCE_PLANS;
+}

packages/testing/containers/pull-test-images.ts

@@ -0,0 +1,113 @@
+#!/usr/bin/env tsx
+/**
+ * Pre-pull test container images in parallel.
+ *
+ * Usage:
+ *   npx tsx pull-test-images.ts              # Pull all images
+ *   npx tsx pull-test-images.ts postgres redis mailpit  # Pull specific images
+ */
+
+import { exec } from 'child_process';
+import { promisify } from 'util';
+
+import { TEST_CONTAINER_IMAGES } from './test-containers';
+
+const execAsync = promisify(exec);
+
+type ImageKey = keyof typeof TEST_CONTAINER_IMAGES;
+
+interface PullResult {
+	image: string;
+	duration: string;
+	success: boolean;
+	cached: number;
+	pulled: number;
+}
+
+async function pullImage(image: string): Promise<PullResult> {
+	const imageStart = Date.now();
+	try {
+		const { stdout, stderr } = await execAsync(`docker pull ${image}`);
+		const output = stdout + stderr;
+
+		// Count cached vs pulled layers
+		const cached = (output.match(/Already exists/g) ?? []).length;
+		const pulled = (output.match(/Pull complete/g) ?? []).length;
+
+		const duration = ((Date.now() - imageStart) / 1000).toFixed(1);
+		return { image, duration, success: true, cached, pulled };
+	} catch (error) {
+		const duration = ((Date.now() - imageStart) / 1000).toFixed(1);
+		const message = error instanceof Error ? error.message : String(error);
+		console.error(`   ⚠️  Pull failed for ${image}: ${message}`);
+		return { image, duration, success: false, cached: 0, pulled: 0 };
+	}
+}
+
+function isValidImageKey(key: string): key is ImageKey {
+	return key in TEST_CONTAINER_IMAGES;
+}
+
+function getRequestedImages(args: string[]): string[] {
+	if (args.length === 0) {
+		return Object.values(TEST_CONTAINER_IMAGES);
+	}
+
+	const invalid = args.filter((arg) => !isValidImageKey(arg));
+	if (invalid.length > 0) {
+		console.error(`❌ Unknown image(s): ${invalid.join(', ')}`);
+		console.error(`   Valid: ${Object.keys(TEST_CONTAINER_IMAGES).join(', ')}`);
+		process.exit(1);
+	}
+
+	return args.filter(isValidImageKey).map((key) => TEST_CONTAINER_IMAGES[key]);
+}
+
+async function main() {
+	const args = process.argv.slice(2);
+	const images = getRequestedImages(args);
+
+	const mode = args.length > 0 ? `${args.length} specified` : 'all';
+	console.log(`🐳 Pre-pulling test container images (${mode})...`);
+	const startTime = Date.now();
+
+	// Filter out local images and start all pulls in parallel
+	const imagesToPull = images.filter((image) => {
+		if (image.endsWith(':local')) {
+			console.log(`⏭️  Skipping ${image} (local build)`);
+			return false;
+		}
+		return true;
+	});
+
+	for (const image of imagesToPull) {
+		console.log(`🔄 Starting pull: ${image}`);
+	}
+
+	const pullPromises = imagesToPull.map(pullImage);
+
+	const results = await Promise.all(pullPromises);
+
+	// Check for failures
+	const failures = results.filter((r) => !r.success);
+	if (failures.length > 0 && process.env.STRICT_IMAGE_PULL === 'true') {
+		console.error(`❌ Failed to pull ${failures.length} image(s)`);
+		process.exit(1);
+	}
+
+	const totalTime = ((Date.now() - startTime) / 1000).toFixed(1);
+	const totalCached = results.reduce((sum, r) => sum + r.cached, 0);
+	const totalPulled = results.reduce((sum, r) => sum + r.pulled, 0);
+
+	console.log('\n' + '='.repeat(60));
+	console.log('📊 Pull Summary:');
+	results.forEach(({ image, duration, success, cached, pulled }) => {
+		const layers = cached + pulled > 0 ? ` (${cached} cached, ${pulled} pulled)` : '';
+		console.log(`  ${success ? '✅' : '❌'} ${image}: ${duration}s${layers}`);
+	});
+	console.log('='.repeat(60));
+	console.log(`📦 Layers: ${totalCached} cached, ${totalPulled} pulled`);
+	console.log(`✅ Total time: ${totalTime}s (parallel)`);
+}
+
+void main();

packages/testing/containers/service-stack.ts

@@ -0,0 +1,31 @@
+import type { ServiceName } from './services/types';
+import { createN8NStack, type N8NStack } from './stack';
+
+export interface ServiceStackOptions {
+	services: ServiceName[];
+	projectName?: string;
+}
+
+/**
+ * Creates a stack with only services (no n8n containers).
+ * Useful for integration tests that need databases/services but not full n8n.
+ *
+ * @example
+ * const stack = await createServiceStack({ services: ['postgres'] });
+ * const pgContainer = stack.serviceResults.postgres?.container;
+ * const host = pgContainer.getHost();
+ * const port = pgContainer.getMappedPort(5432);
+ * await stack.stop();
+ */
+export async function createServiceStack(options: ServiceStackOptions): Promise<N8NStack> {
+	const { services, projectName } = options;
+
+	return await createN8NStack({
+		mains: 0,
+		workers: 0,
+		postgres: services.includes('postgres'),
+		services,
+		projectName,
+		external: true,
+	});
+}

packages/testing/containers/services/cloudflared.ts

@@ -0,0 +1,95 @@
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { createSilentLogConsumer } from '../helpers/utils';
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import { EXTERNAL_HOST, type Service, type ServiceResult, type StartContext } from './types';
+
+export interface CloudflaredMeta {
+	publicUrl: string;
+	proxyHops: number;
+}
+
+export type CloudflaredResult = ServiceResult<CloudflaredMeta>;
+
+const METRICS_PORT = 2000;
+
+function getTunnelTarget(ctx: StartContext): string {
+	if (ctx.external) {
+		return `${EXTERNAL_HOST}:5678`;
+	}
+	if (ctx.needsLoadBalancer) {
+		return `${ctx.projectName}-caddy-lb:80`;
+	}
+	return `${ctx.projectName}-n8n:5678`;
+}
+
+export const cloudflared: Service<CloudflaredResult> = {
+	description: 'Cloudflare Tunnel',
+	dependsOn: ['loadBalancer'],
+
+	shouldStart: (ctx) => ctx.config.services?.includes('cloudflared') ?? false,
+
+	getOptions(ctx) {
+		const proxyHops = ctx.needsLoadBalancer ? 2 : 1;
+		return { tunnelTarget: getTunnelTarget(ctx), proxyHops };
+	},
+
+	env(result) {
+		return {
+			WEBHOOK_URL: result.meta.publicUrl,
+			N8N_PROXY_HOPS: String(result.meta.proxyHops),
+		};
+	},
+
+	async start(
+		network,
+		projectName,
+		config?: unknown,
+		ctx?: StartContext,
+	): Promise<CloudflaredResult> {
+		const { tunnelTarget, proxyHops } = config as { tunnelTarget: string; proxyHops: number };
+		const { consumer, throwWithLogs } = createSilentLogConsumer();
+
+		try {
+			let builder = new GenericContainer(TEST_CONTAINER_IMAGES.cloudflared)
+				.withNetwork(network)
+				.withNetworkAliases('cloudflared')
+				.withName(`${projectName}-cloudflared`)
+				.withExposedPorts(METRICS_PORT)
+				.withCommand([
+					'tunnel',
+					'--url',
+					`http://${tunnelTarget}`,
+					'--metrics',
+					`0.0.0.0:${METRICS_PORT}`,
+					'--no-autoupdate',
+				])
+				.withWaitStrategy(Wait.forHttp('/quicktunnel', METRICS_PORT).forStatusCode(200))
+				.withLabels({
+					'com.docker.compose.project': projectName,
+					'com.docker.compose.service': 'cloudflared',
+				})
+				.withReuse()
+				.withLogConsumer(consumer);
+
+			// On Linux, host.docker.internal is not available without explicit mapping
+			if (ctx?.external) {
+				builder = builder.withExtraHosts([{ host: EXTERNAL_HOST, ipAddress: 'host-gateway' }]);
+			}
+
+			const container = await builder.start();
+
+			const hostPort = container.getMappedPort(METRICS_PORT);
+			const response = await fetch(`http://${container.getHost()}:${hostPort}/quicktunnel`);
+			const data = (await response.json()) as { hostname: string };
+			const publicUrl = `https://${data.hostname}`;
+
+			return {
+				container,
+				meta: { publicUrl, proxyHops },
+			};
+		} catch (error) {
+			return throwWithLogs(error);
+		}
+	},
+};

packages/testing/containers/services/gitea.ts

@@ -0,0 +1,258 @@
+import type { StartedNetwork, StartedTestContainer } from 'testcontainers';
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { createSilentLogConsumer } from '../helpers/utils';
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { HelperContext, Service, ServiceResult } from './types';
+
+const HOSTNAME = 'gitea';
+const HTTP_PORT = 3000;
+const SSH_PORT = 22;
+const DEFAULT_ADMIN = 'giteaadmin';
+const DEFAULT_PASSWORD = 'giteapassword';
+const DEFAULT_EMAIL = 'admin@example.com';
+const DEFAULT_REPO = 'n8n-test-repo';
+const DEFAULT_BRANCHES = ['development', 'staging', 'production'];
+
+export interface GiteaMeta {
+	apiUrl: string;
+	adminUsername: string;
+	adminPassword: string;
+	defaultRepo: string;
+}
+
+export type GiteaResult = ServiceResult<GiteaMeta>;
+
+export const gitea: Service<GiteaResult> = {
+	description: 'Git server (Gitea)',
+
+	async start(network: StartedNetwork, projectName: string): Promise<GiteaResult> {
+		const { consumer, throwWithLogs } = createSilentLogConsumer();
+
+		try {
+			const container = await new GenericContainer(TEST_CONTAINER_IMAGES.gitea)
+				.withNetwork(network)
+				.withNetworkAliases(HOSTNAME)
+				.withExposedPorts(HTTP_PORT, SSH_PORT)
+				.withEnvironment({
+					GITEA__database__DB_TYPE: 'sqlite3',
+					GITEA__server__DOMAIN: HOSTNAME,
+					GITEA__server__ROOT_URL: `http://${HOSTNAME}:${HTTP_PORT}/`,
+					GITEA__server__SSH_DOMAIN: HOSTNAME,
+					GITEA__security__INSTALL_LOCK: 'true',
+					GITEA__security__SECRET_KEY: 'gitea-test-secret-key',
+					GITEA__service__DISABLE_REGISTRATION: 'true',
+				})
+				.withWaitStrategy(Wait.forListeningPorts())
+				.withLabels({
+					'com.docker.compose.project': projectName,
+					'com.docker.compose.service': HOSTNAME,
+				})
+				.withName(`${projectName}-${HOSTNAME}`)
+				.withReuse()
+				.withLogConsumer(consumer)
+				.start();
+
+			// Setup admin user and default repo
+			await addUser(container, DEFAULT_ADMIN, DEFAULT_PASSWORD, DEFAULT_EMAIL, true);
+			await addRepo(container, DEFAULT_REPO, DEFAULT_ADMIN, DEFAULT_PASSWORD);
+
+			// Create default branches
+			for (const branch of DEFAULT_BRANCHES) {
+				await addBranch(container, DEFAULT_REPO, branch, DEFAULT_ADMIN, DEFAULT_PASSWORD);
+			}
+
+			return {
+				container,
+				meta: {
+					apiUrl: `http://${container.getHost()}:${container.getMappedPort(HTTP_PORT)}`,
+					adminUsername: DEFAULT_ADMIN,
+					adminPassword: DEFAULT_PASSWORD,
+					defaultRepo: DEFAULT_REPO,
+				},
+			};
+		} catch (error) {
+			return throwWithLogs(error);
+		}
+	},
+
+	env(result: GiteaResult, external?: boolean): Record<string, string> {
+		return {
+			N8N_SOURCECONTROL_HOST: external ? result.meta.apiUrl : `http://${HOSTNAME}:${HTTP_PORT}`,
+		};
+	},
+};
+
+async function addUser(
+	container: StartedTestContainer,
+	username: string,
+	password: string,
+	email: string,
+	admin = false,
+): Promise<void> {
+	const adminFlag = admin ? '--admin' : '';
+	await container.exec([
+		'bash',
+		'-c',
+		`cd /data/gitea && su git -c "/usr/local/bin/gitea admin user create --config /data/gitea/conf/app.ini --username ${username} --password ${password} --email ${email} ${adminFlag} --must-change-password=false"`,
+	]);
+}
+
+async function addRepo(
+	container: StartedTestContainer,
+	repoName: string,
+	username: string,
+	password: string,
+): Promise<void> {
+	await container.exec([
+		'curl',
+		'-X',
+		'POST',
+		`http://localhost:${HTTP_PORT}/api/v1/user/repos`,
+		'-H',
+		'Content-Type: application/json',
+		'-u',
+		`${username}:${password}`,
+		'-d',
+		`{"name":"${repoName}","private":false,"auto_init":true}`,
+	]);
+}
+
+async function addBranch(
+	container: StartedTestContainer,
+	repoName: string,
+	branchName: string,
+	username: string,
+	password: string,
+	fromBranch = 'main',
+): Promise<void> {
+	await container.exec([
+		'curl',
+		'-X',
+		'POST',
+		`http://localhost:${HTTP_PORT}/api/v1/repos/${username}/${repoName}/branches`,
+		'-H',
+		'Content-Type: application/json',
+		'-u',
+		`${username}:${password}`,
+		'-d',
+		`{"new_branch_name":"${branchName}","old_branch_name":"${fromBranch}"}`,
+	]);
+}
+
+export class GiteaHelper {
+	private readonly container: StartedTestContainer;
+	private readonly meta: GiteaMeta;
+
+	constructor(container: StartedTestContainer, meta: GiteaMeta) {
+		this.container = container;
+		this.meta = meta;
+	}
+
+	get apiUrl(): string {
+		return this.meta.apiUrl;
+	}
+
+	get adminUsername(): string {
+		return this.meta.adminUsername;
+	}
+
+	get adminPassword(): string {
+		return this.meta.adminPassword;
+	}
+
+	get defaultRepo(): string {
+		return this.meta.defaultRepo;
+	}
+
+	async createUser(
+		username: string,
+		password: string,
+		email: string,
+		admin = false,
+	): Promise<void> {
+		await addUser(this.container, username, password, email, admin);
+	}
+
+	async createRepo(repoName: string, username?: string, password?: string): Promise<void> {
+		await addRepo(
+			this.container,
+			repoName,
+			username ?? this.meta.adminUsername,
+			password ?? this.meta.adminPassword,
+		);
+	}
+
+	async createBranch(
+		repoName: string,
+		branchName: string,
+		username?: string,
+		password?: string,
+		fromBranch = 'main',
+	): Promise<void> {
+		await addBranch(
+			this.container,
+			repoName,
+			branchName,
+			username ?? this.meta.adminUsername,
+			password ?? this.meta.adminPassword,
+			fromBranch,
+		);
+	}
+
+	async addSSHKey(
+		keyTitle: string,
+		publicKey: string,
+		username?: string,
+		password?: string,
+	): Promise<void> {
+		await this.container.exec([
+			'curl',
+			'-X',
+			'POST',
+			`http://localhost:${HTTP_PORT}/api/v1/user/keys`,
+			'-H',
+			'Content-Type: application/json',
+			'-u',
+			`${username ?? this.meta.adminUsername}:${password ?? this.meta.adminPassword}`,
+			'-d',
+			`{"title":"${keyTitle}","key":"${publicKey}","read_only":false}`,
+		]);
+	}
+
+	async commitExists(
+		repoName: string,
+		commitHash: string,
+		username?: string,
+		password?: string,
+	): Promise<boolean> {
+		const result = await this.container.exec([
+			'curl',
+			'-s',
+			'-o',
+			'/dev/null',
+			'-w',
+			'%{http_code}',
+			`http://localhost:${HTTP_PORT}/api/v1/repos/${username ?? this.meta.adminUsername}/${repoName}/git/commits/${commitHash}`,
+			'-u',
+			`${username ?? this.meta.adminUsername}:${password ?? this.meta.adminPassword}`,
+		]);
+		// curl writes HTTP status code to stdout, 200 means commit exists
+		const statusCode = result.output.trim();
+		return statusCode === '200';
+	}
+}
+
+export function createGiteaHelper(ctx: HelperContext): GiteaHelper {
+	const result = ctx.serviceResults.gitea as GiteaResult | undefined;
+	if (!result) {
+		throw new Error('Gitea service not found in context');
+	}
+	return new GiteaHelper(result.container, result.meta);
+}
+
+declare module './types' {
+	interface ServiceHelpers {
+		gitea: GiteaHelper;
+	}
+}

packages/testing/containers/services/kafka.ts

@@ -0,0 +1,187 @@
+import { KafkaContainer, type StartedKafkaContainer } from '@testcontainers/kafka';
+import { Kafka, type Producer, type EachMessagePayload } from 'kafkajs';
+import type { StartedNetwork } from 'testcontainers';
+
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { HelperContext, Service, ServiceResult } from './types';
+
+const HOSTNAME = 'kafka';
+
+export interface KafkaMeta {
+	internalBroker: string;
+	externalBroker: string;
+}
+
+export type KafkaResult = ServiceResult<KafkaMeta> & {
+	container: StartedKafkaContainer;
+};
+
+export const kafka: Service<KafkaResult> = {
+	description: 'Apache Kafka broker for message queue testing',
+
+	async start(network: StartedNetwork, projectName: string): Promise<KafkaResult> {
+		const container = await new KafkaContainer(TEST_CONTAINER_IMAGES.kafka)
+			.withNetwork(network)
+			.withNetworkAliases(HOSTNAME)
+			.withLabels({
+				'com.docker.compose.project': projectName,
+				'com.docker.compose.service': HOSTNAME,
+			})
+			.withName(`${projectName}-${HOSTNAME}`)
+			.withKraft()
+			.withReuse()
+			.start();
+
+		return {
+			container,
+			meta: {
+				internalBroker: `${HOSTNAME}:9092`,
+				externalBroker: `${container.getHost()}:${container.getMappedPort(9093)}`,
+			},
+		};
+	},
+
+	env(result: KafkaResult, external?: boolean): Record<string, string> {
+		if (!external) return {};
+		return {
+			KAFKA_BROKER: result.meta.externalBroker,
+		};
+	},
+};
+
+export class KafkaHelper {
+	private readonly kafka: Kafka;
+
+	private producer: Producer | null = null;
+
+	constructor(broker: string) {
+		this.kafka = new Kafka({
+			clientId: 'n8n-test-helper',
+			brokers: [broker],
+		});
+	}
+
+	async createTopic(topic: string, numPartitions = 1): Promise<void> {
+		const admin = this.kafka.admin();
+		try {
+			await admin.connect();
+			await admin.createTopics({
+				topics: [{ topic, numPartitions }],
+			});
+		} finally {
+			await admin.disconnect();
+		}
+	}
+
+	async waitForConsumerGroup(
+		groupId: string,
+		options: { timeoutMs?: number; pollIntervalMs?: number } = {},
+	): Promise<void> {
+		const { timeoutMs = 10000, pollIntervalMs = 500 } = options;
+		const admin = this.kafka.admin();
+		const deadline = Date.now() + timeoutMs;
+
+		try {
+			await admin.connect();
+
+			while (Date.now() < deadline) {
+				const groups = await admin.describeGroups([groupId]);
+				const group = groups.groups[0];
+
+				if (group && group.state === 'Stable' && group.members.length > 0) {
+					return;
+				}
+
+				await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
+			}
+
+			throw new Error(`Consumer group '${groupId}' did not become active within ${timeoutMs}ms`);
+		} finally {
+			await admin.disconnect();
+		}
+	}
+
+	async publish(topic: string, message: string | object, key?: string): Promise<void> {
+		if (!this.producer) {
+			this.producer = this.kafka.producer();
+			await this.producer.connect();
+		}
+
+		const value = typeof message === 'string' ? message : JSON.stringify(message);
+
+		await this.producer.send({
+			topic,
+			messages: [{ key, value }],
+		});
+	}
+
+	async consume(
+		topic: string,
+		options: {
+			groupId?: string;
+			maxMessages?: number;
+			timeoutMs?: number;
+			fromBeginning?: boolean;
+		} = {},
+	): Promise<Array<{ key: string | null; value: string; partition: number; offset: string }>> {
+		const {
+			groupId = `test-consumer-${Date.now()}`,
+			maxMessages = 10,
+			timeoutMs = 5000,
+			fromBeginning = true,
+		} = options;
+
+		const consumer = this.kafka.consumer({ groupId });
+		const messages: Array<{
+			key: string | null;
+			value: string;
+			partition: number;
+			offset: string;
+		}> = [];
+
+		try {
+			await consumer.connect();
+			await consumer.subscribe({ topic, fromBeginning });
+
+			await new Promise<void>((resolve) => {
+				const timeout = setTimeout(() => resolve(), timeoutMs);
+
+				void consumer.run({
+					// kafkajs requires async handler signature, but we don't need to await anything
+					// eslint-disable-next-line @typescript-eslint/require-await
+					eachMessage: async ({ message, partition }: EachMessagePayload) => {
+						messages.push({
+							key: message.key?.toString() ?? null,
+							value: message.value?.toString() ?? '',
+							partition,
+							offset: message.offset,
+						});
+
+						if (messages.length >= maxMessages) {
+							clearTimeout(timeout);
+							resolve();
+						}
+					},
+				});
+			});
+		} finally {
+			await consumer.disconnect();
+		}
+
+		return messages;
+	}
+}
+
+export function createKafkaHelper(ctx: HelperContext): KafkaHelper {
+	const result = ctx.serviceResults.kafka as KafkaResult | undefined;
+	if (!result) {
+		throw new Error('Kafka service not found in context');
+	}
+	return new KafkaHelper(result.meta.externalBroker);
+}
+
+declare module './types' {
+	interface ServiceHelpers {
+		kafka: KafkaHelper;
+	}
+}

packages/testing/containers/services/kent.ts

@@ -0,0 +1,176 @@
+/**
+ * Kent - Sentry's mock server for testing SDK integrations
+ * @see https://github.com/getsentry/kent
+ */
+import { resolve } from 'node:path';
+import { GenericContainer, Wait } from 'testcontainers';
+import type { StartedNetwork } from 'testcontainers';
+
+import type { HelperContext, Service, ServiceResult, ServiceMeta } from './types';
+
+const HOSTNAME = 'kent';
+const PORT = 8000;
+const DOCKERFILE_PATH = resolve(__dirname, '../dockerfiles/kent');
+
+export interface KentMeta extends ServiceMeta {
+	host: string;
+	port: number;
+	apiUrl: string;
+	sentryDsn: string;
+	frontendDsn: string;
+}
+
+export type KentResult = ServiceResult<KentMeta>;
+
+export const kent: Service<KentResult> = {
+	description: 'Sentry mock server for testing',
+
+	async start(network: StartedNetwork, projectName: string): Promise<KentResult> {
+		const container = await GenericContainer.fromDockerfile(DOCKERFILE_PATH)
+			.build('n8n-kent:local', { deleteOnExit: false })
+			.then(
+				async (image) =>
+					await image
+						.withNetwork(network)
+						.withNetworkAliases(HOSTNAME)
+						.withExposedPorts(PORT)
+						.withWaitStrategy(Wait.forListeningPorts())
+						.withLabels({
+							'com.docker.compose.project': projectName,
+							'com.docker.compose.service': HOSTNAME,
+						})
+						.withName(`${projectName}-${HOSTNAME}`)
+						.withReuse()
+						.start(),
+			);
+
+		const mappedPort = container.getMappedPort(PORT);
+		const host = container.getHost();
+
+		return {
+			container,
+			meta: {
+				host: HOSTNAME,
+				port: PORT,
+				apiUrl: `http://${host}:${mappedPort}`,
+				sentryDsn: `http://testkey@${HOSTNAME}:${PORT}/1`,
+				frontendDsn: `http://testkey@${host}:${mappedPort}/1`,
+			},
+		};
+	},
+
+	env(result: KentResult): Record<string, string> {
+		return {
+			N8N_SENTRY_DSN: result.meta.sentryDsn,
+			N8N_FRONTEND_SENTRY_DSN: result.meta.frontendDsn,
+			N8N_SENTRY_TRACES_SAMPLE_RATE: '1.0',
+			ENVIRONMENT: 'test',
+			DEPLOYMENT_NAME: 'e2e-test-deployment',
+		};
+	},
+};
+
+// ==================== Types ====================
+
+export type EventSource = 'backend' | 'frontend' | 'task_runner' | 'unknown';
+export type EventType = 'error' | 'transaction' | 'session' | 'unknown';
+
+export interface KentEventFilter {
+	source?: EventSource;
+	type?: EventType;
+	messageContains?: string;
+}
+
+export interface KentEvent {
+	event_id: string;
+	project_id: number;
+	payload: {
+		body: {
+			sdk?: { name: string; version: string };
+			platform?: string;
+			type?: string;
+			transaction?: string;
+			tags?: Record<string, string>;
+			user?: { id?: string; email?: string; username?: string; ip_address?: string };
+			exception?: { values: Array<{ type: string; value: string }> };
+			spans?: unknown[];
+			[key: string]: unknown;
+		};
+	};
+}
+
+// ==================== Helper ====================
+
+export class KentHelper {
+	constructor(private readonly apiUrl: string) {}
+
+	async clear(): Promise<void> {
+		const res = await fetch(`${this.apiUrl}/api/flush/`, { method: 'POST' });
+		if (!res.ok) throw new Error(`Kent API error: ${res.status}`);
+	}
+
+	async getEvents(filter?: KentEventFilter): Promise<KentEvent[]> {
+		const res = await fetch(`${this.apiUrl}/api/eventlist/`);
+		if (!res.ok) throw new Error(`Kent API error: ${res.status}`);
+		const { events } = (await res.json()) as { events: Array<{ event_id: string }> };
+		const allEvents = await Promise.all(events.map(async (e) => await this.getEvent(e.event_id)));
+
+		if (!filter) return allEvents;
+
+		return allEvents.filter((event) => {
+			if (filter.source && this.getSource(event) !== filter.source) return false;
+			if (filter.type && this.getType(event) !== filter.type) return false;
+			if (filter.messageContains && !this.getErrorMessage(event).includes(filter.messageContains))
+				return false;
+			return true;
+		});
+	}
+
+	getSource(event: KentEvent): EventSource {
+		const sdk = event.payload.body.sdk?.name ?? '';
+		if (sdk.includes('vue') || sdk.includes('browser')) return 'frontend';
+		if (sdk === 'sentry.javascript.node' || event.payload.body.platform === 'node') {
+			return event.payload.body.tags?.server_type === 'task_runner' ? 'task_runner' : 'backend';
+		}
+		if ('sid' in event.payload.body) return 'frontend';
+		return 'unknown';
+	}
+
+	getType(event: KentEvent): EventType {
+		const body = event.payload.body;
+		if ('sid' in body) return 'session';
+		if (body.exception) return 'error';
+		if (body.type === 'transaction' || Array.isArray(body.spans)) return 'transaction';
+		return 'unknown';
+	}
+
+	getErrorMessage(event: KentEvent): string {
+		return event.payload.body.exception?.values?.[0]?.value ?? '';
+	}
+
+	getTags(event: KentEvent): Record<string, string> | undefined {
+		return event.payload.body.tags;
+	}
+
+	getUser(event: KentEvent): KentEvent['payload']['body']['user'] {
+		return event.payload.body.user;
+	}
+
+	private async getEvent(eventId: string): Promise<KentEvent> {
+		const res = await fetch(`${this.apiUrl}/api/event/${eventId}`);
+		if (!res.ok) throw new Error(`Kent API error: ${res.status}`);
+		return (await res.json()) as KentEvent;
+	}
+}
+
+export function createKentHelper(ctx: HelperContext): KentHelper {
+	const result = ctx.serviceResults.kent as KentResult | undefined;
+	if (!result) throw new Error('Kent service not found. Add "kent" to your services array.');
+	return new KentHelper(result.meta.apiUrl);
+}
+
+declare module './types' {
+	interface ServiceHelpers {
+		kent: KentHelper;
+	}
+}

packages/testing/containers/services/keycloak.ts

@@ -0,0 +1,540 @@
+import getPort from 'get-port';
+import { setTimeout as wait } from 'node:timers/promises';
+import type { StartedNetwork, StartedTestContainer } from 'testcontainers';
+import { GenericContainer, Wait } from 'testcontainers';
+import { Agent, request as undiciRequest } from 'undici';
+
+import { createSilentLogConsumer } from '../helpers/utils';
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { FileToMount, HelperContext, Service, ServiceResult } from './types';
+
+const HOSTNAME = 'keycloak';
+const HTTPS_PORT = 8443;
+
+const KEYCLOAK_TEST_REALM = 'test';
+const KEYCLOAK_TEST_CLIENT_ID = 'n8n-e2e';
+const KEYCLOAK_TEST_CLIENT_SECRET = 'n8n-test-secret';
+const KEYCLOAK_TEST_USER_EMAIL = 'test@n8n.io';
+const KEYCLOAK_TEST_USER_PASSWORD = 'testpassword';
+const KEYCLOAK_TEST_USER_FIRSTNAME = 'Test';
+const KEYCLOAK_TEST_USER_LASTNAME = 'User';
+
+const KEYCLOAK_ADMIN_USER = 'admin';
+const KEYCLOAK_ADMIN_PASSWORD = 'admin';
+const KEYCLOAK_CERT_PATH = '/tmp/keycloak-ca.pem';
+const N8N_KEYCLOAK_CERT_PATH = '/tmp/keycloak-ca.pem';
+
+export interface KeycloakConfig {
+	n8nCallbackUrl: string;
+}
+
+export interface KeycloakMeta {
+	discoveryUrl: string;
+	internalDiscoveryUrl: string;
+	certPem: string;
+	hostPort: number;
+	clientId: string;
+	clientSecret: string;
+	testUser: {
+		email: string;
+		password: string;
+		firstName: string;
+		lastName: string;
+	};
+	n8nFilesToMount: FileToMount[];
+}
+
+export type KeycloakResult = ServiceResult<KeycloakMeta>;
+
+function generateRealmJson(callbackUrl: string): string {
+	// Derive the n8n base URL from the OIDC callback URL
+	const n8nBaseUrl = callbackUrl.split('/rest/')[0];
+	return JSON.stringify({
+		realm: KEYCLOAK_TEST_REALM,
+		enabled: true,
+		sslRequired: 'none',
+		registrationAllowed: false,
+		loginWithEmailAllowed: true,
+		duplicateEmailsAllowed: false,
+		resetPasswordAllowed: false,
+		editUsernameAllowed: false,
+		bruteForceProtected: false,
+		clients: [
+			{
+				clientId: KEYCLOAK_TEST_CLIENT_ID,
+				enabled: true,
+				clientAuthenticatorType: 'client-secret',
+				secret: KEYCLOAK_TEST_CLIENT_SECRET,
+				redirectUris: [
+					callbackUrl,
+					`${callbackUrl}/*`,
+					// Allow the n8n OAuth2 credential callback for dynamic credential authorization flow
+					`${n8nBaseUrl}/rest/oauth2-credential/callback`,
+				],
+				webOrigins: ['*'],
+				standardFlowEnabled: true,
+				directAccessGrantsEnabled: true,
+				publicClient: false,
+				protocol: 'openid-connect',
+			},
+		],
+		users: [
+			{
+				username: 'testuser',
+				enabled: true,
+				email: KEYCLOAK_TEST_USER_EMAIL,
+				emailVerified: true,
+				firstName: KEYCLOAK_TEST_USER_FIRSTNAME,
+				lastName: KEYCLOAK_TEST_USER_LASTNAME,
+				credentials: [
+					{
+						type: 'password',
+						value: KEYCLOAK_TEST_USER_PASSWORD,
+						temporary: false,
+					},
+				],
+			},
+		],
+	});
+}
+
+/**
+ * Generates a shell script that creates a keystore with self-signed cert using Java keytool,
+ * exports the certificate to PEM format, and starts Keycloak with HTTPS.
+ */
+function generateStartupScript(): string {
+	return `#!/bin/bash
+set -e
+
+# Generate self-signed certificate using Java keytool (available in Keycloak image)
+keytool -genkeypair \\
+  -storepass password \\
+  -storetype PKCS12 \\
+  -keyalg RSA \\
+  -keysize 2048 \\
+  -dname "CN=localhost" \\
+  -alias server \\
+  -ext "SAN=DNS:localhost,DNS:keycloak,IP:127.0.0.1" \\
+  -keystore /opt/keycloak/conf/server.keystore
+
+# Export the certificate to PEM format for Node.js NODE_EXTRA_CA_CERTS
+keytool -exportcert \\
+  -alias server \\
+  -keystore /opt/keycloak/conf/server.keystore \\
+  -rfc \\
+  -file ${KEYCLOAK_CERT_PATH} \\
+  -storepass password
+
+exec /opt/keycloak/bin/kc.sh start-dev \\
+  --import-realm \\
+  --https-key-store-file=/opt/keycloak/conf/server.keystore \\
+  --https-key-store-password=password \\
+  --hostname=https://localhost:\${KEYCLOAK_HOST_PORT} \\
+  --hostname-backchannel-dynamic=true
+`;
+}
+
+async function extractCertificate(
+	container: StartedTestContainer,
+	timeoutMs: number = 30000,
+): Promise<string> {
+	const startTime = Date.now();
+	const retryIntervalMs = 500;
+
+	while (Date.now() - startTime < timeoutMs) {
+		try {
+			const certResult = await container.exec(['cat', KEYCLOAK_CERT_PATH]);
+			if (certResult.exitCode === 0 && certResult.output.includes('BEGIN CERTIFICATE')) {
+				return certResult.output;
+			}
+		} catch {
+			// Retry on error
+		}
+		await wait(retryIntervalMs);
+	}
+
+	throw new Error(
+		`Failed to extract Keycloak certificate from ${KEYCLOAK_CERT_PATH} within ${timeoutMs}ms`,
+	);
+}
+
+async function waitForKeycloakReady(
+	port: number,
+	certPem: string,
+	timeoutMs: number = 60000,
+): Promise<void> {
+	const startTime = Date.now();
+	const url = `https://localhost:${port}/realms/${KEYCLOAK_TEST_REALM}/.well-known/openid-configuration`;
+	const retryIntervalMs = 2000;
+
+	const agent = new Agent({
+		connect: { ca: certPem },
+	});
+
+	try {
+		while (Date.now() - startTime < timeoutMs) {
+			try {
+				const response = await fetch(url, {
+					// @ts-expect-error - dispatcher is an undici-specific option
+					dispatcher: agent,
+				});
+				if (response.ok) {
+					return;
+				}
+			} catch {
+				// Retry on connection errors
+			}
+
+			await wait(retryIntervalMs);
+		}
+
+		throw new Error(
+			`Keycloak discovery endpoint at ${url} did not become ready within ${timeoutMs / 1000} seconds`,
+		);
+	} finally {
+		await agent.close();
+	}
+}
+
+export const keycloak: Service<KeycloakResult> = {
+	description: 'Keycloak OIDC provider',
+
+	getOptions(ctx) {
+		const port = ctx.allocatedPorts.loadBalancer ?? ctx.allocatedPorts.main;
+		return { n8nCallbackUrl: `http://localhost:${port}/rest/sso/oidc/callback` } as KeycloakConfig;
+	},
+
+	async verifyFromN8n(result, n8nContainers) {
+		const { setTimeout: wait } = await import('node:timers/promises');
+		const timeoutMs = 30000;
+		const retryIntervalMs = 1000;
+
+		for (const container of n8nContainers) {
+			const startTime = Date.now();
+			let verified = false;
+
+			while (Date.now() - startTime < timeoutMs) {
+				try {
+					const execResult = await container.exec([
+						'wget',
+						'--no-check-certificate',
+						'-q',
+						'-O',
+						'-',
+						result.meta.internalDiscoveryUrl,
+					]);
+					if (execResult.exitCode === 0) {
+						verified = true;
+						break;
+					}
+				} catch {
+					// Retry
+				}
+				await wait(retryIntervalMs);
+			}
+
+			if (!verified) {
+				throw new Error(
+					`Keycloak verification failed: ${container.getName()} could not reach ${result.meta.internalDiscoveryUrl} within ${timeoutMs}ms`,
+				);
+			}
+		}
+	},
+
+	async start(
+		network: StartedNetwork,
+		projectName: string,
+		config?: unknown,
+	): Promise<KeycloakResult> {
+		const { n8nCallbackUrl } = config as KeycloakConfig;
+		const { consumer, throwWithLogs } = createSilentLogConsumer();
+
+		// Allocate a fixed host port for Keycloak
+		const allocatedHostPort = await getPort();
+
+		const realmJson = generateRealmJson(n8nCallbackUrl);
+		const startupScript = generateStartupScript();
+
+		try {
+			const container = await new GenericContainer(TEST_CONTAINER_IMAGES.keycloak)
+				.withNetwork(network)
+				.withNetworkAliases(HOSTNAME)
+				.withExposedPorts({ container: HTTPS_PORT, host: allocatedHostPort })
+				.withEnvironment({
+					KEYCLOAK_ADMIN: KEYCLOAK_ADMIN_USER,
+					KEYCLOAK_ADMIN_PASSWORD,
+					KC_HEALTH_ENABLED: 'true',
+					KC_METRICS_ENABLED: 'false',
+					KEYCLOAK_HOST_PORT: String(allocatedHostPort),
+				})
+				.withCopyContentToContainer([
+					{ content: realmJson, target: '/opt/keycloak/data/import/realm.json' },
+					{ content: startupScript, target: '/startup.sh', mode: 0o755 },
+				])
+				.withEntrypoint(['/bin/bash', '/startup.sh'])
+				.withWaitStrategy(
+					Wait.forLogMessage(/Running the server in development mode/).withStartupTimeout(120000),
+				)
+				.withLabels({
+					'com.docker.compose.project': projectName,
+					'com.docker.compose.service': HOSTNAME,
+				})
+				.withName(`${projectName}-${HOSTNAME}`)
+				.withLogConsumer(consumer)
+				.withReuse()
+				.start();
+
+			const discoveryUrl = `https://localhost:${allocatedHostPort}/realms/${KEYCLOAK_TEST_REALM}/.well-known/openid-configuration`;
+			const internalDiscoveryUrl = `https://${HOSTNAME}:${HTTPS_PORT}/realms/${KEYCLOAK_TEST_REALM}/.well-known/openid-configuration`;
+
+			const certPem = await extractCertificate(container);
+			await waitForKeycloakReady(allocatedHostPort, certPem);
+
+			return {
+				container,
+				meta: {
+					discoveryUrl,
+					internalDiscoveryUrl,
+					certPem,
+					hostPort: allocatedHostPort,
+					clientId: KEYCLOAK_TEST_CLIENT_ID,
+					clientSecret: KEYCLOAK_TEST_CLIENT_SECRET,
+					testUser: {
+						email: KEYCLOAK_TEST_USER_EMAIL,
+						password: KEYCLOAK_TEST_USER_PASSWORD,
+						firstName: KEYCLOAK_TEST_USER_FIRSTNAME,
+						lastName: KEYCLOAK_TEST_USER_LASTNAME,
+					},
+					n8nFilesToMount: [{ content: certPem, target: N8N_KEYCLOAK_CERT_PATH }],
+				},
+			};
+		} catch (error) {
+			return throwWithLogs(error);
+		}
+	},
+
+	env(result: KeycloakResult, external?: boolean): Record<string, string> {
+		if (external) {
+			return {
+				N8N_OIDC_DISCOVERY_URL: result.meta.discoveryUrl,
+				N8N_OIDC_CLIENT_ID: result.meta.clientId,
+				N8N_OIDC_CLIENT_SECRET: result.meta.clientSecret,
+			};
+		}
+		return {
+			NODE_EXTRA_CA_CERTS: N8N_KEYCLOAK_CERT_PATH,
+			NO_PROXY: `localhost,127.0.0.1,${HOSTNAME},host.docker.internal`,
+		};
+	},
+};
+
+export class KeycloakHelper {
+	private readonly meta: KeycloakMeta;
+
+	constructor(_container: StartedTestContainer, meta: KeycloakMeta) {
+		this.meta = meta;
+	}
+
+	get discoveryUrl(): string {
+		return this.meta.discoveryUrl;
+	}
+
+	get internalDiscoveryUrl(): string {
+		return this.meta.internalDiscoveryUrl;
+	}
+
+	get certPem(): string {
+		return this.meta.certPem;
+	}
+
+	get hostPort(): number {
+		return this.meta.hostPort;
+	}
+
+	get realm(): string {
+		return KEYCLOAK_TEST_REALM;
+	}
+
+	get clientId(): string {
+		return this.meta.clientId;
+	}
+
+	get clientSecret(): string {
+		return this.meta.clientSecret;
+	}
+
+	get testUser() {
+		return this.meta.testUser;
+	}
+
+	/**
+	 * Obtain an access token for a user via the Resource Owner Password Credentials (ROPC) grant.
+	 * Keycloak's test realm has directAccessGrantsEnabled=true, so no browser redirect is needed.
+	 */
+	async getAccessToken(email: string, password: string): Promise<string> {
+		const tokenEndpoint = `https://localhost:${this.meta.hostPort}/realms/${KEYCLOAK_TEST_REALM}/protocol/openid-connect/token`;
+		const agent = new Agent({ connect: { ca: this.meta.certPem } });
+
+		const body = new URLSearchParams({
+			grant_type: 'password',
+			client_id: KEYCLOAK_TEST_CLIENT_ID,
+			client_secret: KEYCLOAK_TEST_CLIENT_SECRET,
+			username: email,
+			password,
+			scope: 'openid',
+		});
+
+		try {
+			const response = await fetch(tokenEndpoint, {
+				method: 'POST',
+				headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
+				body: body.toString(),
+				// @ts-expect-error - dispatcher is an undici-specific option
+				dispatcher: agent,
+			});
+
+			if (!response.ok) {
+				const text = await response.text();
+				throw new Error(`Keycloak token request failed (${response.status}): ${text}`);
+			}
+
+			const data = (await response.json()) as { access_token: string };
+			return data.access_token;
+		} finally {
+			await agent.close();
+		}
+	}
+
+	/**
+	 * Programmatically completes the OAuth2 authorization code flow for the test user.
+	 * Uses undici (with the Keycloak CA cert) to:
+	 *   1. GET the Keycloak authorization page
+	 *   2. Extract the login form action URL
+	 *   3. POST test user credentials to Keycloak
+	 *
+	 * Returns the n8n OAuth2 callback URL (with `code` and `state` query params).
+	 * The caller should then GET this URL using the n8n API request context (which holds
+	 * the n8n session cookie) so that n8n exchanges the code for tokens and stores them.
+	 */
+	async completeAuthorizationCodeFlow(authorizationUrl: string): Promise<string> {
+		const agent = new Agent({ connect: { ca: this.meta.certPem } });
+
+		try {
+			// Step 1: GET the Keycloak authorization page (HTML with login form).
+			// Use undiciRequest to access raw set-cookie headers for forwarding.
+			const authPageResult = await undiciRequest(authorizationUrl, {
+				method: 'GET',
+				dispatcher: agent,
+			});
+
+			if (authPageResult.statusCode < 200 || authPageResult.statusCode >= 300) {
+				await authPageResult.body.text();
+				throw new Error(
+					`Failed to load Keycloak authorization page: HTTP ${authPageResult.statusCode}`,
+				);
+			}
+
+			// Extract session cookies from the response to forward with the login POST.
+			// Keycloak sets cookies like AUTH_SESSION_ID, KC_RESTART that are required
+			// for the login form submission to succeed.
+			const rawCookies = authPageResult.headers['set-cookie'];
+			const cookieHeader = (Array.isArray(rawCookies) ? rawCookies : [rawCookies])
+				.filter(Boolean)
+				.map((c) => (c as string).split(';')[0])
+				.join('; ');
+
+			const html = await authPageResult.body.text();
+
+			// Step 2: Extract the Keycloak login form action URL.
+			// Keycloak's login form action always contains 'login-actions/authenticate'.
+			const rawFormAction = html.match(/action="([^"]*login-actions\/authenticate[^"]*)"/)?.[1];
+			if (!rawFormAction) {
+				throw new Error('Could not find Keycloak login form action in authorization page HTML');
+			}
+			const formAction = rawFormAction.replace(/&amp;/g, '&');
+
+			// Step 3: POST credentials with session cookies — Keycloak responds with 302
+			// to the n8n callback URL. undiciRequest does NOT follow redirects by default.
+			const loginBody = new URLSearchParams({
+				username: this.meta.testUser.email,
+				password: this.meta.testUser.password,
+			});
+
+			const loginHeaders: Record<string, string> = {
+				'Content-Type': 'application/x-www-form-urlencoded',
+			};
+			if (cookieHeader) {
+				loginHeaders['Cookie'] = cookieHeader;
+			}
+
+			const { headers, body } = await undiciRequest(formAction, {
+				method: 'POST',
+				headers: loginHeaders,
+				body: loginBody.toString(),
+				dispatcher: agent,
+			});
+
+			// Consume the body to prevent resource leaks
+			await body.text();
+
+			const location = headers.location;
+			const redirectUrl = Array.isArray(location) ? location[0] : location;
+
+			if (!redirectUrl) {
+				throw new Error(
+					'Keycloak did not redirect after login. ' +
+						'Ensure the OAuth2 credential callback URL is registered in Keycloak redirectUris.',
+				);
+			}
+
+			return redirectUrl; // e.g. http://localhost:{n8n_port}/rest/oauth2-credential/callback?code=...&state=...
+		} finally {
+			await agent.close();
+		}
+	}
+
+	async waitForFromContainer(
+		n8nContainer: StartedTestContainer,
+		timeoutMs: number = 30000,
+	): Promise<void> {
+		const startTime = Date.now();
+		const retryIntervalMs = 1000;
+
+		while (Date.now() - startTime < timeoutMs) {
+			try {
+				const result = await n8nContainer.exec([
+					'wget',
+					'--no-check-certificate',
+					'-q',
+					'-O',
+					'-',
+					this.meta.internalDiscoveryUrl,
+				]);
+				if (result.exitCode === 0) {
+					return;
+				}
+			} catch {
+				// Retry on error
+			}
+			await wait(retryIntervalMs);
+		}
+
+		throw new Error(
+			`Keycloak discovery endpoint not reachable from n8n container within ${timeoutMs}ms: ${this.meta.internalDiscoveryUrl}`,
+		);
+	}
+}
+
+export function createKeycloakHelper(ctx: HelperContext): KeycloakHelper {
+	const result = ctx.serviceResults.keycloak as KeycloakResult | undefined;
+	if (!result) {
+		throw new Error('Keycloak service not found in context');
+	}
+	return new KeycloakHelper(result.container, result.meta);
+}
+
+declare module './types' {
+	interface ServiceHelpers {
+		keycloak: KeycloakHelper;
+	}
+}

packages/testing/containers/services/load-balancer.ts

@@ -0,0 +1,105 @@
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { createSilentLogConsumer } from '../helpers/utils';
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { Service, ServiceResult } from './types';
+
+export interface LoadBalancerConfig {
+	mainCount: number;
+	hostPort?: number;
+}
+
+export interface LoadBalancerMeta {
+	hostPort: number;
+	baseUrl: string;
+}
+
+export type LoadBalancerResult = ServiceResult<LoadBalancerMeta>;
+
+function buildCaddyConfig(upstreamServers: string[]): string {
+	const backends = upstreamServers.join(' ');
+	return `
+:80 {
+  # Reverse proxy with load balancing
+  reverse_proxy ${backends} {
+    # Use first available backend for simpler debugging
+    lb_policy first
+
+    # Health check
+    health_uri /healthz
+    health_interval 10s
+
+    # Timeouts
+    transport http {
+      dial_timeout 60s
+      read_timeout 60s
+      write_timeout 60s
+    }
+  }
+
+  # Set max request body size
+  request_body {
+    max_size 50MB
+  }
+}`;
+}
+
+export const loadBalancer: Service<LoadBalancerResult> = {
+	description: 'Caddy load balancer',
+	shouldStart: (ctx) => ctx.needsLoadBalancer,
+
+	getOptions(ctx) {
+		return {
+			mainCount: ctx.mains,
+			hostPort: ctx.allocatedPorts.loadBalancer,
+		} as LoadBalancerConfig;
+	},
+
+	env(result) {
+		return {
+			WEBHOOK_URL: result.meta.baseUrl,
+			N8N_PROXY_HOPS: '1',
+		};
+	},
+
+	async start(network, projectName, config?: unknown): Promise<LoadBalancerResult> {
+		const { mainCount, hostPort } = config as LoadBalancerConfig;
+		const { consumer, throwWithLogs } = createSilentLogConsumer();
+
+		// Generate upstream server addresses
+		const upstreamServers = Array.from(
+			{ length: mainCount },
+			(_, index) => `${projectName}-n8n-main-${index + 1}:5678`,
+		);
+
+		const caddyConfig = buildCaddyConfig(upstreamServers);
+
+		try {
+			const container = await new GenericContainer(TEST_CONTAINER_IMAGES.caddy)
+				.withNetwork(network)
+				.withExposedPorts(hostPort ? { container: 80, host: hostPort } : 80)
+				.withCopyContentToContainer([{ content: caddyConfig, target: '/etc/caddy/Caddyfile' }])
+				.withWaitStrategy(Wait.forListeningPorts())
+				.withLabels({
+					'com.docker.compose.project': projectName,
+					'com.docker.compose.service': 'caddy-lb',
+				})
+				.withName(`${projectName}-caddy-lb`)
+				.withReuse()
+				.withLogConsumer(consumer)
+				.start();
+
+			const actualHostPort = container.getMappedPort(80);
+
+			return {
+				container,
+				meta: {
+					hostPort: actualHostPort,
+					baseUrl: `http://localhost:${actualHostPort}`,
+				},
+			};
+		} catch (error) {
+			return throwWithLogs(error);
+		}
+	},
+};

packages/testing/containers/services/localstack.ts

@@ -0,0 +1,257 @@
+import {
+	CreateSecretCommand,
+	DeleteSecretCommand,
+	GetSecretValueCommand,
+	ListSecretsCommand,
+	SecretsManagerClient as AwsSecretsManagerClient,
+} from '@aws-sdk/client-secrets-manager';
+import type { StartedNetwork } from 'testcontainers';
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { createSilentLogConsumer } from '../helpers/utils';
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { HelperContext, Service, ServiceResult } from './types';
+
+const HOSTNAME = 'localstack';
+const EDGE_PORT = 4566;
+const DEFAULT_REGION = 'us-east-1';
+
+export interface LocalStackMeta {
+	endpoint: string;
+	internalEndpoint: string;
+}
+
+export type LocalStackResult = ServiceResult<LocalStackMeta>;
+
+interface LocalStackHealthResponse {
+	services?: Record<string, string>;
+}
+
+export const localstack: Service<LocalStackResult> = {
+	description: 'AWS service emulator (LocalStack)',
+
+	async start(network: StartedNetwork, projectName: string): Promise<LocalStackResult> {
+		const { consumer, throwWithLogs } = createSilentLogConsumer();
+
+		try {
+			const container = await new GenericContainer(TEST_CONTAINER_IMAGES.localstack)
+				.withNetwork(network)
+				.withNetworkAliases(HOSTNAME)
+				.withExposedPorts(EDGE_PORT)
+				.withEnvironment({
+					SERVICES: 'secretsmanager',
+					DEFAULT_REGION,
+					// Disable LocalStack Pro features we don't need
+					SKIP_SSL_CERT_DOWNLOAD: '1',
+				})
+				.withWaitStrategy(
+					Wait.forAll([
+						Wait.forListeningPorts(),
+						Wait.forHttp('/_localstack/health', EDGE_PORT)
+							.forStatusCode(200)
+							.forResponsePredicate((body: string) => {
+								try {
+									const health = JSON.parse(body) as LocalStackHealthResponse;
+									// LocalStack returns 'available' for ready services
+									return health.services?.secretsmanager === 'available';
+								} catch {
+									return false;
+								}
+							})
+							.withStartupTimeout(120000),
+					]),
+				)
+				.withLabels({
+					'com.docker.compose.project': projectName,
+					'com.docker.compose.service': HOSTNAME,
+				})
+				.withName(`${projectName}-${HOSTNAME}`)
+				.withReuse()
+				.withLogConsumer(consumer)
+				.start();
+
+			const hostPort = container.getMappedPort(EDGE_PORT);
+
+			return {
+				container,
+				meta: {
+					endpoint: `http://${container.getHost()}:${hostPort}`,
+					internalEndpoint: `http://${HOSTNAME}:${EDGE_PORT}`,
+				},
+			};
+		} catch (error) {
+			return throwWithLogs(error);
+		}
+	},
+
+	env(result: LocalStackResult, external?: boolean): Record<string, string> {
+		return {
+			AWS_ENDPOINT_URL: external ? result.meta.endpoint : result.meta.internalEndpoint,
+			AWS_ACCESS_KEY_ID: 'test',
+			AWS_SECRET_ACCESS_KEY: 'test',
+			AWS_DEFAULT_REGION: DEFAULT_REGION,
+		};
+	},
+};
+
+/**
+ * Client for interacting with AWS Secrets Manager via LocalStack.
+ * Uses the official AWS SDK for proper API compatibility.
+ */
+export class SecretsManagerClient {
+	private readonly client: AwsSecretsManagerClient;
+
+	constructor(endpoint: string) {
+		this.client = new AwsSecretsManagerClient({
+			endpoint,
+			region: DEFAULT_REGION,
+			credentials: {
+				accessKeyId: 'test',
+				secretAccessKey: 'test',
+			},
+		});
+	}
+
+	/**
+	 * Create a secret.
+	 * @param name - Secret name
+	 * @param value - Secret value (string or object that will be JSON-serialized)
+	 */
+	async createSecret(name: string, value: string | Record<string, unknown>): Promise<void> {
+		const secretString = typeof value === 'string' ? value : JSON.stringify(value);
+
+		await this.client.send(
+			new CreateSecretCommand({
+				Name: name,
+				SecretString: secretString,
+			}),
+		);
+	}
+
+	/**
+	 * Get a secret value.
+	 * @param name - Secret name
+	 * @returns The secret string value
+	 */
+	async getSecret(name: string): Promise<string> {
+		const response = await this.client.send(
+			new GetSecretValueCommand({
+				SecretId: name,
+			}),
+		);
+
+		if (!response.SecretString) {
+			throw new Error(`Secret '${name}' has no string value`);
+		}
+		return response.SecretString;
+	}
+
+	/**
+	 * Delete a secret.
+	 * @param name - Secret name
+	 * @param forceDelete - If true, deletes immediately without recovery window
+	 */
+	async deleteSecret(name: string, forceDelete = true): Promise<void> {
+		try {
+			await this.client.send(
+				new DeleteSecretCommand({
+					SecretId: name,
+					ForceDeleteWithoutRecovery: forceDelete,
+				}),
+			);
+		} catch (error) {
+			// Ignore "not found" errors during cleanup
+			if (error instanceof Error && error.name !== 'ResourceNotFoundException') {
+				throw error;
+			}
+		}
+	}
+
+	/**
+	 * List all secret names.
+	 * @returns Array of secret names
+	 */
+	async listSecrets(): Promise<string[]> {
+		const names: string[] = [];
+		let nextToken: string | undefined;
+
+		do {
+			const response = await this.client.send(
+				new ListSecretsCommand({
+					NextToken: nextToken,
+				}),
+			);
+
+			for (const secret of response.SecretList ?? []) {
+				if (secret.Name) names.push(secret.Name);
+			}
+
+			nextToken = response.NextToken;
+		} while (nextToken);
+
+		return names;
+	}
+
+	/**
+	 * Delete all secrets. Useful for cleanup between tests.
+	 */
+	async clear(): Promise<void> {
+		const secrets = await this.listSecrets();
+		await Promise.all(secrets.map(async (name) => await this.deleteSecret(name)));
+	}
+
+	/**
+	 * Wait for a secret to exist. Useful for eventual consistency scenarios.
+	 * @param name - Secret name to wait for
+	 * @param options - Timeout and polling options
+	 * @returns The secret value once it exists
+	 */
+	async waitForSecret(
+		name: string,
+		options: { timeoutMs?: number; pollMs?: number } = {},
+	): Promise<string> {
+		const { timeoutMs = 10000, pollMs = 200 } = options;
+		const deadline = Date.now() + timeoutMs;
+
+		while (Date.now() < deadline) {
+			try {
+				return await this.getSecret(name);
+			} catch {
+				// Secret doesn't exist yet, keep polling
+			}
+			await new Promise((resolve) => setTimeout(resolve, pollMs));
+		}
+
+		throw new Error(`Secret '${name}' not found within ${timeoutMs}ms`);
+	}
+}
+
+/**
+ * Helper for interacting with LocalStack AWS services in tests.
+ *
+ * Access individual services via properties:
+ * - `localstack.secretsManager` - AWS Secrets Manager operations
+ *
+ * Future services can be added as needed (S3, SQS, etc.)
+ */
+export class LocalStackHelper {
+	readonly secretsManager: SecretsManagerClient;
+
+	constructor(endpoint: string) {
+		this.secretsManager = new SecretsManagerClient(endpoint);
+	}
+}
+
+export function createLocalStackHelper(ctx: HelperContext): LocalStackHelper {
+	const result = ctx.serviceResults.localstack as LocalStackResult | undefined;
+	if (!result) {
+		throw new Error('LocalStack service not found in context');
+	}
+	return new LocalStackHelper(result.meta.endpoint);
+}
+
+declare module './types' {
+	interface ServiceHelpers {
+		localstack: LocalStackHelper;
+	}
+}

packages/testing/containers/services/mailpit.ts

@@ -0,0 +1,230 @@
+import type { StartedNetwork } from 'testcontainers';
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { createSilentLogConsumer } from '../helpers/utils';
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { HelperContext, Service, ServiceResult } from './types';
+
+const HOSTNAME = 'mailpit';
+const SMTP_PORT = 1025;
+const HTTP_PORT = 8025;
+
+type MailpitAddress = {
+	Address: string;
+	Name?: string;
+};
+
+export type MailpitMessageSummary = {
+	ID: string;
+	MessageID: string;
+	Read: boolean;
+	From: MailpitAddress;
+	To: MailpitAddress[];
+	Cc: MailpitAddress[] | null;
+	Bcc: MailpitAddress[] | null;
+	ReplyTo: MailpitAddress[];
+	Subject: string;
+	Created: string;
+	Username: string;
+	Tags: string[];
+	Size: number;
+	Attachments: number;
+	Snippet: string;
+};
+
+export type MailpitMessage = MailpitMessageSummary & {
+	Text?: string;
+	HTML?: string;
+	Inline?: Array<{
+		PartID: string;
+		FileName: string;
+		ContentType: string;
+		ContentID: string;
+		Size: number;
+	}>;
+	Attachments?: Array<{
+		PartID: string;
+		FileName: string;
+		ContentType: string;
+		ContentID: string;
+		Size: number;
+	}>;
+};
+
+export type MailpitQuery = {
+	to?: string | RegExp;
+	subject?: string | RegExp;
+};
+
+type MailpitListResponse = {
+	total: number;
+	unread: number;
+	count: number;
+	messages_count: number;
+	messages_unread: number;
+	start: number;
+	tags: string[];
+	messages: MailpitMessageSummary[];
+};
+
+export interface MailpitMeta {
+	apiBaseUrl: string;
+}
+
+export type MailpitResult = ServiceResult<MailpitMeta>;
+
+export const mailpit: Service<MailpitResult> = {
+	description: 'Email testing server',
+
+	async start(network: StartedNetwork, projectName: string): Promise<MailpitResult> {
+		const { consumer, throwWithLogs } = createSilentLogConsumer();
+
+		try {
+			const container = await new GenericContainer(TEST_CONTAINER_IMAGES.mailpit)
+				.withNetwork(network)
+				.withNetworkAliases(HOSTNAME)
+				.withExposedPorts(SMTP_PORT, HTTP_PORT)
+				.withEnvironment({
+					MP_UI_BIND_ADDR: `0.0.0.0:${HTTP_PORT}`,
+					MP_SMTP_BIND_ADDR: `0.0.0.0:${SMTP_PORT}`,
+				})
+				.withWaitStrategy(
+					Wait.forAll([
+						Wait.forListeningPorts(),
+						Wait.forHttp('/api/v1/info', HTTP_PORT).forStatusCode(200).withStartupTimeout(30000),
+					]),
+				)
+				.withLabels({
+					'com.docker.compose.project': projectName,
+					'com.docker.compose.service': HOSTNAME,
+				})
+				.withName(`${projectName}-${HOSTNAME}`)
+				.withReuse()
+				.withLogConsumer(consumer)
+				.start();
+
+			return {
+				container,
+				meta: {
+					apiBaseUrl: `http://${container.getHost()}:${container.getMappedPort(HTTP_PORT)}`,
+				},
+			};
+		} catch (error) {
+			return throwWithLogs(error);
+		}
+	},
+
+	env(result: MailpitResult, external?: boolean): Record<string, string> {
+		return {
+			N8N_EMAIL_MODE: 'smtp',
+			N8N_SMTP_HOST: external ? result.container.getHost() : HOSTNAME,
+			N8N_SMTP_PORT: external
+				? String(result.container.getMappedPort(SMTP_PORT))
+				: String(SMTP_PORT),
+			N8N_SMTP_SSL: 'false',
+			N8N_SMTP_SENDER: 'test@n8n.local',
+		};
+	},
+};
+
+export class MailpitHelper {
+	private readonly apiBaseUrl: string;
+
+	/** SMTP host that n8n should use to send email (internal hostname in container mode, localhost in local mode) */
+	readonly smtpHost: string;
+
+	/** SMTP port that n8n should use to send email (1025 in container mode, mapped port in local mode) */
+	readonly smtpPort: number;
+
+	constructor(apiBaseUrl: string, smtpHost = HOSTNAME, smtpPort = SMTP_PORT) {
+		this.apiBaseUrl = apiBaseUrl;
+		this.smtpHost = smtpHost;
+		this.smtpPort = smtpPort;
+	}
+
+	async clear(): Promise<void> {
+		const res = await fetch(`${this.apiBaseUrl}/api/v1/messages`, { method: 'DELETE' });
+		if (!res.ok) {
+			throw new Error(`Mailpit clear failed: ${res.status} ${res.statusText}`);
+		}
+	}
+
+	async list(): Promise<MailpitMessageSummary[]> {
+		const res = await fetch(`${this.apiBaseUrl}/api/v1/messages`);
+		if (!res.ok) {
+			throw new Error(`Mailpit list failed: ${res.status} ${res.statusText}`);
+		}
+
+		const data = (await res.json()) as MailpitListResponse;
+		return data.messages || [];
+	}
+
+	async get(id: string): Promise<MailpitMessage> {
+		const res = await fetch(`${this.apiBaseUrl}/api/v1/message/${id}`);
+		if (!res.ok) {
+			throw new Error(`Mailpit get failed: ${res.status} ${res.statusText}`);
+		}
+
+		return (await res.json()) as MailpitMessage;
+	}
+
+	async waitForMessage(
+		query: MailpitQuery,
+		options: { timeoutMs?: number; pollMs?: number } = {},
+	): Promise<MailpitMessageSummary> {
+		const { timeoutMs = 10000, pollMs = 200 } = options;
+		const deadline = Date.now() + timeoutMs;
+
+		const messageMatches = (message: MailpitMessageSummary): boolean => {
+			if (query.to) {
+				const hasMatchingRecipient = message.To.some((recipient) =>
+					typeof query.to === 'string'
+						? recipient.Address === query.to
+						: query.to!.test(recipient.Address),
+				);
+				if (!hasMatchingRecipient) return false;
+			}
+
+			if (query.subject) {
+				const subjectMatches =
+					typeof query.subject === 'string'
+						? message.Subject === query.subject
+						: query.subject.test(message.Subject);
+				if (!subjectMatches) return false;
+			}
+
+			return true;
+		};
+
+		while (Date.now() < deadline) {
+			const messages = await this.list();
+			const match = messages.find(messageMatches);
+
+			if (match) {
+				return match;
+			}
+
+			await new Promise((resolve) => setTimeout(resolve, pollMs));
+		}
+
+		const queryParts = [];
+		if (query.to) queryParts.push(`to: ${query.to}`);
+		if (query.subject) queryParts.push(`subject: ${query.subject}`);
+
+		throw new Error(`Mail not received within ${timeoutMs}ms. Query: ${queryParts.join(', ')}`);
+	}
+}
+
+export function createMailpitHelper(ctx: HelperContext): MailpitHelper {
+	const result = ctx.serviceResults.mailpit as MailpitResult | undefined;
+	if (!result) {
+		throw new Error('Mailpit service not found in context');
+	}
+	return new MailpitHelper(result.meta.apiBaseUrl);
+}
+
+declare module './types' {
+	interface ServiceHelpers {
+		mailpit: MailpitHelper;
+	}
+}

packages/testing/containers/services/mysql.ts

@@ -0,0 +1,68 @@
+import { MySqlContainer, type StartedMySqlContainer } from '@testcontainers/mysql';
+import type { StartedNetwork } from 'testcontainers';
+
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { Service, ServiceResult } from './types';
+
+const HOSTNAME = 'mysql';
+
+export interface MySqlMeta {
+	database: string;
+	username: string;
+	password: string;
+	port: number;
+	internalHost: string;
+	externalHost: string;
+	externalPort: number;
+}
+
+export type MySqlResult = ServiceResult<MySqlMeta> & {
+	container: StartedMySqlContainer;
+};
+
+export const mysqlService: Service<MySqlResult> = {
+	description: 'MySQL database for integration testing',
+
+	async start(network: StartedNetwork, projectName: string): Promise<MySqlResult> {
+		const container = await new MySqlContainer(TEST_CONTAINER_IMAGES.mysql)
+			.withNetwork(network)
+			.withNetworkAliases(HOSTNAME)
+			.withDatabase('n8n_test')
+			.withUsername('n8n_user')
+			.withRootPassword('root_password')
+			.withUserPassword('test_password')
+			.withStartupTimeout(60_000)
+			.withLabels({
+				'com.docker.compose.project': projectName,
+				'com.docker.compose.service': HOSTNAME,
+			})
+			.withName(`${projectName}-${HOSTNAME}`)
+			.withReuse()
+			.start();
+
+		return {
+			container,
+			meta: {
+				database: container.getDatabase(),
+				username: container.getUsername(),
+				password: container.getUserPassword(),
+				port: 3306,
+				internalHost: HOSTNAME,
+				externalHost: container.getHost(),
+				externalPort: container.getPort(),
+			},
+		};
+	},
+
+	env(result: MySqlResult, external?: boolean): Record<string, string> {
+		if (!external) return {};
+		return {
+			DB_TYPE: 'mysqldb',
+			DB_MYSQLDB_HOST: result.meta.externalHost,
+			DB_MYSQLDB_PORT: String(result.meta.externalPort),
+			DB_MYSQLDB_DATABASE: result.meta.database,
+			DB_MYSQLDB_USER: result.meta.username,
+			DB_MYSQLDB_PASSWORD: result.meta.password,
+		};
+	},
+};

packages/testing/containers/services/n8n.ts

@@ -0,0 +1,242 @@
+import type { StartedNetwork, StartedTestContainer } from 'testcontainers';
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { DockerImageNotFoundError } from '../docker-image-not-found-error';
+import { createElapsedLogger, createSilentLogConsumer } from '../helpers/utils';
+import { N8nImagePullPolicy } from '../n8n-image-pull-policy';
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { FileToMount } from './types';
+
+const N8N_IMAGE = TEST_CONTAINER_IMAGES.n8n;
+
+const BASE_ENV: Record<string, string> = {
+	N8N_LOG_LEVEL: 'debug',
+	N8N_ENCRYPTION_KEY: process.env.N8N_ENCRYPTION_KEY ?? 'test-encryption-key',
+	E2E_TESTS: 'false',
+	QUEUE_HEALTH_CHECK_ACTIVE: 'true',
+	N8N_DIAGNOSTICS_ENABLED: 'false',
+	N8N_METRICS: 'true',
+	NODE_ENV: 'development',
+	N8N_DYNAMIC_BANNERS_ENABLED: 'false',
+	N8N_LICENSE_TENANT_ID: process.env.N8N_LICENSE_TENANT_ID ?? '1001',
+	N8N_LICENSE_ACTIVATION_KEY: process.env.N8N_LICENSE_ACTIVATION_KEY ?? '',
+	N8N_LICENSE_CERT: process.env.N8N_LICENSE_CERT ?? '',
+	N8N_RUNNERS_MODE: 'external',
+	N8N_RUNNERS_AUTH_TOKEN: 'test',
+	N8N_RUNNERS_BROKER_LISTEN_ADDRESS: '0.0.0.0',
+	// Expose V8 garbage collector for memory profiling in performance tests
+	NODE_OPTIONS: '--expose-gc',
+};
+
+const MAIN_WAIT_STRATEGY = Wait.forAll([
+	Wait.forListeningPorts(),
+	Wait.forHttp('/healthz/readiness', 5678).forStatusCode(200).withStartupTimeout(30000),
+	Wait.forLogMessage('Editor is now accessible via').withStartupTimeout(30000),
+]);
+
+const WORKER_WAIT_STRATEGY = Wait.forAll([
+	Wait.forListeningPorts(),
+	Wait.forLogMessage('n8n worker is now ready').withStartupTimeout(30000),
+]);
+
+export interface N8NInstancesOptions {
+	mains: number;
+	workers: number;
+	projectName: string;
+	network: StartedNetwork;
+	serviceEnvironment: Record<string, string>;
+	userEnvironment?: Record<string, string>;
+	usePostgres: boolean;
+	baseUrl?: string;
+	allocatedPort?: number;
+	resourceQuota?: { memory?: number; cpu?: number };
+	filesToMount?: FileToMount[];
+}
+
+export interface N8NInstancesResult {
+	containers: StartedTestContainer[];
+	environment: Record<string, string>;
+}
+
+function computeEnvironment(options: N8NInstancesOptions): Record<string, string> {
+	const {
+		mains,
+		workers,
+		usePostgres,
+		baseUrl,
+		serviceEnvironment,
+		userEnvironment = {},
+	} = options;
+
+	const isQueueMode = mains > 1 || workers > 0;
+
+	const env: Record<string, string> = {
+		...BASE_ENV,
+		...serviceEnvironment,
+		...userEnvironment,
+	};
+
+	if (!usePostgres) {
+		env.DB_TYPE = 'sqlite';
+	}
+
+	if (isQueueMode) {
+		env.EXECUTIONS_MODE = 'queue';
+		env.OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS = 'true';
+
+		if (mains > 1) {
+			if (!process.env.N8N_LICENSE_ACTIVATION_KEY && !process.env.N8N_LICENSE_CERT) {
+				throw new Error(
+					'N8N_LICENSE_ACTIVATION_KEY or N8N_LICENSE_CERT is required for multi-main instances',
+				);
+			}
+			env.N8N_MULTI_MAIN_SETUP_ENABLED = 'true';
+		}
+	}
+
+	if (mains === 1 && baseUrl && !serviceEnvironment.WEBHOOK_URL) {
+		env.WEBHOOK_URL = baseUrl;
+		env.N8N_PORT = '5678';
+	}
+
+	return env;
+}
+
+interface InstanceConfig {
+	name: string;
+	isWorker: boolean;
+	instanceNumber: number;
+	networkAlias?: string;
+	hostPort?: number;
+}
+
+interface SharedConfig {
+	projectName: string;
+	environment: Record<string, string>;
+	network: StartedNetwork;
+	resourceQuota?: { memory?: number; cpu?: number };
+	filesToMount?: FileToMount[];
+}
+
+async function createContainer(
+	instance: InstanceConfig,
+	shared: SharedConfig,
+): Promise<StartedTestContainer> {
+	const { name, isWorker, instanceNumber, networkAlias, hostPort } = instance;
+	const { projectName, environment, network, resourceQuota, filesToMount } = shared;
+	const { consumer, throwWithLogs } = createSilentLogConsumer();
+
+	let container = new GenericContainer(N8N_IMAGE)
+		.withEnvironment(environment)
+		.withLabels({
+			'com.docker.compose.project': projectName,
+			'com.docker.compose.service': isWorker ? 'n8n-worker' : 'n8n-main',
+			instance: instanceNumber.toString(),
+		})
+		.withPullPolicy(new N8nImagePullPolicy(N8N_IMAGE))
+		.withName(name)
+		.withLogConsumer(consumer)
+		.withReuse()
+		.withNetwork(network);
+
+	if (filesToMount?.length) {
+		container = container.withCopyContentToContainer(filesToMount);
+	}
+
+	if (resourceQuota) {
+		container = container.withResourcesQuota(resourceQuota);
+	}
+
+	if (networkAlias) {
+		container = container.withNetworkAliases(networkAlias);
+	}
+
+	const waitStrategy = isWorker ? WORKER_WAIT_STRATEGY : MAIN_WAIT_STRATEGY;
+	const ports = hostPort ? [{ container: 5678, host: hostPort }, 5679] : [5678, 5679];
+
+	container = container.withExposedPorts(...ports).withWaitStrategy(waitStrategy);
+
+	if (isWorker) {
+		container = container.withCommand(['worker']);
+	}
+
+	try {
+		return await container.start();
+	} catch (error: unknown) {
+		if (error instanceof Error && 'statusCode' in error) {
+			const statusCode = (error as Error & { statusCode: number }).statusCode;
+			if (statusCode === 404) {
+				throw new DockerImageNotFoundError(name, error);
+			}
+		}
+		console.error(`Container "${name}" failed to start:`, error);
+		return throwWithLogs(error);
+	}
+}
+
+export async function createN8NInstances(
+	options: N8NInstancesOptions,
+): Promise<N8NInstancesResult> {
+	const { mains, workers, projectName, network, allocatedPort, resourceQuota, filesToMount } =
+		options;
+
+	const log = createElapsedLogger('n8n-instances');
+	const environment = computeEnvironment(options);
+	const containers: StartedTestContainer[] = [];
+
+	const shared: SharedConfig = {
+		projectName,
+		environment,
+		network,
+		resourceQuota,
+		filesToMount,
+	};
+
+	const instances: InstanceConfig[] = [
+		...Array.from({ length: mains }, (_, i) => {
+			const num = i + 1;
+			const name = mains > 1 ? `${projectName}-n8n-main-${num}` : `${projectName}-n8n`;
+			return {
+				name,
+				isWorker: false,
+				instanceNumber: num,
+				networkAlias: name,
+				hostPort: num === 1 ? allocatedPort : undefined,
+			};
+		}),
+		...Array.from({ length: workers }, (_, i) => ({
+			name: `${projectName}-n8n-worker-${i + 1}`,
+			isWorker: true,
+			instanceNumber: i + 1,
+		})),
+	];
+
+	// Service-only mode: no n8n containers needed
+	if (instances.length === 0) {
+		log('No n8n instances requested (service-only mode)');
+		return { containers, environment };
+	}
+
+	// Start main 1 first (handles DB migrations/setup)
+	const [main1, ...remaining] = instances;
+	log(`Starting main 1: ${main1.name} (DB setup)`);
+	containers.push(await createContainer(main1, shared));
+	log('main 1 ready');
+
+	// Start remaining instances in parallel
+	if (remaining.length > 0) {
+		log(`Starting ${remaining.length} remaining instances in parallel...`);
+		const parallelContainers = await Promise.all(
+			remaining.map(async (instance) => {
+				const type = instance.isWorker ? 'worker' : 'main';
+				log(`Starting ${type} ${instance.instanceNumber}: ${instance.name}`);
+				const container = await createContainer(instance, shared);
+				log(`${type} ${instance.instanceNumber} ready`);
+				return container;
+			}),
+		);
+		containers.push(...parallelContainers);
+	}
+
+	return { containers, environment };
+}

packages/testing/containers/services/ngrok.ts

@@ -0,0 +1,106 @@
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { createSilentLogConsumer } from '../helpers/utils';
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import { EXTERNAL_HOST, type Service, type ServiceResult, type StartContext } from './types';
+
+export interface NgrokMeta {
+	publicUrl: string;
+	proxyHops: number;
+}
+
+export type NgrokResult = ServiceResult<NgrokMeta>;
+
+const API_PORT = 4040;
+
+function getTunnelTarget(ctx: StartContext): string {
+	if (ctx.external) {
+		return `${EXTERNAL_HOST}:5678`;
+	}
+	if (ctx.needsLoadBalancer) {
+		return `${ctx.projectName}-caddy-lb:80`;
+	}
+	return `${ctx.projectName}-n8n:5678`;
+}
+
+export const ngrok: Service<NgrokResult> = {
+	description: 'ngrok Tunnel',
+	dependsOn: ['loadBalancer'],
+
+	shouldStart: (ctx) => ctx.config.services?.includes('ngrok') ?? false,
+
+	getOptions(ctx) {
+		const proxyHops = ctx.needsLoadBalancer ? 2 : 1;
+		return { tunnelTarget: getTunnelTarget(ctx), proxyHops };
+	},
+
+	env(result) {
+		return {
+			WEBHOOK_URL: result.meta.publicUrl,
+			N8N_PROXY_HOPS: String(result.meta.proxyHops),
+		};
+	},
+
+	async start(network, projectName, config?: unknown, ctx?: StartContext): Promise<NgrokResult> {
+		const { tunnelTarget, proxyHops } = config as { tunnelTarget: string; proxyHops: number };
+		const { consumer, throwWithLogs } = createSilentLogConsumer();
+
+		const authToken = process.env.NGROK_AUTHTOKEN;
+		if (!authToken) {
+			throw new Error(
+				'NGROK_AUTHTOKEN environment variable is required. ' +
+					'Get a free token at https://dashboard.ngrok.com/get-started/your-authtoken',
+			);
+		}
+
+		try {
+			let builder = new GenericContainer(TEST_CONTAINER_IMAGES.ngrok)
+				.withNetwork(network)
+				.withNetworkAliases('ngrok')
+				.withName(`${projectName}-ngrok`)
+				.withExposedPorts(API_PORT)
+				.withEnvironment({
+					NGROK_AUTHTOKEN: authToken,
+				})
+				.withCommand(['http', `http://${tunnelTarget}`, '--log', 'stdout'])
+				.withWaitStrategy(Wait.forLogMessage(/started tunnel/i))
+				.withLabels({
+					'com.docker.compose.project': projectName,
+					'com.docker.compose.service': 'ngrok',
+				})
+				.withReuse()
+				.withLogConsumer(consumer);
+
+			// On Linux, host.docker.internal is not available without explicit mapping
+			if (ctx?.external) {
+				builder = builder.withExtraHosts([{ host: EXTERNAL_HOST, ipAddress: 'host-gateway' }]);
+			}
+
+			const container = await builder.start();
+
+			const hostPort = container.getMappedPort(API_PORT);
+			const host = container.getHost();
+
+			// ngrok API returns tunnel info at /api/tunnels
+			const response = await fetch(`http://${host}:${hostPort}/api/tunnels`);
+			const data = (await response.json()) as {
+				tunnels: Array<{ public_url: string; proto: string }>;
+			};
+
+			// Find the https tunnel
+			const httpsTunnel = data.tunnels.find((t) => t.proto === 'https');
+			const publicUrl = httpsTunnel?.public_url ?? data.tunnels[0]?.public_url;
+
+			if (!publicUrl) {
+				throw new Error('Failed to get ngrok public URL from API');
+			}
+
+			return {
+				container,
+				meta: { publicUrl, proxyHops },
+			};
+		} catch (error) {
+			return throwWithLogs(error);
+		}
+	},
+};

packages/testing/containers/services/observability.ts

@@ -0,0 +1,48 @@
+/**
+ * Combined observability helper that provides unified access to logs and metrics.
+ * The actual services are in victoria-logs.ts and victoria-metrics.ts.
+ */
+import type { HelperContext } from './types';
+import { LogsHelper, type VictoriaLogsResult, escapeLogsQL } from './victoria-logs';
+import { MetricsHelper, type VictoriaMetricsResult } from './victoria-metrics';
+
+export { escapeLogsQL };
+export { LogsHelper, type LogEntry, type LogQueryOptions } from './victoria-logs';
+export {
+	MetricsHelper,
+	type MetricResult,
+	type WaitForMetricOptions,
+	type ScrapeTarget,
+} from './victoria-metrics';
+
+export class ObservabilityHelper {
+	readonly logs: LogsHelper;
+	readonly metrics: MetricsHelper;
+	readonly syslog: VictoriaLogsResult['meta']['syslog'];
+
+	constructor(logsMeta: VictoriaLogsResult['meta'], metricsMeta: VictoriaMetricsResult['meta']) {
+		this.logs = new LogsHelper(logsMeta.queryEndpoint);
+		this.metrics = new MetricsHelper(metricsMeta.queryEndpoint);
+		this.syslog = logsMeta.syslog;
+	}
+}
+
+export function createObservabilityHelper(ctx: HelperContext): ObservabilityHelper {
+	const logsResult = ctx.serviceResults.victoriaLogs as VictoriaLogsResult | undefined;
+	const metricsResult = ctx.serviceResults.victoriaMetrics as VictoriaMetricsResult | undefined;
+
+	if (!logsResult) {
+		throw new Error('VictoriaLogs service not found in context');
+	}
+	if (!metricsResult) {
+		throw new Error('VictoriaMetrics service not found in context');
+	}
+
+	return new ObservabilityHelper(logsResult.meta, metricsResult.meta);
+}
+
+declare module './types' {
+	interface ServiceHelpers {
+		observability: ObservabilityHelper;
+	}
+}

packages/testing/containers/services/postgres.ts

@@ -0,0 +1,68 @@
+import { PostgreSqlContainer } from '@testcontainers/postgresql';
+import type { StartedNetwork } from 'testcontainers';
+
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { Service, ServiceResult } from './types';
+
+const HOSTNAME = 'postgres';
+
+export interface PostgresMeta {
+	database: string;
+	username: string;
+	password: string;
+}
+
+export type PostgresResult = ServiceResult<PostgresMeta>;
+
+export const postgres: Service<PostgresResult> = {
+	description: 'PostgreSQL database',
+	shouldStart: (ctx) => ctx.usePostgres,
+
+	async start(network: StartedNetwork, projectName: string): Promise<PostgresResult> {
+		const container = await new PostgreSqlContainer(TEST_CONTAINER_IMAGES.postgres)
+			.withNetwork(network)
+			.withNetworkAliases(HOSTNAME)
+			.withDatabase('n8n_db')
+			.withUsername('n8n_user')
+			.withPassword('test_password')
+			.withStartupTimeout(30000)
+			.withLabels({
+				'com.docker.compose.project': projectName,
+				'com.docker.compose.service': HOSTNAME,
+			})
+			.withName(`${projectName}-${HOSTNAME}`)
+			.withAddedCapabilities('NET_ADMIN') // Allows us to drop IP tables and block traffic
+			.withTmpFs({ '/var/lib/postgresql': 'rw' })
+			.withCommand([
+				'postgres',
+				'-c',
+				'fsync=off',
+				'-c',
+				'synchronous_commit=off',
+				'-c',
+				'full_page_writes=off',
+			])
+			.withReuse()
+			.start();
+
+		return {
+			container,
+			meta: {
+				database: container.getDatabase(),
+				username: container.getUsername(),
+				password: container.getPassword(),
+			},
+		};
+	},
+
+	env(result: PostgresResult, external?: boolean): Record<string, string> {
+		return {
+			DB_TYPE: 'postgresdb',
+			DB_POSTGRESDB_HOST: external ? result.container.getHost() : HOSTNAME,
+			DB_POSTGRESDB_PORT: external ? String(result.container.getMappedPort(5432)) : '5432',
+			DB_POSTGRESDB_DATABASE: result.meta.database,
+			DB_POSTGRESDB_USER: result.meta.username,
+			DB_POSTGRESDB_PASSWORD: result.meta.password,
+		};
+	},
+};

packages/testing/containers/services/proxy.ts

@@ -0,0 +1,372 @@
+import crypto from 'crypto';
+import { promises as fs } from 'fs';
+import type { Expectation, RequestDefinition } from 'mockserver-client';
+import { mockServerClient } from 'mockserver-client';
+import type { HttpRequest, HttpResponse } from 'mockserver-client/mockServer';
+import type {
+	MockServerClient,
+	PathOrRequestDefinition,
+	RequestResponse,
+} from 'mockserver-client/mockServerClient';
+import { join } from 'path';
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { createSilentLogConsumer } from '../helpers/utils';
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { HelperContext, Service, ServiceResult } from './types';
+
+const HOSTNAME = 'proxyserver';
+const PORT = 1080;
+
+export interface ProxyMeta {
+	host: string;
+	port: number;
+	internalUrl: string;
+}
+
+export type ProxyResult = ServiceResult<ProxyMeta>;
+
+export const proxy: Service<ProxyResult> = {
+	description: 'HTTP proxy server',
+
+	extraEnv(result: ProxyResult, external?: boolean): Record<string, string> {
+		const url = external
+			? `http://${result.container.getHost()}:${result.container.getMappedPort(PORT)}`
+			: result.meta.internalUrl;
+		return {
+			HTTP_PROXY: url,
+			HTTPS_PROXY: url,
+			NODE_TLS_REJECT_UNAUTHORIZED: '0',
+		};
+	},
+
+	async start(network, projectName): Promise<ProxyResult> {
+		const { consumer, throwWithLogs } = createSilentLogConsumer();
+
+		try {
+			const container = await new GenericContainer(TEST_CONTAINER_IMAGES.mockserver)
+				.withNetwork(network)
+				.withNetworkAliases(HOSTNAME)
+				.withExposedPorts(PORT)
+				.withWaitStrategy(Wait.forLogMessage(`INFO ${PORT} started on port: ${PORT}`))
+				.withLabels({
+					'com.docker.compose.project': projectName,
+					'com.docker.compose.service': HOSTNAME,
+				})
+				.withName(`${projectName}-${HOSTNAME}`)
+				.withReuse()
+				.withLogConsumer(consumer)
+				.start();
+
+			return {
+				container,
+				meta: {
+					host: HOSTNAME,
+					port: PORT,
+					internalUrl: `http://${HOSTNAME}:${PORT}`,
+				},
+			};
+		} catch (error) {
+			return throwWithLogs(error);
+		}
+	},
+
+	env(result: ProxyResult, external?: boolean): Record<string, string> {
+		return {
+			N8N_PROXY_HOST: external ? result.container.getHost() : result.meta.host,
+			N8N_PROXY_PORT: external
+				? String(result.container.getMappedPort(PORT))
+				: String(result.meta.port),
+		};
+	},
+};
+
+// --- ProxyServer helper (MockServer API client) ---
+
+export type RequestMade = {
+	httpRequest?: HttpRequest;
+	httpResponse?: HttpResponse;
+	timestamp?: string;
+};
+
+export interface ProxyServerRequest {
+	method: string;
+	path: string;
+	queryStringParameters?: Record<string, string[]>;
+	headers?: Record<string, string[]>;
+	body?: string | { type?: string; [key: string]: unknown };
+}
+
+export interface ProxyServerResponse {
+	statusCode: number;
+	headers?: Record<string, string[]>;
+	body?: string;
+	delay?: {
+		timeUnit: 'MICROSECONDS' | 'MILLISECONDS' | 'SECONDS' | 'MINUTES';
+		value: number;
+	};
+}
+
+export interface ProxyServerExpectation {
+	httpRequest: ProxyServerRequest;
+	httpResponse: ProxyServerResponse;
+	times?: {
+		remainingTimes?: number;
+		unlimited?: boolean;
+	};
+}
+
+export interface RequestLog {
+	method: string;
+	path: string;
+	headers: Record<string, string[]>;
+	queryStringParameters?: Record<string, string[]>;
+	body?: string;
+	timestamp: string;
+}
+
+export class ProxyServer {
+	private client: MockServerClient;
+
+	url: string;
+
+	private expectationsDir: string;
+
+	constructor(proxyServerUrl: string, expectationsDir = './expectations') {
+		this.url = proxyServerUrl;
+		this.expectationsDir = expectationsDir;
+		const parsedURL = new URL(proxyServerUrl);
+		this.client = mockServerClient(parsedURL.hostname, parseInt(parsedURL.port, 10));
+	}
+
+	async loadExpectations(
+		folderName: string,
+		options: { strictBodyMatching?: boolean } = {},
+	): Promise<void> {
+		try {
+			const targetDir = join(this.expectationsDir, folderName);
+			const files = await fs.readdir(targetDir);
+			const jsonFiles = files.filter((file) => file.endsWith('.json'));
+			const expectations: Expectation[] = [];
+
+			for (const file of jsonFiles) {
+				try {
+					const filePath = join(targetDir, file);
+					const fileContent = await fs.readFile(filePath, 'utf8');
+					const expectation = JSON.parse(fileContent) as Expectation;
+
+					if (
+						options.strictBodyMatching &&
+						expectation.httpRequest &&
+						'body' in expectation.httpRequest
+					) {
+						(expectation.httpRequest as { body: { matchType: string } }).body.matchType = 'STRICT';
+					}
+
+					expectations.push(expectation);
+				} catch (parseError) {
+					console.log(`Error parsing expectation from ${file}:`, parseError);
+				}
+			}
+
+			if (expectations.length > 0) {
+				console.log('Loading expectations:', expectations.length);
+				await this.client.mockAnyResponse(expectations);
+			}
+		} catch (error) {
+			console.log('Error loading expectations:', error);
+		}
+	}
+
+	async createExpectation(expectation: ProxyServerExpectation): Promise<RequestResponse> {
+		try {
+			return await this.client.mockAnyResponse({
+				httpRequest: expectation.httpRequest,
+				httpResponse: expectation.httpResponse,
+				times: expectation.times,
+			});
+		} catch (error) {
+			throw new Error(
+				`Failed to create expectation: ${error instanceof Error ? error.message : String(error)}`,
+			);
+		}
+	}
+
+	async verifyRequest(request: RequestDefinition, numberOfRequests: number): Promise<boolean> {
+		try {
+			await this.client.verify(request, numberOfRequests, numberOfRequests);
+			return true;
+		} catch (error) {
+			console.log('error', error);
+			return false;
+		}
+	}
+
+	async clearAllExpectations(): Promise<void> {
+		try {
+			await this.client.clear('', 'ALL');
+		} catch (error) {
+			throw new Error(`Failed to clear ProxyServer: ${JSON.stringify(error)}`);
+		}
+	}
+
+	async createGetExpectation(
+		path: string,
+		responseBody: unknown,
+		queryParams?: Record<string, string>,
+		statusCode: number = 200,
+	): Promise<RequestResponse> {
+		const queryStringParameters = queryParams
+			? Object.entries(queryParams).reduce<Record<string, string[]>>((acc, [key, value]) => {
+					acc[key] = [value];
+					return acc;
+				}, {})
+			: undefined;
+
+		return await this.createExpectation({
+			httpRequest: {
+				method: 'GET',
+				path,
+				...(queryStringParameters && { queryStringParameters }),
+			},
+			httpResponse: {
+				statusCode,
+				headers: {
+					'Content-Type': ['application/json'],
+				},
+				body: JSON.stringify(responseBody),
+			},
+		});
+	}
+
+	async wasRequestMade(request: RequestDefinition, numberOfRequests = 1): Promise<boolean> {
+		return await this.verifyRequest(request, numberOfRequests);
+	}
+
+	async getAllRequestsMade(): Promise<RequestMade[]> {
+		// @ts-expect-error mockserver types seem to be messed up
+		return await this.client.retrieveRecordedRequestsAndResponses('');
+	}
+
+	async recordExpectations(
+		folderName: string,
+		options?: {
+			pathOrRequestDefinition?: PathOrRequestDefinition;
+			host?: string;
+			dedupe?: boolean;
+			raw?: boolean;
+			transform?: (expectation: Expectation) => Expectation;
+		},
+	): Promise<void> {
+		try {
+			const recordedExpectations = await this.client.retrieveRecordedExpectations(
+				options?.pathOrRequestDefinition,
+			);
+
+			const targetDir = join(this.expectationsDir, folderName);
+
+			await fs.mkdir(targetDir, { recursive: true });
+			const seenRequests = new Set<string>();
+
+			for (const expectation of recordedExpectations) {
+				if (
+					!expectation.httpRequest ||
+					!(
+						'method' in expectation.httpRequest &&
+						typeof expectation.httpRequest.method === 'string' &&
+						typeof expectation.httpRequest.path === 'string'
+					)
+				) {
+					continue;
+				}
+
+				const headers = (expectation.httpRequest.headers ?? {}) as Record<string, unknown>;
+				const hostHeader = 'Host' in headers ? (headers.Host as string | string[]) : undefined;
+				const hostName = Array.isArray(hostHeader) ? hostHeader[0] : (hostHeader ?? 'unknown-host');
+
+				if (options?.host && typeof hostName === 'string' && !hostName.includes(options.host)) {
+					continue;
+				}
+
+				const method = expectation.httpRequest.method;
+				let requestForProcessing: Record<string, unknown> | HttpRequest;
+
+				if (options?.raw) {
+					requestForProcessing = expectation.httpRequest;
+				} else {
+					const cleanedRequest: Record<string, unknown> = {
+						method: expectation.httpRequest.method,
+						path: expectation.httpRequest.path,
+					};
+
+					if (method === 'GET') {
+						if (expectation.httpRequest.queryStringParameters) {
+							cleanedRequest.queryStringParameters = expectation.httpRequest.queryStringParameters;
+						}
+					} else if (method === 'POST' || method === 'PUT') {
+						if (expectation.httpRequest.body) {
+							cleanedRequest.body = expectation.httpRequest.body;
+						}
+					}
+
+					requestForProcessing = cleanedRequest;
+				}
+
+				if (options?.dedupe) {
+					const dedupeKey = JSON.stringify(requestForProcessing);
+
+					if (seenRequests.has(dedupeKey)) {
+						continue;
+					}
+
+					seenRequests.add(dedupeKey);
+				}
+
+				let processedExpectation: Expectation = {
+					...expectation,
+					httpRequest: requestForProcessing,
+					times: {
+						unlimited: true,
+					},
+				};
+
+				if (options?.transform) {
+					processedExpectation = options.transform(processedExpectation);
+				}
+
+				const hash = crypto
+					.createHash('sha256')
+					.update(JSON.stringify(requestForProcessing))
+					.digest('hex')
+					.substring(0, 8);
+
+				const filename = `${Date.now()}-${hostName}-${method}-${expectation.httpRequest.path.replace(/[^a-zA-Z0-9]/g, '_')}-${hash}.json`;
+				processedExpectation.id = filename;
+				const filePath = join(targetDir, filename);
+
+				await fs.writeFile(filePath, JSON.stringify(processedExpectation, null, 2));
+			}
+		} catch (error) {
+			throw new Error(`Failed to record expectations: ${JSON.stringify(error)}`);
+		}
+	}
+
+	async getActiveExpectations() {
+		return await this.client.retrieveActiveExpectations({ method: 'GET' });
+	}
+}
+
+export function createProxyHelper(ctx: HelperContext): ProxyServer {
+	const result = ctx.serviceResults.proxy as ProxyResult | undefined;
+	if (!result) {
+		throw new Error('Proxy service not found in context');
+	}
+	const url = `http://${result.container.getHost()}:${result.container.getMappedPort(PORT)}`;
+	return new ProxyServer(url);
+}
+
+declare module './types' {
+	interface ServiceHelpers {
+		proxy: ProxyServer;
+	}
+}

packages/testing/containers/services/redis.ts

@@ -0,0 +1,56 @@
+import { RedisContainer } from '@testcontainers/redis';
+import type { StartedNetwork } from 'testcontainers';
+
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { Service, ServiceResult } from './types';
+
+const HOSTNAME = 'redis';
+
+export interface RedisMeta {
+	host: string;
+	port: number;
+}
+
+export type RedisResult = ServiceResult<RedisMeta>;
+
+export const redis: Service<RedisResult> = {
+	description: 'Redis',
+	shouldStart: (ctx) => ctx.isQueueMode,
+
+	async start(network: StartedNetwork, projectName: string): Promise<RedisResult> {
+		const container = await new RedisContainer(TEST_CONTAINER_IMAGES.redis)
+			.withNetwork(network)
+			.withNetworkAliases(HOSTNAME)
+			.withLabels({
+				'com.docker.compose.project': projectName,
+				'com.docker.compose.service': HOSTNAME,
+			})
+			.withName(`${projectName}-${HOSTNAME}`)
+			.withReuse()
+			.start();
+
+		return {
+			container,
+			meta: {
+				host: HOSTNAME,
+				port: 6379,
+			},
+		};
+	},
+
+	env(result: RedisResult, external?: boolean): Record<string, string> {
+		const host = external ? result.container.getHost() : HOSTNAME;
+		const port = external ? String(result.container.getMappedPort(6379)) : '6379';
+		return {
+			// In container mode, EXECUTIONS_MODE is set by the stack based on worker count.
+			// In external/local mode, redis implies the user wants queue mode.
+			...(external ? { EXECUTIONS_MODE: 'queue' } : {}),
+			QUEUE_BULL_REDIS_HOST: host,
+			QUEUE_BULL_REDIS_PORT: port,
+			N8N_CACHE_ENABLED: 'true',
+			N8N_CACHE_BACKEND: 'redis',
+			N8N_CACHE_REDIS_HOST: host,
+			N8N_CACHE_REDIS_PORT: port,
+		};
+	},
+};

packages/testing/containers/services/registry.ts

@@ -0,0 +1,54 @@
+import { cloudflared } from './cloudflared';
+import { gitea, createGiteaHelper } from './gitea';
+import { kafka, createKafkaHelper } from './kafka';
+import { kent, createKentHelper } from './kent';
+import { keycloak, createKeycloakHelper } from './keycloak';
+import { loadBalancer } from './load-balancer';
+import { localstack, createLocalStackHelper } from './localstack';
+import { mailpit, createMailpitHelper } from './mailpit';
+import { mysqlService } from './mysql';
+import { ngrok } from './ngrok';
+import { createObservabilityHelper } from './observability';
+import { postgres } from './postgres';
+import { proxy, createProxyHelper } from './proxy';
+import { redis } from './redis';
+import { taskRunner } from './task-runner';
+import { tracing, createTracingHelper } from './tracing';
+import type { Service, ServiceName, ServiceResult, HelperFactories } from './types';
+import { vector } from './vector';
+import { victoriaLogs } from './victoria-logs';
+import { victoriaMetrics } from './victoria-metrics';
+
+/** Service registry - must include all ServiceName entries */
+export const services: Record<ServiceName, Service<ServiceResult>> = {
+	postgres,
+	redis,
+	mailpit,
+	gitea,
+	keycloak,
+	victoriaLogs,
+	victoriaMetrics,
+	vector,
+	tracing,
+	proxy,
+	taskRunner,
+	loadBalancer,
+	cloudflared,
+	ngrok,
+	kafka,
+	mysql: mysqlService,
+	localstack,
+	kent,
+};
+
+export const helperFactories: Partial<HelperFactories> = {
+	mailpit: createMailpitHelper,
+	gitea: createGiteaHelper,
+	keycloak: createKeycloakHelper,
+	observability: createObservabilityHelper,
+	tracing: createTracingHelper,
+	proxy: createProxyHelper,
+	kafka: createKafkaHelper,
+	localstack: createLocalStackHelper,
+	kent: createKentHelper,
+};

packages/testing/containers/services/task-runner.ts

@@ -0,0 +1,71 @@
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { createSilentLogConsumer } from '../helpers/utils';
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import { EXTERNAL_HOST, type Service, type ServiceResult } from './types';
+
+export interface TaskRunnerConfig {
+	taskBrokerUri: string;
+}
+
+export interface TaskRunnerMeta {
+	taskBrokerUri: string;
+}
+
+export type TaskRunnerResult = ServiceResult<TaskRunnerMeta>;
+
+export const taskRunner: Service<TaskRunnerResult> = {
+	description: 'Task Runner',
+	shouldStart: (ctx) => ctx.mains > 0 || ctx.workers > 0,
+
+	getOptions(ctx) {
+		if (ctx.external) {
+			return { taskBrokerUri: `http://${EXTERNAL_HOST}:5679` } as TaskRunnerConfig;
+		}
+		const { workers, mains, projectName } = ctx;
+		const taskBrokerHost =
+			workers > 0
+				? `${projectName}-n8n-worker-1`
+				: mains > 1
+					? `${projectName}-n8n-main-1`
+					: `${projectName}-n8n`;
+		return { taskBrokerUri: `http://${taskBrokerHost}:5679` } as TaskRunnerConfig;
+	},
+
+	async start(network, projectName, config?: unknown): Promise<TaskRunnerResult> {
+		const { taskBrokerUri } = config as TaskRunnerConfig;
+		const { consumer, throwWithLogs } = createSilentLogConsumer();
+
+		try {
+			const container = await new GenericContainer(TEST_CONTAINER_IMAGES.taskRunner)
+				.withNetwork(network)
+				.withNetworkAliases(`${projectName}-task-runner`)
+				.withExposedPorts(5680)
+				.withEnvironment({
+					N8N_RUNNERS_AUTH_TOKEN: 'test',
+					N8N_RUNNERS_LAUNCHER_LOG_LEVEL: 'debug',
+					N8N_RUNNERS_TASK_BROKER_URI: taskBrokerUri,
+					N8N_RUNNERS_MAX_CONCURRENCY: '5',
+					N8N_RUNNERS_AUTO_SHUTDOWN_TIMEOUT: '0', // Disabled in tests to prevent cold-start delays
+				})
+				.withWaitStrategy(Wait.forListeningPorts())
+				.withLabels({
+					'com.docker.compose.project': projectName,
+					'com.docker.compose.service': 'task-runner',
+				})
+				.withName(`${projectName}-task-runner`)
+				.withReuse()
+				.withLogConsumer(consumer)
+				.start();
+
+			return {
+				container,
+				meta: {
+					taskBrokerUri,
+				},
+			};
+		} catch (error) {
+			return throwWithLogs(error);
+		}
+	},
+};

packages/testing/containers/services/tracing.ts

@@ -0,0 +1,167 @@
+import type { StartedNetwork, StartedTestContainer } from 'testcontainers';
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { HelperContext, Service, ServiceResult } from './types';
+
+const JAEGER_OTLP_PORT = 4318;
+const JAEGER_UI_PORT = 16686;
+const N8N_TRACER_INGEST_PORT = 8889;
+const N8N_TRACER_HEALTH_PORT = 8888;
+const JAEGER_HOSTNAME = 'jaeger';
+const N8N_TRACER_HOSTNAME = 'n8n-tracer';
+
+export interface TracingConfig {
+	deploymentMode?: 'scaling';
+}
+
+export interface TracingMeta {
+	jaeger: {
+		uiUrl: string;
+		internalOtlpEndpoint: string;
+	};
+	tracer: {
+		internalIngestEndpoint: string;
+		ingestUrl: string;
+	};
+}
+
+export type TracingResult = ServiceResult<TracingMeta> & {
+	containers: StartedTestContainer[];
+};
+
+export interface TracerWebhookConfig {
+	url: string;
+	method: 'POST';
+	label: string;
+	subscribedEvents: string[];
+}
+
+export const tracing: Service<TracingResult> = {
+	description: 'Tracing stack (Jaeger + n8n-tracer)',
+
+	async start(
+		network: StartedNetwork,
+		projectName: string,
+		config?: unknown,
+	): Promise<TracingResult> {
+		const { deploymentMode = 'scaling' } = (config as TracingConfig) ?? {};
+
+		// Start Jaeger first (OTLP receiver)
+		const jaegerContainer = await new GenericContainer(TEST_CONTAINER_IMAGES.jaeger)
+			.withName(`${projectName}-jaeger`)
+			.withNetwork(network)
+			.withNetworkAliases(JAEGER_HOSTNAME)
+			.withLabels({
+				'com.docker.compose.project': projectName,
+				'com.docker.compose.service': 'jaeger',
+			})
+			.withExposedPorts(JAEGER_UI_PORT, JAEGER_OTLP_PORT)
+			.withEnvironment({
+				COLLECTOR_OTLP_ENABLED: 'true',
+				COLLECTOR_OTLP_HTTP_HOST_PORT: '0.0.0.0:4318',
+			})
+			.withWaitStrategy(
+				Wait.forHttp('/', JAEGER_UI_PORT).forStatusCode(200).withStartupTimeout(60000),
+			)
+			.withReuse()
+			.start();
+
+		const jaegerUiPort = jaegerContainer.getMappedPort(JAEGER_UI_PORT);
+		const internalOtlpEndpoint = `http://${JAEGER_HOSTNAME}:${JAEGER_OTLP_PORT}`;
+
+		// Start n8n-tracer pointing to Jaeger
+		const tracerContainer = await new GenericContainer(TEST_CONTAINER_IMAGES.n8nTracer)
+			.withName(`${projectName}-n8n-tracer`)
+			.withNetwork(network)
+			.withNetworkAliases(N8N_TRACER_HOSTNAME)
+			.withLabels({
+				'com.docker.compose.project': projectName,
+				'com.docker.compose.service': 'n8n-tracer',
+			})
+			.withExposedPorts(N8N_TRACER_INGEST_PORT, N8N_TRACER_HEALTH_PORT)
+			.withEnvironment({
+				N8N_DEPLOYMENT_MODE: deploymentMode,
+				OTEL_EXPORTER_OTLP_ENDPOINT: internalOtlpEndpoint,
+				HTTP_INGEST_PORT: String(N8N_TRACER_INGEST_PORT),
+				HEALTH_PORT: String(N8N_TRACER_HEALTH_PORT),
+			})
+			.withWaitStrategy(
+				Wait.forHttp('/health', N8N_TRACER_HEALTH_PORT)
+					.forStatusCode(200)
+					.withStartupTimeout(60000),
+			)
+			.withReuse()
+			.start();
+
+		const internalIngestEndpoint = `http://${N8N_TRACER_HOSTNAME}:${N8N_TRACER_INGEST_PORT}`;
+
+		return {
+			container: jaegerContainer, // Primary container
+			containers: [jaegerContainer, tracerContainer],
+			meta: {
+				jaeger: {
+					uiUrl: `http://localhost:${jaegerUiPort}`,
+					internalOtlpEndpoint,
+				},
+				tracer: {
+					internalIngestEndpoint,
+					ingestUrl: `${internalIngestEndpoint}/ingest`,
+				},
+			},
+		};
+	},
+
+	env(): Record<string, string> {
+		return {
+			N8N_LOG_OUTPUT: 'console',
+		};
+	},
+};
+
+export class TracingHelper {
+	private readonly meta: TracingMeta;
+
+	constructor(meta: TracingMeta) {
+		this.meta = meta;
+	}
+
+	get jaegerUiUrl(): string {
+		return this.meta.jaeger.uiUrl;
+	}
+
+	get internalOtlpEndpoint(): string {
+		return this.meta.jaeger.internalOtlpEndpoint;
+	}
+
+	get internalIngestEndpoint(): string {
+		return this.meta.tracer.internalIngestEndpoint;
+	}
+
+	get ingestUrl(): string {
+		return this.meta.tracer.ingestUrl;
+	}
+
+	getWebhookConfig(label = 'n8n-tracer'): TracerWebhookConfig {
+		return {
+			url: this.meta.tracer.ingestUrl,
+			method: 'POST',
+			label,
+			subscribedEvents: ['*'],
+		};
+	}
+}
+
+export function createTracingHelper(ctx: HelperContext): TracingHelper {
+	const result = ctx.serviceResults.tracing as TracingResult | undefined;
+	if (!result) {
+		throw new Error('Tracing service not found in context');
+	}
+	return new TracingHelper(result.meta);
+}
+
+declare module './types' {
+	interface ServiceHelpers {
+		tracing: TracingHelper;
+	}
+}

packages/testing/containers/services/types.ts

@@ -0,0 +1,113 @@
+import type { StartedTestContainer, StartedNetwork } from 'testcontainers';
+
+/** Hostname that containers use to reach the host machine (Docker Desktop built-in) */
+export const EXTERNAL_HOST = 'host.docker.internal';
+
+export const SERVICE_NAMES = [
+	'postgres',
+	'redis',
+	'mailpit',
+	'gitea',
+	'keycloak',
+	'victoriaLogs',
+	'victoriaMetrics',
+	'vector',
+	'tracing',
+	'proxy',
+	'taskRunner',
+	'loadBalancer',
+	'cloudflared',
+	'kafka',
+	'ngrok',
+	'mysql',
+	'localstack',
+	'kent',
+] as const;
+
+export type ServiceName = (typeof SERVICE_NAMES)[number];
+
+export interface FileToMount {
+	content: string;
+	target: string;
+}
+
+export interface ServiceMeta {
+	/**
+	 * Files to mount into n8n containers. Use when n8n needs files that can't
+	 * be passed via environment (e.g., NODE_EXTRA_CA_CERTS requires a file path).
+	 * See keycloak.ts for usage example.
+	 */
+	n8nFilesToMount?: FileToMount[];
+}
+
+export interface ServiceResult<TMeta = unknown> {
+	container: StartedTestContainer;
+	meta: TMeta;
+}
+
+export interface StartContext {
+	config: StackConfig;
+	projectName: string;
+	mains: number;
+	workers: number;
+	isQueueMode: boolean;
+	usePostgres: boolean;
+	needsLoadBalancer: boolean;
+	/** When true, services should target host.testcontainers.internal instead of Docker-internal hostnames */
+	external: boolean;
+	environment: Record<string, string>;
+	serviceResults: Partial<Record<ServiceName, ServiceResult>>;
+	allocatedPorts: { main?: number; loadBalancer?: number };
+	baseUrl?: string;
+}
+
+export interface StackConfig {
+	mains?: number;
+	workers?: number;
+	postgres?: boolean;
+	env?: Record<string, string>;
+	projectName?: string;
+	resourceQuota?: { memory?: number; cpu?: number };
+	services?: readonly ServiceName[];
+	/** When true, services target host machine instead of Docker-internal n8n */
+	external?: boolean;
+}
+
+export interface Service<TResult extends ServiceResult = ServiceResult> {
+	/** @example 'Redis' */
+	readonly description: string;
+	/** @example ['victoriaLogs'] // vector depends on victoriaLogs */
+	readonly dependsOn?: readonly ServiceName[];
+	/** @example (ctx) => ctx.isQueueMode // redis auto-starts in queue mode */
+	shouldStart?(ctx: StartContext): boolean;
+	/** @example (ctx) => ({ taskBrokerUri: `http://${ctx.projectName}-n8n:5679` }) */
+	getOptions?(ctx: StartContext): unknown;
+	/** Starts container, returns connection details for env() */
+	start(
+		network: StartedNetwork,
+		projectName: string,
+		options?: unknown,
+		ctx?: StartContext,
+	): Promise<TResult>;
+	/** @param external When true, returns host-compatible values using mapped ports (for local dev) */
+	env?(result: TResult, external?: boolean): Record<string, string>;
+	/** @param external When true, returns host-compatible values using mapped ports (for local dev) */
+	extraEnv?(result: TResult, external?: boolean): Record<string, string>;
+	/** Verifies service is reachable from inside n8n containers */
+	verifyFromN8n?(result: TResult, n8nContainers: StartedTestContainer[]): Promise<void>;
+}
+
+export interface HelperContext {
+	containers: StartedTestContainer[];
+	findContainer(pattern: RegExp): StartedTestContainer | undefined;
+	serviceResults: Partial<Record<ServiceName, ServiceResult>>;
+}
+
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type
+export interface ServiceHelpers {}
+
+export type HelperFactory<T> = (ctx: HelperContext) => T;
+
+export type HelperFactories = {
+	[K in keyof ServiceHelpers]: HelperFactory<ServiceHelpers[K]>;
+};

packages/testing/containers/services/vector.ts

@@ -0,0 +1,99 @@
+import type { StartedNetwork } from 'testcontainers';
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { Service, ServiceResult, StartContext } from './types';
+import type { VictoriaLogsResult } from './victoria-logs';
+
+const VICTORIA_LOGS_HOSTNAME = 'victoria-logs';
+const VICTORIA_LOGS_HTTP_PORT = 9428;
+
+function generateVectorConfig(projectName: string, victoriaLogsEndpoint: string): string {
+	return `
+# Disable healthcheck to allow Vector to start while Docker socket becomes available
+[healthchecks]
+enabled = false
+
+[sources.docker_logs]
+type = "docker_logs"
+include_labels = ["com.docker.compose.project=${projectName}"]
+
+[transforms.format_for_victorialogs]
+type = "remap"
+inputs = ["docker_logs"]
+source = '''
+._msg = .message
+._time = .timestamp
+.project = "${projectName}"
+.service = .label."com.docker.compose.service" || "unknown"
+.container = .container_name || "unknown"
+._stream = .stream || "unknown"
+del(.message)
+del(.timestamp)
+del(.label)
+del(.source_type)
+del(.stream)
+'''
+
+[sinks.victoria_logs]
+type = "http"
+inputs = ["format_for_victorialogs"]
+uri = "${victoriaLogsEndpoint}/insert/jsonline"
+method = "post"
+framing.method = "newline_delimited"
+encoding.codec = "json"
+`;
+}
+
+export type VectorResult = ServiceResult<Record<string, never>>;
+
+export const vector: Service<VectorResult> = {
+	description: 'Vector log collector',
+	dependsOn: ['victoriaLogs'],
+
+	async start(
+		network: StartedNetwork,
+		projectName: string,
+		_config?: unknown,
+		ctx?: StartContext,
+	): Promise<VectorResult> {
+		// Get the VictoriaLogs internal endpoint from the already-started service
+		const victoriaLogsResult = ctx?.serviceResults.victoriaLogs as VictoriaLogsResult | undefined;
+		const logsInternalEndpoint =
+			victoriaLogsResult?.meta.internalEndpoint ??
+			`http://${VICTORIA_LOGS_HOSTNAME}:${VICTORIA_LOGS_HTTP_PORT}`;
+
+		const vectorConfig = generateVectorConfig(projectName, logsInternalEndpoint);
+
+		const container = await new GenericContainer(TEST_CONTAINER_IMAGES.vector)
+			.withName(`${projectName}-vector`)
+			.withNetwork(network)
+			.withNetworkAliases('vector')
+			.withLabels({
+				'com.docker.compose.project': projectName,
+				'com.docker.compose.service': 'vector',
+			})
+			.withBindMounts([
+				{
+					source: '/var/run/docker.sock',
+					target: '/var/run/docker.sock',
+					mode: 'ro',
+				},
+			])
+			.withCopyContentToContainer([
+				{
+					content: vectorConfig,
+					target: '/etc/vector/vector.toml',
+				},
+			])
+			.withCommand(['--config', '/etc/vector/vector.toml'])
+			.withWaitStrategy(Wait.forLogMessage(/Vector has started/, 1).withStartupTimeout(60000))
+			.withReuse()
+			.start();
+
+		return {
+			container,
+			meta: {},
+		};
+	},
+};

packages/testing/containers/services/victoria-logs.ts

@@ -0,0 +1,156 @@
+import type { StartedNetwork } from 'testcontainers';
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { HelperContext, Service, ServiceResult } from './types';
+
+const VICTORIA_LOGS_HTTP_PORT = 9428;
+const VICTORIA_LOGS_SYSLOG_PORT = 514;
+const VICTORIA_LOGS_HOSTNAME = 'victoria-logs';
+const SYSLOG_FACILITY_LOCAL0 = 16; // RFC 5424
+
+export interface VictoriaLogsMeta {
+	queryEndpoint: string;
+	internalEndpoint: string;
+	syslog: {
+		host: string;
+		port: number;
+		protocol: 'tcp' | 'udp';
+		facility: number;
+		appName: string;
+	};
+}
+
+export type VictoriaLogsResult = ServiceResult<VictoriaLogsMeta>;
+
+export const victoriaLogs: Service<VictoriaLogsResult> = {
+	description: 'VictoriaLogs',
+
+	async start(network: StartedNetwork, projectName: string): Promise<VictoriaLogsResult> {
+		const container = await new GenericContainer(TEST_CONTAINER_IMAGES.victoriaLogs)
+			.withName(`${projectName}-victoria-logs`)
+			.withNetwork(network)
+			.withNetworkAliases(VICTORIA_LOGS_HOSTNAME)
+			.withLabels({
+				'com.docker.compose.project': projectName,
+				'com.docker.compose.service': 'victoria-logs',
+			})
+			.withExposedPorts(VICTORIA_LOGS_HTTP_PORT, VICTORIA_LOGS_SYSLOG_PORT)
+			.withCommand([
+				'-storageDataPath=/victoria-logs-data',
+				'-retentionPeriod=1d',
+				`-syslog.listenAddr.tcp=:${VICTORIA_LOGS_SYSLOG_PORT}`,
+			])
+			.withWaitStrategy(
+				Wait.forHttp('/health', VICTORIA_LOGS_HTTP_PORT)
+					.forStatusCode(200)
+					.withStartupTimeout(60000),
+			)
+			.withReuse()
+			.start();
+
+		const httpPort = container.getMappedPort(VICTORIA_LOGS_HTTP_PORT);
+
+		return {
+			container,
+			meta: {
+				queryEndpoint: `http://localhost:${httpPort}`,
+				internalEndpoint: `http://${VICTORIA_LOGS_HOSTNAME}:${VICTORIA_LOGS_HTTP_PORT}`,
+				syslog: {
+					host: VICTORIA_LOGS_HOSTNAME,
+					port: VICTORIA_LOGS_SYSLOG_PORT,
+					protocol: 'tcp',
+					facility: SYSLOG_FACILITY_LOCAL0,
+					appName: 'n8n',
+				},
+			},
+		};
+	},
+
+	env(): Record<string, string> {
+		return {
+			N8N_LOG_OUTPUT: 'console',
+		};
+	},
+};
+
+export interface LogEntry {
+	_time: string;
+	_msg: string;
+	message: string;
+	[key: string]: string | undefined;
+}
+
+export interface LogQueryOptions {
+	limit?: number;
+	start?: string;
+	end?: string;
+	timeoutMs?: number;
+	intervalMs?: number;
+}
+
+export class LogsHelper {
+	constructor(private readonly endpoint: string) {}
+
+	async exportAll(options: LogQueryOptions = {}): Promise<string> {
+		const logs = await this.query('*', { limit: 10000, ...options });
+		return logs.map((log) => JSON.stringify(log)).join('\n');
+	}
+
+	async query(query: string, options: LogQueryOptions = {}): Promise<LogEntry[]> {
+		const params = new URLSearchParams({ query });
+		if (options.limit) params.set('limit', String(options.limit));
+		if (options.start) params.set('start', options.start);
+		if (options.end) params.set('end', options.end);
+
+		const response = await fetch(`${this.endpoint}/select/logsql/query?${params}`);
+		if (!response.ok) {
+			throw new Error(`VictoriaLogs query failed: ${response.status}`);
+		}
+
+		const text = await response.text();
+		if (!text.trim()) return [];
+
+		return text
+			.trim()
+			.split('\n')
+			.filter(Boolean)
+			.map((line) => {
+				try {
+					const entry = JSON.parse(line) as LogEntry;
+					entry.message = entry._msg;
+					return entry;
+				} catch {
+					throw new Error(`Failed to parse VictoriaLogs line: ${line}`);
+				}
+			});
+	}
+
+	async waitForLog(query: string, options: LogQueryOptions = {}): Promise<LogEntry | null> {
+		const { setTimeout: wait } = await import('node:timers/promises');
+		const deadline = Date.now() + (options.timeoutMs ?? 30000);
+		const interval = options.intervalMs ?? 1000;
+
+		while (Date.now() < deadline) {
+			const logs = await this.query(query, options);
+			if (logs.length > 0) return logs[0];
+			await wait(interval);
+		}
+		return null;
+	}
+}
+
+export function createLogsHelper(ctx: HelperContext): LogsHelper {
+	const result = ctx.serviceResults.victoriaLogs as VictoriaLogsResult | undefined;
+	if (!result) {
+		throw new Error('VictoriaLogs service not found in context');
+	}
+	return new LogsHelper(result.meta.queryEndpoint);
+}
+
+/**
+ * Escape special characters in LogsQL queries.
+ */
+export function escapeLogsQL(str: string): string {
+	return str.replace(/["\\]/g, '\\$&');
+}

packages/testing/containers/services/victoria-metrics.ts

@@ -0,0 +1,223 @@
+import type { StartedNetwork } from 'testcontainers';
+import { GenericContainer, Wait } from 'testcontainers';
+
+import { TEST_CONTAINER_IMAGES } from '../test-containers';
+import type { HelperContext, Service, ServiceResult, StartContext } from './types';
+
+const VICTORIA_METRICS_HTTP_PORT = 8428;
+const VICTORIA_METRICS_HOSTNAME = 'victoria-metrics';
+
+export interface ScrapeTarget {
+	job: string;
+	instance: string;
+	host: string;
+	port: number;
+}
+
+export interface VictoriaMetricsConfig {
+	scrapeTargets: ScrapeTarget[];
+}
+
+export interface VictoriaMetricsMeta {
+	queryEndpoint: string;
+	internalEndpoint: string;
+}
+
+export type VictoriaMetricsResult = ServiceResult<VictoriaMetricsMeta>;
+
+function generateScrapeConfig(targets: ScrapeTarget[]): string {
+	const jobGroups = new Map<string, ScrapeTarget[]>();
+	for (const target of targets) {
+		const existing = jobGroups.get(target.job) ?? [];
+		existing.push(target);
+		jobGroups.set(target.job, existing);
+	}
+
+	const scrapeConfigs: string[] = [];
+	for (const [jobName, jobTargets] of jobGroups) {
+		const targetConfigs = jobTargets
+			.map(
+				(t) => `      - targets: ['${t.host}:${t.port}']
+        labels:
+          instance: '${t.instance}'`,
+			)
+			.join('\n');
+
+		scrapeConfigs.push(`  - job_name: '${jobName}'
+    static_configs:
+${targetConfigs}
+    metrics_path: '/metrics'
+    scrape_interval: '5s'`);
+	}
+
+	return `
+global:
+  scrape_interval: 15s
+
+scrape_configs:
+${scrapeConfigs.join('\n')}
+`;
+}
+
+export const victoriaMetrics: Service<VictoriaMetricsResult> = {
+	description: 'VictoriaMetrics',
+
+	getOptions(ctx: StartContext): VictoriaMetricsConfig {
+		const { mains, workers, projectName } = ctx;
+		const scrapeTargets: ScrapeTarget[] = [];
+
+		for (let i = 1; i <= mains; i++) {
+			const hostname = mains > 1 ? `${projectName}-n8n-main-${i}` : `${projectName}-n8n`;
+			scrapeTargets.push({
+				job: 'n8n-main',
+				instance: `n8n-main-${i}`,
+				host: hostname,
+				port: 5678,
+			});
+		}
+		for (let i = 1; i <= workers; i++) {
+			scrapeTargets.push({
+				job: 'n8n-worker',
+				instance: `n8n-worker-${i}`,
+				host: `${projectName}-n8n-worker-${i}`,
+				port: 5678,
+			});
+		}
+
+		return { scrapeTargets };
+	},
+
+	async start(
+		network: StartedNetwork,
+		projectName: string,
+		config?: unknown,
+	): Promise<VictoriaMetricsResult> {
+		const { scrapeTargets = [] } = (config as VictoriaMetricsConfig) ?? {};
+		const scrapeConfig = generateScrapeConfig(scrapeTargets);
+
+		const container = await new GenericContainer(TEST_CONTAINER_IMAGES.victoriaMetrics)
+			.withName(`${projectName}-victoria-metrics`)
+			.withNetwork(network)
+			.withNetworkAliases(VICTORIA_METRICS_HOSTNAME)
+			.withLabels({
+				'com.docker.compose.project': projectName,
+				'com.docker.compose.service': 'victoria-metrics',
+			})
+			.withExposedPorts(VICTORIA_METRICS_HTTP_PORT)
+			.withCommand([
+				'-storageDataPath=/victoria-metrics-data',
+				'-retentionPeriod=1d',
+				'-promscrape.config=/etc/prometheus/prometheus.yml',
+			])
+			.withCopyContentToContainer([
+				{
+					content: scrapeConfig,
+					target: '/etc/prometheus/prometheus.yml',
+				},
+			])
+			.withWaitStrategy(
+				Wait.forHttp('/health', VICTORIA_METRICS_HTTP_PORT)
+					.forStatusCode(200)
+					.withStartupTimeout(60000),
+			)
+			.withReuse()
+			.start();
+
+		const httpPort = container.getMappedPort(VICTORIA_METRICS_HTTP_PORT);
+
+		return {
+			container,
+			meta: {
+				queryEndpoint: `http://localhost:${httpPort}`,
+				internalEndpoint: `http://${VICTORIA_METRICS_HOSTNAME}:${VICTORIA_METRICS_HTTP_PORT}`,
+			},
+		};
+	},
+
+	env(): Record<string, string> {
+		return {
+			N8N_METRICS_ENABLED: 'true',
+		};
+	},
+};
+
+export interface MetricResult {
+	labels: Record<string, string>;
+	value: number;
+}
+
+export interface WaitForMetricOptions {
+	timeoutMs?: number;
+	intervalMs?: number;
+	predicate?: (values: MetricResult[]) => boolean;
+}
+
+export class MetricsHelper {
+	constructor(private readonly endpoint: string) {}
+
+	async exportAll(options: { start?: string; end?: string } = {}): Promise<string> {
+		const params = new URLSearchParams({
+			'match[]': '{__name__=~".+"}',
+		});
+		if (options.start) params.set('start', options.start);
+		if (options.end) params.set('end', options.end);
+
+		const response = await fetch(`${this.endpoint}/api/v1/export?${params}`);
+		if (!response.ok) {
+			throw new Error(`VictoriaMetrics export failed: ${response.status}`);
+		}
+
+		return await response.text();
+	}
+
+	async query(query: string): Promise<MetricResult[]> {
+		const response = await fetch(`${this.endpoint}/api/v1/query?${new URLSearchParams({ query })}`);
+		if (!response.ok) {
+			throw new Error(`VictoriaMetrics query failed: ${response.status}`);
+		}
+
+		const data = (await response.json()) as {
+			status: string;
+			data?: { result: Array<{ metric: Record<string, string>; value: [number, string] }> };
+			error?: string;
+		};
+
+		if (data.status !== 'success') {
+			throw new Error(`VictoriaMetrics error: ${data.error}`);
+		}
+
+		return (data.data?.result ?? []).map((r) => ({
+			labels: r.metric,
+			value: parseFloat(r.value[1]),
+		}));
+	}
+
+	async waitForMetric(
+		query: string,
+		options: WaitForMetricOptions = {},
+	): Promise<MetricResult | null> {
+		const { setTimeout: wait } = await import('node:timers/promises');
+		const deadline = Date.now() + (options.timeoutMs ?? 30000);
+		const interval = options.intervalMs ?? 1000;
+		const predicate = options.predicate ?? ((v) => v.length > 0);
+
+		while (Date.now() < deadline) {
+			try {
+				const values = await this.query(query);
+				if (predicate(values)) return values[0] ?? null;
+			} catch {
+				// Ignore transient errors during polling
+			}
+			await wait(interval);
+		}
+		return null;
+	}
+}
+
+export function createMetricsHelper(ctx: HelperContext): MetricsHelper {
+	const result = ctx.serviceResults.victoriaMetrics as VictoriaMetricsResult | undefined;
+	if (!result) {
+		throw new Error('VictoriaMetrics service not found in context');
+	}
+	return new MetricsHelper(result.meta.queryEndpoint);
+}

packages/testing/containers/stack.ts

@@ -0,0 +1,372 @@
+import getPort from 'get-port';
+import type { StartedNetwork, StartedTestContainer, StoppedTestContainer } from 'testcontainers';
+import { Network } from 'testcontainers';
+
+import { createElapsedLogger, pollContainerHttpEndpoint } from './helpers/utils';
+import { waitForNetworkQuiet } from './network-stabilization';
+import type { LoadBalancerResult } from './services/load-balancer';
+import { createN8NInstances } from './services/n8n';
+import { helperFactories, services } from './services/registry';
+import type {
+	FileToMount,
+	HelperContext,
+	HelperFactories,
+	Service,
+	ServiceHelpers,
+	ServiceName,
+	ServiceResult,
+	StackConfig,
+	StartContext,
+} from './services/types';
+import { createTelemetryRecorder } from './telemetry';
+
+const SERVICE_REGISTRY: Record<ServiceName, Service> = services;
+
+export type N8NConfig = StackConfig;
+
+export interface N8NStack {
+	baseUrl: string;
+	projectName: string;
+	stop: () => Promise<void>;
+	containers: StartedTestContainer[];
+	serviceResults: Partial<Record<ServiceName, ServiceResult>>;
+	services: ServiceHelpers;
+	logs: ServiceHelpers['observability']['logs'];
+	metrics: ServiceHelpers['observability']['metrics'];
+	findContainers: (namePattern: string | RegExp) => StartedTestContainer[];
+	stopContainer: (namePattern: string | RegExp) => Promise<StoppedTestContainer | null>;
+	/** Direct URLs to each main instance (bypasses load balancer). Index 0 = main-1, etc. */
+	mainUrls: string[];
+}
+
+function shouldServiceStart(name: ServiceName, service: Service, ctx: StartContext): boolean {
+	// Explicitly requested services always start
+	if (ctx.config.services?.includes(name)) return true;
+	if (service.shouldStart) {
+		return service.shouldStart(ctx);
+	}
+	return false;
+}
+
+function groupByDependencyLevel(serviceNames: ServiceName[]): ServiceName[][] {
+	const levels: ServiceName[][] = [];
+	const assigned = new Set<ServiceName>();
+
+	while (assigned.size < serviceNames.length) {
+		const currentLevel: ServiceName[] = [];
+		for (const name of serviceNames) {
+			if (assigned.has(name)) continue;
+			const service = SERVICE_REGISTRY[name];
+			const deps = service?.dependsOn ?? [];
+			if (deps.every((dep) => !serviceNames.includes(dep) || assigned.has(dep))) {
+				currentLevel.push(name);
+			}
+		}
+		if (currentLevel.length === 0) {
+			throw new Error('Circular dependency detected in services');
+		}
+		levels.push(currentLevel);
+		currentLevel.forEach((name) => assigned.add(name));
+	}
+
+	return levels;
+}
+
+export async function createN8NStack(config: N8NConfig = {}): Promise<N8NStack> {
+	const {
+		mains = 1,
+		workers = 0,
+		postgres: usePostgresConfig = false,
+		env = {},
+		projectName,
+		resourceQuota,
+		services: enabledServices = [],
+		external = false,
+	} = config;
+
+	const log = createElapsedLogger('stack');
+
+	const isQueueMode = mains > 1 || workers > 0;
+	const needsLoadBalancer = mains > 1;
+	const usePostgres = usePostgresConfig || isQueueMode || enabledServices.includes('keycloak');
+	const uniqueProjectName = projectName ?? `n8n-stack-${Math.random().toString(36).substring(7)}`;
+
+	let allocatedMainPort: number | undefined;
+	let allocatedLbPort: number | undefined;
+
+	if (needsLoadBalancer) {
+		allocatedLbPort = await getPort();
+	} else {
+		allocatedMainPort = await getPort();
+	}
+
+	const containers: StartedTestContainer[] = [];
+	const serviceResults: Record<string, ServiceResult> = {};
+	let environment: Record<string, string> = {};
+
+	log(`Starting: ${uniqueProjectName}`);
+
+	const telemetry = createTelemetryRecorder(config);
+
+	let network: StartedNetwork;
+	try {
+		const networkStart = performance.now();
+		network = await new Network().start();
+		telemetry.recordNetwork(Math.round(performance.now() - networkStart));
+	} catch (error) {
+		const message = error instanceof Error ? error.message : String(error);
+		telemetry.flush(false, `Network creation failed: ${message}`);
+		throw error;
+	}
+
+	try {
+		const ctx: StartContext = {
+			config: { ...config, postgres: usePostgres },
+			projectName: uniqueProjectName,
+			mains,
+			workers,
+			isQueueMode,
+			usePostgres,
+			needsLoadBalancer,
+			external,
+			environment,
+			serviceResults,
+			allocatedPorts: {
+				main: allocatedMainPort,
+				loadBalancer: allocatedLbPort,
+			},
+		};
+
+		// Step 1: Start services (parallel by dependency level)
+		const allServiceNames = Object.keys(SERVICE_REGISTRY) as ServiceName[];
+		const servicesToStart = allServiceNames.filter((name) =>
+			shouldServiceStart(name, SERVICE_REGISTRY[name], ctx),
+		);
+		const dependencyLevels = groupByDependencyLevel(servicesToStart);
+
+		for (const level of dependencyLevels) {
+			const levelNames = level.map((name) => SERVICE_REGISTRY[name].description).join(', ');
+
+			const levelPromises = level.map(async (name) => {
+				const service = SERVICE_REGISTRY[name];
+				const options = service.getOptions?.(ctx);
+				const serviceStart = performance.now();
+				try {
+					const result = await service.start(network, uniqueProjectName, options, ctx);
+					telemetry.recordService(name, Math.round(performance.now() - serviceStart));
+					return { name, service, result };
+				} catch (error) {
+					telemetry.recordService(name, Math.round(performance.now() - serviceStart));
+					const message = error instanceof Error ? error.message : String(error);
+					throw new Error(`Service "${service.description}" (${name}) failed to start: ${message}`);
+				}
+			});
+
+			const results = await Promise.all(levelPromises);
+
+			for (const { name, service, result } of results) {
+				// Some services (e.g., tracing) return multiple containers
+				const serviceContainers =
+					'containers' in result && Array.isArray(result.containers)
+						? (result.containers as StartedTestContainer[])
+						: [result.container];
+				containers.push(...serviceContainers);
+				serviceResults[name] = result;
+
+				if (service.env) {
+					environment = { ...environment, ...service.env(result) };
+				}
+				if (service.extraEnv) {
+					environment = { ...environment, ...service.extraEnv(result) };
+				}
+			}
+
+			ctx.environment = environment;
+			ctx.serviceResults = serviceResults;
+
+			log(`Services ready: ${levelNames}`);
+		}
+
+		// Step 2: Start n8n (main 1 first for DB setup, then rest in parallel)
+		const lbResult = serviceResults.loadBalancer as LoadBalancerResult | undefined;
+		const baseUrl = lbResult?.meta.baseUrl ?? `http://localhost:${allocatedMainPort}`;
+
+		const filesToMount: FileToMount[] = Object.values(serviceResults).flatMap((result) => {
+			const meta = result.meta as { n8nFilesToMount?: FileToMount[] } | undefined;
+			return meta?.n8nFilesToMount ?? [];
+		});
+
+		const n8nStartupStart = performance.now();
+		const n8nResult = await createN8NInstances({
+			mains,
+			workers,
+			projectName: uniqueProjectName,
+			network,
+			serviceEnvironment: environment,
+			userEnvironment: env,
+			usePostgres,
+			baseUrl: needsLoadBalancer ? undefined : baseUrl,
+			allocatedPort: needsLoadBalancer ? undefined : allocatedMainPort,
+			resourceQuota,
+			filesToMount,
+		});
+		containers.push(...n8nResult.containers);
+		telemetry.recordN8nStartup(
+			Math.round(performance.now() - n8nStartupStart),
+			n8nResult.containers.length,
+		);
+		log(`n8n ready: ${mains} main(s), ${workers} worker(s)`);
+
+		if (lbResult) {
+			await pollContainerHttpEndpoint(lbResult.container, '/healthz/readiness');
+			log('Load balancer ready');
+		}
+
+		ctx.baseUrl = baseUrl;
+
+		// Build direct main URLs (bypassing load balancer)
+		const mainUrls: string[] = [];
+		for (let i = 1; i <= mains; i++) {
+			const mainNamePattern = mains > 1 ? `-n8n-main-${i}` : '-n8n';
+			const mainContainer = containers.find((c) => c.getName().endsWith(mainNamePattern));
+			if (mainContainer) {
+				const mainPort = mainContainer.getMappedPort(5678);
+				mainUrls.push(`http://localhost:${mainPort}`);
+			}
+		}
+		log(`Direct main URLs: ${mainUrls.join(', ')}`);
+
+		// Run verification hooks (e.g. keycloak connectivity check)
+		const n8nContainers = containers.filter((c) => {
+			const name = c.getName();
+			return name.includes('-n8n-main-') || name.endsWith('-n8n');
+		});
+
+		const verifications: string[] = [];
+		for (const name of servicesToStart) {
+			const service = SERVICE_REGISTRY[name];
+			if (service.verifyFromN8n && serviceResults[name]) {
+				await service.verifyFromN8n(serviceResults[name], n8nContainers);
+				verifications.push(service.description);
+			}
+		}
+		if (verifications.length > 0) {
+			log(`Verified: ${verifications.join(', ')}`);
+		}
+
+		await waitForNetworkQuiet();
+		telemetry.flush(true);
+
+		const helperCtx: HelperContext = {
+			containers,
+			findContainer: (pattern: RegExp) => containers.find((c) => pattern.test(c.getName())),
+			serviceResults,
+		};
+		const helperCache: Partial<ServiceHelpers> = {};
+
+		const servicesProxy = new Proxy({} as ServiceHelpers, {
+			get: <K extends keyof ServiceHelpers>(
+				_target: ServiceHelpers,
+				prop: K,
+			): ServiceHelpers[K] => {
+				if (prop in helperCache) {
+					return helperCache[prop]!;
+				}
+
+				const factory = (helperFactories as HelperFactories)[prop];
+				if (!factory) {
+					throw new Error(
+						`No helper factory found for service: ${String(prop)}. ` +
+							`Available helpers: ${Object.keys(helperFactories).join(', ')}`,
+					);
+				}
+
+				const helper = factory(helperCtx);
+				helperCache[prop] = helper;
+				return helper;
+			},
+			has: (_target, prop) => prop in helperFactories,
+			ownKeys: () => Object.keys(helperFactories),
+			getOwnPropertyDescriptor: (_target, prop) => {
+				if (prop in helperFactories) {
+					return { enumerable: true, configurable: true };
+				}
+				return undefined;
+			},
+		});
+
+		return {
+			baseUrl,
+			projectName: uniqueProjectName,
+			stop: async () => await stopN8NStack(containers, network, uniqueProjectName),
+			containers,
+			serviceResults,
+			services: servicesProxy,
+			get logs() {
+				return servicesProxy.observability.logs;
+			},
+			get metrics() {
+				return servicesProxy.observability.metrics;
+			},
+			findContainers(namePattern: string | RegExp): StartedTestContainer[] {
+				const regex = typeof namePattern === 'string' ? new RegExp(namePattern) : namePattern;
+				return containers.filter((container) => regex.test(container.getName()));
+			},
+			async stopContainer(namePattern: string | RegExp): Promise<StoppedTestContainer | null> {
+				const regex = typeof namePattern === 'string' ? new RegExp(namePattern) : namePattern;
+				const container = containers.find((c) => regex.test(c.getName()));
+				return container ? await container.stop() : null;
+			},
+			mainUrls,
+		};
+	} catch (error) {
+		const message = error instanceof Error ? error.message : String(error);
+		telemetry.flush(false, message);
+		throw error;
+	}
+}
+
+function getErrorMessage(error: unknown): string {
+	if (error instanceof Error) {
+		return error.message;
+	}
+	return String(error);
+}
+
+async function stopN8NStack(
+	containers: StartedTestContainer[],
+	network: StartedNetwork,
+	uniqueProjectName: string,
+): Promise<void> {
+	const errors: Error[] = [];
+	try {
+		const stopPromises = containers.reverse().map(async (container) => {
+			try {
+				await container.stop();
+			} catch (error) {
+				errors.push(
+					new Error(`Failed to stop container ${container.getId()}: ${getErrorMessage(error)}`),
+				);
+			}
+		});
+		await Promise.allSettled(stopPromises);
+
+		try {
+			await network.stop();
+		} catch (error) {
+			errors.push(
+				new Error(`Failed to stop network ${network.getName()}: ${getErrorMessage(error)}`),
+			);
+		}
+
+		if (errors.length > 0) {
+			console.warn(
+				`Some cleanup operations failed for stack ${uniqueProjectName}:`,
+				errors.map((e) => e.message).join(', '),
+			);
+		}
+	} catch (error) {
+		console.error(`Critical error during cleanup for stack ${uniqueProjectName}:`, error);
+		throw error;
+	}
+}

packages/testing/containers/telemetry.ts

@@ -0,0 +1,239 @@
+import { spawn } from 'child_process';
+import * as os from 'os';
+
+import type { StackConfig } from './services/types';
+
+export interface StackTelemetryRecord {
+	/** ISO timestamp when stack creation started */
+	timestamp: string;
+
+	/** Git context from environment */
+	git: {
+		sha: string;
+		branch: string;
+		pr?: number;
+	};
+
+	/** CI context from GitHub Actions environment */
+	ci: {
+		runId?: string;
+		job?: string;
+		workflow?: string;
+		attempt?: number;
+	};
+
+	/** Runner info - provider detected from env vars, specs from runtime */
+	runner: {
+		provider: 'github' | 'blacksmith' | 'local';
+		cpuCores: number;
+		memoryGb: number;
+	};
+
+	/** Stack configuration */
+	stack: {
+		type: 'single' | 'queue' | 'multi-main';
+		mains: number;
+		workers: number;
+		postgres: boolean;
+		services: string[];
+	};
+
+	/** Timing metrics in milliseconds */
+	timing: {
+		total: number;
+		network: number;
+		n8nStartup: number;
+		services: Record<string, number>;
+	};
+
+	/** Container counts */
+	containers: {
+		total: number;
+		services: number;
+		n8n: number;
+	};
+
+	/** Outcome */
+	success: boolean;
+	errorMessage?: string;
+}
+
+function getGitContext(): StackTelemetryRecord['git'] {
+	const ref = process.env.GITHUB_REF ?? '';
+	const prMatch = ref.match(/refs\/pull\/(\d+)/);
+
+	return {
+		sha: process.env.GITHUB_SHA?.slice(0, 8) ?? 'local',
+		branch: process.env.GITHUB_HEAD_REF ?? process.env.GITHUB_REF_NAME ?? 'local',
+		pr: prMatch ? parseInt(prMatch[1], 10) : undefined,
+	};
+}
+
+function getCIContext(): StackTelemetryRecord['ci'] {
+	return {
+		runId: process.env.GITHUB_RUN_ID,
+		job: process.env.GITHUB_JOB,
+		workflow: process.env.GITHUB_WORKFLOW,
+		attempt: process.env.GITHUB_RUN_ATTEMPT
+			? parseInt(process.env.GITHUB_RUN_ATTEMPT, 10)
+			: undefined,
+	};
+}
+
+function getRunnerProvider(): StackTelemetryRecord['runner']['provider'] {
+	// Not in CI = local development
+	if (!process.env.CI) {
+		return 'local';
+	}
+	// GitHub-hosted runners explicitly identify themselves
+	if (process.env.RUNNER_ENVIRONMENT === 'github-hosted') {
+		return 'github';
+	}
+	// Everything else in CI is Blacksmith
+	return 'blacksmith';
+}
+
+function getRunnerInfo(): StackTelemetryRecord['runner'] {
+	const cpus = os.cpus();
+	return {
+		provider: getRunnerProvider(),
+		cpuCores: cpus.length,
+		memoryGb: Math.round((os.totalmem() / (1024 * 1024 * 1024)) * 10) / 10,
+	};
+}
+
+function inferStackType(config: StackConfig): 'single' | 'queue' | 'multi-main' {
+	if ((config.mains ?? 1) > 1) return 'multi-main';
+	if ((config.workers ?? 0) > 0) return 'queue';
+	return 'single';
+}
+
+function inferPostgres(config: StackConfig): boolean {
+	const isQueueMode = (config.mains ?? 1) > 1 || (config.workers ?? 0) > 0;
+	const hasKeycloak = config.services?.includes('keycloak') ?? false;
+	return (config.postgres ?? false) || isQueueMode || hasKeycloak;
+}
+
+export class TelemetryRecorder {
+	private startTimestamp = Date.now();
+	private startPerf = performance.now();
+	private networkTime = 0;
+	private n8nStartupTime = 0;
+	private serviceTimings: Record<string, number> = {};
+	private serviceCount = 0;
+	private n8nCount = 0;
+
+	constructor(private config: StackConfig) {}
+
+	recordNetwork(durationMs: number): void {
+		this.networkTime = durationMs;
+	}
+
+	recordService(name: string, durationMs: number): void {
+		this.serviceTimings[name] = durationMs;
+		this.serviceCount++;
+	}
+
+	recordN8nStartup(durationMs: number, count: number): void {
+		this.n8nStartupTime = durationMs;
+		this.n8nCount = count;
+	}
+
+	private buildRecord(success: boolean, errorMessage?: string): StackTelemetryRecord {
+		return {
+			timestamp: new Date(this.startTimestamp).toISOString(),
+			git: getGitContext(),
+			ci: getCIContext(),
+			runner: getRunnerInfo(),
+			stack: {
+				type: inferStackType(this.config),
+				mains: this.config.mains ?? 1,
+				workers: this.config.workers ?? 0,
+				postgres: inferPostgres(this.config),
+				services: [...(this.config.services ?? [])],
+			},
+			timing: {
+				total: Math.round(performance.now() - this.startPerf),
+				network: this.networkTime,
+				n8nStartup: this.n8nStartupTime,
+				services: { ...this.serviceTimings },
+			},
+			containers: {
+				total: this.serviceCount + this.n8nCount,
+				services: this.serviceCount,
+				n8n: this.n8nCount,
+			},
+			success,
+			errorMessage,
+		};
+	}
+
+	/**
+	 * Flush telemetry - outputs based on environment configuration
+	 *
+	 * Output modes:
+	 * - CONTAINER_TELEMETRY_WEBHOOK: Send via detached process (non-blocking, survives parent exit)
+	 * - CONTAINER_TELEMETRY_VERBOSE=1: Full breakdown + elapsed logs (via utils.ts)
+	 * - Default: One-liner summary only
+	 */
+	flush(success: boolean, errorMessage?: string): void {
+		const record = this.buildRecord(success, errorMessage);
+
+		const isVerbose = process.env.CONTAINER_TELEMETRY_VERBOSE === '1';
+		const webhookUrl = process.env.CONTAINER_TELEMETRY_WEBHOOK;
+
+		if (webhookUrl) {
+			this.sendToWebhook(record, webhookUrl);
+		}
+
+		if (isVerbose) {
+			console.log(JSON.stringify(record, null, 2));
+		} else {
+			this.printSummaryLine(record);
+		}
+	}
+
+	private printSummaryLine(record: StackTelemetryRecord): void {
+		const time = this.formatMs(record.timing.total);
+		const containers = record.containers.total;
+		const services = record.stack.services.length;
+
+		if (record.success) {
+			const serviceInfo = services > 0 ? `, ${services} services` : '';
+			console.log(`\x1b[32m✓\x1b[0m Stack ready (${time}, ${containers} containers${serviceInfo})`);
+		} else {
+			console.log(`\x1b[31m✗\x1b[0m Stack failed: ${record.errorMessage} (${time})`);
+		}
+	}
+
+	/**
+	 * Send telemetry via a detached child process.
+	 * The process runs independently and survives parent exit, ensuring delivery
+	 * even when the main process throws/exits immediately after flush().
+	 */
+	private sendToWebhook(record: StackTelemetryRecord, webhookUrl: string): void {
+		const payload = JSON.stringify(record);
+		const script = `
+			fetch(process.argv[1], {
+				method: 'POST',
+				headers: { 'Content-Type': 'application/json' },
+				body: process.argv[2]
+			}).catch(() => process.exit(1));
+		`;
+
+		const child = spawn(process.execPath, ['-e', script, webhookUrl, payload], {
+			detached: true,
+			stdio: 'ignore',
+		});
+		child.unref();
+	}
+
+	private formatMs(ms: number): string {
+		if (ms < 1000) return `${ms.toFixed(0)}ms`;
+		return `${(ms / 1000).toFixed(2)}s`;
+	}
+}
+
+export function createTelemetryRecorder(config: StackConfig): TelemetryRecorder {
+	return new TelemetryRecorder(config);
+}

packages/testing/containers/test-containers.ts

@@ -0,0 +1,120 @@
+/**
+ * Single source of truth for all test container images.
+ *
+ * All images can be overridden via environment variables:
+ *   TEST_IMAGE_<KEY> where KEY is the SCREAMING_SNAKE_CASE version of the image key
+ *   e.g., TEST_IMAGE_POSTGRES=postgres:16 overrides the postgres image
+ *   e.g., TEST_IMAGE_N8N=n8nio/n8n:latest overrides the n8n image
+ *   e.g., TEST_IMAGE_TASK_RUNNER=n8nio/runners:latest overrides the task runner image
+ *
+ * For n8n image, shorthand syntax is supported:
+ *   TEST_IMAGE_N8N=stable         → n8nio/n8n:stable
+ *   TEST_IMAGE_N8N=n8n:stable     → n8nio/n8n:stable
+ *   TEST_IMAGE_N8N=n8nio/n8n:stable → n8nio/n8n:stable
+ *
+ * Task runner image derivation:
+ *   When TEST_IMAGE_TASK_RUNNER is not set, the image is derived from the n8n image:
+ *   TEST_IMAGE_N8N=n8nio/n8n:nightly              → taskRunner=n8nio/runners:nightly
+ *   TEST_IMAGE_N8N=ghcr.io/n8n-io/n8n:pr-123      → taskRunner=ghcr.io/n8n-io/runners:pr-123
+ *
+ * N8N_DOCKER_IMAGE is also supported for backwards compatibility.
+ */
+
+/** Default images - override via TEST_IMAGE_<KEY> env vars */
+const DEFAULT_IMAGES = {
+	postgres: 'postgres:18-alpine',
+	redis: 'redis:alpine',
+	caddy: 'caddy:alpine',
+	n8n: 'n8nio/n8n:local',
+	taskRunner: 'n8nio/runners:local',
+	mailpit: 'axllent/mailpit:latest',
+	mockserver: 'mockserver/mockserver:5.15.0',
+	gitea: 'gitea/gitea:1.25.1',
+	keycloak: 'keycloak/keycloak:26.4',
+	victoriaLogs: 'victoriametrics/victoria-logs:v1.21.0-victorialogs',
+	victoriaMetrics: 'victoriametrics/victoria-metrics:v1.115.0',
+	vector: 'timberio/vector:0.52.0-alpine',
+	n8nTracer: 'ghcr.io/ivov/n8n-tracer:0.1.0',
+	jaeger: 'jaegertracing/all-in-one:1.76.0',
+	cloudflared: 'cloudflare/cloudflared:2025.1.1',
+	ngrok: 'ngrok/ngrok:alpine',
+	kafka: 'confluentinc/cp-kafka:8.0.3',
+	mysql: 'mysql:9.6.0',
+	localstack: 'localstack/localstack:4.13.1',
+} as const;
+
+/** Convert camelCase to SCREAMING_SNAKE_CASE for env var names */
+function toEnvVarName(key: string): string {
+	return key.replace(/([a-z])([A-Z])/g, '$1_$2').toUpperCase();
+}
+
+/** Normalize n8n image shorthand: "stable" → "n8nio/n8n:stable" */
+function normalizeN8nImage(image: string): string {
+	if (image.includes('/')) return image;
+	if (image.includes(':')) return `n8nio/${image}`;
+	return `n8nio/n8n:${image}`;
+}
+
+/** Parse "ghcr.io/n8n-io/n8n:pr-123" or "n8nio/n8n:nightly" into components */
+function parseImage(image: string): { registry?: string; org: string; tag: string } {
+	const [imagePath, tag = 'latest'] = image.split(':');
+	const parts = imagePath.split('/');
+
+	if (parts.length === 3) {
+		return { registry: parts[0], org: parts[1], tag };
+	}
+	return { org: parts[0], tag };
+}
+
+/** Derive runners image from n8n image components */
+function buildRunnersImage({ registry, org, tag }: ReturnType<typeof parseImage>): string {
+	return registry ? `${registry}/${org}/runners:${tag}` : `${org}/runners:${tag}`;
+}
+
+let resolvedN8nImage: string | undefined;
+
+/** Get image with TEST_IMAGE_<KEY> env var override support */
+function getImage<K extends keyof typeof DEFAULT_IMAGES>(key: K): string {
+	const envVar = `TEST_IMAGE_${toEnvVarName(key)}`;
+	let value = process.env[envVar];
+
+	if (key === 'n8n' && !value) {
+		value = process.env.N8N_DOCKER_IMAGE;
+	}
+
+	if (key === 'taskRunner' && !value) {
+		resolvedN8nImage ??= getImage('n8n');
+		return buildRunnersImage(parseImage(resolvedN8nImage));
+	}
+
+	value = value ?? DEFAULT_IMAGES[key];
+
+	if (key === 'n8n') {
+		resolvedN8nImage = normalizeN8nImage(value);
+		return resolvedN8nImage;
+	}
+
+	return value;
+}
+
+export const TEST_CONTAINER_IMAGES = {
+	postgres: getImage('postgres'),
+	redis: getImage('redis'),
+	caddy: getImage('caddy'),
+	n8n: getImage('n8n'),
+	taskRunner: getImage('taskRunner'),
+	mailpit: getImage('mailpit'),
+	mockserver: getImage('mockserver'),
+	gitea: getImage('gitea'),
+	keycloak: getImage('keycloak'),
+	victoriaLogs: getImage('victoriaLogs'),
+	victoriaMetrics: getImage('victoriaMetrics'),
+	vector: getImage('vector'),
+	n8nTracer: getImage('n8nTracer'),
+	jaeger: getImage('jaeger'),
+	cloudflared: getImage('cloudflared'),
+	kafka: getImage('kafka'),
+	mysql: getImage('mysql'),
+	ngrok: getImage('ngrok'),
+	localstack: getImage('localstack'),
+} as const;

packages/testing/containers/tsconfig.json

@@ -0,0 +1,12 @@
+{
+	"extends": "../../../tsconfig.json",
+	"compilerOptions": {
+		"sourceMap": false,
+		"declaration": false,
+		"lib": ["esnext", "dom"],
+		"types": ["node"]
+	},
+	"include": ["**/*.ts"],
+	"exclude": ["**/dist/**/*", "**/node_modules/**/*"],
+	"references": [{ "path": "../../workflow/tsconfig.build.esm.json" }]
+}