first commit
Security: Sync from Public / sync-from-public (push) Has been cancelled
Test: Benchmark Nightly / build (push) Has been cancelled
Test: Benchmark Nightly / Notify Cats on failure (push) Has been cancelled
CI: Python / Checks (push) Has been cancelled
Test: Evals Python / Workflow Comparison Python (push) Has been cancelled
Util: Check Docs URLs / check-docs-urls (push) Has been cancelled
Test: Visual Storybook / Cloudflare Pages (push) Has been cancelled
Test: E2E Performance / build-and-test-performance (push) Has been cancelled
Test: Workflows Nightly / Run Workflow Tests (push) Has been cancelled
Util: Cleanup CI Docker Images / Delete stale CI images (push) Has been cancelled
Test: Benchmark Destroy Env / build (push) Has been cancelled
Util: Update Node Popularity / update-popularity (push) Has been cancelled
Test: E2E Coverage Weekly / Coverage Tests (push) Has been cancelled

This commit is contained in:
2026-03-17 16:22:57 +03:30
commit 3d5eaf9445
15349 changed files with 2847338 additions and 0 deletions
@@ -0,0 +1,366 @@
# Backend module
A backend module is a self-contained unit of backend functionality tied to a specific n8n feature.
Benefits of modularity:
- **Organization:** Keep code structured, focused and discoverable
- **Independence**: Reduce conflicts among teams working independently
- **Decoupling**: Prevent features from becoming entangled with internals
- **Simplicity:** Make logic easier to maintain, test and reason about
- **Efficiency**: Load and run only modules that are explicitly enabled
## File structure
To set up a backend module, run this command at monorepo root:
```sh
pnpm setup-backend-module
```
Your new module will be located at `packages/cli/src/modules/my-feature`. Rename `my-feature` in the dirname and in the filenames to your actual feature name. Use kebab-case.
A modules file structure is as follows:
```sh
.
├── __tests__
│   ├── my-feature.controller.test.ts
│   └── my-feature.service.test.ts
├── my-feature.entity.ts # DB model
├── my-feature.repository.ts # DB access
├── my-feature.config.ts # env vars
├── my-feature.constants.ts # constants
├── my-feature.controller.ts # HTTP REST routes
├── my-feature.service.ts # business logic
└── my-feature.module.ts # entrypoint
```
This is only a template - your module may not need all of these files, or it may need more than these and also subdirs to keep things organized. Infixes are currently not required (except for the `.module.ts` entrypoint), but infixes are strongly recommended as they make the contents searchable and discoverable.
Backend modules currently live at `packages/cli/src/modules`, so imports can be:
- from inside the module dir
- from common packages like `@n8n/db`, `@n8n/backend-common`, `@n8n/backend-test-utils`, etc.
- from `cli`
- from third-party libs available in, or added to, `cli`
Modules are managed via env vars:
- To enable a module (activate it on instance startup), use the env var `N8N_ENABLED_MODULES`.
- To disable a module (skip it on instance startup), use the env var `N8N_DISABLED_MODULES`.
- Some modules are **default modules** so they are always enabled unless specifically disabled. To enable a module by default, add it [here](https://github.com/n8n-io/n8n/blob/c0360e52afe9db37d4dd6e00955fa42b0c851904/packages/%40n8n/backend-common/src/modules/module-registry.ts#L26).
Modules that are under a license flag are automatically skipped on startup if the instance is not licensed to use the feature.
## Entrypoint
The `.module.ts` file is the entrypoint of the module and uses the `@BackendModule()` decorator:
```ts
@BackendModule({ name: 'my-feature' })
export class MyFeatureModule implements ModuleInterface {
async init() {
await import('./my-feature.controller');
const { MyFeatureService } = await import('./my-feature.service');
Container.get(MyFeatureService).start();
}
@OnShutdown()
async shutdown() {
const { MyFeatureService } = await import('./my-feature.service');
await Container.get(MyFeatureService).shutdown();
}
async entities() {
const { MyEntity } = await import('./my-feature.entity.ts');
return [MyEntity]
}
async settings() {
const { MyFeatureService } = await import('./my-feature.service');
return Container.get(MyFeatureService).settings();
}
async context() {
const { MyFeatureService } = await import('./my-feature.service');
return { myFeatureProxy: Container.get(MyFeatureService) };
}
}
```
The entrypoint is responsible for providing:
- **initialization logic**, e.g. in insights, start compaction timers,
- **shutdown logic**, e.g. in insights, stop compaction timers,
- **database entities** to register with `typeorm`, e.g. in insights, the three database entities `InsightsMetadata`, `InsightsByPeriod` and `InsightsRaw`
- **settings** to send to the client for adjusting the UI, e.g. in insights, `{ summary: true, dashboard: false }`
- **context** to merge an object into the workflow execution context `WorkflowExecuteAdditionalData`. This allows you to make module functionality available to `core`, namespaced under the module name. For now, you will also need to manually [update the type](https://github.com/n8n-io/n8n/blob/master/packages/core/src/execution-engine/index.ts#L7) of `WorkflowExecuteAdditionalData` to reflect the resulting context.
A module entrypoint may or may not need to implement all of these methods.
Why do we use dynamic imports in entrypoint methods? `await import('...')` ensures we load module-specific logic **only when needed**, so that n8n instances which do not have a module enabled, or do not have licensed access to it, do not pay for the performance cost. Linting enforces that relative imports in the entrypoint use dynamic imports. Loading on demand is also the reason why the entrypoint does not use dependency injection, and forbids it via linting.
A module may be fully behind a license flag:
```ts
@BackendModule({
name: 'external-secrets',
licenseFlag: 'feat:externalSecrets'
})
export class ExternalSecretsModule implements ModuleInterface {
// This module will be activated only if the license flag is true.
}
```
A module may be restricted to specific instance types:
```ts
@BackendModule({
name: 'my-feature',
instanceTypes: ['main', 'webhook']
})
export class MyFeatureModule implements ModuleInterface {
// This module will only be initialized on main and webhook instances,
// not on worker instances.
}
```
If `instanceTypes` is omitted, the module will be initialized on all instance types (`main`, `webhook`, and `worker`).
If a module is only _partially_ behind a license flag, e.g. insights, then use the `@Licensed()` decorator instead:
```ts
class InsightsController {
constructor(private readonly insightsService: InsightsService) {}
@Get('/by-workflow')
@Licensed('feat:insights:viewDashboard')
async getInsightsByWorkflow(
_req: AuthenticatedRequest,
_res: Response,
@Query payload: ListInsightsWorkflowQueryDto,
) {
const dateRangeAndMaxAgeInDays = this.getMaxAgeInDaysAndGranularity({
dateRange: payload.dateRange ?? 'week',
});
return await this.insightsService.getInsightsByWorkflow({
maxAgeInDays: dateRangeAndMaxAgeInDays.maxAgeInDays,
skip: payload.skip,
take: payload.take,
sortBy: payload.sortBy,
});
}
}
```
Module-level decorators to be aware of:
- `@OnShutdown()` to register a method to be called during the instance shutdown sequence
## Controller
To register a controller with the server, simply import the controller file in the module entrypoint:
```ts
@BackendModule({ name: 'my-feature' })
export class MyFeatureModule implements ModuleInterface {
async init() {
await import('./my-feature.controller');
}
}
```
A controller must handle only request-response orchestration, delegating business logic to services.
```ts
@RestController('/my-feature')
export class MyFeatureController {
constructor(private readonly myFeatureService: MyFeatureService) {}
@Get('/summary')
async getSummary(_req: AuthenticatedRequest, _res: Response) {
return await this.myFeatureService.getSummary();
}
}
```
Controller-level decorators to be aware of:
- `@RestController()` to define a REST controller
- `@Get()`, `@Post()`, `@Put()`, etc. to define controller routes
- `@Query()`, `@Body()`, etc. to validate requests
- `@GlobalScope()` to gate a method depending on a user's instance-wide permissions
- `@ProjectScope()` to gate a method depending on a user's project-specific permissions
- `@Licensed()` to gate a method depending on license state
## Services
Services handle business logic, delegating database access to repositories.
To kickstart a service during module init, run the relevant method in the module entrypoint:
```ts
@BackendModule({ name: 'my-feature' })
export class MyFeatureModule implements ModuleInterface {
async init() {
const { MyFeatureService } = await import('./my-feature.service');
Container.get(MyFeatureService).start();
}
}
```
A module typically has one service, but it may have as many as needed.
```ts
@Service()
export class MyFeatureService {
private intervalId?: NodeJS.Timeout;
constructor(
private readonly myFeatureRepository: MyFeatureRepository,
private readonly logger: Logger,
private readonly config: MyFeatureConfig,
) {
this.logger = this.logger.scoped('my-feature');
}
start() {
this.logger.debug('Starting feature work...');
this.intervalId = setInterval(
() => {
this.logger.debug('Running scheduled task...');
},
this.config.taskInterval * 60 * 1000,
);
}
```
Service-level decorators to be aware of:
- `@Service()` to make a service usable by the dependency injection container
- `@OnLifecycleEvent()` to register a class method to be called on an execution lifecycle event, e.g. `nodeExecuteBefore`, `nodeExecuteAfter`, `workflowExecuteBefore`, and `workflowExecuteAfter`
- `@OnPubSubEvent()` to register a class method to be called on receiving a message via Redis pubsub
- `@OnLeaderTakeover()` and `@OnLeaderStepdown` to register a class method to be called on leadership transition in a multi-main setup
## Repositories
Repositories handle database access using `typeorm`, operating on database models ("entities").
```ts
@Service()
export class MyFeatureRepository extends Repository<MyFeatureEntity> {
constructor(dataSource: DataSource) {
super(MyFeatureEntity, dataSource.manager);
}
async getSummary() {
return await /* typeorm query on entities */;
}
}
```
Repository-level decorators to be aware of:
- `@Service()` to make a repository usable by the dependency injection container
## Entities
Entities are database models, typically representing tables in the database.
```ts
@Entity()
export class MyFeatureEntity extends BaseEntity {
@Column()
name: string;
@Column()
count: number;
}
```
We recommend using the `.entity.ts` infix, but omit the `-Entity` suffix from the class name to make it less verbose. (The `-Entity` suffix is being used in the setup example only for consistency with other placeholders.)
Entities must be registered with `typeorm` in the module entrypoint:
```ts
class MyFeatureModule implements ModuleInterface {
async entities() {
const { MyFeatureEntity } = await import('./my-feature.entity');
return [MyFeatureEntity];
}
}
```
Entity-level decorators to be aware of:
- `@Entity()` to define an entity
## Migrations
As an exception, migrations remain centralized at `@n8n/db/src/migrations`, because conditionally running migrations would introduce unwanted complexity at this time. This means that schema changes from modules are _always_ applied to the database, even when modules are disabled.
## Configuration
Module-specific configuration is defined in the the `.config.ts` file:
```ts
@Config
export class MyFeatureConfig {
/**
* How often in minutes to run some task.
* @default 30
*/
@Env('N8N_MY_FEATURE_TASK_INTERVAL')
taskInterval: number = 30;
}
```
Config-level decorators to be aware of:
- `@Env()` to define environment variables
- `@Config()` to register a config class and make it usable by the dependency injection container
## CLI commands
Occasionally, a module may need to define a module-specific CLI command. To do so, set up a `.command.ts` file and use the `@Command()` decorator. Currently there are no module-specific commands, so use any of the existing global CLI commands at `packages/cli/src/commands` as reference.
## Tests
Place unit and integration tests for a backend module at `packages/cli/src/modules/{featureName}/__tests__`. Use the `.test.ts` infix.
Currently, testing utilities live partly at `cli` and partly at `@n8n/backend-test-utils`. In future, all testing utilities will be moved to common packages, to make modules more decoupled from `cli`.
## Future work
1. A few aspects of modules continue to be defined outside a module's dir:
- Add a license flag to `LICENSE_FEATURES` at `packages/@n8n/constants/src/index.ts`
- Add a logging scope to `LOG_SCOPES` at `packages/@n8n/config/src/configs/logging.config.ts`
- Add a license check to `LicenseState` at `packages/@n8n/backend-common/src/license-state.ts`
- Add a migration (as discussed above) at `packages/@n8n/db/src/migrations`
- Add request payload validation using `zod` at `@n8n/api-types`
- Add a module to default modules at `packages/@n8n/backend-common/src/modules/module-registry.ts`
2. License events (e.g. expiration) currently do not trigger module shutdown or initialization at runtime.
3. Some core functionality is yet to be moved from `cli` into common packages. This is not a blocker for module adoption, but this is desirable so that (a) modules become decoupled from `cli` in the long term, and (b) future external extensions can access some of that functionality.
4. Existing features that are not modules (e.g. LDAP) should be turned into modules over time.
## FAQs
- **What is a good example of a backend module?** Our first backend module is the `insights` module at `packages/@n8n/modules/insights`.
- **My feature is already a separate _package_ at `packages/@n8n/{feature}`. How does this work with modules?** If your feature is already fully decoupled from `cli`, or if you know in advance that your feature will have zero dependencies on `cli`, then you already stand to gain most of the benefits of modularity. In this case, you can add a thin module to `cli` containing an entrypoint to your feature imported from your package, so that your feature is loaded only when needed.
- **Does all new functionality need to be added as a module?** If your feature relies heavily on internals, e.g. workflow archival, then a module may not be a good fit. Consider a module first, but use your best judgment. Reach out if unsure.
- **Are backend modules meant for use by external contributors?** No, they are meant for features developed by the core team.
- **How do I hot reload a module?** Modules are part of `cli` so you can use the usual `watch` command.
- **How do modules interoperate with each other?** This is not supported at this time. Reach out if you need this.
- **I have a use case that is not covered by modules. What should I do?** Modules live in `cli` so any imports from `cli` remain available, i.e. aim for decoupling but do not consider it a blocker for progress. Reach out if you think the module system needs expanding.
@@ -0,0 +1,11 @@
import { Config, Env } from '@n8n/config';
@Config
export class MyFeatureConfig {
/**
* How often in minutes to run some task.
* @default 30
*/
@Env('N8N_MY_FEATURE_TASK_INTERVAL')
taskInterval: number = 30;
}
@@ -0,0 +1,14 @@
import { AuthenticatedRequest } from '@n8n/db';
import { Get, RestController } from '@n8n/decorators';
import { MyFeatureService } from './my-feature.service';
@RestController('/my-feature')
export class MyFeatureController {
constructor(private readonly myFeatureService: MyFeatureService) {}
@Get('/summary')
async getSummary(_req: AuthenticatedRequest, _res: Response) {
return await this.myFeatureService.getSummary();
}
}
@@ -0,0 +1,10 @@
import { BaseEntity, Column, Entity } from '@n8n/typeorm';
@Entity()
export class MyFeatureEntity extends BaseEntity {
@Column()
name: string;
@Column()
count: number;
}
@@ -0,0 +1,32 @@
import type { ModuleInterface } from '@n8n/decorators';
import { BackendModule, OnShutdown } from '@n8n/decorators';
import { Container } from '@n8n/di';
@BackendModule({ name: 'my-feature' })
export class MyFeatureModule implements ModuleInterface {
async init() {
await import('./my-feature.controller');
const { MyFeatureService } = await import('./my-feature.service');
Container.get(MyFeatureService).start();
}
@OnShutdown()
async shutdown() {
const { MyFeatureService } = await import('./my-feature.service');
await Container.get(MyFeatureService).shutdown();
}
async entities() {
const { MyFeatureEntity } = await import('./my-feature.entity');
return [MyFeatureEntity];
}
async context() {
const { MyFeatureService } = await import('./my-feature.service');
return { myFeatureProxy: Container.get(MyFeatureService) };
}
}
@@ -0,0 +1,15 @@
import { Service } from '@n8n/di';
import { DataSource, Repository } from '@n8n/typeorm';
import { MyFeatureEntity } from './my-feature.entity';
@Service()
export class MyFeatureRepository extends Repository<MyFeatureEntity> {
constructor(dataSource: DataSource) {
super(MyFeatureEntity, dataSource.manager);
}
async getSummary() {
return await Promise.resolve({});
}
}
@@ -0,0 +1,43 @@
import { Logger } from '@n8n/backend-common';
import { Service } from '@n8n/di';
import { MyFeatureConfig } from './my-feature.config';
import { MyFeatureRepository } from './my-feature.repository';
@Service()
export class MyFeatureService {
private intervalId?: NodeJS.Timeout;
constructor(
private readonly myFeatureRepository: MyFeatureRepository,
private readonly logger: Logger,
private readonly config: MyFeatureConfig,
) {
this.logger = this.logger.scoped('my-feature' /* Add scope to logging.config.ts */);
}
start() {
this.logger.debug('Starting feature work...');
this.intervalId = setInterval(
() => {
this.logger.debug('Running scheduled task...');
},
this.config.taskInterval * 60 * 1000,
);
}
async shutdown() {
this.logger.debug('Shutting down...');
if (this.intervalId) {
clearInterval(this.intervalId);
}
await Promise.resolve();
}
async getSummary() {
return await this.myFeatureRepository.getSummary();
}
}
@@ -0,0 +1,22 @@
import { testDb, testModules } from '@n8n/backend-test-utils';
import { MyFeatureService } from '../my-feature.service';
beforeAll(async () => {
await testModules.loadModules(['my-feature']);
await testDb.init();
});
beforeEach(async () => {
await testDb.truncate(['MyFeatureUnit']);
});
afterAll(async () => {
await testDb.terminate();
});
describe('start', () => {
it('should do something', () => {
expect(true).toBeTruthy();
});
});
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env zx
import { readFile } from 'node:fs/promises';
import path from 'node:path';
const MODULE_NAME = 'my-feature';
const moduleDir = `./packages/cli/src/modules/${MODULE_NAME}`;
const testsDir = `${moduleDir}/__tests__`;
const templateDir = './scripts/backend-module';
await $`mkdir -p ${moduleDir}`;
await $`mkdir -p ${testsDir}`;
const templateFiles = [
{ template: `${MODULE_NAME}.config.template`, target: `${moduleDir}/${MODULE_NAME}.config.ts` },
{
template: `${MODULE_NAME}.controller.template`,
target: `${moduleDir}/${MODULE_NAME}.controller.ts`,
},
{ template: `${MODULE_NAME}.entity.template`, target: `${moduleDir}/${MODULE_NAME}.entity.ts` },
{ template: `${MODULE_NAME}.module.template`, target: `${moduleDir}/${MODULE_NAME}.module.ts` },
{
template: `${MODULE_NAME}.repository.template`,
target: `${moduleDir}/${MODULE_NAME}.repository.ts`,
},
{
template: `${MODULE_NAME}.service.test.template`,
target: `${testsDir}/${MODULE_NAME}.service.test.ts`,
},
{ template: `${MODULE_NAME}.service.template`, target: `${moduleDir}/${MODULE_NAME}.service.ts` },
];
await Promise.all(
templateFiles.map(async ({ template, target }) => {
const content = await readFile(path.join(templateDir, template), 'utf8');
await fs.writeFile(target, content);
}),
);
console.log(`Backend module setup done at: ${moduleDir}`);
+12
View File
@@ -0,0 +1,12 @@
const { npm_config_user_agent: UA } = process.env;
const [packageManager] = (UA ?? '').split(' ');
const [name, version] = packageManager.split('/');
if (name !== 'pnpm') {
const suggestion = '\033[1;92mpnpm\033[0;31m';
console.error('\033[0;31m');
console.error('╭───────────────────────────────────────────╮');
console.error(`\tPlease use ${suggestion} instead of ${name} \t`);
console.error('╰───────────────────────────────────────────╯');
console.error('\033[0m');
process.exit(1);
}
+319
View File
@@ -0,0 +1,319 @@
#!/usr/bin/env node
/**
* This script is used to build the n8n application for production.
* It will:
* 1. Clean the previous build output
* 2. Run pnpm install and build
* 3. Prepare for deployment - clean package.json files
* 4. Create a pruned production deployment in 'compiled'
*/
import { $, echo, fs, chalk } from 'zx';
import path from 'path';
// Check if running in a CI environment
const isCI = process.env.CI === 'true';
// Check if test controller should be excluded (CI + flag not set)
const excludeTestController =
process.env.CI === 'true' && process.env.INCLUDE_TEST_CONTROLLER !== 'true';
// Disable verbose output and force color only if not in CI
$.verbose = !isCI;
process.env.FORCE_COLOR = isCI ? '0' : '1';
const scriptDir = path.dirname(new URL(import.meta.url).pathname);
const isInScriptsDir = path.basename(scriptDir) === 'scripts';
const rootDir = isInScriptsDir ? path.join(scriptDir, '..') : scriptDir;
// #region ===== Configuration =====
const config = {
compiledAppDir: path.join(rootDir, 'compiled'),
compiledTaskRunnerDir: path.join(rootDir, 'dist', 'task-runner-javascript'),
cliDir: path.join(rootDir, 'packages', 'cli'),
rootDir: rootDir,
};
// Define backend patches to keep during deployment
const PATCHES_TO_KEEP = ['pdfjs-dist', 'pkce-challenge', 'bull'];
// #endregion ===== Configuration =====
// #region ===== Helper Functions =====
const timers = new Map();
function startTimer(name) {
timers.set(name, Date.now());
}
function getElapsedTime(name) {
const start = timers.get(name);
if (!start) return 0;
return Math.floor((Date.now() - start) / 1000);
}
function formatDuration(seconds) {
const hours = Math.floor(seconds / 3600);
const minutes = Math.floor((seconds % 3600) / 60);
const secs = seconds % 60;
if (hours > 0) return `${hours}h ${minutes}m ${secs}s`;
if (minutes > 0) return `${minutes}m ${secs}s`;
return `${secs}s`;
}
function printHeader(title) {
echo('');
echo(chalk.blue.bold(`===== ${title} =====`));
}
function printDivider() {
echo(chalk.gray('-----------------------------------------------'));
}
// #endregion ===== Helper Functions =====
// #region ===== Main Build Process =====
printHeader('n8n Build & Production Preparation');
echo(`INFO: Output Directory: ${config.compiledAppDir}`);
printDivider();
startTimer('total_build');
// 0. Clean Previous Build Output
echo(chalk.yellow(`INFO: Cleaning previous output directory: ${config.compiledAppDir}...`));
await fs.remove(config.compiledAppDir);
echo(
chalk.yellow(
`INFO: Cleaning previous task runner output directory: ${config.compiledTaskRunnerDir}...`,
),
);
await fs.remove(config.compiledTaskRunnerDir);
printDivider();
// 1. Local Application Pre-build
echo(chalk.yellow('INFO: Starting local application pre-build...'));
startTimer('package_build');
echo(chalk.yellow('INFO: Running pnpm install and build...'));
try {
const installProcess = $`cd ${config.rootDir} && pnpm install --frozen-lockfile`;
installProcess.pipe(process.stdout);
await installProcess;
const buildProcess = $`cd ${config.rootDir} && pnpm build --summarize`;
buildProcess.pipe(process.stdout);
await buildProcess;
// Generate third-party licenses for production builds
// Skip with N8N_SKIP_LICENSES=true for CI test builds
if (process.env.N8N_SKIP_LICENSES !== 'true') {
echo(chalk.yellow('INFO: Generating third-party licenses...'));
try {
const licenseProcess = $`cd ${config.rootDir} && node scripts/generate-third-party-licenses.mjs`;
licenseProcess.pipe(process.stdout);
await licenseProcess;
echo(chalk.green('✅ Third-party licenses generated successfully'));
} catch (error) {
echo(chalk.yellow('⚠️ Warning: Third-party license generation failed, continuing build...'));
echo(chalk.red(`ERROR: License generation failed: ${error.message}`));
}
} else {
echo(chalk.gray('INFO: Skipping license generation (N8N_SKIP_LICENSES=true)'));
}
echo(chalk.green('✅ pnpm install and build completed'));
} catch (error) {
console.error(chalk.red('\n🛑 BUILD PROCESS FAILED!'));
console.error(chalk.red('An error occurred during the build process:'));
process.exit(1);
}
const packageBuildTime = getElapsedTime('package_build');
echo(chalk.green(`✅ Package build completed in ${formatDuration(packageBuildTime)}`));
printDivider();
// 2. Prepare for deployment - clean package.json files
echo(chalk.yellow('INFO: Performing pre-deploy cleanup on package.json files...'));
// Find and backup package.json files
const packageJsonFiles = await $`cd ${config.rootDir} && find . -name "package.json" \
-not -path "./node_modules/*" \
-not -path "*/node_modules/*" \
-not -path "./compiled/*" \
-type f`.lines();
// Backup all package.json files
// This is only needed locally, not in CI
if (process.env.CI !== 'true') {
for (const file of packageJsonFiles) {
if (file) {
const fullPath = path.join(config.rootDir, file);
await fs.copy(fullPath, `${fullPath}.bak`);
}
}
}
// Run FE trim script
await $`cd ${config.rootDir} && node .github/scripts/trim-fe-packageJson.js`;
echo(chalk.yellow('INFO: Performing selective patch cleanup...'));
const packageJsonPath = path.join(config.rootDir, 'package.json');
if (await fs.pathExists(packageJsonPath)) {
try {
// 1. Read the package.json file
const packageJsonContent = await fs.readFile(packageJsonPath, 'utf8');
let packageJson = JSON.parse(packageJsonContent);
// 2. Modify the patchedDependencies directly in JavaScript
if (packageJson.pnpm && packageJson.pnpm.patchedDependencies) {
const filteredPatches = {};
for (const [key, value] of Object.entries(packageJson.pnpm.patchedDependencies)) {
// Check if the key (patch name) starts with any of the allowed patches
const shouldKeep = PATCHES_TO_KEEP.some((patchPrefix) => key.startsWith(patchPrefix));
if (shouldKeep) {
filteredPatches[key] = value;
}
}
packageJson.pnpm.patchedDependencies = filteredPatches;
}
// 3. Write the modified package.json back
await fs.writeFile(packageJsonPath, JSON.stringify(packageJson, null, 2), 'utf8');
echo(chalk.green('✅ Kept backend patches: ' + PATCHES_TO_KEEP.join(', ')));
echo(
chalk.gray(
`Removed FE/dev patches that are not in the list of backend patches to keep: ${PATCHES_TO_KEEP.join(', ')}`,
),
);
} catch (error) {
echo(chalk.red(`ERROR: Failed to cleanup patches in package.json: ${error.message}`));
process.exit(1);
}
}
echo(chalk.yellow(`INFO: Creating pruned production deployment in '${config.compiledAppDir}'...`));
startTimer('package_deploy');
await fs.ensureDir(config.compiledAppDir);
if (excludeTestController) {
const cliPackagePath = path.join(config.rootDir, 'packages/cli/package.json');
const content = await fs.readFile(cliPackagePath, 'utf8');
const packageJson = JSON.parse(content);
packageJson.files.push('!dist/**/e2e.*');
await fs.writeFile(cliPackagePath, JSON.stringify(packageJson, null, 2));
echo(chalk.gray(' - Excluded test controller from packages/cli/package.json'));
}
await $`cd ${config.rootDir} && NODE_ENV=production DOCKER_BUILD=true pnpm --filter=n8n --prod --legacy deploy --no-optional ./compiled`;
await fs.ensureDir(config.compiledTaskRunnerDir);
echo(
chalk.yellow(
`INFO: Creating JavaScript task runner deployment in '${config.compiledTaskRunnerDir}'...`,
),
);
await $`cd ${config.rootDir} && NODE_ENV=production DOCKER_BUILD=true pnpm --filter=@n8n/task-runner --prod --legacy deploy --no-optional ${config.compiledTaskRunnerDir}`;
const packageDeployTime = getElapsedTime('package_deploy');
// Restore package.json files
// This is only needed locally, not in CI
if (process.env.CI !== 'true') {
for (const file of packageJsonFiles) {
if (file) {
const fullPath = path.join(config.rootDir, file);
const backupPath = `${fullPath}.bak`;
if (await fs.pathExists(backupPath)) {
await fs.move(backupPath, fullPath, { overwrite: true });
}
}
}
}
// Calculate output size
const compiledAppOutputSize = (await $`du -sh ${config.compiledAppDir} | cut -f1`).stdout.trim();
const compiledTaskRunnerOutputSize = (
await $`du -sh ${config.compiledTaskRunnerDir} | cut -f1`
).stdout.trim();
// Generate build manifests
const buildManifest = {
buildTime: new Date().toISOString(),
artifactSize: compiledAppOutputSize,
buildDuration: {
packageBuild: packageBuildTime,
packageDeploy: packageDeployTime,
total: getElapsedTime('total_build'),
},
};
// Copy third-party licenses if they exist
const licensesSourcePath = path.join(config.cliDir, 'THIRD_PARTY_LICENSES.md');
if (await fs.pathExists(licensesSourcePath)) {
await fs.copy(licensesSourcePath, path.join(config.compiledAppDir, 'THIRD_PARTY_LICENSES.md'));
}
await fs.writeJson(path.join(config.compiledAppDir, 'build-manifest.json'), buildManifest, {
spaces: 2,
});
const taskRunnerbuildManifest = {
buildTime: new Date().toISOString(),
artifactSize: compiledTaskRunnerOutputSize,
buildDuration: {
packageBuild: packageBuildTime,
packageDeploy: packageDeployTime,
total: getElapsedTime('total_build'),
},
};
await fs.writeJson(
path.join(config.compiledTaskRunnerDir, 'build-manifest.json'),
taskRunnerbuildManifest,
{
spaces: 2,
},
);
echo(chalk.green(`✅ Package deployment completed in ${formatDuration(packageDeployTime)}`));
echo(`INFO: Size of ${config.compiledAppDir}: ${compiledAppOutputSize}`);
printDivider();
// Calculate total time
const totalBuildTime = getElapsedTime('total_build');
// #endregion ===== Main Build Process =====
// #region ===== Final Output =====
echo('');
echo(chalk.green.bold('================ BUILD SUMMARY ================'));
echo(chalk.green(`✅ n8n built successfully!`));
echo('');
echo(chalk.blue('📦 Build Output:'));
echo(chalk.green(' n8n:'));
echo(` Directory: ${path.resolve(config.compiledAppDir)}`);
echo(` Size: ${compiledAppOutputSize}`);
echo('');
echo(chalk.green(' task-runner-javascript:'));
echo(` Directory: ${path.resolve(config.compiledTaskRunnerDir)}`);
echo(` Size: ${compiledTaskRunnerOutputSize}`);
echo('');
echo(chalk.blue('⏱️ Build Times:'));
echo(` Package Build: ${formatDuration(packageBuildTime)}`);
echo(` Package Deploy: ${formatDuration(packageDeployTime)}`);
echo(chalk.gray(' -----------------------------'));
echo(chalk.bold(` Total Time: ${formatDuration(totalBuildTime)}`));
echo('');
echo(chalk.blue('📋 Build Manifests:'));
echo(` ${path.resolve(config.compiledAppDir)}/build-manifest.json`);
echo(` ${path.resolve(config.compiledTaskRunnerDir)}/build-manifest.json`);
echo(chalk.green.bold('=============================================='));
// #endregion ===== Final Output =====
// Exit with success
process.exit(0);
+33
View File
@@ -0,0 +1,33 @@
#!/usr/bin/env node
import { readFileSync } from 'node:fs';
import { existsSync } from 'node:fs';
const isPackageJson = (file) => file.endsWith('package.json');
const files = process.argv.slice(2).filter((file) => file && isPackageJson(file) && existsSync(file));
let foundError = false;
for (const file of files) {
try {
const content = readFileSync(file, 'utf8');
if (content.includes('"workspace:^"')) {
if (!foundError) {
console.log('');
console.log("ERROR: Found 'workspace:^' in package.json files.");
console.log('');
console.log("Use 'workspace:*' instead to pin exact versions.");
console.log("Using 'workspace:^' causes npm to resolve semver ranges when users");
console.log("install from npm, which can lead to version mismatches between");
console.log("@n8n/* packages and break n8n startup.");
console.log('');
}
foundError = true;
}
} catch (error) {
// Ignore read errors for individual files
}
}
if (foundError) {
process.exit(1);
}
+294
View File
@@ -0,0 +1,294 @@
#!/usr/bin/env node
/**
* Build n8n and runners Docker images locally
*
* This script simulates the CI build process for local testing.
* Default output: 'n8nio/n8n:local' and 'n8nio/runners:local'
* Override with IMAGE_BASE_NAME and IMAGE_TAG environment variables.
*/
import { $, echo, fs, chalk, os } from 'zx';
import { fileURLToPath } from 'url';
import path from 'path';
// Disable verbose mode for cleaner output
$.verbose = false;
process.env.FORCE_COLOR = '1';
// #region ===== Helper Functions =====
/**
* Get Docker platform string based on host architecture
* @returns {string} Platform string (e.g., 'linux/amd64')
*/
function getDockerPlatform() {
const arch = os.arch();
const dockerArch = {
x64: 'amd64',
arm64: 'arm64',
}[arch];
if (!dockerArch) {
throw new Error(`Unsupported architecture: ${arch}. Only x64 and arm64 are supported.`);
}
return `linux/${dockerArch}`;
}
/**
* Format duration in seconds
* @param {number} ms - Duration in milliseconds
* @returns {string} Formatted duration
*/
function formatDuration(ms) {
return `${Math.floor(ms / 1000)}s`;
}
/**
* Get Docker image size
* @param {string} imageName - Full image name with tag
* @returns {Promise<string>} Image size or 'Unknown'
*/
async function getImageSize(imageName) {
try {
const { stdout } = await $`docker images ${imageName} --format "{{.Size}}"`;
return stdout.trim();
} catch {
return 'Unknown';
}
}
/**
* Check if a command exists
* @param {string} command - Command to check
* @returns {Promise<boolean>} True if command exists
*/
async function commandExists(command) {
try {
await $`command -v ${command}`;
return true;
} catch {
return false;
}
}
const SupportedContainerEngines = /** @type {const} */ (['docker', 'podman']);
/**
* Detect if the local `docker` CLI is actually Podman via the docker shim.
* @returns {Promise<boolean>}
*/
async function isDockerPodmanShim() {
try {
const { stdout } = await $`docker version`;
return stdout.toLowerCase().includes('podman');
} catch {
return false;
}
}
/**
* @returns {Promise<(typeof SupportedContainerEngines[number])>}
*/
async function getContainerEngine() {
// Allow explicit override via env var
const override = process.env.CONTAINER_ENGINE?.toLowerCase();
if (override && /** @type {readonly string[]} */ (SupportedContainerEngines).includes(override)) {
return /** @type {typeof SupportedContainerEngines[number]} */ (override);
}
const hasDocker = await commandExists('docker');
const hasPodman = await commandExists('podman');
if (hasDocker) {
// If docker is actually a Podman shim, use podman path to avoid unsupported flags like --load
if (hasPodman && (await isDockerPodmanShim())) {
return 'podman';
}
return 'docker';
}
if (hasPodman) return 'podman';
throw new Error('No supported container engine found. Please install Docker or Podman.');
}
// #endregion ===== Helper Functions =====
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const isInScriptsDir = path.basename(__dirname) === 'scripts';
const rootDir = isInScriptsDir ? path.join(__dirname, '..') : __dirname;
const config = {
n8n: {
dockerfilePath: path.join(rootDir, 'docker/images/n8n/Dockerfile'),
imageBaseName: process.env.IMAGE_BASE_NAME || 'n8nio/n8n',
imageTag: process.env.IMAGE_TAG || 'local',
get fullImageName() {
return `${this.imageBaseName}:${this.imageTag}`;
},
},
runners: {
dockerfilePath: path.join(rootDir, 'docker/images/runners/Dockerfile'),
imageBaseName: process.env.RUNNERS_IMAGE_BASE_NAME || 'n8nio/runners',
get imageTag() {
// Runners use the same tag as n8n for consistency
return config.n8n.imageTag;
},
get fullImageName() {
return `${this.imageBaseName}:${this.imageTag}`;
},
},
buildContext: rootDir,
compiledAppDir: path.join(rootDir, 'compiled'),
compiledTaskRunnerDir: path.join(rootDir, 'dist', 'task-runner-javascript'),
};
// #region ===== Main Build Process =====
const platform = getDockerPlatform();
async function main() {
echo(chalk.blue.bold('===== Docker Build for n8n & Runners ====='));
echo(`INFO: n8n Image: ${config.n8n.fullImageName}`);
echo(`INFO: Runners Image: ${config.runners.fullImageName}`);
echo(`INFO: Platform: ${platform}`);
echo(chalk.gray('-'.repeat(47)));
await checkPrerequisites();
const n8nBuildTime = await buildDockerImage({
name: 'n8n',
dockerfilePath: config.n8n.dockerfilePath,
fullImageName: config.n8n.fullImageName,
});
const runnersBuildTime = await buildDockerImage({
name: 'runners',
dockerfilePath: config.runners.dockerfilePath,
fullImageName: config.runners.fullImageName,
});
// Get image details
const n8nImageSize = await getImageSize(config.n8n.fullImageName);
const runnersImageSize = await getImageSize(config.runners.fullImageName);
const imageStats = [
{
imageName: config.n8n.fullImageName,
platform,
size: n8nImageSize,
buildTime: n8nBuildTime,
},
{
imageName: config.runners.fullImageName,
platform,
size: runnersImageSize,
buildTime: runnersBuildTime,
},
];
// Write docker build manifest for telemetry collection
const dockerManifest = {
buildTime: new Date().toISOString(),
platform,
images: imageStats.map(({ imageName, size, buildTime }) => ({
imageName,
size,
buildTime,
})),
};
await fs.writeJson(path.join(config.buildContext, 'docker-build-manifest.json'), dockerManifest, {
spaces: 2,
});
// Display summary
displaySummary(imageStats);
}
async function checkPrerequisites() {
if (!(await fs.pathExists(config.compiledAppDir))) {
echo(chalk.red(`Error: Compiled app directory not found at ${config.compiledAppDir}`));
echo(chalk.yellow('Please run build-n8n.mjs first!'));
process.exit(1);
}
if (!(await fs.pathExists(config.compiledTaskRunnerDir))) {
echo(chalk.red(`Error: Task runner directory not found at ${config.compiledTaskRunnerDir}`));
echo(chalk.yellow('Please run build-n8n.mjs first!'));
process.exit(1);
}
// Ensure at least one supported container engine is available
if (!(await commandExists('docker')) && !(await commandExists('podman'))) {
echo(chalk.red('Error: Neither Docker nor Podman is installed or in PATH'));
process.exit(1);
}
}
async function buildDockerImage({ name, dockerfilePath, fullImageName }) {
const startTime = Date.now();
const containerEngine = await getContainerEngine();
// Push directly if image name contains a registry (e.g., ghcr.io/...)
// This avoids the slow --load step (export/import tarball) when pushing to a registry
const shouldPush = fullImageName.includes('/') && fullImageName.split('/').length > 2;
echo(chalk.yellow(`INFO: Building ${name} Docker image using ${containerEngine}...`));
if (shouldPush) {
echo(chalk.yellow(`INFO: Registry detected - pushing directly to ${fullImageName}`));
}
try {
if (containerEngine === 'podman') {
const { stdout } = await $`podman build \
--platform ${platform} \
--build-arg TARGETPLATFORM=${platform} \
-t ${fullImageName} \
-f ${dockerfilePath} \
${config.buildContext}`;
echo(stdout);
} else {
// Use docker buildx build to leverage Blacksmith's layer caching when running in CI.
// The setup-docker-builder action creates a buildx builder with sticky disk cache.
// In CI, push directly to registry to avoid slow --load (export/import tarball).
// Locally, use --load to make image available in local daemon.
const outputFlag = shouldPush ? '--push' : '--load';
const { stdout } = await $`docker buildx build \
--platform ${platform} \
--build-arg TARGETPLATFORM=${platform} \
-t ${fullImageName} \
-f ${dockerfilePath} \
--provenance=false \
${outputFlag} \
${config.buildContext}`;
echo(stdout);
}
return formatDuration(Date.now() - startTime);
} catch (error) {
echo(chalk.red(`ERROR: ${name} Docker build failed: ${error.stderr || error.message}`));
process.exit(1);
}
}
function displaySummary(images) {
echo('');
echo(chalk.green.bold('═'.repeat(54)));
echo(chalk.green.bold(' DOCKER BUILD COMPLETE'));
echo(chalk.green.bold('═'.repeat(54)));
for (const { imageName, platform, size, buildTime } of images) {
echo(chalk.green(`✅ Image built: ${imageName}`));
echo(` Platform: ${platform}`);
echo(` Size: ${size}`);
echo(` Build time: ${buildTime}`);
echo('');
}
echo(chalk.green.bold('═'.repeat(54)));
}
// #endregion ===== Main Build Process =====
main().catch((error) => {
echo(chalk.red(`Unexpected error: ${error.message}`));
process.exit(1);
});
+17
View File
@@ -0,0 +1,17 @@
import { accessSync, constants } from 'node:fs';
import { execSync } from 'node:child_process';
const ZX_PATH = 'node_modules/.bin/zx';
if (!zxExists()) {
execSync('pnpm --frozen-lockfile --filter n8n-monorepo install', { stdio: 'inherit' });
}
function zxExists() {
try {
accessSync(ZX_PATH, constants.F_OK);
return true;
} catch {
return false;
}
}
+65
View File
@@ -0,0 +1,65 @@
#!/usr/bin/env node
import fs from 'fs';
import path from 'path';
import { execSync } from 'child_process';
const prettier = path.resolve('node_modules', '.bin', 'prettier');
const biome = path.resolve('node_modules', '.bin', 'biome');
[prettier, biome].forEach((bin) => {
if (!fs.existsSync(bin)) {
throw new Error(
[`${path.basename(bin)} not found at path: ${bin}`, 'Please run `pnpm i` first'].join('\n'),
);
}
});
const prettierConfig = path.resolve('.prettierrc.js');
const biomeConfig = path.resolve('biome.jsonc');
const ignore = path.resolve('.prettierignore');
const ROOT_DIRS_TO_SKIP = ['.git', 'node_modules', 'packages', '.turbo'];
const EXTENSIONS_TO_FORMAT_WITH_PRETTIER = ['.yml'];
const EXTENSIONS_TO_FORMAT_WITH_BIOME = ['.js', '.json', '.ts'];
const isDir = (path) => fs.lstatSync(path).isDirectory();
const isPrettierTarget = (path) =>
EXTENSIONS_TO_FORMAT_WITH_PRETTIER.some((ext) => path.endsWith(ext));
const isBiomeTarget = (path) => EXTENSIONS_TO_FORMAT_WITH_BIOME.some((ext) => path.endsWith(ext));
const biomeTargets = [];
const prettierTargets = [];
const walk = (dir) => {
fs.readdirSync(dir).forEach((entry) => {
const entryPath = path.resolve(dir, entry);
if (isDir(entryPath)) walk(entryPath);
if (isPrettierTarget(entryPath)) prettierTargets.push(entryPath);
if (isBiomeTarget(entryPath)) biomeTargets.push(entryPath);
});
};
fs.readdirSync('.').forEach((cur) => {
if (ROOT_DIRS_TO_SKIP.includes(cur)) return;
if (isDir(cur)) walk(cur);
if (isPrettierTarget(cur)) prettierTargets.push(cur);
if (isBiomeTarget(cur)) biomeTargets.push(cur);
});
execSync(
[
prettier,
'--config',
prettierConfig,
'--ignore-path',
ignore,
'--write',
prettierTargets.join(' '),
].join(' '),
);
execSync(
[biome, 'format', '--write', `--config-path=${biomeConfig}`, biomeTargets.join(' ')].join(' '),
);
+306
View File
@@ -0,0 +1,306 @@
#!/usr/bin/env node
/**
* Third-Party License Generator for n8n
*
* Generates THIRD_PARTY_LICENSES.md by scanning all dependencies using license-checker,
* extracting license information, and formatting it into a markdown report.
*
* Usage: node scripts/generate-third-party-licenses.mjs
*/
import { $, echo, fs, chalk } from 'zx';
import path from 'path';
import os from 'os';
import { fileURLToPath } from 'url';
// Disable verbose zx output
$.verbose = false;
const scriptDir = path.dirname(fileURLToPath(import.meta.url));
const rootDir = path.join(scriptDir, '..');
const config = {
tempLicenseFile: 'licenses.json',
outputFile: 'THIRD_PARTY_LICENSES.md',
invalidLicenseFiles: ['readme.md', 'readme.txt', 'readme', 'package.json', 'changelog.md', 'history.md'],
validLicenseFiles: ['license', 'licence', 'copying', 'copyright', 'unlicense'],
paths: {
root: rootDir,
cliRoot: path.join(rootDir, 'packages', 'cli'),
formatConfig: path.join(scriptDir, 'third-party-license-format.json'),
tempLicenses: path.join(os.tmpdir(), 'licenses.json'),
output: path.join(rootDir, 'packages', 'cli', 'THIRD_PARTY_LICENSES.md'),
},
};
// #region ===== Helper Functions =====
async function generateLicenseData() {
echo(chalk.yellow('📊 Running license-checker...'));
try {
$.cwd = config.paths.root;
await $`pnpm exec license-checker --json --customPath ${config.paths.formatConfig}`.pipe(
fs.createWriteStream(config.paths.tempLicenses),
);
echo(chalk.green('✅ License data collected'));
return config.paths.tempLicenses;
} catch (error) {
echo(chalk.red('❌ Failed to run license-checker'));
throw error;
}
}
async function readLicenseData(filePath) {
try {
const data = await fs.readFile(filePath, 'utf-8');
const parsed = JSON.parse(data);
echo(chalk.green(`✅ Parsed ${Object.keys(parsed).length} packages`));
return parsed;
} catch (error) {
echo(chalk.red('❌ Failed to parse license data'));
throw error;
}
}
function parsePackageKey(packageKey) {
const lastAtIndex = packageKey.lastIndexOf('@');
return {
packageName: packageKey.substring(0, lastAtIndex),
version: packageKey.substring(lastAtIndex + 1),
};
}
function shouldExcludePackage(packageName) {
const n8nPatterns = [
/^@n8n\//, // @n8n/package
/^@n8n_/, // @n8n_io/package
/^n8n-/, // n8n-package
/-n8n/ // package-n8n
];
return n8nPatterns.some(pattern => pattern.test(packageName));
}
function isValidLicenseFile(filePath) {
if (!filePath) return false;
const fileName = path.basename(filePath).toLowerCase();
// Exclude non-license files
const isInvalidFile = config.invalidLicenseFiles.some((invalid) =>
fileName === invalid || fileName.endsWith(invalid)
);
if (isInvalidFile) return false;
// Must contain license-related keywords
return config.validLicenseFiles.some((valid) => fileName.includes(valid));
}
function getFallbackLicenseText(licenseType, packages = []) {
const fallbacks = {
'CC-BY-3.0': 'Creative Commons Attribution 3.0 Unported License\n\nFull license text available at: https://creativecommons.org/licenses/by/3.0/legalcode',
'LGPL-3.0-or-later': 'GNU Lesser General Public License v3.0 or later\n\nFull license text available at: https://www.gnu.org/licenses/lgpl-3.0.html',
'PSF': 'Python Software Foundation License\n\nFull license text available at: https://docs.python.org/3/license.html',
'(MIT OR CC0-1.0)': 'Licensed under MIT OR CC0-1.0\n\nMIT License full text available at: https://opensource.org/licenses/MIT\nCC0 1.0 Universal full text available at: https://creativecommons.org/publicdomain/zero/1.0/legalcode',
'UNKNOWN': `License information not available for the following packages:\n${packages.map(pkg => `- ${pkg.name} ${pkg.version}`).join('\n')}\n\nPlease check individual package repositories for license details.`,
};
// Check for custom licenses that start with "Custom:"
if (licenseType.startsWith('Custom:')) {
return `Custom license. See: ${licenseType.replace('Custom: ', '')}`;
}
return fallbacks[licenseType] || null;
}
function cleanLicenseText(text) {
return text
.replaceAll('\\n', '\n')
.replaceAll('\\"', '"')
.replaceAll('\r\n', '\n')
.trim();
}
function addPackageToGroup(licenseGroups, licenseType, packageInfo) {
if (!licenseGroups.has(licenseType)) {
licenseGroups.set(licenseType, []);
}
licenseGroups.get(licenseType).push(packageInfo);
}
function processLicenseText(licenseTexts, licenseType, pkg) {
if (!licenseTexts.has(licenseType)) {
licenseTexts.set(licenseType, null);
}
if (!licenseTexts.get(licenseType) && pkg.licenseText?.trim() && isValidLicenseFile(pkg.licenseFile)) {
licenseTexts.set(licenseType, cleanLicenseText(pkg.licenseText));
}
}
function applyFallbackLicenseTexts(licenseTexts, licenseGroups) {
const missingTexts = [];
const fallbacksUsed = [];
for (const [licenseType, text] of licenseTexts.entries()) {
if (!text || !text.trim()) {
const packagesForLicense = licenseGroups.get(licenseType) || [];
const fallback = getFallbackLicenseText(licenseType, packagesForLicense);
if (fallback) {
licenseTexts.set(licenseType, fallback);
fallbacksUsed.push(licenseType);
} else {
missingTexts.push(licenseType);
}
}
}
return { missingTexts, fallbacksUsed };
}
function logProcessingResults(processedCount, licenseGroupCount, fallbacksUsed, missingTexts) {
echo(chalk.cyan(`📦 Processed ${processedCount} packages in ${licenseGroupCount} license groups`));
if (fallbacksUsed.length > 0) {
echo(chalk.blue(`️ Used fallback texts for: ${fallbacksUsed.join(', ')}`));
}
if (missingTexts.length > 0) {
echo(chalk.yellow(`⚠️ Still missing license texts for: ${missingTexts.join(', ')}`));
} else {
echo(chalk.green(`✅ All license types have texts`));
}
}
function processPackages(packages) {
const licenseGroups = new Map();
const licenseTexts = new Map();
let processedCount = 0;
for (const [packageKey, pkg] of Object.entries(packages)) {
const { packageName, version } = parsePackageKey(packageKey);
if (shouldExcludePackage(packageName)) {
continue;
}
const licenseType = pkg.licenses || 'Unknown';
processedCount++;
// Group packages by license
addPackageToGroup(licenseGroups, licenseType, {
name: packageName,
version,
repository: pkg.repository,
copyright: pkg.copyright,
});
// Store license text (use first non-empty occurrence)
processLicenseText(licenseTexts, licenseType, pkg);
}
// Apply fallback license texts for missing ones
const { missingTexts, fallbacksUsed } = applyFallbackLicenseTexts(licenseTexts, licenseGroups);
logProcessingResults(processedCount, licenseGroups.size, fallbacksUsed, missingTexts);
return { licenseGroups, licenseTexts, processedCount };
}
// #endregion ===== Helper Functions =====
// #region ===== Document Generation =====
function createPackageSection(licenseType, packages) {
const sortedPackages = [...packages].sort((a, b) => a.name.localeCompare(b.name));
let section = `## ${licenseType}\n\n`;
for (const pkg of sortedPackages) {
section += `* ${pkg.name} ${pkg.version}`;
if (pkg.copyright) {
section += `, ${pkg.copyright}`;
}
section += '\n';
}
section += '\n';
return section;
}
function createLicenseTextSection(licenseType, licenseText) {
let section = `## ${licenseType} License Text\n\n`;
if (licenseText && licenseText.trim()) {
section += `\`\`\`\n${licenseText}\n\`\`\`\n\n`;
} else {
section += `${licenseType} license text not available.\n\n`;
}
return section;
}
function createDocumentHeader() {
return `# Third-Party Licenses
This file lists third-party software components included in n8n and their respective license terms.
The n8n software includes open source packages, libraries, and modules, each of which is subject to its own license. The following sections list those dependencies and provide required attributions and license texts.
`;
}
function buildMarkdownDocument(packages) {
const { licenseGroups, licenseTexts, processedCount } = processPackages(packages);
let document = createDocumentHeader();
const sortedLicenseTypes = [...licenseGroups.keys()].sort();
// First: Add all package sections
for (const licenseType of sortedLicenseTypes) {
const packages = licenseGroups.get(licenseType);
document += createPackageSection(licenseType, packages);
}
// Second: Add license texts section
document += '# License Texts\n\n';
for (const licenseType of sortedLicenseTypes) {
const licenseText = licenseTexts.get(licenseType);
document += createLicenseTextSection(licenseType, licenseText);
}
return { content: document, processedCount };
}
// #endregion ===== Document Generation =====
async function generateThirdPartyLicenses() {
echo(chalk.blue('🚀 Generating third-party licenses for n8n...'));
try {
const licensesJsonPath = await generateLicenseData();
const packages = await readLicenseData(licensesJsonPath);
echo(chalk.yellow('📝 Building markdown document...'));
const { content, processedCount } = buildMarkdownDocument(packages);
await fs.ensureDir(config.paths.cliRoot);
await fs.writeFile(config.paths.output, content);
// Clean up temporary file
await fs.remove(licensesJsonPath);
echo(chalk.green('\n🎉 License generation completed successfully!'));
echo(chalk.green(`📄 Output: ${config.paths.output}`));
echo(chalk.green(`📦 Packages: ${processedCount}`));
} catch (error) {
echo(chalk.red(`\n❌ Generation failed: ${error.message}`));
process.exit(1);
}
}
generateThirdPartyLicenses();
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env node
/**
* Created to ease the running of binaries on cross-platform teams.
* Enabled writing startup scripts once, but defaulting to platform specific runners.
*
* Usage: node scripts/os-normalize.mjs --dir packages/cli/bin n8n
* Usage (with args): node scripts/os-normalize.mjs --dir packages/cli/bin -- n8n --help
* */
import { $, argv, cd, chalk, echo, usePowerShell, fs } from 'zx';
const isWindows = process.platform === 'win32';
/**
* @param { string } baseName
* */
function normalizeCommand(baseName) {
if (!isWindows) {
return `./${baseName}`;
}
const candidates = [`${baseName}.cmd`, `${baseName}.exe`, baseName];
const found = candidates.find((c) => fs.existsSync(c));
return found ? `./${found}` : `./${baseName}.cmd`; // last resort: try .cmd anyway
}
function determineShell() {
if (!isWindows) {
return;
}
usePowerShell();
}
function printUsage() {
echo(chalk.red('Usage: node scripts/os-normalize.mjs --dir <dir> <run>'));
echo(
chalk.red('Usage (with args): node scripts/os-normalize.mjs --dir <dir> -- <run> [args...]'),
);
}
const { dir = '.' } = argv;
const [run, ...args] = argv._;
if (!dir || !run) {
printUsage();
process.exit(2);
}
determineShell();
$.verbose = true;
cd(dir);
const cmd = normalizeCommand(run);
echo(chalk.cyan(`$ Running (dir: ${dir}) ${cmd} ${args.join(' ')}`));
await $({ stdio: 'inherit' })`${cmd} ${args}`;
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env node
import { execSync } from 'node:child_process';
// Skip lefthook install in CI or Docker build
if (process.env.CI || process.env.DOCKER_BUILD) {
process.exit(0);
}
execSync('pnpm lefthook install', { stdio: 'inherit' });
+34
View File
@@ -0,0 +1,34 @@
// Resets the repository by deleting all untracked files except for few exceptions.
import { $, argv, echo, fs } from 'zx';
$.verbose = true;
process.env.FORCE_COLOR = '1';
const excludePatterns = ['/.vscode/', '/.idea/', '.env', '/.claude/'];
const excludeFlags = excludePatterns.map((exclude) => ['-e', exclude]).flat();
echo(
`This will delete all untracked files except for those matching the following patterns: ${excludePatterns.map((x) => `"${x}"`).join(', ')}.`,
);
const skipConfirmation = argv.force || argv.f;
if (!skipConfirmation) {
const answer = await question('❓ Do you want to continue? (y/n) ');
if (!['y', 'Y', ''].includes(answer)) {
echo('Aborting...');
process.exit(0);
}
}
echo('🧹 Cleaning untracked files...');
await $({ verbose: false })`git clean -fxd ${excludeFlags}`;
// In case node_modules is not removed by git clean
fs.removeSync('node_modules');
echo('⏬ Running pnpm install...');
await $`pnpm install`;
echo('🏗️ Running pnpm build...');
await $`pnpm build`;
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env node
/**
* This script is used to scan the n8n docker image for vulnerabilities.
* It uses Trivy to scan the image.
*/
import { $, echo, fs, chalk } from 'zx';
import path from 'path';
$.verbose = false;
process.env.FORCE_COLOR = '1';
const scriptDir = path.dirname(new URL(import.meta.url).pathname);
const isInScriptsDir = path.basename(scriptDir) === 'scripts';
const rootDir = isInScriptsDir ? path.join(scriptDir, '..') : scriptDir;
const assertPathWithinRoot = (envVar, defaultRelPath) => {
const resolved = path.resolve(process.env[envVar] || path.join(rootDir, defaultRelPath));
if (!resolved.startsWith(rootDir + path.sep) && resolved !== rootDir) {
echo(chalk.red(`Error: ${envVar} must resolve within the repository root`));
process.exit(1);
}
return resolved;
};
// #region ===== Configuration =====
const config = {
imageBaseName: process.env.IMAGE_BASE_NAME || 'n8nio/n8n',
imageTag: process.env.IMAGE_TAG || 'local',
trivyImage: process.env.TRIVY_IMAGE || 'aquasec/trivy:latest',
severity: process.env.TRIVY_SEVERITY || 'CRITICAL,HIGH,MEDIUM,LOW',
outputFormat: process.env.TRIVY_FORMAT || 'table',
outputFile: process.env.TRIVY_OUTPUT || null,
scanTimeout: process.env.TRIVY_TIMEOUT || '10m',
ignoreUnfixed: process.env.TRIVY_IGNORE_UNFIXED === 'true',
scanners: process.env.TRIVY_SCANNERS || 'vuln',
quiet: process.env.TRIVY_QUIET === 'true',
rootDir: rootDir,
vexFile: assertPathWithinRoot('TRIVY_VEX', 'security/vex.openvex.json'),
ignorePolicyFile: assertPathWithinRoot('TRIVY_IGNORE_POLICY', 'security/trivy-ignore-policy.rego'),
};
config.fullImageName = `${config.imageBaseName}:${config.imageTag}`;
const printHeader = (title) =>
!config.quiet && echo(`\n${chalk.blue.bold(`===== ${title} =====`)}`);
const printSummary = (status, time, message) => {
if (config.quiet) return;
echo('\n' + chalk.blue.bold('===== Scan Summary ====='));
echo(status === 'success' ? chalk.green.bold(message) : chalk.yellow.bold(message));
echo(chalk[status === 'success' ? 'green' : 'yellow'](` Scan time: ${time}s`));
if (config.outputFile) {
const resolvedPath = path.isAbsolute(config.outputFile)
? config.outputFile
: path.join(config.rootDir, config.outputFile);
echo(chalk[status === 'success' ? 'green' : 'yellow'](` Report saved to: ${resolvedPath}`));
}
echo('\n' + chalk.gray('Scan Configuration:'));
echo(chalk.gray(` • Target Image: ${config.fullImageName}`));
echo(chalk.gray(` • Severity Levels: ${config.severity}`));
echo(chalk.gray(` • Scanners: ${config.scanners}`));
echo(chalk.gray(` • VEX file: ${config.vexFile}`));
echo(chalk.gray(` • Ignore policy: ${config.ignorePolicyFile}`));
if (config.ignoreUnfixed) echo(chalk.gray(` • Ignored unfixed: yes`));
echo(chalk.blue.bold('========================'));
};
// #endregion ===== Configuration =====
// #region ===== Main Process =====
(async () => {
printHeader('Trivy Security Scan for n8n Image');
try {
await $`command -v docker`;
} catch {
echo(chalk.red('Error: Docker is not installed or not in PATH'));
process.exit(1);
}
try {
await $`docker image inspect ${config.fullImageName} > /dev/null 2>&1`;
} catch {
echo(chalk.red(`Error: Docker image '${config.fullImageName}' not found`));
echo(chalk.yellow('Please run dockerize-n8n.mjs first!'));
process.exit(1);
}
// Pull latest Trivy image silently
try {
await $`docker pull ${config.trivyImage} > /dev/null 2>&1`;
} catch {
// Silent fallback to cached version
}
// Build Trivy command
const trivyArgs = [
'run',
'--rm',
'-v',
'/var/run/docker.sock:/var/run/docker.sock',
'-v',
`${config.vexFile}:/vex.openvex.json:ro`,
'-v',
`${config.ignorePolicyFile}:/trivy-ignore-policy.rego:ro`,
config.trivyImage,
'image',
'--severity',
config.severity,
'--format',
config.outputFormat,
'--timeout',
config.scanTimeout,
'--scanners',
config.scanners,
'--no-progress',
'--vex',
'/vex.openvex.json',
'--ignore-policy',
'/trivy-ignore-policy.rego',
];
if (config.ignoreUnfixed) trivyArgs.push('--ignore-unfixed');
if (config.quiet && config.outputFormat === 'table') trivyArgs.push('--quiet');
// Handle output file - resolve relative to root directory
if (config.outputFile) {
const outputPath = path.isAbsolute(config.outputFile)
? config.outputFile
: path.join(config.rootDir, config.outputFile);
await fs.ensureDir(path.dirname(outputPath));
trivyArgs.push('--output', '/tmp/trivy-output', '-v', `${outputPath}:/tmp/trivy-output`);
}
trivyArgs.push(config.fullImageName);
// Run the scan
const startTime = Date.now();
try {
const result = await $`docker ${trivyArgs}`;
// Print Trivy output first
if (!config.outputFile && result.stdout) {
echo(result.stdout);
}
// Then print our summary
const scanTime = Math.floor((Date.now() - startTime) / 1000);
printSummary('success', scanTime, '✅ Security scan completed successfully');
process.exit(0);
} catch (error) {
const scanTime = Math.floor((Date.now() - startTime) / 1000);
// Trivy returns exit code 1 when vulnerabilities are found
if (error.exitCode === 1) {
// Print Trivy output first
if (!config.outputFile && error.stdout) {
echo(error.stdout);
}
// Then print our summary
printSummary('warning', scanTime, '⚠️ Vulnerabilities found!');
process.exit(1);
} else {
echo(chalk.red(`❌ Scan failed: ${error.message}`));
process.exit(error.exitCode || 1);
}
}
})();
// #endregion ===== Main Process =====
+12
View File
@@ -0,0 +1,12 @@
{
"name": "",
"version": "",
"description": "",
"licenses": "",
"copyright": "",
"licenseFile": "",
"licenseText": "",
"licenseModified": "",
"repository": "",
"url": ""
}