first commit
Security: Sync from Public / sync-from-public (push) Has been cancelled
Test: Benchmark Nightly / build (push) Has been cancelled
Test: Benchmark Nightly / Notify Cats on failure (push) Has been cancelled
CI: Python / Checks (push) Has been cancelled
Test: Evals Python / Workflow Comparison Python (push) Has been cancelled
Util: Check Docs URLs / check-docs-urls (push) Has been cancelled
Test: Visual Storybook / Cloudflare Pages (push) Has been cancelled
Test: E2E Performance / build-and-test-performance (push) Has been cancelled
Test: Workflows Nightly / Run Workflow Tests (push) Has been cancelled
Util: Cleanup CI Docker Images / Delete stale CI images (push) Has been cancelled
Test: Benchmark Destroy Env / build (push) Has been cancelled
Util: Update Node Popularity / update-popularity (push) Has been cancelled
Test: E2E Coverage Weekly / Coverage Tests (push) Has been cancelled

This commit is contained in:
2026-03-17 16:22:57 +03:30
commit 3d5eaf9445
15349 changed files with 2847338 additions and 0 deletions
@@ -0,0 +1,154 @@
export const CORRECTNESS_PROMPT = `You are an expert factual evaluator assessing the accuracy of answers compared to established ground truths.
Evaluate the factual correctness of a given output compared to the provided ground truth on a scale from 1 to 5. Use detailed reasoning to thoroughly analyze all claims before determining the final score.
# Scoring Criteria
- 5: Highly similar - The output and ground truth are nearly identical, with only minor, insignificant differences.
- 4: Somewhat similar - The output is largely similar to the ground truth but has few noticeable differences.
- 3: Moderately similar - There are some evident differences, but the core essence is captured in the output.
- 2: Slightly similar - The output only captures a few elements of the ground truth and contains several differences.
- 1: Not similar - The output is significantly different from the ground truth, with few or no matching elements.
# Evaluation Steps
1. Identify and list the key elements present in both the output and the ground truth.
2. Compare these key elements to evaluate their similarities and differences, considering both content and structure.
3. Analyze the semantic meaning conveyed by both the output and the ground truth, noting any significant deviations.
4. Consider factual accuracy of specific details, including names, dates, numbers, and relationships.
5. Assess whether the output maintains the factual integrity of the ground truth, even if phrased differently.
6. Determine the overall level of similarity and accuracy according to the defined criteria.
# Output Format
Provide:
- A detailed analysis of the comparison (extended reasoning)
- A one-sentence summary highlighting key differences (not similarities)
- The final similarity score as an integer (1, 2, 3, 4, or 5)
Always follow the JSON format below and return nothing else:
{
"extended_reasoning": "<detailed step-by-step analysis of factual accuracy and similarity>",
"reasoning_summary": "<one sentence summary focusing on key differences>",
"score": <number: integer from 1 to 5>
}
# Examples
**Example 1:**
Input:
- Output: "The cat sat on the mat."
- Ground Truth: "The feline is sitting on the rug."
Expected Output:
{
"extended_reasoning": "I need to compare 'The cat sat on the mat' with 'The feline is sitting on the rug.' First, let me identify the key elements: both describe an animal ('cat' vs 'feline') in a position ('sat' vs 'sitting') on a surface ('mat' vs 'rug'). The subject is semantically identical - 'cat' and 'feline' refer to the same animal. The action is also semantically equivalent - 'sat' and 'sitting' both describe the same position, though one is past tense and one is present continuous. The location differs in specific wording ('mat' vs 'rug') but both refer to floor coverings that serve the same function. The basic structure and meaning of both sentences are preserved, though they use different vocabulary and slightly different tense. The core information being conveyed is the same, but there are noticeable wording differences.",
"reasoning_summary": "The sentences differ in vocabulary choice ('cat' vs 'feline', 'mat' vs 'rug') and verb tense ('sat' vs 'is sitting').",
"score": 3
}
**Example 2:**
Input:
- Output: "The quick brown fox jumps over the lazy dog."
- Ground Truth: "A fast brown animal leaps over a sleeping canine."
Expected Output:
{
"extended_reasoning": "I need to compare 'The quick brown fox jumps over the lazy dog' with 'A fast brown animal leaps over a sleeping canine.' Starting with the subjects: 'quick brown fox' vs 'fast brown animal'. Both describe the same entity (a fox is a type of animal) with the same attributes (quick/fast and brown). The action is described as 'jumps' vs 'leaps', which are synonymous verbs describing the same motion. The object in both sentences is a dog, described as 'lazy' in one and 'sleeping' in the other, which are related concepts (a sleeping dog could be perceived as lazy). The structure follows the same pattern: subject + action + over + object. The sentences convey the same scene with slightly different word choices that maintain the core meaning. The level of specificity differs slightly ('fox' vs 'animal', 'dog' vs 'canine'), but the underlying information and imagery remain very similar.",
"reasoning_summary": "The sentences use different but synonymous terminology ('quick' vs 'fast', 'jumps' vs 'leaps', 'lazy' vs 'sleeping') and varying levels of specificity ('fox' vs 'animal', 'dog' vs 'canine').",
"score": 4
}
# Notes
- Focus primarily on factual accuracy and semantic similarity, not writing style or phrasing differences.
- Identify specific differences rather than making general assessments.
- Pay special attention to dates, numbers, names, locations, and causal relationships when present.
- Consider the significance of each difference in the context of the overall information.
- Be consistent in your scoring approach across different evaluations.`;
export const CORRECTNESS_INPUT_PROMPT: string[] = [
`Output: {actual_answer}
Ground truth: {expected_answer}`,
'Requires the placeholders <code>{actual_answer}</code> and <code>{expected_answer}</code>',
];
export const HELPFULNESS_PROMPT = `You are an expert evaluator assessing the helpfulness of responses to user queries.
Evaluate how helpful and useful a given response is to the user's question or request on a scale from 1 to 5. Consider whether the response addresses the user's needs, provides actionable information, and is relevant to their query.
# Scoring Criteria
- 5: Extremely helpful - The response fully addresses the user's needs, provides comprehensive and actionable information, and goes above and beyond to be useful.
- 4: Very helpful - The response addresses most of the user's needs, provides useful information, and is highly relevant.
- 3: Moderately helpful - The response addresses some of the user's needs, provides some useful information, but may lack completeness or depth.
- 2: Slightly helpful - The response provides minimal useful information and only partially addresses the user's needs.
- 1: Not helpful - The response fails to address the user's needs, provides no useful information, or is irrelevant.
# Evaluation Steps
1. Analyze the user's question or request to understand what they're looking for.
2. Evaluate how well the response addresses the specific needs expressed in the query.
3. Assess the completeness and quality of the information provided.
4. Consider the relevance and applicability of the response to the user's situation.
5. Evaluate whether the response provides actionable insights or next steps.
6. Determine the overall helpfulness according to the defined criteria.
# Output Format
Provide:
- A detailed analysis of the response's helpfulness (extended reasoning)
- A one-sentence summary highlighting the key strengths or weaknesses
- The final helpfulness score as an integer (1, 2, 3, 4, or 5)
Always follow the JSON format below and return nothing else:
{
"extended_reasoning": "<detailed step-by-step analysis of the response's helpfulness>",
"reasoning_summary": "<one sentence summary of the response's helpfulness>",
"score": <number: integer from 1 to 5>
}
# Examples
**Example 1:**
Input:
- Query: "How do I fix a leaky faucet?"
- Response: "A leaky faucet is usually caused by a worn washer or O-ring. Turn off the water supply, remove the handle, replace the washer or O-ring, and reassemble. If the leak persists, you may need to replace the entire cartridge."
Expected Output:
{
"extended_reasoning": "The user asked for help fixing a leaky faucet, which is a practical home maintenance question. The response directly addresses the query by identifying the most common cause (worn washer or O-ring) and provides a clear step-by-step solution. It includes important safety information (turning off water supply) and offers a backup solution if the initial fix doesn't work. The response is concise, actionable, and comprehensive for this common problem.",
"reasoning_summary": "The response provides a complete, actionable solution with clear steps and troubleshooting advice.",
"score": 5
}
**Example 2:**
Input:
- Query: "What's the weather like?"
- Response: "Weather can be sunny, rainy, cloudy, or snowy depending on various atmospheric conditions."
Expected Output:
{
"extended_reasoning": "The user asked about the weather, which typically implies they want current weather conditions for their location or a specific place. However, the response provides only generic information about weather types without addressing the specific query. It doesn't provide current conditions, forecasts, or ask for location clarification. The response is factually correct but completely unhelpful for the user's actual need.",
"reasoning_summary": "The response provides generic weather information instead of addressing the user's likely need for current conditions.",
"score": 1
}
# Notes
- Focus on practical utility and how well the response serves the user's actual needs
- Consider completeness, accuracy, and actionability of the information
- Pay attention to whether the response asks for clarification when needed
- Evaluate whether the response is appropriately detailed for the query complexity`;
export const HELPFULNESS_INPUT_PROMPT: string[] = [
`Query: {user_query}
Response: {actual_answer}`,
'Requires the placeholders <code>{user_query}</code> and <code>{actual_answer}</code>',
];
@@ -0,0 +1,499 @@
import type { INodeProperties } from 'n8n-workflow';
import { DEFAULT_EVALUATION_METRIC } from 'n8n-workflow';
import {
CORRECTNESS_PROMPT,
CORRECTNESS_INPUT_PROMPT,
HELPFULNESS_PROMPT,
HELPFULNESS_INPUT_PROMPT,
} from './CannedMetricPrompts.ee';
import { document, sheet } from '../../Google/Sheet/GoogleSheetsTrigger.node';
export const setInputsProperties: INodeProperties[] = [
{
displayName:
'For adding columns from your dataset to the evaluation results. Anything you add here will be displayed in the evaluations tab, not on the Google Sheet or Data table.',
name: 'setInputsNotice',
type: 'notice',
default: '',
displayOptions: {
show: {
operation: ['setInputs'],
},
},
},
{
displayName: 'Inputs',
name: 'inputs',
placeholder: 'Add Input',
type: 'fixedCollection',
typeOptions: {
multipleValueButtonText: 'Add Input',
multipleValues: true,
},
default: {},
options: [
{
displayName: 'Filter',
name: 'values',
values: [
{
displayName: 'Name',
name: 'inputName',
type: 'string',
default: '',
requiresDataPath: 'single',
},
{
displayName: 'Value',
name: 'inputValue',
type: 'string',
default: '',
},
],
},
],
displayOptions: {
show: {
operation: ['setInputs'],
},
},
},
];
export const setOutputProperties: INodeProperties[] = [
{
displayName: 'Credentials',
name: 'credentials',
type: 'credentials',
default: '',
displayOptions: { hide: { source: ['dataTable'] } },
},
{
...document,
displayName: 'Document Containing Dataset',
displayOptions: {
show: {
operation: ['setOutputs'],
},
hide: { source: ['dataTable'] },
},
},
{
...sheet,
displayName: 'Sheet Containing Dataset',
displayOptions: {
show: {
operation: ['setOutputs'],
},
hide: { source: ['dataTable'] },
},
},
{
// eslint-disable-next-line n8n-nodes-base/node-param-display-name-miscased
displayName: 'Data table',
name: 'dataTableId',
type: 'resourceLocator',
default: { mode: 'list', value: '' },
required: true,
modes: [
{
displayName: 'From List',
name: 'list',
type: 'list',
typeOptions: {
searchListMethod: 'dataTableSearch',
searchable: true,
skipCredentialsCheckInRLC: true,
},
},
{
displayName: 'ID',
name: 'id',
type: 'string',
},
],
displayOptions: {
show: { source: ['dataTable'] },
},
},
{
displayName: 'Outputs',
name: 'outputs',
placeholder: 'Add Output',
type: 'fixedCollection',
typeOptions: {
multipleValueButtonText: 'Add Output',
multipleValues: true,
},
default: {},
options: [
{
displayName: 'Filter',
name: 'values',
values: [
{
displayName: 'Name',
name: 'outputName',
type: 'string',
default: '',
requiresDataPath: 'single',
},
{
displayName: 'Value',
name: 'outputValue',
type: 'string',
default: '',
},
],
},
],
displayOptions: {
show: {
operation: ['setOutputs'],
},
},
},
];
export const setCheckIfEvaluatingProperties: INodeProperties[] = [
{
displayName:
'Routes to the evaluation branch if the execution started from an evaluation trigger. Otherwise routes to the normal branch.',
name: 'notice',
type: 'notice',
default: '',
displayOptions: {
show: {
operation: ['checkIfEvaluating'],
},
},
},
];
const correctnessFields: INodeProperties[] = [
{
displayName: 'Expected Answer',
name: 'expectedAnswer',
type: 'string',
default: '',
description: 'The expected output defined in your evaluation dataset, used as ground truth',
displayOptions: {
show: {
operation: ['setMetrics'],
metric: ['correctness', 'stringSimilarity', 'categorization'],
},
},
},
{
displayName: 'Actual Answer',
name: 'actualAnswer',
type: 'string',
default: '',
description: 'The real response generated by AI (e.g. an agent or LLM in the workflow)',
displayOptions: {
show: {
operation: ['setMetrics'],
metric: ['correctness', 'stringSimilarity', 'categorization'],
},
},
},
];
const helpfulnessFields: INodeProperties[] = [
{
displayName: 'User Query',
name: 'userQuery',
type: 'string',
default: '',
description: 'The original input or question submitted by the user',
displayOptions: {
show: {
operation: ['setMetrics'],
metric: ['helpfulness'],
},
},
},
{
displayName: 'Response',
name: 'actualAnswer',
type: 'string',
default: '',
description: 'The response generated by AI (e.g. an agent or LLM in the workflow)',
displayOptions: {
show: {
operation: ['setMetrics'],
metric: ['helpfulness'],
},
},
},
];
function promptFieldForMetric(metric: string, prompt: string): INodeProperties[] {
return [
{
displayName: 'Prompt',
name: 'prompt',
type: 'string',
default: prompt,
description: `Instruction used to guide the model in scoring the actual answers ${metric} against the expected answer`,
typeOptions: {
rows: 4,
},
displayOptions: {
show: {
operation: ['setMetrics'],
metric: [metric],
},
},
},
];
}
function optionsForMetric(
metric: string,
prompt: string[],
defaultName: string,
): INodeProperties[] {
return [
{
displayName: 'Options',
name: 'options',
type: 'collection',
default: {},
placeholder: 'Add Option',
options: [
{
displayName: 'Metric Name',
name: 'metricName',
type: 'string',
default: defaultName,
description: 'Set this parameter if you want to set a custom name to the metric',
},
// eslint-disable-next-line n8n-nodes-base/node-param-default-missing
{
displayName: 'Input Prompt',
name: 'inputPrompt',
type: 'string',
default: prompt[0] ?? '',
typeOptions: {
rows: 4,
},
hint: prompt[1],
},
],
displayOptions: {
show: {
operation: ['setMetrics'],
metric: [metric],
},
},
},
];
}
function optionsForMetricBasic(metric: string, defaultName: string): INodeProperties[] {
return [
{
displayName: 'Options',
name: 'options',
type: 'collection',
default: {},
placeholder: 'Add Option',
options: [
{
displayName: 'Metric Name',
name: 'metricName',
type: 'string',
default: defaultName,
},
],
displayOptions: {
show: {
operation: ['setMetrics'],
metric: [metric],
},
},
},
];
}
const toolsUsedFields: INodeProperties[] = [
{
displayName: 'Expected Tools',
name: 'expectedTools',
type: 'string',
default: '',
description: 'Enter the name(s) of the tool(s) you expect the AI to call (separated by commas)',
placeholder: 'Get Events, Send Email, Search Database',
displayOptions: {
show: {
operation: ['setMetrics'],
metric: ['toolsUsed'],
},
},
},
{
displayName: 'Intermediate Steps (of Agent)',
name: 'intermediateSteps',
type: 'string',
default: '',
hint: 'Map the <code>intermediateSteps</code> field here. To see it, enable returning intermediate steps in the agents options',
displayOptions: {
show: {
operation: ['setMetrics'],
metric: ['toolsUsed'],
},
},
},
];
export const setMetricsProperties: INodeProperties[] = [
{
displayName:
'Metrics measure the quality of an execution. They will be displayed in the evaluations tab, not on the Google Sheet or Data table.',
// "Calculate the score(s) for the evaluation, then map them into this node. They will be displayed in the evaluations tab, not the Google Sheet. <a href='https://docs.n8n.io/advanced-ai/evaluations/metric-based-evaluations/#2-calculate-metrics' target='_blank'>View metric examples</a>",
name: 'notice',
type: 'notice',
default: '',
displayOptions: {
show: {
operation: ['setMetrics'],
},
},
},
{
displayName: 'Metric',
name: 'metric',
type: 'hidden',
default: 'customMetrics',
displayOptions: {
show: {
operation: ['setMetrics'],
'@version': [4.6],
},
},
},
{
displayName: 'Metric',
name: 'metric',
type: 'options',
noDataExpression: true,
// eslint-disable-next-line n8n-nodes-base/node-param-options-type-unsorted-items
options: [
{
// eslint-disable-next-line n8n-nodes-base/node-param-display-name-miscased
name: 'Correctness (AI-based)',
value: 'correctness',
description:
'Whether the answers meaning is consistent with a reference answer. Uses a scale of 1 (worst) to 5 (best).',
},
{
// eslint-disable-next-line n8n-nodes-base/node-param-display-name-miscased
name: 'Helpfulness (AI-based)',
value: 'helpfulness',
description:
'Whether the response addresses the query. Uses a scale of 1 (worst) to 5 (best).',
},
{
name: 'String Similarity',
value: 'stringSimilarity',
description:
'How close the answer is to a reference answer, measured character-by-character (edit distance). Returns a score between 0 and 1.',
},
{
name: 'Categorization',
value: 'categorization',
description:
'Whether the answer exactly matches the reference answer. Returns 1 if so and 0 otherwise.',
},
{
name: 'Tools Used',
value: 'toolsUsed',
description: 'Whether tool(s) were used or not. Returns a score between 0 and 1.',
},
{
name: 'Custom Metrics',
value: 'customMetrics',
description: 'Define your own metric(s)',
},
],
default: DEFAULT_EVALUATION_METRIC,
displayOptions: {
show: {
operation: ['setMetrics'],
'@version': [{ _cnd: { gte: 4.7 } }],
},
},
},
...correctnessFields,
...helpfulnessFields,
...toolsUsedFields,
...promptFieldForMetric('correctness', CORRECTNESS_PROMPT),
...promptFieldForMetric('helpfulness', HELPFULNESS_PROMPT),
{
displayName:
"Calculate the custom metrics before this node, then map them below. <a href='https://docs.n8n.io/advanced-ai/evaluations/metric-based-evaluations/#2-calculate-metrics' target='_blank'>View metric examples</a>",
name: 'notice',
type: 'notice',
default: '',
displayOptions: {
show: {
operation: ['setMetrics'],
metric: ['customMetrics'],
},
},
},
{
displayName: 'Metrics to Return',
name: 'metrics',
type: 'assignmentCollection',
default: {
assignments: [
{
name: '',
value: '',
type: 'number',
},
],
},
typeOptions: {
assignment: {
disableType: true,
defaultType: 'number',
},
},
displayOptions: {
show: {
operation: ['setMetrics'],
metric: ['customMetrics'],
},
},
},
...optionsForMetric('correctness', CORRECTNESS_INPUT_PROMPT, 'Correctness'),
...optionsForMetric('helpfulness', HELPFULNESS_INPUT_PROMPT, 'Helpfulness'),
...optionsForMetricBasic('categorization', 'Categorization'),
...optionsForMetricBasic('stringSimilarity', 'String similarity'),
...optionsForMetricBasic('toolsUsed', 'Tools Used'),
];
export const sourcePicker: INodeProperties = {
displayName: 'Source',
name: 'source',
type: 'options',
options: [
{
// eslint-disable-next-line n8n-nodes-base/node-param-display-name-miscased
name: 'Data table',
value: 'dataTable',
description: 'Load the test dataset from a local Data table',
},
{
name: 'Google Sheets',
value: 'googleSheets',
description: 'Load the test dataset from a Google Sheets document',
},
],
default: 'dataTable',
description: 'Where to get the test dataset from',
};
@@ -0,0 +1,15 @@
{
"node": "n8n-nodes-base.evaluation",
"nodeVersion": "1.0",
"codexVersion": "1.0",
"categories": ["Utility"],
"resources": {
"primaryDocumentation": [
{
"url": "https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.evaluation/"
}
],
"generic": []
},
"alias": ["Test", "Metrics", "Evals", "Set Output", "Set Metrics"]
}
@@ -0,0 +1,140 @@
/* eslint-disable n8n-nodes-base/node-filename-against-convention */
import type {
IExecuteFunctions,
INodeType,
INodeTypeDescription,
INodeExecutionData,
} from 'n8n-workflow';
import { metricRequiresModelConnection } from 'n8n-workflow'; // See packages/workflow/src/evaluation-helpers.ts
import {
setCheckIfEvaluatingProperties,
setInputsProperties,
setMetricsProperties,
setOutputProperties,
sourcePicker,
} from './Description.node';
import { authentication } from '../../Google/Sheet/v2/actions/versionDescription';
import { listSearch, loadOptions, credentialTest } from '../methods';
import {
checkIfEvaluating,
setMetrics,
getInputConnectionTypes,
getOutputConnectionTypes,
setOutputs,
setInputs,
} from '../utils/evaluationUtils';
export class Evaluation implements INodeType {
description: INodeTypeDescription = {
displayName: 'Evaluation',
icon: 'fa:check-double',
name: 'evaluation',
group: ['transform'],
version: [4.6, 4.7, 4.8],
description: 'Runs an evaluation',
eventTriggerDescription: '',
subtitle: '={{$parameter["operation"]}}',
defaults: {
name: 'Evaluation',
color: '#c3c9d5',
},
// Pass function explicitly since expression context doesn't allow imports in getInputConnectionTypes
inputs: `={{(${getInputConnectionTypes})($parameter, ${metricRequiresModelConnection})}}`,
outputs: `={{(${getOutputConnectionTypes})($parameter)}}`,
credentials: [
{
name: 'googleApi',
required: true,
displayOptions: {
show: {
authentication: ['serviceAccount'],
operation: ['setOutputs'],
},
},
testedBy: 'googleApiCredentialTest',
},
{
name: 'googleSheetsOAuth2Api',
required: true,
displayOptions: {
show: {
authentication: ['oAuth2'],
operation: ['setOutputs'],
},
},
},
],
properties: [
{
displayName: 'Operation',
name: 'operation',
type: 'options',
noDataExpression: true,
options: [
{
name: 'Set Inputs',
value: 'setInputs',
},
{
name: 'Set Outputs',
value: 'setOutputs',
},
{
name: 'Set Metrics',
value: 'setMetrics',
},
{
name: 'Check If Evaluating',
value: 'checkIfEvaluating',
},
],
default: 'setOutputs',
},
{
...sourcePicker,
default: 'dataTable',
displayOptions: {
show: { '@version': [{ _cnd: { gte: 4.8 } }], operation: ['setOutputs'] },
},
},
{
...sourcePicker,
default: 'googleSheets',
displayOptions: {
show: { '@version': [{ _cnd: { lte: 4.7 } }], operation: ['setOutputs'] },
},
},
{
...authentication,
displayOptions: {
hide: {
source: ['dataTable'],
},
},
},
...setInputsProperties,
...setOutputProperties,
...setMetricsProperties,
...setCheckIfEvaluatingProperties,
],
};
methods = { loadOptions, listSearch, credentialTest };
async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
const operation = this.getNodeParameter('operation', 0);
if (operation === 'setOutputs') {
return await setOutputs.call(this);
} else if (operation === 'setInputs') {
return setInputs.call(this);
} else if (operation === 'setMetrics') {
return await setMetrics.call(this);
} else if (operation === 'checkIfEvaluating') {
return await checkIfEvaluating.call(this);
}
throw new Error('Unsupported Operation');
}
}
@@ -0,0 +1,15 @@
{
"node": "n8n-nodes-base.evaluationTrigger",
"nodeVersion": "1.0",
"codexVersion": "1.0",
"categories": ["Utility"],
"resources": {
"primaryDocumentation": [
{
"url": "https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.evaluationtrigger/"
}
],
"generic": []
},
"alias": ["Test", "Metrics", "Evals", "Set Output", "Set Metrics"]
}
@@ -0,0 +1,387 @@
/* eslint-disable n8n-nodes-base/node-filename-against-convention */
import type {
INodeType,
INodeTypeDescription,
IExecuteFunctions,
INodeExecutionData,
NodeExecutionWithMetadata,
} from 'n8n-workflow';
import { NodeConnectionTypes, NodeOperationError } from 'n8n-workflow';
import {
getSelectFields as dataTableFilters,
getSelectFilter as getDataTableFilter,
} from '../../DataTable/common/selectMany';
import { document, sheet } from '../../Google/Sheet/GoogleSheetsTrigger.node';
import { readFilter as googleSheetFilters } from '../../Google/Sheet/v2/actions/sheet/read.operation';
import { authentication } from '../../Google/Sheet/v2/actions/versionDescription';
import type { ILookupValues } from '../../Google/Sheet/v2/helpers/GoogleSheets.types';
import { sourcePicker } from '../Evaluation/Description.node';
import { listSearch, loadOptions, credentialTest } from '../methods';
import {
getGoogleSheet,
getResults,
getRowsLeft,
getNumberOfRowsLeftFiltered,
getSheet,
} from '../utils/evaluationTriggerUtils';
export const DEFAULT_STARTING_ROW = 2;
const MAX_ROWS = 1000;
export class EvaluationTrigger implements INodeType {
description: INodeTypeDescription = {
displayName: 'Evaluation Trigger',
icon: 'fa:check-double',
name: 'evaluationTrigger',
group: ['trigger'],
version: [4.6, 4.7],
description: 'Run a test dataset through your workflow to check performance',
eventTriggerDescription: '',
defaults: {
name: 'When fetching a dataset row',
color: '#c3c9d5',
},
inputs: [],
outputs: [NodeConnectionTypes.Main],
properties: [
{
...sourcePicker,
default: 'dataTable',
displayOptions: { show: { '@version': [{ _cnd: { gte: 4.7 } }] } },
},
{
...sourcePicker,
default: 'googleSheets',
displayOptions: { show: { '@version': [{ _cnd: { lte: 4.6 } }] } },
},
{
displayName:
'Pulls a test dataset from a Google Sheet. The workflow will run once for each row, in sequence. Tips for wiring this node up <a href="https://docs.n8n.io/advanced-ai/evaluations/tips-and-common-issues/#combining-multiple-triggers">here</a>.',
name: 'notice',
type: 'notice',
default: '',
displayOptions: { hide: { source: ['dataTable'] } },
},
{
displayName: 'Credentials',
name: 'credentials',
type: 'credentials',
default: '',
displayOptions: { hide: { source: ['dataTable'] } },
},
{
...authentication,
displayOptions: {
hide: {
source: ['dataTable'],
},
},
},
{
...document,
displayName: 'Document Containing Dataset',
hint: 'Example dataset format <a href="https://docs.google.com/spreadsheets/d/1vD_IdeFUg7sHsK9okL6Doy1rGOkWTnPJV3Dro4FBUsY/edit?gid=0#gid=0">here</a>',
displayOptions: { hide: { source: ['dataTable'] } },
},
{
...sheet,
displayName: 'Sheet Containing Dataset',
displayOptions: { hide: { source: ['dataTable'] } },
},
{
// eslint-disable-next-line n8n-nodes-base/node-param-display-name-miscased
displayName: 'Data table',
name: 'dataTableId',
type: 'resourceLocator',
default: { mode: 'list', value: '' },
required: true,
modes: [
{
displayName: 'From List',
name: 'list',
type: 'list',
typeOptions: {
searchListMethod: 'dataTableSearch',
searchable: true,
skipCredentialsCheckInRLC: true,
},
},
{
displayName: 'ID',
name: 'id',
type: 'string',
},
],
displayOptions: { show: { source: ['dataTable'] } },
},
{
displayName: 'Limit Rows',
name: 'limitRows',
type: 'boolean',
default: false,
noDataExpression: true,
description: 'Whether to limit number of rows to process',
},
{
displayName: 'Max Rows to Process',
name: 'maxRows',
type: 'number',
default: 10,
description: 'Maximum number of rows to process',
noDataExpression: false,
displayOptions: { show: { limitRows: [true] } },
},
{ ...googleSheetFilters, displayOptions: { hide: { source: ['dataTable'] } } },
{
displayName: 'Filter Rows',
name: 'filterRows',
type: 'boolean',
default: false,
noDataExpression: true,
description: 'Whether to filter rows to process',
displayOptions: { show: { source: ['dataTable'] } },
},
...dataTableFilters({ show: { filterRows: [true] } }),
],
codex: {
alias: ['Test', 'Metrics', 'Evals', 'Set Output', 'Set Metrics'],
},
credentials: [
{
name: 'googleApi',
required: true,
displayOptions: {
show: {
authentication: ['serviceAccount'],
},
},
testedBy: 'googleApiCredentialTest',
},
{
name: 'googleSheetsOAuth2Api',
required: true,
displayOptions: {
show: {
authentication: ['oAuth2'],
},
},
},
],
};
methods = {
loadOptions,
listSearch,
credentialTest,
};
async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
const inputData = this.getInputData();
const source = this.getNodeParameter('source', 0) as string;
const previousRunRowNumber = inputData?.[0]?.json?.row_number;
const previousRunRowsLeft = inputData?.[0]?.json?._rowsLeft;
if (source === 'dataTable') {
const maxRows = this.getNodeParameter('limitRows', 0, false)
? (this.getNodeParameter('maxRows', 0, MAX_ROWS) as number)
: MAX_ROWS;
if (this.helpers.getDataTableProxy === undefined) {
throw new NodeOperationError(
this.getNode(),
'Attempted to use Data table node but the module is disabled',
);
}
const currentIndex =
typeof previousRunRowNumber === 'number' && previousRunRowsLeft !== 0
? previousRunRowNumber + 1
: 0;
const dataTableId = this.getNodeParameter('dataTableId', 0, undefined, {
extractValue: true,
}) as string;
const dataTableProxy = await this.helpers.getDataTableProxy(dataTableId);
const filter = await getDataTableFilter(this, 0);
const previousRunRowId = inputData?.[0]?.json?.row_id;
let effectiveFilter = filter;
if (typeof previousRunRowId === 'number' && previousRunRowsLeft !== 0) {
effectiveFilter = {
type: 'and',
filters: [
...filter.filters,
{
columnName: 'id',
condition: 'gt',
value: previousRunRowId,
},
],
};
}
const { data, count } = await dataTableProxy.getManyRowsAndCount({
skip: 0,
take: 1,
filter: effectiveFilter,
});
if (data.length === 0) {
throw new NodeOperationError(this.getNode(), 'No row found');
}
const effectiveTotal = Math.min(count, maxRows);
const rowsLeft = Math.max(0, effectiveTotal - 1);
const currentRow = {
json: {
...data[0],
row_number: currentIndex,
row_id: data[0].id,
_rowsLeft: rowsLeft,
},
} satisfies INodeExecutionData;
return [[currentRow]];
} else if (source === 'googleSheets') {
const maxRows = this.getNodeParameter('limitRows', 0, false)
? (this.getNodeParameter('maxRows', 0, MAX_ROWS) as number) + 1
: MAX_ROWS;
const firstDataRow =
typeof previousRunRowNumber === 'number' && previousRunRowsLeft !== 0
? previousRunRowNumber + 1
: DEFAULT_STARTING_ROW;
const rangeOptions = {
rangeDefinition: 'specifyRange',
headerRow: 1,
firstDataRow,
};
const googleSheetInstance = getGoogleSheet.call(this);
const googleSheet = await getSheet.call(this, googleSheetInstance);
const allRows = await getResults.call(
this,
[],
googleSheetInstance,
googleSheet,
rangeOptions,
);
const hasFilter = this.getNodeParameter('filtersUI.values', 0, []) as ILookupValues[];
if (hasFilter.length > 0) {
const currentRow = allRows[0];
const currentRowNumber = currentRow.json?.row_number as number;
if (currentRow === undefined) {
throw new NodeOperationError(this.getNode(), 'No row found');
}
const rowsLeft = await getNumberOfRowsLeftFiltered.call(
this,
googleSheetInstance,
googleSheet.title,
currentRowNumber + 1,
maxRows,
);
currentRow.json._rowsLeft = rowsLeft;
return [[currentRow]];
} else {
const currentRow = allRows.find(
(row) => (row?.json?.row_number as number) === firstDataRow,
);
const rowsLeft = await getRowsLeft.call(
this,
googleSheetInstance,
googleSheet.title,
`${googleSheet.title}!${firstDataRow}:${maxRows}`,
);
if (currentRow === undefined) {
throw new NodeOperationError(this.getNode(), 'No row found');
}
currentRow.json._rowsLeft = rowsLeft;
return [[currentRow]];
}
}
throw new NodeOperationError(this.getNode(), `Unknown source "${source}"`);
}
customOperations = {
dataset: {
async getRows(
this: IExecuteFunctions,
): Promise<INodeExecutionData[][] | NodeExecutionWithMetadata[][] | null> {
try {
const source = this.getNodeParameter('source', 0) as string;
if (source === 'dataTable') {
const maxRows = this.getNodeParameter('limitRows', 0, false)
? (this.getNodeParameter('maxRows', 0, MAX_ROWS) as number)
: MAX_ROWS;
if (this.helpers.getDataTableProxy === undefined) {
throw new NodeOperationError(
this.getNode(),
'Attempted to use Data table node but the module is disabled',
);
}
const dataTableId = this.getNodeParameter('dataTableId', 0, undefined, {
extractValue: true,
}) as string;
const dataTableProxy = await this.helpers.getDataTableProxy(dataTableId);
const filter = await getDataTableFilter(this, 0);
const { data } = await dataTableProxy.getManyRowsAndCount({
skip: 0,
take: maxRows,
filter,
});
const result: INodeExecutionData[] = data.map((row, i) => ({
json: {
...row,
row_id: row.id,
row_number: i,
},
pairedItem: { item: 0 },
}));
return [result];
} else if (source === 'googleSheets') {
const maxRows = this.getNodeParameter('limitRows', 0, false)
? (this.getNodeParameter('maxRows', 0, MAX_ROWS) as number) + 1
: MAX_ROWS;
const googleSheetInstance = getGoogleSheet.call(this);
const googleSheet = await getSheet.call(this, googleSheetInstance);
const results = await getResults.call(this, [], googleSheetInstance, googleSheet, {});
const result = results.slice(0, maxRows - 1);
return [result];
}
throw new NodeOperationError(this.getNode(), `Unknown source "${source}"`);
} catch (error) {
throw new NodeOperationError(this.getNode(), error);
}
},
},
};
}
@@ -0,0 +1,3 @@
export * as loadOptions from './loadOptions';
export * as listSearch from './listSearch';
export * as credentialTest from './../../Google/Sheet/v2/methods/credentialTest';
@@ -0,0 +1,13 @@
import type { ILoadOptionsFunctions, INodeListSearchResult } from 'n8n-workflow';
import { tableSearch } from '../../DataTable/common/methods';
export * from './../../Google/Sheet/v2/methods/listSearch';
export async function dataTableSearch(
this: ILoadOptionsFunctions,
filterString?: string,
prevPaginationToken?: string,
): Promise<INodeListSearchResult> {
return await tableSearch.call(this, filterString, prevPaginationToken);
}
@@ -0,0 +1,18 @@
import type { ILoadOptionsFunctions, INodePropertyOptions } from 'n8n-workflow';
import { getSheetHeaderRow } from '../../Google/Sheet/v2/methods/loadOptions';
export { getConditionsForColumn, getDataTableColumns } from '../../DataTable/common/methods';
export async function getSheetHeaderRowWithGeneratedColumnNames(
this: ILoadOptionsFunctions,
): Promise<INodePropertyOptions[]> {
const returnData = await getSheetHeaderRow.call(this);
return returnData.map((column, i) => {
if (column.value !== '') return column;
const indexBasedValue = `col_${i + 1}`;
return {
name: indexBasedValue,
value: indexBasedValue,
};
});
}
@@ -0,0 +1,488 @@
import { mock } from 'jest-mock-extended';
import {
type IDataTableProjectService,
NodeOperationError,
type AssignmentCollectionValue,
type IExecuteFunctions,
type INodeTypes,
type NodeParameterValueType,
} from 'n8n-workflow';
import { GoogleSheet } from '../../Google/Sheet/v2/helpers/GoogleSheet';
import { Evaluation } from '../Evaluation/Evaluation.node.ee';
describe('Test Evaluation', () => {
const sheetName = 'Sheet5';
const spreadsheetId = '1oqFpPgEPTGDw7BPkp1SfPXq3Cb3Hyr1SROtf-Ec4zvA';
const mockDataTable = mock<IDataTableProjectService>({
getColumns: jest.fn(),
addColumn: jest.fn(),
updateRows: jest.fn(),
});
const mockExecuteFunctions = mock<IExecuteFunctions>({
helpers: { getDataTableProxy: jest.fn().mockResolvedValue(mockDataTable) },
});
beforeEach(() => {
(mockExecuteFunctions.getInputData as jest.Mock).mockReturnValue([{ json: {} }]);
(mockExecuteFunctions.getNode as jest.Mock).mockReturnValue({ typeVersion: 4.6 });
(mockExecuteFunctions.getParentNodes as jest.Mock).mockReturnValue([
{ type: 'n8n-nodes-base.evaluationTrigger', name: 'Evaluation' },
]);
(mockExecuteFunctions.evaluateExpression as jest.Mock).mockReturnValue({
row_number: 23,
foo: 1,
bar: 2,
_rowsLeft: 2,
});
});
afterEach(() => jest.clearAllMocks());
describe('Test Evaluation Node for Set Output', () => {
describe('Data tables', () => {
test('should have data table methods defined', async () => {
const evaluationNode = new Evaluation();
expect(evaluationNode.methods.listSearch.dataTableSearch).toBeDefined();
expect(evaluationNode.methods.loadOptions.getConditionsForColumn).toBeDefined();
expect(evaluationNode.methods.loadOptions.getDataTableColumns).toBeDefined();
});
test('should throw error if output values is empty', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'dataTable',
'outputs.values': [],
dataTableId: 'mockDataTableId',
operation: 'setOutputs',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
await expect(new Evaluation().execute.call(mockExecuteFunctions)).rejects.toThrow(
'No outputs to set',
);
expect(mockDataTable.getColumns).not.toHaveBeenCalled();
expect(mockDataTable.addColumn).not.toHaveBeenCalled();
expect(mockDataTable.updateRows).not.toHaveBeenCalled();
});
test('should return empty when there is no parent evaluation trigger', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'dataTable',
'outputs.values': [{ outputName: 'bob', outputValue: 'clam' }],
dataTableId: 'mockDataTableId',
operation: 'setOutputs',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
mockExecuteFunctions.getParentNodes.mockReturnValue([]);
const result = await new Evaluation().execute.call(mockExecuteFunctions);
expect(result).toEqual([[{ json: {} }]]);
expect(mockDataTable.getColumns).not.toHaveBeenCalled();
expect(mockDataTable.addColumn).not.toHaveBeenCalled();
expect(mockDataTable.updateRows).not.toHaveBeenCalled();
});
test('should update rows and return input data with existing columns', async () => {
(mockExecuteFunctions.evaluateExpression as jest.Mock).mockReturnValue({
row_id: 23,
row_number: 23,
foo: 1,
bar: 2,
_rowsLeft: 2,
});
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'dataTable',
'outputs.values': [{ outputName: 'foo', outputValue: 'clam' }],
dataTableId: 'mockDataTableId',
operation: 'setOutputs',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
mockDataTable.getColumns.mockResolvedValue([
{
id: '1',
index: 0,
name: 'foo',
type: 'string',
dataTableId: 'mockDataTableId',
},
]);
await new Evaluation().execute.call(mockExecuteFunctions);
expect(mockDataTable.getColumns).toHaveBeenCalled();
expect(mockDataTable.addColumn).not.toHaveBeenCalled();
expect(mockDataTable.updateRows).toHaveBeenCalledWith({
filter: {
type: 'and',
filters: [{ columnName: 'id', condition: 'eq', value: 23 }],
},
data: { foo: 'clam' },
});
});
test('should update rows and return input data with new columns', async () => {
(mockExecuteFunctions.evaluateExpression as jest.Mock).mockReturnValue({
row_id: 23,
row_number: 23,
foo: 1,
bar: 2,
_rowsLeft: 2,
});
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'dataTable',
'outputs.values': [
{ outputName: 'foo', outputValue: 'clam' },
{ outputName: 'bar', outputValue: 'baz' },
],
dataTableId: 'mockDataTableId',
operation: 'setOutputs',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
mockDataTable.getColumns.mockResolvedValue([
{
id: '1',
index: 0,
name: 'foo',
type: 'string',
dataTableId: 'mockDataTableId',
},
]);
await new Evaluation().execute.call(mockExecuteFunctions);
expect(mockDataTable.getColumns).toHaveBeenCalled();
expect(mockDataTable.addColumn).toHaveBeenCalledWith({
name: 'bar',
type: 'string',
});
expect(mockDataTable.updateRows).toHaveBeenCalledWith({
filter: {
type: 'and',
filters: [{ columnName: 'id', condition: 'eq', value: 23 }],
},
data: { foo: 'clam', bar: 'baz' },
});
});
});
describe('Google Sheets', () => {
jest.spyOn(GoogleSheet.prototype, 'spreadsheetGetSheet').mockImplementation(async () => {
return { sheetId: 1, title: sheetName };
});
jest.spyOn(GoogleSheet.prototype, 'updateRows').mockImplementation(async () => {
return { sheetId: 1, title: sheetName };
});
jest.spyOn(GoogleSheet.prototype, 'batchUpdate').mockImplementation(async () => {
return { sheetId: 1, title: sheetName };
});
test('credential test for googleApi should be in methods', async () => {
const evaluationNode = new Evaluation();
expect(evaluationNode.methods.credentialTest.googleApiCredentialTest).toBeDefined();
});
test('should throw error if output values is empty', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
'outputs.values': [],
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
operation: 'setOutputs',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
await expect(new Evaluation().execute.call(mockExecuteFunctions)).rejects.toThrow(
'No outputs to set',
);
expect(GoogleSheet.prototype.updateRows).not.toBeCalled();
expect(GoogleSheet.prototype.batchUpdate).not.toBeCalled();
});
test('should update rows and return input data for existing headers', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'googleSheets',
'outputs.values': [{ outputName: 'foo', outputValue: 'clam' }],
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
operation: 'setOutputs',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
await new Evaluation().execute.call(mockExecuteFunctions);
expect(GoogleSheet.prototype.updateRows).toHaveBeenCalledWith(
sheetName,
[['foo', 'bar']],
'RAW',
1,
);
expect(GoogleSheet.prototype.batchUpdate).toHaveBeenCalledWith(
[
{
range: 'Sheet5!A23',
values: [['clam']],
},
],
'RAW',
);
});
test('should return empty when there is no parent evaluation trigger', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'googleSheets',
'outputs.values': [{ outputName: 'bob', outputValue: 'clam' }],
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
operation: 'setOutputs',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
mockExecuteFunctions.getParentNodes.mockReturnValue([]);
const result = await new Evaluation().execute.call(mockExecuteFunctions);
expect(result).toEqual([[{ json: {} }]]);
expect(GoogleSheet.prototype.updateRows).not.toBeCalled();
expect(GoogleSheet.prototype.batchUpdate).not.toBeCalled();
});
test('should update rows and return input data for new headers', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'googleSheets',
'outputs.values': [{ outputName: 'bob', outputValue: 'clam' }],
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
operation: 'setOutputs',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
await new Evaluation().execute.call(mockExecuteFunctions);
expect(GoogleSheet.prototype.updateRows).toHaveBeenCalledWith(
sheetName,
[['foo', 'bar', 'bob']],
'RAW',
1,
);
expect(GoogleSheet.prototype.batchUpdate).toHaveBeenCalledWith(
[
{
range: 'Sheet5!C23',
values: [['clam']],
},
],
'RAW',
);
});
});
});
describe('Test Evaluation Node for Set Metrics', () => {
const nodeTypes = mock<INodeTypes>();
const evaluationMetricsNode = new Evaluation();
let mockExecuteFunction: IExecuteFunctions;
function getMockExecuteFunction(metrics: AssignmentCollectionValue['assignments']) {
return {
getInputData: jest.fn().mockReturnValue([{}]),
getNodeParameter: jest.fn((param: string, _: number) => {
if (param === 'metrics') {
return { assignments: metrics };
}
if (param === 'operation') {
return 'setMetrics';
}
if (param === 'metric') {
return 'customMetrics';
}
return param;
}),
getNode: jest.fn().mockReturnValue({
typeVersion: 1,
}),
} as unknown as IExecuteFunctions;
}
beforeAll(() => {
mockExecuteFunction = getMockExecuteFunction([
{
id: '1',
name: 'Accuracy',
value: 0.95,
type: 'number',
},
{
id: '2',
name: 'Latency',
value: 100,
type: 'number',
},
]);
nodeTypes.getByName.mockReturnValue(evaluationMetricsNode);
jest.clearAllMocks();
});
describe('execute', () => {
it('should output the defined metrics', async () => {
const result = await evaluationMetricsNode.execute.call(mockExecuteFunction);
expect(result).toHaveLength(1);
expect(result[0]).toHaveLength(1);
const outputItem = result[0][0].json;
expect(outputItem).toEqual({
Accuracy: 0.95,
Latency: 100,
});
});
it('should handle no metrics defined', async () => {
mockExecuteFunction = getMockExecuteFunction([]);
const result = await evaluationMetricsNode.execute.call(mockExecuteFunction);
expect(result).toHaveLength(1);
expect(result[0]).toHaveLength(1);
expect(result[0][0].json).toEqual({});
});
it('should convert string values to numbers', async () => {
const mockExecuteWithStringValues = getMockExecuteFunction([
{
id: '1',
name: 'Accuracy',
value: '0.95',
type: 'number',
},
{
id: '2',
name: 'Latency',
value: '100',
type: 'number',
},
]);
const result = await evaluationMetricsNode.execute.call(mockExecuteWithStringValues);
expect(result).toHaveLength(1);
expect(result[0]).toHaveLength(1);
const outputItem = result[0][0].json;
expect(outputItem).toEqual({
Accuracy: 0.95,
Latency: 100,
});
});
it('should throw error for non-numeric string values', async () => {
const mockExecuteWithInvalidValue = getMockExecuteFunction([
{
id: '1',
name: 'Accuracy',
value: 'not-a-number',
type: 'number',
},
]);
await expect(
evaluationMetricsNode.execute.call(mockExecuteWithInvalidValue),
).rejects.toThrow(NodeOperationError);
});
});
});
describe('Test Evaluation Node for Check If Evaluating', () => {
beforeEach(() => {
(mockExecuteFunctions.getInputData as jest.Mock).mockReturnValue([{ json: {} }]);
(mockExecuteFunctions.getNode as jest.Mock).mockReturnValue({ typeVersion: 4.6 });
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
operation: 'checkIfEvaluating',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
});
afterEach(() => jest.clearAllMocks());
test('should return output in normal branch if normal execution', async () => {
(mockExecuteFunctions.getParentNodes as jest.Mock).mockReturnValue([]);
const result = await new Evaluation().execute.call(mockExecuteFunctions);
expect(result).toEqual([[], [{ json: {} }]]);
});
test('should return output in evaluation branch if evaluation execution', async () => {
(mockExecuteFunctions.getParentNodes as jest.Mock).mockReturnValue([
{ type: 'n8n-nodes-base.evaluationTrigger', name: 'Evaluation' },
]);
const result = await new Evaluation().execute.call(mockExecuteFunctions);
expect(result).toEqual([[{ json: {} }], []]);
});
});
});
@@ -0,0 +1,736 @@
import { mock, mockDeep } from 'jest-mock-extended';
import type { IExecuteFunctions, NodeParameterValueType } from 'n8n-workflow';
import { GoogleSheet } from '../../Google/Sheet/v2/helpers/GoogleSheet';
import { EvaluationTrigger } from '../EvaluationTrigger/EvaluationTrigger.node.ee';
import * as utils from '../utils/evaluationTriggerUtils';
describe('Evaluation Trigger Node', () => {
const sheetName = 'Sheet5';
const spreadsheetId = '1oqFpPgEPTGDw7BPkp1SfPXq3Cb3Hyr1SROtf-Ec4zvA';
let mockExecuteFunctions = mock<IExecuteFunctions>({
getInputData: jest.fn().mockReturnValue([{ json: {} }]),
getNode: jest.fn().mockReturnValue({ typeVersion: 4.6 }),
});
let mockDataTable: { getManyRowsAndCount: jest.Mock; getColumns: jest.Mock };
describe('execute', () => {
describe('without filters', () => {
beforeEach(() => {
jest.resetAllMocks();
mockExecuteFunctions = mock<IExecuteFunctions>({
getInputData: jest.fn().mockReturnValue([{ json: {} }]),
getNode: jest.fn().mockReturnValue({ typeVersion: 4.6 }),
});
jest.spyOn(GoogleSheet.prototype, 'spreadsheetGetSheet').mockImplementation(async () => {
return { sheetId: 1, title: sheetName };
});
// Mocks getResults() and getRowsLeft()
jest.spyOn(GoogleSheet.prototype, 'getData').mockImplementation(async (range: string) => {
if (range === `${sheetName}!1:1`) {
return [['Header1', 'Header2']];
} else if (range === `${sheetName}!2:1000`) {
return [
['Header1', 'Header2'],
['Value1', 'Value2'],
['Value3', 'Value4'],
];
} else if (range === `${sheetName}!2:2`) {
// getRowsLeft with limit
return [];
} else if (range === sheetName) {
return [
['Header1', 'Header2'],
['Value1', 'Value2'],
['Value3', 'Value4'],
];
} else {
return [];
}
});
});
test('credential test for googleApi should be in methods', async () => {
const evaluationTrigger = new EvaluationTrigger();
expect(evaluationTrigger.methods.credentialTest.googleApiCredentialTest).toBeDefined();
});
test('should return a single row from google sheet', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'googleSheets',
options: {},
'filtersUI.values': [],
combineFilters: 'AND',
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
const result = await new EvaluationTrigger().execute.call(mockExecuteFunctions);
expect(result).toEqual([
[
{
json: {
row_number: 2,
Header1: 'Value1',
Header2: 'Value2',
_rowsLeft: 2,
},
pairedItem: {
item: 0,
},
},
],
]);
});
test('should return the next row from google sheet', async () => {
mockExecuteFunctions.getInputData.mockReturnValue([
{
json: {
row_number: 2,
Header1: 'Value1',
Header2: 'Value2',
_rowsLeft: 1,
},
pairedItem: {
item: 0,
input: undefined,
},
},
]);
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'googleSheets',
options: {},
'filtersUI.values': [],
combineFilters: 'AND',
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
const result = await new EvaluationTrigger().execute.call(mockExecuteFunctions);
expect(result).toEqual([
[
{
json: {
row_number: 3,
Header1: 'Value3',
Header2: 'Value4',
_rowsLeft: 0,
},
pairedItem: {
item: 0,
},
},
],
]);
});
test('should return the first row from google sheet if no rows left', async () => {
mockExecuteFunctions.getInputData.mockReturnValue([
{
json: {
row_number: 3,
Header1: 'Value3',
Header2: 'Value4',
_rowsLeft: 0,
},
pairedItem: {
item: 0,
input: undefined,
},
},
]);
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'googleSheets',
options: {},
'filtersUI.values': [],
combineFilters: 'AND',
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
const result = await new EvaluationTrigger().execute.call(mockExecuteFunctions);
expect(result).toEqual([
[
{
json: {
row_number: 2,
Header1: 'Value1',
Header2: 'Value2',
_rowsLeft: 2,
},
pairedItem: {
item: 0,
},
},
],
]);
});
test('should return a single row from google sheet with limit', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'googleSheets',
options: {},
'filtersUI.values': [],
combineFilters: 'AND',
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
limitRows: true,
maxRows: 1,
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
const result = await new EvaluationTrigger().execute.call(mockExecuteFunctions);
expect(result).toEqual([
[
{
json: {
row_number: 2,
Header1: 'Value1',
Header2: 'Value2',
_rowsLeft: 0,
},
pairedItem: {
item: 0,
},
},
],
]);
});
});
describe('with filters', () => {
beforeEach(() => {
jest.resetAllMocks();
mockExecuteFunctions = mock<IExecuteFunctions>({
getInputData: jest.fn().mockReturnValue([{ json: {} }]),
getNode: jest.fn().mockReturnValue({ typeVersion: 4.6 }),
});
jest.spyOn(GoogleSheet.prototype, 'spreadsheetGetSheet').mockImplementation(async () => {
return { sheetId: 1, title: sheetName };
});
});
test('should return a single row from google sheet using filter', async () => {
jest
.spyOn(GoogleSheet.prototype, 'getData')
.mockResolvedValueOnce([
// operationResult
['Header1', 'Header2'],
['Value1', 'Value2'],
['Value3', 'Value4'],
])
.mockResolvedValueOnce([
// rowsLeft
['Header1', 'Header2'],
['Value1', 'Value2'],
['Value3', 'Value4'],
]);
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'googleSheets',
limitRows: true,
maxRows: 2,
'filtersUI.values': [{ lookupColumn: 'Header1', lookupValue: 'Value1' }],
options: {},
combineFilters: 'AND',
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
jest.spyOn(utils, 'getRowsLeft').mockResolvedValue(0);
const evaluationTrigger = new EvaluationTrigger();
const result = await evaluationTrigger.execute.call(mockExecuteFunctions);
expect(result).toEqual([
[
{
json: {
row_number: 2,
Header1: 'Value1',
Header2: 'Value2',
_rowsLeft: 0,
},
pairedItem: {
item: 0,
},
},
],
]);
});
});
describe('Data tables with filters', () => {
beforeEach(() => {
jest.resetAllMocks();
mockDataTable = {
getManyRowsAndCount: jest.fn(),
getColumns: jest.fn().mockResolvedValue([{ name: 'processed', type: 'number' }]),
};
mockExecuteFunctions = mockDeep<IExecuteFunctions>({
getNode: jest.fn().mockReturnValue({ typeVersion: 4.7 }),
helpers: {
getDataTableProxy: jest.fn().mockResolvedValue(mockDataTable),
},
});
});
test('should process rows sequentially with filters when dataset changes', async () => {
// Simulate the user's scenario: 5 rows with processed=1, updating to processed=2 after each execution
// With each execution, one row is processed and thus no longer matches the filter
mockDataTable.getManyRowsAndCount
.mockResolvedValueOnce({
data: [{ id: 1, processed: 1 }],
count: 5,
})
.mockResolvedValueOnce({
data: [{ id: 2, processed: 1 }],
count: 4,
})
.mockResolvedValueOnce({
data: [{ id: 3, processed: 1 }],
count: 3,
});
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'dataTable',
limitRows: false,
dataTableId: 'mockDataTableId',
'filters.conditions': [
{
keyName: 'processed',
condition: 'eq',
keyValue: '1',
},
],
matchType: 'anyCondition',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
const evaluationTrigger = new EvaluationTrigger();
// First execution - no previous data
mockExecuteFunctions.getInputData.mockReturnValue([{ json: {} }]);
const result1 = await evaluationTrigger.execute.call(mockExecuteFunctions);
expect(result1[0][0].json.row_id).toBe(1);
expect(result1[0][0].json.row_number).toBe(0);
expect(result1[0][0].json._rowsLeft).toBe(4);
// Verify first call used user filter only (no id filter yet)
expect(mockDataTable.getManyRowsAndCount).toHaveBeenNthCalledWith(1, {
skip: 0,
take: 1,
filter: {
type: 'or',
filters: [
{
columnName: 'processed',
condition: 'eq',
value: '1',
},
],
},
});
// Second execution - previous row was id=1
mockExecuteFunctions.getInputData.mockReturnValue(result1[0]);
const result2 = await evaluationTrigger.execute.call(mockExecuteFunctions);
expect(result2[0][0].json.row_id).toBe(2);
expect(result2[0][0].json.row_number).toBe(1);
// Verify second call includes id > 1 filter
expect(mockDataTable.getManyRowsAndCount).toHaveBeenNthCalledWith(2, {
skip: 0,
take: 1,
filter: {
type: 'and',
filters: [
{
columnName: 'processed',
condition: 'eq',
value: '1',
},
{
columnName: 'id',
condition: 'gt',
value: 1,
},
],
},
});
// Third execution - previous row was id=2
mockExecuteFunctions.getInputData.mockReturnValue(result2[0]);
const result3 = await evaluationTrigger.execute.call(mockExecuteFunctions);
expect(result3[0][0].json.row_id).toBe(3);
expect(result3[0][0].json.row_number).toBe(2);
// Verify third call includes id > 2 filter
expect(mockDataTable.getManyRowsAndCount).toHaveBeenNthCalledWith(3, {
skip: 0,
take: 1,
filter: {
type: 'and',
filters: [
{
columnName: 'processed',
condition: 'eq',
value: '1',
},
{
columnName: 'id',
condition: 'gt',
value: 2,
},
],
},
});
});
});
});
describe('customOperations.dataset.getRows', () => {
describe('Data tables', () => {
beforeEach(() => {
jest.resetAllMocks();
mockDataTable = {
getManyRowsAndCount: jest.fn().mockResolvedValue({
data: [
{ id: 1, field1: 'value1', field2: 'value2' },
{ id: 2, field1: 'value3', field2: 'value4' },
],
}),
getColumns: jest.fn().mockResolvedValue([
{ name: 'field1', type: 'string' },
{ name: 'field2', type: 'string' },
]),
};
mockExecuteFunctions = mockDeep<IExecuteFunctions>({
getNode: jest.fn().mockReturnValue({ typeVersion: 4.7 }),
helpers: {
getDataTableProxy: jest.fn().mockResolvedValue(mockDataTable),
},
});
});
test('should return the rows with limits applied, without filters', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'dataTable',
limitRows: true,
maxRows: 2,
dataTableId: 'mockDataTableId',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
const result = await new EvaluationTrigger().customOperations.dataset.getRows.call(
mockExecuteFunctions,
);
expect(mockDataTable.getManyRowsAndCount).toHaveBeenCalledWith({
skip: 0,
take: 2,
filter: { filters: [], type: 'or' },
});
expect(result).toEqual([
[
{
json: {
id: 1,
row_id: 1,
row_number: 0,
field1: 'value1',
field2: 'value2',
},
pairedItem: {
item: 0,
},
},
{
json: {
id: 2,
row_id: 2,
row_number: 1,
field1: 'value3',
field2: 'value4',
},
pairedItem: {
item: 0,
},
},
],
]);
});
test('should return the rows with limits applied, with filters', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'dataTable',
limitRows: true,
maxRows: 2,
dataTableId: 'mockDataTableId',
'filters.conditions': [
{
keyName: 'field1',
condition: 'like',
keyValue: '1',
},
{
keyName: 'field2',
condition: 'eq',
keyValue: 'value4',
},
],
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
await new EvaluationTrigger().customOperations.dataset.getRows.call(mockExecuteFunctions);
expect(mockDataTable.getManyRowsAndCount).toHaveBeenCalledWith({
skip: 0,
take: 2,
filter: {
filters: [
{
columnName: 'field1',
condition: 'like',
value: '1',
},
{
columnName: 'field2',
condition: 'eq',
value: 'value4',
},
],
type: 'or',
},
});
});
});
describe('Google Sheets', () => {
beforeEach(() => {
jest.resetAllMocks();
mockExecuteFunctions = mock<IExecuteFunctions>({
getNode: jest.fn().mockReturnValue({ typeVersion: 4.6 }),
});
jest.spyOn(GoogleSheet.prototype, 'spreadsheetGetSheet').mockImplementation(async () => {
return { sheetId: 1, title: sheetName };
});
// Mocks getResults() and getRowsLeft()
jest.spyOn(GoogleSheet.prototype, 'getData').mockImplementation(async (range: string) => {
if (range === `${sheetName}!1:1`) {
return [['Header1', 'Header2']];
} else if (range === `${sheetName}!2:1000`) {
return [
['Header1', 'Header2'],
['Value1', 'Value2'],
['Value3', 'Value4'],
];
} else if (range === `${sheetName}!2:2`) {
// getRowsLeft with limit
return [];
} else if (range === sheetName) {
return [
['Header1', 'Header2'],
['Value1', 'Value2'],
['Value3', 'Value4'],
];
} else {
return [];
}
});
});
test('should return the sheet with limits applied, without filters', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'googleSheets',
options: {},
'filtersUI.values': [],
combineFilters: 'AND',
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
limitRows: true,
maxRows: 2,
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
const result = await new EvaluationTrigger().customOperations.dataset.getRows.call(
mockExecuteFunctions,
);
expect(result).toEqual([
[
{
json: {
row_number: 2,
Header1: 'Value1',
Header2: 'Value2',
},
pairedItem: {
item: 0,
},
},
{
json: {
row_number: 3,
Header1: 'Value3',
Header2: 'Value4',
},
pairedItem: {
item: 0,
},
},
],
]);
});
test('should return all relevant rows from google sheet using filters', async () => {
mockExecuteFunctions.getInputData.mockReturnValue([{ json: {} }]);
jest
.spyOn(GoogleSheet.prototype, 'getData')
.mockResolvedValueOnce([
// operationResult
['Header1', 'Header2'],
['Value1', 'Value2'],
['Value3', 'Value4'],
['Value1', 'Value4'],
])
.mockResolvedValueOnce([
// rowsLeft
['Header1', 'Header2'],
['Value1', 'Value2'],
['Value3', 'Value4'],
['Value1', 'Value4'],
]);
mockExecuteFunctions.getNodeParameter.mockImplementation(
(key: string, _: number, fallbackValue?: string | number | boolean | object) => {
const mockParams: { [key: string]: unknown } = {
source: 'googleSheets',
'filtersUI.values': [{ lookupColumn: 'Header1', lookupValue: 'Value1' }],
options: {},
combineFilters: 'AND',
documentId: {
mode: 'id',
value: spreadsheetId,
},
sheetName,
sheetMode: 'id',
};
return (mockParams[key] ?? fallbackValue) as NodeParameterValueType;
},
);
jest.spyOn(utils, 'getRowsLeft').mockResolvedValue(0);
const evaluationTrigger = new EvaluationTrigger();
const result =
await evaluationTrigger.customOperations.dataset.getRows.call(mockExecuteFunctions);
expect(result).toEqual([
[
{
json: { row_number: 2, Header1: 'Value1', Header2: 'Value2' },
pairedItem: {
item: 0,
},
},
{
json: { row_number: 4, Header1: 'Value1', Header2: 'Value4' },
pairedItem: {
item: 0,
},
},
],
]);
});
});
});
});
@@ -0,0 +1,102 @@
import type { IExecuteFunctions, INodeExecutionData } from 'n8n-workflow';
import { readSheet } from '../../Google/Sheet/v2/actions/utils/readOperation';
import { GoogleSheet } from '../../Google/Sheet/v2/helpers/GoogleSheet';
import { getFilteredResults } from '../utils/evaluationTriggerUtils';
jest.mock('../../Google/Sheet/v2/actions/utils/readOperation', () => ({
readSheet: jest.fn(),
}));
describe('getFilteredResults', () => {
let mockThis: IExecuteFunctions;
let mockGoogleSheet: GoogleSheet;
beforeEach(() => {
// Mock the `this` context
mockThis = {
getNode: jest.fn().mockReturnValue({ typeVersion: 1 }),
} as unknown as IExecuteFunctions;
// Mock the GoogleSheet instance
mockGoogleSheet = new GoogleSheet('mockSpreadsheetId', mockThis);
// Reset mocks before each test
jest.clearAllMocks();
});
it('should return filtered results based on endingRow', async () => {
// Arrange
const mockOperationResult: INodeExecutionData[] = [];
const mockResult = { title: 'Sheet1', sheetId: 1 };
const startingRow = 1;
const endingRow = 3;
(readSheet as jest.Mock).mockResolvedValue([
{ json: { row_number: 1, data: 'Row 1' } },
{ json: { row_number: 2, data: 'Row 2' } },
{ json: { row_number: 3, data: 'Row 3' } },
{ json: { row_number: 4, data: 'Row 4' } },
]);
// Act
const result = await getFilteredResults.call(
mockThis,
mockOperationResult,
mockGoogleSheet,
mockResult,
startingRow,
endingRow,
);
// Assert
expect(readSheet).toHaveBeenCalledWith(
mockGoogleSheet,
'Sheet1',
0,
mockOperationResult,
1,
[],
undefined,
{
rangeDefinition: 'specifyRange',
headerRow: 1,
firstDataRow: startingRow,
includeHeadersWithEmptyCells: true,
},
);
expect(result).toEqual([
{ json: { row_number: 1, data: 'Row 1' } },
{ json: { row_number: 2, data: 'Row 2' } },
{ json: { row_number: 3, data: 'Row 3' } },
]);
});
it('should return an empty array if no rows match the filter', async () => {
// Arrange
const mockOperationResult: INodeExecutionData[] = [];
const mockResult = { title: 'Sheet1', sheetId: 1 };
const startingRow = 1;
const endingRow = 0;
(readSheet as jest.Mock).mockResolvedValue([
{ json: { row_number: 1, data: 'Row 1' } },
{ json: { row_number: 2, data: 'Row 2' } },
]);
// Act
const result = await getFilteredResults.call(
mockThis,
mockOperationResult,
mockGoogleSheet,
mockResult,
startingRow,
endingRow,
);
// Assert
expect(readSheet).toHaveBeenCalled();
expect(result).toEqual([]);
});
});
@@ -0,0 +1,497 @@
import type { IExecuteFunctions } from 'n8n-workflow';
import { UserError } from 'n8n-workflow';
import { setInputs, setOutputs } from '../utils/evaluationUtils';
jest.mock('../utils/evaluationTriggerUtils', () => ({
getGoogleSheet: jest.fn(),
getSheet: jest.fn(),
}));
import { getGoogleSheet, getSheet } from '../utils/evaluationTriggerUtils';
import { mockDeep } from 'jest-mock-extended';
describe('setInputs', () => {
const mockThis = (options: Partial<any> = {}) =>
mockDeep<IExecuteFunctions>({
getNode: jest.fn().mockReturnValue({ name: 'EvalNode' }),
getParentNodes: jest
.fn()
.mockReturnValue([{ name: 'EvalTrigger', type: 'n8n-nodes-base.evaluationTrigger' }]),
evaluateExpression: jest.fn().mockReturnValue(true),
getNodeParameter: jest.fn().mockReturnValue([
{ inputName: 'foo', inputValue: 'bar' },
{ inputName: 'baz', inputValue: 'qux' },
]),
getInputData: jest.fn().mockReturnValue([{ json: { test: 1 } }]),
addExecutionHints: jest.fn(),
getMode: jest.fn().mockReturnValue('evaluation'),
...options,
});
it('should return input data with evaluationData when inputs are provided', () => {
const context = mockThis();
const result = setInputs.call(context);
expect(result).toHaveLength(1);
expect(result[0][0].evaluationData).toEqual({ foo: 'bar', baz: 'qux' });
});
it('should throw UserError if no input fields are provided', () => {
const context = mockThis({
getNodeParameter: jest.fn().mockReturnValue([]),
});
expect(() => setInputs.call(context)).toThrow(UserError);
});
it('should add execution hints and return input data if not started from evaluation trigger', () => {
const context = mockThis({
getParentNodes: jest.fn().mockReturnValue([]),
getInputData: jest.fn().mockReturnValue([{ json: { test: 2 } }]),
});
const result = setInputs.call(context);
expect(context.addExecutionHints).toHaveBeenCalledWith(
expect.objectContaining({
message: expect.stringContaining('No inputs were set'),
}),
);
expect(result).toEqual([[{ json: { test: 2 } }]]);
});
it('should add execution hints and return input data if evalTriggerOutput is falsy', () => {
const context = mockThis({
evaluateExpression: jest.fn().mockReturnValue(undefined),
getInputData: jest.fn().mockReturnValue([{ json: { test: 3 } }]),
});
const result = setInputs.call(context);
expect(context.addExecutionHints).toHaveBeenCalled();
expect(result).toEqual([[{ json: { test: 3 } }]]);
});
});
describe('setOutputs', () => {
describe('common', () => {
const mockThis = (options: Partial<IExecuteFunctions> = {}) =>
mockDeep<IExecuteFunctions>({
getNode: jest.fn().mockReturnValue({ name: 'EvalNode' }),
getParentNodes: jest
.fn()
.mockReturnValue([{ name: 'EvalTrigger', type: 'n8n-nodes-base.evaluationTrigger' }]),
evaluateExpression: jest.fn().mockImplementation((expr: string) => {
if (expr.includes('isExecuted')) return true;
if (expr.includes('first().json')) return { row_id: 1, inputField: 'inputValue' };
return true;
}),
getNodeParameter: jest.fn().mockImplementation((param: string) => {
if (param === 'outputs.values') {
return [{ outputName: 'result', outputValue: 'success' }];
}
}),
getInputData: jest.fn().mockReturnValue([{ json: { test: 1 } }]),
addExecutionHints: jest.fn(),
getMode: jest.fn().mockReturnValue('evaluation'),
...options,
});
it('should throw UserError if no output fields are provided', async () => {
const context = mockThis({
getNodeParameter: jest.fn().mockReturnValue([]),
});
await expect(setOutputs.call(context)).rejects.toThrow(UserError);
await expect(setOutputs.call(context)).rejects.toThrow('No outputs to set');
});
it('should add execution hints and return input data if not started from evaluation trigger', async () => {
const context = mockThis({
getParentNodes: jest.fn().mockReturnValue([]),
getInputData: jest.fn().mockReturnValue([{ json: { test: 2 } }]),
});
const result = await setOutputs.call(context);
expect(context.addExecutionHints).toHaveBeenCalledWith(
expect.objectContaining({
message: expect.stringContaining("execution didn't start from an evaluation trigger"),
}),
);
expect(result).toEqual([[{ json: { test: 2 } }]]);
expect(getGoogleSheet).not.toHaveBeenCalled();
});
it('should add execution hints and return input data if evalTriggerOutput is falsy', async () => {
const context = mockThis({
evaluateExpression: jest.fn().mockImplementation((expr: string) => {
if (expr.includes('isExecuted')) return false;
return true;
}),
getInputData: jest.fn().mockReturnValue([{ json: { test: 3 } }]),
});
const result = await setOutputs.call(context);
expect(context.addExecutionHints).toHaveBeenCalledWith(
expect.objectContaining({
message: expect.stringContaining("execution didn't start from an evaluation trigger"),
}),
);
expect(result).toEqual([[{ json: { test: 3 } }]]);
});
});
describe('Data tables', () => {
const outputValues = [
{ outputName: 'result', outputValue: 'success' },
{ outputName: 'score', outputValue: 95 },
{ outputName: 'active', outputValue: true },
{ outputName: 'timestamp', outputValue: new Date('2025-09-18T12:34:56Z') },
{ outputName: 'data', outputValue: { key: 'value' } },
];
const mockDataTable = {
updateRows: jest.fn(),
getColumns: jest.fn().mockReturnValue(outputValues.map((o) => ({ name: o.outputName }))),
addColumn: jest.fn(),
};
const mockThis = (options: Partial<IExecuteFunctions> = {}) =>
mockDeep<IExecuteFunctions>({
getNode: jest.fn().mockReturnValue({ name: 'EvalNode' }),
getParentNodes: jest
.fn()
.mockReturnValue([{ name: 'EvalTrigger', type: 'n8n-nodes-base.evaluationTrigger' }]),
evaluateExpression: jest.fn().mockImplementation((expr: string) => {
if (expr.includes('isExecuted')) return true;
if (expr.includes('first().json')) return { row_id: 1, inputField: 'inputValue' };
return true;
}),
getNodeParameter: jest.fn().mockImplementation((param: string) => {
if (param === 'outputs.values') {
return outputValues;
} else if (param === 'source') {
return 'dataTable';
} else if (param === 'dataTableId') {
return 'mockDataTableId';
}
}),
getInputData: jest.fn().mockReturnValue([{ json: { test: 1 } }]),
addExecutionHints: jest.fn(),
getMode: jest.fn().mockReturnValue('evaluation'),
helpers: {
getDataTableProxy: jest.fn().mockResolvedValue(mockDataTable),
},
...options,
});
beforeEach(() => {
jest.clearAllMocks();
});
it('should set outputs to Data table and return evaluation data', async () => {
const context = mockThis();
const result = await setOutputs.call(context);
expect(mockDataTable.getColumns).toHaveBeenCalled();
expect(mockDataTable.addColumn).not.toHaveBeenCalled();
expect(mockDataTable.updateRows).toHaveBeenCalledWith({
filter: {
type: 'and',
filters: [
{
columnName: 'id',
condition: 'eq',
value: 1,
},
],
},
data: {
result: 'success',
score: 95,
active: true,
timestamp: new Date('2025-09-18T12:34:56'),
data: '{"key":"value"}',
},
});
expect(result).toHaveLength(1);
expect(result[0][0].evaluationData).toEqual({
result: 'success',
score: 95,
active: true,
timestamp: new Date('2025-09-18T12:34:56'),
data: { key: 'value' },
});
});
it('should set outputs to Data table, correct subsequent row', async () => {
const context = mockThis({
evaluateExpression: jest.fn().mockImplementation((expr: string) => {
if (expr.includes('isExecuted')) return true;
if (expr.includes('first().json')) return { row_id: 3, inputField: 'inputValue' };
return true;
}),
});
await setOutputs.call(context);
expect(mockDataTable.updateRows).toHaveBeenCalledWith(
expect.objectContaining({
filter: {
type: 'and',
filters: [
{
columnName: 'id',
condition: 'eq',
value: 3,
},
],
},
}),
);
});
it("should create columns if they don't exist, string", async () => {
const context = mockThis({
getNodeParameter: jest.fn().mockImplementation((param: string) => {
if (param === 'outputs.values') {
return [...outputValues, { outputName: 'new_column', outputValue: 'new_value' }];
} else if (param === 'source') {
return 'dataTable';
} else if (param === 'dataTableId') {
return 'mockDataTableId';
}
}),
});
const result = await setOutputs.call(context);
expect(mockDataTable.getColumns).toHaveBeenCalled();
expect(mockDataTable.addColumn).toHaveBeenCalledWith({
name: 'new_column',
type: 'string',
});
expect(mockDataTable.updateRows).toHaveBeenCalledWith({
filter: {
type: 'and',
filters: [
{
columnName: 'id',
condition: 'eq',
value: 1,
},
],
},
data: {
result: 'success',
score: 95,
active: true,
timestamp: new Date('2025-09-18T12:34:56'),
data: '{"key":"value"}',
new_column: 'new_value',
},
});
expect(result).toHaveLength(1);
expect(result[0][0].evaluationData).toEqual({
result: 'success',
score: 95,
active: true,
timestamp: new Date('2025-09-18T12:34:56'),
data: { key: 'value' },
new_column: 'new_value',
});
});
it("should create columns if they don't exist, number", async () => {
const context = mockThis({
getNodeParameter: jest.fn().mockImplementation((param: string) => {
if (param === 'outputs.values') {
return [...outputValues, { outputName: 'new_column', outputValue: 123.45 }];
} else if (param === 'source') {
return 'dataTable';
} else if (param === 'dataTableId') {
return 'mockDataTableId';
}
}),
});
await setOutputs.call(context);
expect(mockDataTable.addColumn).toHaveBeenCalledWith({
name: 'new_column',
type: 'number',
});
});
it("should create columns if they don't exist, boolean", async () => {
const context = mockThis({
getNodeParameter: jest.fn().mockImplementation((param: string) => {
if (param === 'outputs.values') {
return [...outputValues, { outputName: 'new_column', outputValue: true }];
} else if (param === 'source') {
return 'dataTable';
} else if (param === 'dataTableId') {
return 'mockDataTableId';
}
}),
});
await setOutputs.call(context);
expect(mockDataTable.addColumn).toHaveBeenCalledWith({
name: 'new_column',
type: 'boolean',
});
});
it("should create columns if they don't exist, date", async () => {
const context = mockThis({
getNodeParameter: jest.fn().mockImplementation((param: string) => {
if (param === 'outputs.values') {
return [...outputValues, { outputName: 'new_column', outputValue: new Date() }];
} else if (param === 'source') {
return 'dataTable';
} else if (param === 'dataTableId') {
return 'mockDataTableId';
}
}),
});
await setOutputs.call(context);
expect(mockDataTable.addColumn).toHaveBeenCalledWith({
name: 'new_column',
type: 'date',
});
});
it("should create columns if they don't exist, null", async () => {
const context = mockThis({
getNodeParameter: jest.fn().mockImplementation((param: string) => {
if (param === 'outputs.values') {
return [...outputValues, { outputName: 'new_column', outputValue: null }];
} else if (param === 'source') {
return 'dataTable';
} else if (param === 'dataTableId') {
return 'mockDataTableId';
}
}),
});
await setOutputs.call(context);
expect(mockDataTable.addColumn).toHaveBeenCalledWith({
name: 'new_column',
type: 'string',
});
});
});
describe('Google Sheets', () => {
const mockGoogleSheetInstance = {
updateRows: jest.fn(),
prepareDataForUpdatingByRowNumber: jest.fn().mockReturnValue({
updateData: [{ range: 'Sheet1!A2:C2', values: [['foo', 'bar']] }],
}),
batchUpdate: jest.fn(),
};
const mockSheet = {
title: 'Sheet1',
};
const mockThis = (options: Partial<any> = {}) =>
mockDeep<IExecuteFunctions>({
getNode: jest.fn().mockReturnValue({ name: 'EvalNode' }),
getParentNodes: jest
.fn()
.mockReturnValue([{ name: 'EvalTrigger', type: 'n8n-nodes-base.evaluationTrigger' }]),
evaluateExpression: jest.fn().mockImplementation((expr) => {
if (expr.includes('isExecuted')) return true;
if (expr.includes('first().json')) return { row_number: 2, inputField: 'inputValue' };
return true;
}),
getNodeParameter: jest.fn().mockImplementation((param: string) => {
if (param === 'outputs.values') {
return [
{ outputName: 'result', outputValue: 'success' },
{ outputName: 'score', outputValue: '95' },
];
} else if (param === 'source') {
return 'googleSheets';
}
}),
getInputData: jest.fn().mockReturnValue([{ json: { test: 1 } }]),
addExecutionHints: jest.fn(),
getMode: jest.fn().mockReturnValue('evaluation'),
...options,
});
beforeEach(() => {
jest.clearAllMocks();
(getGoogleSheet as jest.Mock).mockReturnValue(mockGoogleSheetInstance);
(getSheet as jest.Mock).mockResolvedValue(mockSheet);
});
it('should set outputs to Google Sheet and return evaluation data', async () => {
const context = mockThis();
const result = await setOutputs.call(context);
expect(getGoogleSheet).toHaveBeenCalled();
expect(getSheet).toHaveBeenCalledWith(mockGoogleSheetInstance);
expect(mockGoogleSheetInstance.updateRows).toHaveBeenCalledWith(
'Sheet1',
[['inputField', 'result', 'score']],
'RAW',
1,
);
expect(mockGoogleSheetInstance.prepareDataForUpdatingByRowNumber).toHaveBeenCalledWith(
[{ row_number: 2, result: 'success', score: '95' }],
'Sheet1!A:Z',
[['inputField', 'result', 'score']],
);
expect(mockGoogleSheetInstance.batchUpdate).toHaveBeenCalledWith(
[{ range: 'Sheet1!A2:C2', values: [['foo', 'bar']] }],
'RAW',
);
expect(result).toHaveLength(1);
expect(result[0][0].evaluationData).toEqual({ result: 'success', score: '95' });
});
it('should handle row_number as string "row_number" by using 1', async () => {
const context = mockThis({
evaluateExpression: jest.fn().mockImplementation((expr) => {
if (expr.includes('isExecuted')) return true;
if (expr.includes('first().json'))
return { row_number: 'row_number', inputField: 'inputValue' };
return true;
}),
});
const result = await setOutputs.call(context);
expect(mockGoogleSheetInstance.prepareDataForUpdatingByRowNumber).toHaveBeenCalledWith(
[{ row_number: 1, result: 'success', score: '95' }],
'Sheet1!A:Z',
[['inputField', 'result', 'score']],
);
expect(result).toHaveLength(1);
});
it('should add new column names that are not in existing columns', async () => {
const context = mockThis({
evaluateExpression: jest.fn().mockImplementation((expr) => {
if (expr.includes('isExecuted')) return true;
if (expr.includes('first().json')) return { row_number: 2, existingCol: 'value' };
return true;
}),
getNodeParameter: jest.fn().mockImplementation((param) => {
if (param === 'outputs.values') {
return [{ outputName: 'newCol', outputValue: 'newValue' }];
} else if (param === 'source') {
return 'googleSheets';
}
}),
});
const result = await setOutputs.call(context);
expect(mockGoogleSheetInstance.updateRows).toHaveBeenCalledWith(
'Sheet1',
[['existingCol', 'newCol']],
'RAW',
1,
);
expect(result).toHaveLength(1);
});
});
});
@@ -0,0 +1,63 @@
/* eslint-disable n8n-nodes-base/node-param-display-name-miscased */
import { type ILoadOptionsFunctions } from 'n8n-workflow';
import { getSheetHeaderRow } from '../../Google/Sheet/v2/methods/loadOptions';
import { getSheetHeaderRowWithGeneratedColumnNames } from '../methods/loadOptions';
jest.mock('../../Google/Sheet/v2/methods/loadOptions', () => ({
getSheetHeaderRow: jest.fn(),
}));
describe('getSheetHeaderRowWithGeneratedColumnNames', () => {
let mockThis: ILoadOptionsFunctions;
beforeEach(() => {
mockThis = {
getNodeParameter: jest.fn(),
getCredentials: jest.fn(),
} as unknown as ILoadOptionsFunctions;
jest.clearAllMocks();
});
it('should return column names as-is if they are not empty', async () => {
(getSheetHeaderRow as jest.Mock).mockResolvedValue([
{ name: 'Column1', value: 'Column1' },
{ name: 'Column2', value: 'Column2' },
]);
const result = await getSheetHeaderRowWithGeneratedColumnNames.call(mockThis);
expect(getSheetHeaderRow).toHaveBeenCalled();
expect(result).toEqual([
{ name: 'Column1', value: 'Column1' },
{ name: 'Column2', value: 'Column2' },
]);
});
it('should generate column names for empty values', async () => {
(getSheetHeaderRow as jest.Mock).mockResolvedValue([
{ name: '', value: '' },
{ name: 'Column2', value: 'Column2' },
{ name: '', value: '' },
]);
const result = await getSheetHeaderRowWithGeneratedColumnNames.call(mockThis);
expect(getSheetHeaderRow).toHaveBeenCalled();
expect(result).toEqual([
{ name: 'col_1', value: 'col_1' },
{ name: 'Column2', value: 'Column2' },
{ name: 'col_3', value: 'col_3' },
]);
});
it('should handle an empty header row gracefully', async () => {
(getSheetHeaderRow as jest.Mock).mockResolvedValue([]);
const result = await getSheetHeaderRowWithGeneratedColumnNames.call(mockThis);
expect(getSheetHeaderRow).toHaveBeenCalled();
expect(result).toEqual([]);
});
});
@@ -0,0 +1,791 @@
import { mock } from 'jest-mock-extended';
import { NodeOperationError } from 'n8n-workflow';
import type { IExecuteFunctions, INode, AssignmentCollectionValue } from 'n8n-workflow';
import type { BaseLanguageModel } from '@langchain/core/language_models/base';
import { ChatPromptTemplate } from '@langchain/core/prompts';
import type { Runnable } from '@langchain/core/runnables';
import { metricHandlers } from '../utils/metricHandlers';
// Mock the validateEntry function
jest.mock('../../Set/v2/helpers/utils', () => ({
validateEntry: jest.fn((name: string, _type: string, value: any) => ({
name,
value,
})),
}));
describe('metricHandlers', () => {
let mockExecuteFunctions: jest.Mocked<IExecuteFunctions>;
let mockNode: INode;
beforeEach(() => {
mockExecuteFunctions = mock<IExecuteFunctions>();
mockNode = {
id: 'test-node',
name: 'Test Node',
type: 'n8n-nodes-base.evaluation',
typeVersion: 1,
position: [0, 0],
parameters: {},
};
mockExecuteFunctions.getNode.mockReturnValue(mockNode);
});
afterEach(() => {
jest.clearAllMocks();
});
describe('customMetrics', () => {
it('should process valid custom metrics', async () => {
const metricsData: AssignmentCollectionValue = {
assignments: [
{ id: '1', name: 'Metric1', value: 5, type: 'number' },
{ id: '2', name: 'Metric2', value: '10', type: 'number' },
{ id: '3', name: 'Metric3', value: 7.5, type: 'number' },
],
};
mockExecuteFunctions.getNodeParameter.mockReturnValue(metricsData);
const result = await metricHandlers.customMetrics.call(mockExecuteFunctions, 0);
expect(result).toEqual({
Metric1: 5,
Metric2: 10,
Metric3: 7.5,
});
});
it('should throw error for non-numeric values', async () => {
const metricsData: AssignmentCollectionValue = {
assignments: [{ id: '1', name: 'Metric1', value: 'not-a-number', type: 'number' }],
};
mockExecuteFunctions.getNodeParameter.mockReturnValue(metricsData);
await expect(metricHandlers.customMetrics.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should throw error for missing metric name', async () => {
const metricsData: AssignmentCollectionValue = {
assignments: [{ id: '1', name: '', value: 5, type: 'number' }],
};
mockExecuteFunctions.getNodeParameter.mockReturnValue(metricsData);
await expect(metricHandlers.customMetrics.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should handle empty assignments array', async () => {
const metricsData: AssignmentCollectionValue = {
assignments: [],
};
mockExecuteFunctions.getNodeParameter.mockReturnValue(metricsData);
const result = await metricHandlers.customMetrics.call(mockExecuteFunctions, 0);
expect(result).toEqual({});
});
it('should handle undefined assignments', async () => {
const metricsData: AssignmentCollectionValue = { assignments: [] };
mockExecuteFunctions.getNodeParameter.mockReturnValue(metricsData);
const result = await metricHandlers.customMetrics.call(mockExecuteFunctions, 0);
expect(result).toEqual({});
});
});
describe('toolsUsed', () => {
it('should return correct tool usage metrics', async () => {
const expectedTools = 'calculator, search';
const intermediateSteps = [
{ action: { tool: 'calculator' } },
{ action: { tool: 'calculator' } },
{ action: { tool: 'search' } },
];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
if (paramName === 'options.metricName') return 'Tools Used';
return undefined;
});
const result = await metricHandlers.toolsUsed.call(mockExecuteFunctions, 0);
expect(result).toEqual({
'Tools Used': 1,
});
});
it('should return 0 for unused tools', async () => {
const expectedTools = 'calculator, search';
const intermediateSteps = [{ action: { tool: 'calculator' } }];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
if (paramName === 'options.metricName') return 'Tools Used';
return undefined;
});
const result = await metricHandlers.toolsUsed.call(mockExecuteFunctions, 0);
expect(result).toEqual({
'Tools Used': 0.5,
});
});
it('should handle tool names with spaces and special characters', async () => {
const expectedTools = 'Get Events, Send Email, Search Database';
const intermediateSteps = [
{ action: { tool: 'Get_Events' } },
{ action: { tool: 'Send_Email' } },
{ action: { tool: 'Search_Database' } },
];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
if (paramName === 'options.metricName') return 'Tools Used';
return undefined;
});
const result = await metricHandlers.toolsUsed.call(mockExecuteFunctions, 0);
expect(result).toEqual({
'Tools Used': 1,
});
});
it('should work case-insensitively', async () => {
const expectedTools = 'Get Events, send email, SEARCH DATABASE';
const intermediateSteps = [
{ action: { tool: 'get_events' } },
{ action: { tool: 'SEND_EMAIL' } },
{ action: { tool: 'Search_Database' } },
];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
if (paramName === 'options.metricName') return 'Tools Used';
return undefined;
});
const result = await metricHandlers.toolsUsed.call(mockExecuteFunctions, 0);
expect(result).toEqual({
'Tools Used': 1,
});
});
it('should handle mixed case and format variations', async () => {
const expectedTools = 'calculator tool, Search Engine, data-processor';
const intermediateSteps = [
{ action: { tool: 'Calculator_Tool' } },
{ action: { tool: 'search_engine' } },
// data-processor is not used, so partial match
];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
if (paramName === 'options.metricName') return 'Tools Used';
return undefined;
});
const result = await metricHandlers.toolsUsed.call(mockExecuteFunctions, 0);
// 2 out of 3 tools used = 2/3 ≈ 0.6667
expect(result).toEqual({
'Tools Used': 2 / 3,
});
});
it('should throw error for missing expected tools', async () => {
const expectedTools = '';
const intermediateSteps: any[] = [];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
return undefined;
});
await expect(metricHandlers.toolsUsed.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should throw error for undefined expected tools', async () => {
const expectedTools = undefined;
const intermediateSteps: any[] = [];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
return undefined;
});
await expect(metricHandlers.toolsUsed.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
describe('intermediate steps validation', () => {
it('should throw error for missing intermediate steps parameter', async () => {
const expectedTools = 'calculator';
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return undefined;
return undefined;
});
await expect(metricHandlers.toolsUsed.call(mockExecuteFunctions, 0)).rejects.toThrow(
new NodeOperationError(mockNode, 'Intermediate steps missing', {
description:
"Make sure to enable returning intermediate steps in your agent node's options, then map them in here",
}),
);
});
it('should throw error for empty object intermediate steps', async () => {
const expectedTools = 'calculator';
const intermediateSteps = {};
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
return undefined;
});
await expect(metricHandlers.toolsUsed.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should throw error for string intermediate steps', async () => {
const expectedTools = 'calculator';
const intermediateSteps = 'not an array';
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
return undefined;
});
await expect(metricHandlers.toolsUsed.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should throw error for null intermediate steps', async () => {
const expectedTools = 'calculator';
const intermediateSteps = null;
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
return undefined;
});
await expect(metricHandlers.toolsUsed.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should handle empty array intermediate steps gracefully', async () => {
const expectedTools = 'calculator, search';
const intermediateSteps: any[] = [];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
if (paramName === 'options.metricName') return 'Tools Used';
return undefined;
});
const result = await metricHandlers.toolsUsed.call(mockExecuteFunctions, 0);
expect(result).toEqual({
'Tools Used': 0,
});
});
it('should handle malformed intermediate steps objects', async () => {
const expectedTools = 'calculator, search';
const intermediateSteps = [
{ action: { tool: 'calculator' } }, // valid
{ action: {} }, // missing tool property
{ notAction: { tool: 'search' } }, // wrong structure
{}, // completely empty
];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
if (paramName === 'options.metricName') return 'Tools Used';
return undefined;
});
const result = await metricHandlers.toolsUsed.call(mockExecuteFunctions, 0);
// Only 'calculator' should match (1 out of 2 expected tools)
expect(result).toEqual({
'Tools Used': 0.5,
});
});
it('should handle intermediate steps with null/undefined tool names', async () => {
const expectedTools = 'calculator, search';
const intermediateSteps = [
{ action: { tool: 'calculator' } }, // valid
{ action: { tool: null } }, // null tool
{ action: { tool: undefined } }, // undefined tool
{ action: { tool: '' } }, // empty string tool
];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
if (paramName === 'options.metricName') return 'Tools Used';
return undefined;
});
const result = await metricHandlers.toolsUsed.call(mockExecuteFunctions, 0);
// Only 'calculator' should match (1 out of 2 expected tools)
expect(result).toEqual({
'Tools Used': 0.5,
});
});
it('should handle intermediate steps with non-string tool names', async () => {
const expectedTools = 'calculator, search';
const intermediateSteps = [
{ action: { tool: 'calculator' } }, // valid
{ action: { tool: 123 } }, // number
{ action: { tool: { name: 'search' } } }, // object
{ action: { tool: ['search'] } }, // array
];
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedTools') return expectedTools;
if (paramName === 'intermediateSteps') return intermediateSteps;
if (paramName === 'options.metricName') return 'Tools Used';
return undefined;
});
// This should not throw an error, but might have unexpected behavior
// depending on how the comparison works
const result = await metricHandlers.toolsUsed.call(mockExecuteFunctions, 0);
// Only 'calculator' should match reliably (1 out of 2 expected tools)
expect(result).toEqual({
'Tools Used': 0.5,
});
});
});
});
describe('categorization', () => {
it('should return 1 for exact match', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'expected answer';
if (paramName === 'actualAnswer') return 'expected answer';
if (paramName === 'options.metricName') return 'Categorization';
return undefined;
});
const result = await metricHandlers.categorization.call(mockExecuteFunctions, 0);
expect(result).toEqual({ Categorization: 1 });
});
it('should return 0 for non-match', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'expected answer';
if (paramName === 'actualAnswer') return 'different answer';
if (paramName === 'options.metricName') return 'Categorization';
return undefined;
});
const result = await metricHandlers.categorization.call(mockExecuteFunctions, 0);
expect(result).toEqual({ Categorization: 0 });
});
it('should use custom metric name', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'expected answer';
if (paramName === 'actualAnswer') return 'expected answer';
if (paramName === 'options.metricName') return 'Custom Categorization';
return undefined;
});
const result = await metricHandlers.categorization.call(mockExecuteFunctions, 0);
expect(result).toEqual({ 'Custom Categorization': 1 });
});
it('should handle whitespace trimming', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return ' expected answer ';
if (paramName === 'actualAnswer') return 'expected answer';
if (paramName === 'options.metricName') return 'Categorization';
return undefined;
});
const result = await metricHandlers.categorization.call(mockExecuteFunctions, 0);
expect(result).toEqual({ Categorization: 1 });
});
it('should throw error for missing expected answer', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return '';
if (paramName === 'actualAnswer') return 'actual answer';
return undefined;
});
await expect(metricHandlers.categorization.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should throw error for missing actual answer', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'expected answer';
if (paramName === 'actualAnswer') return '';
return undefined;
});
await expect(metricHandlers.categorization.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
});
describe('stringSimilarity', () => {
it('should return inverted similarity score', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'hello';
if (paramName === 'actualAnswer') return 'helo';
if (paramName === 'options.metricName') return 'String similarity';
return undefined;
});
const result = await metricHandlers.stringSimilarity.call(mockExecuteFunctions, 0);
// Edit distance is 1, longer string length is 5, so similarity = 1 - (1/5) = 0.8
expect(result).toEqual({ 'String similarity': 0.8 });
});
it('should return 1 for identical strings', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'hello';
if (paramName === 'actualAnswer') return 'hello';
if (paramName === 'options.metricName') return 'String similarity';
return undefined;
});
const result = await metricHandlers.stringSimilarity.call(mockExecuteFunctions, 0);
expect(result).toEqual({ 'String similarity': 1 });
});
it('should handle whitespace trimming', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return ' hello ';
if (paramName === 'actualAnswer') return 'hello';
if (paramName === 'options.metricName') return 'String similarity';
return undefined;
});
const result = await metricHandlers.stringSimilarity.call(mockExecuteFunctions, 0);
expect(result).toEqual({ 'String similarity': 1 });
});
it('should return low similarity for very different strings', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'hello';
if (paramName === 'actualAnswer') return 'world';
if (paramName === 'options.metricName') return 'String similarity';
return undefined;
});
const result = await metricHandlers.stringSimilarity.call(mockExecuteFunctions, 0);
// Edit distance is 4, longer string length is 5, so similarity = 1 - (4/5) = 0.2
expect(result['String similarity']).toBeCloseTo(0.2, 2);
});
it('should handle different string lengths', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'hello';
if (paramName === 'actualAnswer') return 'hello world';
if (paramName === 'options.metricName') return 'String similarity';
return undefined;
});
const result = await metricHandlers.stringSimilarity.call(mockExecuteFunctions, 0);
// Edit distance is 6, longer string length is 11, so similarity = 1 - (6/11) ≈ 0.45
expect(result['String similarity']).toBeCloseTo(0.45, 2);
});
it('should throw error for missing expected answer', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return '';
if (paramName === 'actualAnswer') return 'actual answer';
return undefined;
});
await expect(metricHandlers.stringSimilarity.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should throw error for missing actual answer', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'expected answer';
if (paramName === 'actualAnswer') return '';
return undefined;
});
await expect(metricHandlers.stringSimilarity.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
});
describe('helpfulness', () => {
let mockLLM: jest.Mocked<BaseLanguageModel>;
beforeEach(() => {
mockLLM = mock<BaseLanguageModel>();
mockExecuteFunctions.getInputConnectionData.mockResolvedValue(mockLLM);
});
it('should return helpfulness score from LLM', async () => {
const mockResponse = {
extended_reasoning: 'The response is very helpful...',
reasoning_summary: 'Response directly addresses the query',
score: 4,
};
// Mock the LLM with withStructuredOutput
const mockLLMWithStructuredOutput = mock<Runnable>();
mockLLMWithStructuredOutput.invoke.mockResolvedValue(mockResponse);
mockLLM.withStructuredOutput = jest.fn().mockReturnValue(mockLLMWithStructuredOutput);
// Mock ChatPromptTemplate.fromMessages to return a chain that can be piped
const mockChatPromptTemplate = mock<ChatPromptTemplate>();
mockChatPromptTemplate.pipe.mockReturnValue(mockLLMWithStructuredOutput);
// Mock the static method
jest.spyOn(ChatPromptTemplate, 'fromMessages').mockReturnValue(mockChatPromptTemplate);
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'userQuery') return 'What is the capital of France?';
if (paramName === 'actualAnswer') return 'Paris is the capital of France.';
if (paramName === 'prompt') return 'You are an AI assistant...';
if (paramName === 'options.inputPrompt')
return 'Query: {user_query}\\nResponse: {actual_answer}';
if (paramName === 'options.metricName') return 'Helpfulness';
return undefined;
});
const result = await metricHandlers.helpfulness.call(mockExecuteFunctions, 0);
expect(result).toEqual({ Helpfulness: 4 });
});
it('should throw error for missing user query', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'userQuery') return '';
if (paramName === 'actualAnswer') return 'Some response';
return undefined;
});
await expect(metricHandlers.helpfulness.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should throw error for missing actual answer', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'userQuery') return 'Some query';
if (paramName === 'actualAnswer') return '';
return undefined;
});
await expect(metricHandlers.helpfulness.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should throw error when no LLM is connected', async () => {
mockExecuteFunctions.getInputConnectionData.mockResolvedValue(null);
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'userQuery') return 'What is the capital of France?';
if (paramName === 'actualAnswer') return 'Paris is the capital of France.';
return undefined;
});
await expect(metricHandlers.helpfulness.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should handle LLM errors gracefully', async () => {
const mockError = new Error('LLM processing failed');
const mockFinalChain = mock<Runnable>();
mockFinalChain.invoke.mockRejectedValue(mockError);
const mockMiddleChain = mock<Runnable>();
mockMiddleChain.pipe.mockReturnValue(mockFinalChain);
const mockChatPromptTemplate = mock<ChatPromptTemplate>();
mockChatPromptTemplate.pipe.mockReturnValue(mockMiddleChain);
jest.spyOn(ChatPromptTemplate, 'fromMessages').mockReturnValue(mockChatPromptTemplate);
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'userQuery') return 'What is the capital of France?';
if (paramName === 'actualAnswer') return 'Paris is the capital of France.';
if (paramName === 'prompt') return 'You are an AI assistant...';
if (paramName === 'options.inputPrompt')
return 'Query: {user_query}\\nResponse: {actual_answer}';
if (paramName === 'options.metricName') return 'Helpfulness';
return undefined;
});
await expect(metricHandlers.helpfulness.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
});
describe('correctness', () => {
let mockLLM: jest.Mocked<BaseLanguageModel>;
beforeEach(() => {
mockLLM = mock<BaseLanguageModel>();
mockExecuteFunctions.getInputConnectionData.mockResolvedValue(mockLLM);
});
it('should return correctness score from LLM', async () => {
const mockResponse = {
extended_reasoning: 'The response is factually correct...',
reasoning_summary: 'Response matches expected answer',
score: 5,
};
// Mock the LLM with withStructuredOutput
const mockLLMWithStructuredOutput = mock<Runnable>();
mockLLMWithStructuredOutput.invoke.mockResolvedValue(mockResponse);
mockLLM.withStructuredOutput = jest.fn().mockReturnValue(mockLLMWithStructuredOutput);
const mockChatPromptTemplate = mock<ChatPromptTemplate>();
mockChatPromptTemplate.pipe.mockReturnValue(mockLLMWithStructuredOutput);
jest.spyOn(ChatPromptTemplate, 'fromMessages').mockReturnValue(mockChatPromptTemplate);
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'Paris';
if (paramName === 'actualAnswer') return 'Paris is the capital of France.';
if (paramName === 'prompt') return 'You are an AI assistant...';
if (paramName === 'options.inputPrompt')
return 'Expected: {expected_answer}\\nActual: {actual_answer}';
if (paramName === 'options.metricName') return 'Correctness';
return undefined;
});
const result = await metricHandlers.correctness.call(mockExecuteFunctions, 0);
expect(result).toEqual({ Correctness: 5 });
});
it('should throw error for missing expected answer', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return '';
if (paramName === 'actualAnswer') return 'Some response';
return undefined;
});
await expect(metricHandlers.correctness.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should throw error for missing actual answer', async () => {
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'Expected answer';
if (paramName === 'actualAnswer') return '';
return undefined;
});
await expect(metricHandlers.correctness.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should throw error when no LLM is connected', async () => {
mockExecuteFunctions.getInputConnectionData.mockResolvedValue(null);
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'Paris';
if (paramName === 'actualAnswer') return 'Paris is the capital of France.';
return undefined;
});
await expect(metricHandlers.correctness.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
it('should handle LLM errors gracefully', async () => {
const mockError = new Error('LLM processing failed');
const mockFinalChain = mock<Runnable>();
mockFinalChain.invoke.mockRejectedValue(mockError);
const mockMiddleChain = mock<Runnable>();
mockMiddleChain.pipe.mockReturnValue(mockFinalChain);
const mockChatPromptTemplate = mock<ChatPromptTemplate>();
mockChatPromptTemplate.pipe.mockReturnValue(mockMiddleChain);
jest.spyOn(ChatPromptTemplate, 'fromMessages').mockReturnValue(mockChatPromptTemplate);
mockExecuteFunctions.getNodeParameter.mockImplementation((paramName: string) => {
if (paramName === 'expectedAnswer') return 'Paris';
if (paramName === 'actualAnswer') return 'Paris is the capital of France.';
if (paramName === 'prompt') return 'You are an AI assistant...';
if (paramName === 'options.inputPrompt')
return 'Expected: {expected_answer}\\nActual: {actual_answer}';
if (paramName === 'options.metricName') return 'Correctness';
return undefined;
});
await expect(metricHandlers.correctness.call(mockExecuteFunctions, 0)).rejects.toThrow(
NodeOperationError,
);
});
});
});
@@ -0,0 +1,132 @@
import type { IExecuteFunctions, INodeExecutionData, IDataObject } from 'n8n-workflow';
import { readSheet } from '../../Google/Sheet/v2/actions/utils/readOperation';
import { GoogleSheet } from '../../Google/Sheet/v2/helpers/GoogleSheet';
import type { ResourceLocator } from '../../Google/Sheet/v2/helpers/GoogleSheets.types';
import { getSpreadsheetId } from '../../Google/Sheet/v2/helpers/GoogleSheets.utils';
export async function getSheet(
this: IExecuteFunctions,
googleSheet: GoogleSheet,
): Promise<{
title: string;
sheetId: number;
}> {
const sheetWithinDocument = this.getNodeParameter('sheetName', 0, undefined, {
extractValue: true,
}) as string;
const { mode: sheetMode } = this.getNodeParameter('sheetName', 0) as {
mode: ResourceLocator;
};
return await googleSheet.spreadsheetGetSheet(this.getNode(), sheetMode, sheetWithinDocument);
}
export function getGoogleSheet(this: IExecuteFunctions) {
const { mode, value } = this.getNodeParameter('documentId', 0) as IDataObject;
const spreadsheetId = getSpreadsheetId(this.getNode(), mode as ResourceLocator, value as string);
const googleSheet = new GoogleSheet(spreadsheetId, this);
return googleSheet;
}
export async function getFilteredResults(
this: IExecuteFunctions,
operationResult: INodeExecutionData[],
googleSheet: GoogleSheet,
result: { title: string; sheetId: number },
startingRow: number,
endingRow: number,
): Promise<INodeExecutionData[]> {
const sheetName = result.title;
operationResult = await readSheet.call(
this,
googleSheet,
sheetName,
0,
operationResult,
this.getNode().typeVersion,
[],
undefined,
{
rangeDefinition: 'specifyRange',
headerRow: 1,
firstDataRow: startingRow,
includeHeadersWithEmptyCells: true,
},
);
return operationResult.filter((row) => (row?.json?.row_number as number) <= endingRow);
}
export async function getNumberOfRowsLeftFiltered(
this: IExecuteFunctions,
googleSheet: GoogleSheet,
sheetName: string,
startingRow: number,
endingRow: number,
) {
const remainderSheet: INodeExecutionData[] = await readSheet.call(
this,
googleSheet,
sheetName,
0,
[],
this.getNode().typeVersion,
[],
undefined,
{
rangeDefinition: 'specifyRange',
headerRow: 1,
firstDataRow: startingRow,
},
);
return remainderSheet.filter((row) => (row?.json?.row_number as number) <= endingRow).length;
}
export async function getResults(
this: IExecuteFunctions,
operationResult: INodeExecutionData[],
googleSheet: GoogleSheet,
result: { title: string; sheetId: number },
rangeOptions: IDataObject,
): Promise<INodeExecutionData[]> {
const sheetName = result.title;
operationResult = await readSheet.call(
this,
googleSheet,
sheetName,
0,
operationResult,
this.getNode().typeVersion,
[],
undefined,
{ ...rangeOptions, includeHeadersWithEmptyCells: true },
);
return operationResult;
}
export async function getRowsLeft(
this: IExecuteFunctions,
googleSheet: GoogleSheet,
sheetName: string,
rangeString: string,
) {
const remainderSheet: INodeExecutionData[] = await readSheet.call(
this,
googleSheet,
sheetName,
0,
[],
this.getNode().typeVersion,
[],
rangeString,
);
return remainderSheet.length;
}
@@ -0,0 +1,346 @@
import {
UserError,
NodeOperationError,
EVALUATION_TRIGGER_NODE_TYPE,
jsonStringify,
} from 'n8n-workflow';
import type {
INodeParameters,
IDataObject,
IExecuteFunctions,
INodeExecutionData,
JsonObject,
JsonValue,
DataTableColumnJsType,
} from 'n8n-workflow';
import { getGoogleSheet, getSheet } from './evaluationTriggerUtils';
import { metricHandlers } from './metricHandlers';
import { composeReturnItem } from '../../Set/v2/helpers/utils';
import assert from 'node:assert';
function withEvaluationData(this: IExecuteFunctions, data: JsonObject): INodeExecutionData[] {
const inputData = this.getInputData();
if (!inputData.length) {
return inputData;
}
const isEvaluationMode = this.getMode() === 'evaluation';
return [
{
...inputData[0],
// test-runner only looks at first item. Don't need to duplicate the data for each item
evaluationData: isEvaluationMode ? data : undefined,
},
...inputData.slice(1),
];
}
function isOutputsArray(
value: unknown,
): value is Array<{ outputName: string; outputValue: JsonValue }> {
return (
Array.isArray(value) &&
value.every(
(item) =>
typeof item === 'object' &&
item !== null &&
'outputName' in item &&
'outputValue' in item &&
typeof item.outputName === 'string',
)
);
}
export function toDataTableValue(value: JsonValue): DataTableColumnJsType {
if (
typeof value === 'string' ||
typeof value === 'number' ||
typeof value === 'boolean' ||
value instanceof Date ||
value === null
)
return value;
return jsonStringify(value);
}
const toDataTableColumnType = (value: JsonValue) => {
switch (typeof value) {
case 'string':
return 'string';
case 'number':
return 'number';
case 'boolean':
return 'boolean';
case 'object':
if (value instanceof Date) {
return 'date';
}
// this catches null, arrays and objects
return 'string';
default:
return 'string';
}
};
export async function setOutputs(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
const evaluationNode = this.getNode();
const parentNodes = this.getParentNodes(evaluationNode.name);
const evalTrigger = parentNodes.find((node) => node.type === EVALUATION_TRIGGER_NODE_TYPE);
const isEvalTriggerExecuted = evalTrigger
? this.evaluateExpression(`{{ $('${evalTrigger?.name}').isExecuted }}`, 0)
: false;
if (!evalTrigger || !isEvalTriggerExecuted) {
this.addExecutionHints({
message: "No outputs were set since the execution didn't start from an evaluation trigger",
location: 'outputPane',
});
return [this.getInputData()];
}
const outputFields = this.getNodeParameter('outputs.values', 0, []);
assert(
isOutputsArray(outputFields),
'Invalid output fields format. Expected an array of objects with outputName and outputValue properties.',
);
if (outputFields.length === 0) {
throw new UserError('No outputs to set', {
description: 'Add outputs to write back to the Google Sheet using the Add Output button',
});
}
const evaluationTrigger = this.evaluateExpression(
`{{ $('${evalTrigger.name}').first().json }}`,
0,
) as IDataObject;
const rowNumber =
evaluationTrigger.row_number === 'row_number' ? 1 : evaluationTrigger.row_number;
const columnNames = Object.keys(evaluationTrigger).filter(
(key) => key !== 'row_number' && key !== '_rowsLeft',
);
outputFields.forEach(({ outputName }) => {
if (!columnNames.includes(outputName)) {
columnNames.push(outputName);
}
});
const outputs = outputFields.reduce<JsonObject>((acc, { outputName, outputValue }) => {
acc[outputName] = outputValue;
return acc;
}, {});
const source = this.getNodeParameter('source', 0) as string;
if (source === 'dataTable') {
if (this.helpers.getDataTableProxy === undefined) {
throw new NodeOperationError(
this.getNode(),
'Attempted to use Data table node but the module is disabled',
);
}
const dataTableId = this.getNodeParameter('dataTableId', 0, undefined, {
extractValue: true,
}) as string;
const dataTableProxy = await this.helpers.getDataTableProxy(dataTableId);
const rowId = typeof evaluationTrigger.row_id === 'number' ? evaluationTrigger.row_id : 1;
const data = Object.fromEntries(
Object.entries(outputs).map(([k, v]) => [k, toDataTableValue(v)]),
);
const columns = await dataTableProxy.getColumns();
for (const [columnName, value] of Object.entries(outputs)) {
if (columns.find((c) => c.name === columnName)) {
continue;
}
await dataTableProxy.addColumn({
name: columnName,
type: toDataTableColumnType(value),
});
}
await dataTableProxy.updateRows({
filter: {
type: 'and',
filters: [
{
columnName: 'id',
condition: 'eq',
value: rowId,
},
],
},
data,
});
} else if (source === 'googleSheets') {
const googleSheetInstance = getGoogleSheet.call(this);
const googleSheet = await getSheet.call(this, googleSheetInstance);
await googleSheetInstance.updateRows(
googleSheet.title,
[columnNames],
'RAW', // default value for Value Input Mode
1, // header row
);
const preparedData = googleSheetInstance.prepareDataForUpdatingByRowNumber(
[
{
row_number: rowNumber,
...outputs,
},
],
`${googleSheet.title}!A:Z`,
[columnNames],
);
await googleSheetInstance.batchUpdate(
preparedData.updateData,
'RAW', // default value for Value Input Mode
);
} else {
throw new NodeOperationError(this.getNode(), `Unknown source "${source}"`);
}
return [withEvaluationData.call(this, outputs)];
}
function isInputsArray(
value: unknown,
): value is Array<{ inputName: string; inputValue: JsonValue }> {
return (
Array.isArray(value) &&
value.every(
(item) =>
typeof item === 'object' &&
item !== null &&
'inputName' in item &&
'inputValue' in item &&
typeof item.inputName === 'string',
)
);
}
export function setInputs(this: IExecuteFunctions): INodeExecutionData[][] {
const evaluationNode = this.getNode();
const parentNodes = this.getParentNodes(evaluationNode.name);
const evalTrigger = parentNodes.find((node) => node.type === 'n8n-nodes-base.evaluationTrigger');
const isEvalTriggerExecuted = evalTrigger
? this.evaluateExpression(`{{ $('${evalTrigger?.name}').isExecuted }}`, 0)
: false;
if (!evalTrigger || !isEvalTriggerExecuted) {
this.addExecutionHints({
message: "No inputs were set since the execution didn't start from an evaluation trigger",
location: 'outputPane',
});
return [this.getInputData()];
}
const inputFields = this.getNodeParameter('inputs.values', 0, []);
assert(
isInputsArray(inputFields),
'Invalid input fields format. Expected an array of objects with inputName and inputValue properties.',
);
if (inputFields.length === 0) {
throw new UserError('No inputs to set', {
description: 'Add inputs using the Add Input button',
});
}
const inputs = inputFields.reduce<JsonObject>((acc, { inputName, inputValue }) => {
acc[inputName] = inputValue;
return acc;
}, {});
return [withEvaluationData.call(this, inputs)];
}
export async function setMetrics(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
const items = this.getInputData();
const metrics: INodeExecutionData[] = [];
for (let i = 0; i < items.length; i++) {
const metric = this.getNodeParameter('metric', i, {}) as keyof typeof metricHandlers;
if (!metricHandlers.hasOwnProperty(metric)) {
throw new NodeOperationError(this.getNode(), 'Unknown metric');
}
const newData = await metricHandlers[metric].call(this, i);
const newItem: INodeExecutionData = {
json: {},
pairedItem: { item: i },
};
const returnItem = composeReturnItem.call(
this,
i,
newItem,
newData,
{ dotNotation: false, include: 'none' },
1,
);
metrics.push(returnItem);
}
return [metrics];
}
export async function checkIfEvaluating(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
const evaluationExecutionResult: INodeExecutionData[] = [];
const normalExecutionResult: INodeExecutionData[] = [];
const evaluationNode = this.getNode();
const parentNodes = this.getParentNodes(evaluationNode.name);
const evalTrigger = parentNodes.find((node) => node.type === 'n8n-nodes-base.evaluationTrigger');
const isEvalTriggerExecuted = evalTrigger
? this.evaluateExpression(`{{ $('${evalTrigger?.name}').isExecuted }}`, 0)
: false;
if (isEvalTriggerExecuted) {
return [this.getInputData(), normalExecutionResult];
} else {
return [evaluationExecutionResult, this.getInputData()];
}
}
export function getOutputConnectionTypes(parameters: INodeParameters) {
if (parameters.operation === 'checkIfEvaluating') {
return [
{ type: 'main', displayName: 'Evaluation' },
{ type: 'main', displayName: 'Normal' },
];
}
return [{ type: 'main' }];
}
export function getInputConnectionTypes(
parameters: INodeParameters,
metricRequiresModelConnectionFn: (metric: string) => boolean,
) {
if (
parameters.operation === 'setMetrics' &&
metricRequiresModelConnectionFn(parameters.metric as string)
) {
return [
{ type: 'main' },
{ type: 'ai_languageModel', displayName: 'Model', maxConnections: 1 },
];
}
return [{ type: 'main' }];
}
@@ -0,0 +1,361 @@
import {
ChatPromptTemplate,
SystemMessagePromptTemplate,
HumanMessagePromptTemplate,
} from '@langchain/core/prompts';
import type { BaseLanguageModel } from '@langchain/core/language_models/base';
import { distance } from 'fastest-levenshtein';
import { NodeOperationError, nodeNameToToolName } from 'n8n-workflow';
import type {
FieldType,
AssignmentCollectionValue,
IDataObject,
IExecuteFunctions,
} from 'n8n-workflow';
import { z } from 'zod';
import { validateEntry } from '../../Set/v2/helpers/utils';
import {
CORRECTNESS_PROMPT,
CORRECTNESS_INPUT_PROMPT,
HELPFULNESS_PROMPT,
HELPFULNESS_INPUT_PROMPT,
} from '../Evaluation/CannedMetricPrompts.ee';
export const metricHandlers = {
async customMetrics(this: IExecuteFunctions, i: number): Promise<IDataObject> {
const dataToSave = this.getNodeParameter('metrics', i, {}) as AssignmentCollectionValue;
return Object.fromEntries(
(dataToSave?.assignments ?? []).map((assignment) => {
const assignmentValue =
typeof assignment.value === 'number' ? assignment.value : Number(assignment.value);
if (isNaN(assignmentValue)) {
throw new NodeOperationError(
this.getNode(),
`Value for '${assignment.name}' isn't a number`,
{
description: `It's currently '${assignment.value}'. Metrics must be numeric.`,
},
);
}
if (!assignment.name || isNaN(assignmentValue)) {
throw new NodeOperationError(this.getNode(), 'Metric name missing', {
description: 'Make sure each metric you define has a name',
});
}
const { name, value } = validateEntry(
assignment.name,
assignment.type as FieldType,
assignmentValue,
this.getNode(),
i,
false,
1,
);
return [name, value];
}),
);
},
async toolsUsed(this: IExecuteFunctions, i: number): Promise<IDataObject> {
const expectedToolsParam = this.getNodeParameter('expectedTools', i, '');
const expectedToolsString = (expectedToolsParam as string)?.trim() || '';
const expectedTools: string[] = expectedToolsString
? expectedToolsString
.split(',')
.map((tool) => tool.trim())
.filter((tool) => tool !== '')
: [];
const intermediateSteps = this.getNodeParameter('intermediateSteps', i, {}) as Array<{
action: { tool: string };
}>;
if (!expectedTools || expectedTools.length === 0) {
throw new NodeOperationError(this.getNode(), 'Expected tool name missing', {
description:
'Make sure you add at least one expected tool name (comma-separated if multiple)',
});
}
if (!intermediateSteps || !Array.isArray(intermediateSteps)) {
throw new NodeOperationError(this.getNode(), 'Intermediate steps missing', {
description:
"Make sure to enable returning intermediate steps in your agent node's options, then map them in here",
});
}
// Convert user-entered tool names to the format used in intermediate steps (case-insensitive)
const normalizedExpectedTools = expectedTools.map((tool) =>
nodeNameToToolName(tool).toLowerCase(),
);
// Calculate individual tool usage (1 if used, 0 if not used)
const toolUsageScores = normalizedExpectedTools.map((normalizedTool) => {
return intermediateSteps.some((step) => {
// Handle malformed intermediate steps gracefully
if (!step || !step.action || typeof step.action.tool !== 'string') {
return false;
}
return step.action.tool.toLowerCase() === normalizedTool;
})
? 1
: 0;
});
// Calculate the average of all tool usage scores
const averageScore =
toolUsageScores.reduce((sum: number, score: number) => sum + score, 0) /
toolUsageScores.length;
const metricName = this.getNodeParameter('options.metricName', i, 'Tools Used') as string;
return {
[metricName]: averageScore,
};
},
async categorization(this: IExecuteFunctions, i: number): Promise<IDataObject> {
const expectedAnswer = (this.getNodeParameter('expectedAnswer', i, '') as string)
.toString()
.trim();
const actualAnswer = (this.getNodeParameter('actualAnswer', i, '') as string).toString().trim();
if (!expectedAnswer) {
throw new NodeOperationError(this.getNode(), 'Expected answer is missing', {
description: 'Make sure to fill in an expected answer',
});
}
if (!actualAnswer) {
throw new NodeOperationError(this.getNode(), 'Actual answer is missing', {
description: 'Make sure to fill in an actual answer',
});
}
const metricName = this.getNodeParameter('options.metricName', i, 'Categorization') as string;
return {
[metricName]: expectedAnswer === actualAnswer ? 1 : 0,
};
},
async stringSimilarity(this: IExecuteFunctions, i: number): Promise<IDataObject> {
const expectedAnswer = (this.getNodeParameter('expectedAnswer', i, '') as string)
.toString()
.trim();
const actualAnswer = (this.getNodeParameter('actualAnswer', i, '') as string).toString().trim();
if (!expectedAnswer) {
throw new NodeOperationError(this.getNode(), 'Expected answer is missing', {
description: 'Make sure to fill in an expected answer',
});
}
if (!actualAnswer) {
throw new NodeOperationError(this.getNode(), 'Actual answer is missing', {
description: 'Make sure to fill in an actual answer',
});
}
const metricName = this.getNodeParameter(
'options.metricName',
i,
'String similarity',
) as string;
const editDistance = distance(expectedAnswer, actualAnswer);
const longerStringLength = Math.max(expectedAnswer.length, actualAnswer.length);
const similarity = longerStringLength === 0 ? 1 : 1 - editDistance / longerStringLength;
return {
[metricName]: similarity,
};
},
async helpfulness(this: IExecuteFunctions, i: number): Promise<IDataObject> {
const userQuery = (this.getNodeParameter('userQuery', i, '') as string).toString().trim();
const actualAnswer = (this.getNodeParameter('actualAnswer', i, '') as string).toString().trim();
if (!userQuery) {
throw new NodeOperationError(this.getNode(), 'User query is missing', {
description: 'Make sure to fill in the user query in the User Query field',
});
}
if (!actualAnswer) {
throw new NodeOperationError(this.getNode(), 'Response is missing', {
description: 'Make sure to fill in the response to evaluate in the Response field',
});
}
// Get the connected LLM model
const llm = (await this.getInputConnectionData('ai_languageModel', 0)) as BaseLanguageModel;
if (!llm) {
throw new NodeOperationError(this.getNode(), 'No language model connected', {
description: 'Connect a language model to the Model input to use the helpfulness metric',
});
}
// Get the system prompt and input prompt template, using defaults if not provided
const systemPrompt = this.getNodeParameter('prompt', i, HELPFULNESS_PROMPT) as string;
const inputPromptTemplate = this.getNodeParameter(
'options.inputPrompt',
i,
HELPFULNESS_INPUT_PROMPT[0],
) as string;
// Define the expected response schema
const responseSchema = z.object({
extended_reasoning: z
.string()
.describe('detailed step-by-step analysis of the response helpfulness'),
reasoning_summary: z.string().describe('one sentence summary of the response helpfulness'),
score: z
.number()
.int()
.min(1)
.max(5)
.describe('integer from 1 to 5 representing the helpfulness score'),
});
// Create LangChain prompt templates
const systemMessageTemplate = SystemMessagePromptTemplate.fromTemplate('{systemPrompt}');
const humanMessageTemplate = HumanMessagePromptTemplate.fromTemplate(inputPromptTemplate);
// Create the chat prompt template
const chatPrompt = ChatPromptTemplate.fromMessages([
systemMessageTemplate,
humanMessageTemplate,
]);
// Create chain with structured output
if (!llm.withStructuredOutput) {
throw new NodeOperationError(
this.getNode(),
'Language model does not support structured output',
{
description:
'The connected language model does not support structured output. Please use a compatible model.',
},
);
}
const chain = chatPrompt.pipe(
llm.withStructuredOutput<z.infer<typeof responseSchema>>(responseSchema),
);
try {
const response = await chain.invoke({
systemPrompt,
user_query: userQuery,
actual_answer: actualAnswer,
});
const metricName = this.getNodeParameter('options.metricName', i, 'Helpfulness') as string;
// Return the score as the main metric
return {
[metricName]: response.score,
};
} catch (error) {
throw new NodeOperationError(this.getNode(), 'Failed to evaluate helpfulness', {
description: `Error from language model: ${error instanceof Error ? error.message : String(error)}`,
});
}
},
async correctness(this: IExecuteFunctions, i: number): Promise<IDataObject> {
const expectedAnswer = (this.getNodeParameter('expectedAnswer', i, '') as string)
.toString()
.trim();
const actualAnswer = (this.getNodeParameter('actualAnswer', i, '') as string).toString().trim();
if (!expectedAnswer) {
throw new NodeOperationError(this.getNode(), 'Expected answer is missing', {
description: 'Make sure to fill in an expected answer',
});
}
if (!actualAnswer) {
throw new NodeOperationError(this.getNode(), 'Actual answer is missing', {
description: 'Make sure to fill in an actual answer',
});
}
// Get the connected LLM model
const llm = (await this.getInputConnectionData('ai_languageModel', 0)) as BaseLanguageModel;
if (!llm) {
throw new NodeOperationError(this.getNode(), 'No language model connected', {
description: 'Connect a language model to the Model input to use the correctness metric',
});
}
// Get the system prompt and input prompt template, using defaults if not provided
const systemPrompt = this.getNodeParameter('prompt', i, CORRECTNESS_PROMPT) as string;
const inputPromptTemplate = this.getNodeParameter(
'options.inputPrompt',
i,
CORRECTNESS_INPUT_PROMPT[0],
) as string;
// Define the expected response schema
const responseSchema = z.object({
extended_reasoning: z
.string()
.describe('detailed step-by-step analysis of factual accuracy and similarity'),
reasoning_summary: z.string().describe('one sentence summary focusing on key differences'),
score: z
.number()
.int()
.min(1)
.max(5)
.describe('integer from 1 to 5 representing the similarity score'),
});
// Create LangChain prompt templates
const systemMessageTemplate = SystemMessagePromptTemplate.fromTemplate('{systemPrompt}');
const humanMessageTemplate = HumanMessagePromptTemplate.fromTemplate(inputPromptTemplate);
// Create the chat prompt template
const chatPrompt = ChatPromptTemplate.fromMessages([
systemMessageTemplate,
humanMessageTemplate,
]);
// Create chain with structured output
if (!llm.withStructuredOutput) {
throw new NodeOperationError(
this.getNode(),
'Language model does not support structured output',
{
description:
'The connected language model does not support structured output. Please use a compatible model.',
},
);
}
const chain = chatPrompt.pipe(
llm.withStructuredOutput<z.infer<typeof responseSchema>>(responseSchema),
);
try {
const response = await chain.invoke({
systemPrompt,
actual_answer: actualAnswer,
expected_answer: expectedAnswer,
});
const metricName = this.getNodeParameter('options.metricName', i, 'Correctness') as string;
// Return the score as the main metric
return {
[metricName]: response.score,
};
} catch (error) {
throw new NodeOperationError(this.getNode(), 'Failed to evaluate correctness', {
description: `Error from language model: ${error instanceof Error ? error.message : String(error)}`,
});
}
},
};