test: simplify changed test routing

This commit is contained in:
Peter Steinberger
2026-04-26 23:58:09 +01:00
parent 199d5f765f
commit 89ab39ca64
12 changed files with 340 additions and 279 deletions

View File

@@ -41,9 +41,9 @@ Prove the touched surface first. Do not reflexively run the whole suite.
```bash
pnpm changed:lanes --json
pnpm check:changed
pnpm test:changed
pnpm test:changed:focused
pnpm check:changed # changed typecheck/lint/guards; no Vitest
pnpm test:changed # cheap smart changed Vitest targets
OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed
pnpm test <path-or-filter> -- --reporter=verbose
OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test <path-or-filter>
```
@@ -51,6 +51,22 @@ OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test <path-or-filter>
Use targeted file paths whenever possible. Avoid raw `vitest`; use the repo
`pnpm test` wrapper so project routing, workers, and setup stay correct.
## Command Semantics
- `pnpm check` and `pnpm check:changed` do not run Vitest tests. They are for
typecheck, lint, and guard proof.
- `pnpm test` and `pnpm test:changed` run Vitest tests.
- `pnpm test:changed` is intentionally cheap by default: direct test edits,
sibling tests, explicit source mappings, and import-graph dependents.
- `OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed` is the explicit broad
fallback for harness/config/package edits that genuinely need it.
- Do not run extension sweeps just because core changed. If a core edit is for a
specific plugin bug, run that plugin's tests explicitly. If a public SDK or
contract change needs consumer proof, choose the smallest representative
plugin/contract tests first, then broaden only when the risk justifies it.
- The test wrapper prints a short `[test] passed|failed|skipped ... in ...`
line. Vitest's own duration is still the per-shard detail.
## CI Debugging
Start with current run state, not logs for everything:

File diff suppressed because one or more lines are too long

View File

@@ -411,9 +411,9 @@ Think of the suites as “increasing realism” (and increasing flakiness/cost):
- Untargeted `pnpm test` runs twelve smaller shard configs (`core-unit-fast`, `core-unit-src`, `core-unit-security`, `core-unit-ui`, `core-unit-support`, `core-support-boundary`, `core-contracts`, `core-bundled`, `core-runtime`, `agentic`, `auto-reply`, `extensions`) instead of one giant native root-project process. This cuts peak RSS on loaded machines and avoids auto-reply/extension work starving unrelated suites.
- `pnpm test --watch` still uses the native root `vitest.config.ts` project graph, because a multi-shard watch loop is not practical.
- `pnpm test`, `pnpm test:watch`, and `pnpm test:perf:imports` route explicit file/directory targets through scoped lanes first, so `pnpm test extensions/discord/src/monitor/message-handler.preflight.test.ts` avoids paying the full root project startup tax.
- `pnpm test:changed` expands changed git paths into the same scoped lanes when the diff only touches routable source/test files; config/setup edits still fall back to the broad root-project rerun.
- `pnpm check:changed` is the normal smart local gate for narrow work. It classifies the diff into core, core tests, extensions, extension tests, apps, docs, release metadata, live Docker tooling, and tooling, then runs the matching typecheck/lint/test lanes. Public Plugin SDK and plugin-contract changes include one extension validation pass because extensions depend on those core contracts. Release metadata-only version bumps run targeted version/config/root-dependency checks instead of the full suite, with a guard that rejects package changes outside the top-level version field.
- Live Docker ACP harness edits run a focused local gate: shell syntax for the live Docker auth scripts, live Docker scheduler dry-run, ACP bind unit tests, and the ACPX extension tests. `package.json` changes are included only when the diff is limited to `scripts["test:docker:live-*"]`; dependency, export, version, and other package-surface edits still use the broader guards.
- `pnpm test:changed` expands changed git paths into cheap scoped lanes by default: direct test edits, sibling `*.test.ts` files, explicit source mappings, and local import-graph dependents. Config/setup/package edits do not broad-run tests unless you explicitly use `OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed`.
- `pnpm check:changed` is the normal smart local check gate for narrow work. It classifies the diff into core, core tests, extensions, extension tests, apps, docs, release metadata, live Docker tooling, and tooling, then runs the matching typecheck, lint, and guard commands. It does not run Vitest tests; call `pnpm test:changed` or explicit `pnpm test <target>` for test proof. Release metadata-only version bumps run targeted version/config/root-dependency checks, with a guard that rejects package changes outside the top-level version field.
- Live Docker ACP harness edits run focused checks: shell syntax for the live Docker auth scripts and a live Docker scheduler dry-run. `package.json` changes are included only when the diff is limited to `scripts["test:docker:live-*"]`; dependency, export, version, and other package-surface edits still use the broader guards.
- Import-light unit tests from agents, commands, plugins, auto-reply helpers, `plugin-sdk`, and similar pure utility areas route through the `unit-fast` lane, which skips `test/setup-openclaw-runtime.ts`; stateful/runtime-heavy files stay on the existing lanes.
- Selected `plugin-sdk` and `commands` helper source files also map changed-mode runs to explicit sibling tests in those light lanes, so helper edits avoid rerunning the full heavy suite for that directory.
- `auto-reply` has dedicated buckets for top-level core helpers, top-level `reply.*` integration tests, and the `src/auto-reply/reply/**` subtree. CI further splits the reply subtree into agent-runner, dispatch, and commands/state-routing shards so one import-heavy bucket does not own the full Node tail.
@@ -458,10 +458,11 @@ Think of the suites as “increasing realism” (and increasing flakiness/cost):
- The pre-commit hook is formatting-only. It restages formatted files and
does not run lint, typecheck, or tests.
- Run `pnpm check:changed` explicitly before handoff or push when you
need the smart local gate. Public Plugin SDK and plugin-contract
changes include one extension validation pass.
- `pnpm test:changed` routes through scoped lanes when the changed paths
map cleanly to a smaller suite.
need the smart local check gate.
- `pnpm test:changed` routes through cheap scoped lanes by default. Use
`OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed` only when the agent
decides a harness, config, package, or contract edit really needs broader
Vitest coverage.
- `pnpm test:max` and `pnpm test:changed:max` keep the same routing
behavior, just with a higher worker cap.
- Local worker auto-scaling is intentionally conservative and backs off

View File

@@ -10,11 +10,12 @@ title: "Tests"
- `pnpm test:force`: Kills any lingering gateway process holding the default control port, then runs the full Vitest suite with an isolated gateway port so server tests dont collide with a running instance. Use this when a prior gateway run left port 18789 occupied.
- `pnpm test:coverage`: Runs the unit suite with V8 coverage (via `vitest.unit.config.ts`). This is a loaded-file unit coverage gate, not whole-repo all-file coverage. Thresholds are 70% lines/functions/statements and 55% branches. Because `coverage.all` is false, the gate measures files loaded by the unit coverage suite instead of treating every split-lane source file as uncovered.
- `pnpm test:coverage:changed`: Runs unit coverage only for files changed since `origin/main`.
- `pnpm test:changed`: expands changed git paths into scoped Vitest lanes when the diff only touches routable source/test files. Config/setup changes still fall back to the native root projects run so wiring edits rerun broadly when needed.
- `pnpm test:changed:focused`: inner-loop changed test run. It only runs precise targets from direct test edits, sibling `*.test.ts` files, explicit source mappings, and the local import graph. Broad/config/package changes are skipped instead of expanding to the full changed-test fallback.
- `pnpm test:changed`: cheap smart changed test run. It runs precise targets from direct test edits, sibling `*.test.ts` files, explicit source mappings, and the local import graph. Broad/config/package changes are skipped unless they map to precise tests.
- `OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed`: explicit broad changed test run. Use it when a test harness/config/package edit should fall back to Vitest's broader changed-test behavior.
- `pnpm changed:lanes`: shows the architectural lanes triggered by the diff against `origin/main`.
- `pnpm check:changed`: runs the smart changed gate for the diff against `origin/main`. It runs core work with core test lanes, extension work with extension test lanes, test-only work with test typecheck/tests only, expands public Plugin SDK or plugin-contract changes to one extension validation pass, and keeps release metadata-only version bumps on targeted version/config/root-dependency checks.
- `pnpm check:changed`: runs the smart changed check gate for the diff against `origin/main`. It runs typecheck, lint, and guard commands for the affected architectural lanes, but does not run Vitest tests. Use `pnpm test:changed` or explicit `pnpm test <target>` for test proof.
- `pnpm test`: routes explicit file/directory targets through scoped Vitest lanes. Untargeted runs use fixed shard groups and expand to leaf configs for local parallel execution; the extension group always expands to the per-extension shard configs instead of one giant root-project process.
- Test wrapper runs end with a short `[test] passed|failed|skipped ... in ...` summary. Vitest's own duration line stays the per-shard detail.
- Full, extension, and include-pattern shard runs update local timing data in `.artifacts/vitest-shard-timings.json`; later whole-config runs use those timings to balance slow and fast shards. Include-pattern CI shards append the shard name to the timing key, which keeps filtered shard timings visible without replacing whole-config timing data. Set `OPENCLAW_TEST_PROJECTS_TIMINGS=0` to ignore the local timing artifact.
- Selected `plugin-sdk` and `commands` test files now route through dedicated light lanes that keep only `test/setup.ts`, leaving runtime-heavy cases on their existing lanes.
- Source files with sibling tests map to that sibling before falling back to wider directory globs. Helper edits under `test/helpers/channels` and `test/helpers/plugins` use a local import graph to run importing tests instead of broad-running every shard when the dependency path is precise.

View File

@@ -1480,7 +1480,6 @@
"test:build:singleton": "node scripts/test-built-plugin-singleton.mjs",
"test:bundled": "node scripts/run-vitest.mjs run --config test/vitest/vitest.bundled.config.ts",
"test:changed": "node scripts/test-projects.mjs --changed origin/main",
"test:changed:focused": "OPENCLAW_TEST_CHANGED_FOCUSED=1 node scripts/test-projects.mjs --changed origin/main",
"test:changed:max": "OPENCLAW_VITEST_MAX_WORKERS=8 node scripts/test-projects.mjs --changed origin/main",
"test:channels": "node scripts/run-vitest.mjs run --config test/vitest/vitest.channels.config.ts",
"test:contracts": "pnpm test:contracts:channels && pnpm test:contracts:plugins",

View File

@@ -67,7 +67,7 @@ export function createEmptyChangedLanes() {
/**
* @param {string[]} changedPaths
* @param {{ packageJsonChangeKind?: "liveDockerTooling" | null }} [options]
* @param {{ packageJsonChangeKind?: "liveDockerTooling" | "tooling" | null }} [options]
* @returns {ChangedLaneResult}
*/
export function detectChangedLanes(changedPaths, options = {}) {
@@ -80,6 +80,8 @@ export function detectChangedLanes(changedPaths, options = {}) {
let hasNonDocs = false;
const packageJsonIsLiveDockerTooling =
paths.includes("package.json") && options.packageJsonChangeKind === "liveDockerTooling";
const packageJsonIsTooling =
paths.includes("package.json") && options.packageJsonChangeKind === "tooling";
if (paths.length === 0) {
reasons.push("no changed paths");
@@ -88,6 +90,7 @@ export function detectChangedLanes(changedPaths, options = {}) {
if (
!packageJsonIsLiveDockerTooling &&
!packageJsonIsTooling &&
paths.some((changedPath) => RELEASE_METADATA_PATHS.has(changedPath)) &&
paths.every(
(changedPath) => RELEASE_METADATA_PATHS.has(changedPath) || DOCS_PATH_RE.test(changedPath),
@@ -115,6 +118,12 @@ export function detectChangedLanes(changedPaths, options = {}) {
continue;
}
if (changedPath === "package.json" && packageJsonIsTooling) {
lanes.tooling = true;
reasons.push(`${changedPath}: package scripts`);
continue;
}
if (LIVE_DOCKER_TOOLING_PATH_RE.test(changedPath)) {
lanes.liveDockerTooling = true;
reasons.push(`${changedPath}: live Docker tooling surface`);
@@ -195,39 +204,42 @@ export function detectChangedLanes(changedPaths, options = {}) {
}
/**
* @param {{ base: string; head?: string; includeWorktree?: boolean }} params
* @param {{ base: string; head?: string; includeWorktree?: boolean; cwd?: string }} params
* @returns {string[]}
*/
export function listChangedPathsFromGit(params) {
const base = params.base;
const head = params.head ?? "HEAD";
const cwd = params.cwd ?? process.cwd();
if (!base) {
return [];
}
const rangePaths = runGitNameOnlyDiff([`${base}...${head}`]);
const rangePaths = runGitNameOnlyDiff([`${base}...${head}`], cwd);
if (params.includeWorktree === false) {
return rangePaths;
}
return [
...new Set([
...rangePaths,
...runGitNameOnlyDiff(["--cached", "--diff-filter=ACMR"]),
...runGitNameOnlyDiff(["--diff-filter=ACMR"]),
...runGitLsFiles(["--others", "--exclude-standard"]),
...runGitNameOnlyDiff(["--cached", "--diff-filter=ACMR"], cwd),
...runGitNameOnlyDiff(["--diff-filter=ACMR"], cwd),
...runGitLsFiles(["--others", "--exclude-standard"], cwd),
]),
].toSorted((left, right) => left.localeCompare(right));
}
function runGitNameOnlyDiff(extraArgs) {
function runGitNameOnlyDiff(extraArgs, cwd = process.cwd()) {
const output = execFileSync("git", ["diff", "--name-only", ...extraArgs], {
cwd,
stdio: ["ignore", "pipe", "pipe"],
encoding: "utf8",
});
return output.split("\n").map(normalizeChangedPath).filter(Boolean);
}
function runGitLsFiles(extraArgs) {
function runGitLsFiles(extraArgs, cwd = process.cwd()) {
const output = execFileSync("git", ["ls-files", ...extraArgs], {
cwd,
stdio: ["ignore", "pipe", "pipe"],
encoding: "utf8",
});
@@ -245,7 +257,10 @@ export function listStagedChangedPaths() {
export function classifyPackageJsonChangeFromGit(params) {
try {
const { before, after } = readPackageJsonBeforeAfter(params);
return isLiveDockerPackageScriptOnlyChange(before, after) ? "liveDockerTooling" : null;
if (isLiveDockerPackageScriptOnlyChange(before, after)) {
return "liveDockerTooling";
}
return isPackageScriptOnlyChange(before, after) ? "tooling" : null;
} catch {
return null;
}
@@ -265,6 +280,20 @@ export function isLiveDockerPackageScriptOnlyChange(before, after) {
);
}
export function isPackageScriptOnlyChange(before, after) {
const beforePackage = JSON.parse(before);
const afterPackage = JSON.parse(after);
const beforeScripts = extractPackageScripts(beforePackage);
const afterScripts = extractPackageScripts(afterPackage);
const beforeStripped = stripPackageScripts(beforePackage);
const afterStripped = stripPackageScripts(afterPackage);
return (
stableJson(beforeStripped) === stableJson(afterStripped) &&
stableJson(beforeScripts) !== stableJson(afterScripts)
);
}
function readPackageJsonBeforeAfter(params) {
const before = readGitText(params.staged ? "HEAD" : params.base, "package.json");
if (params.staged) {
@@ -317,6 +346,17 @@ function stripLiveDockerPackageScripts(packageJson) {
return clone;
}
function extractPackageScripts(packageJson) {
const scripts = packageJson?.scripts;
return scripts && typeof scripts === "object" && !Array.isArray(scripts) ? scripts : {};
}
function stripPackageScripts(packageJson) {
const clone = JSON.parse(JSON.stringify(packageJson));
delete clone.scripts;
return clone;
}
function stableJson(value) {
if (Array.isArray(value)) {
return `[${value.map(stableJson).join(",")}]`;

View File

@@ -14,12 +14,7 @@ import {
} from "./lib/local-heavy-check-runtime.mjs";
import { runManagedCommand } from "./lib/managed-child-process.mjs";
import { createSparseTsgoSkipEnv } from "./lib/tsgo-sparse-guard.mjs";
import { isCiLikeEnv } from "./lib/vitest-local-scheduling.mjs";
import { resolveChangedTestTargetPlan } from "./test-projects.test-support.mjs";
export const CHANGED_CHECK_VITEST_NO_OUTPUT_TIMEOUT_MS = "600000";
const VITEST_NO_OUTPUT_TIMEOUT_ENV_KEY = "OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS";
const VITEST_NO_OUTPUT_RETRY_ENV_KEY = "OPENCLAW_VITEST_NO_OUTPUT_RETRY";
const LIVE_DOCKER_AUTH_SHELL_TARGETS = [
"scripts/lib/live-docker-auth.sh",
"scripts/test-live-acp-bind-docker.sh",
@@ -39,35 +34,6 @@ export function createChangedCheckChildEnv(baseEnv = process.env) {
};
}
export function createChangedCheckVitestEnv(baseEnv = process.env) {
const resolvedBaseEnv = createChangedCheckChildEnv(baseEnv);
const env = {
...resolvedBaseEnv,
[VITEST_NO_OUTPUT_TIMEOUT_ENV_KEY]:
resolvedBaseEnv[VITEST_NO_OUTPUT_TIMEOUT_ENV_KEY]?.trim() ||
CHANGED_CHECK_VITEST_NO_OUTPUT_TIMEOUT_MS,
[VITEST_NO_OUTPUT_RETRY_ENV_KEY]:
resolvedBaseEnv[VITEST_NO_OUTPUT_RETRY_ENV_KEY]?.trim() || "0",
};
const hasWorkerOverride = Boolean(
(resolvedBaseEnv.OPENCLAW_VITEST_MAX_WORKERS ?? resolvedBaseEnv.OPENCLAW_TEST_WORKERS)?.trim(),
);
const hasParallelOverride = Boolean(resolvedBaseEnv.OPENCLAW_TEST_PROJECTS_PARALLEL?.trim());
const serialOverride = resolvedBaseEnv.OPENCLAW_TEST_PROJECTS_SERIAL?.trim();
if (
!isCiLikeEnv(resolvedBaseEnv) &&
!hasWorkerOverride &&
!hasParallelOverride &&
serialOverride !== "0"
) {
env.OPENCLAW_TEST_PROJECTS_SERIAL = serialOverride || "1";
env.OPENCLAW_VITEST_MAX_WORKERS = "1";
}
return env;
}
export function createChangedCheckPlan(result, options = {}) {
const commands = [];
const baseEnv = createChangedCheckChildEnv(options.env ?? process.env);
@@ -93,10 +59,6 @@ export function createChangedCheckPlan(result, options = {}) {
if (result.docsOnly) {
return {
commands,
testTargets: [],
runChangedTestsBroad: false,
runFullTests: false,
runExtensionTests: false,
summary: "docs-only",
};
}
@@ -118,10 +80,6 @@ export function createChangedCheckPlan(result, options = {}) {
add("root dependency ownership", ["deps:root-ownership:check"]);
return {
commands,
testTargets: [],
runChangedTestsBroad: false,
runFullTests: false,
runExtensionTests: false,
summary: "release metadata",
};
}
@@ -132,10 +90,6 @@ export function createChangedCheckPlan(result, options = {}) {
add("runtime import cycles", ["check:import-cycles"]);
return {
commands,
testTargets: [],
runChangedTestsBroad: false,
runFullTests: true,
runExtensionTests: false,
summary: "all",
};
}
@@ -189,26 +143,10 @@ export function createChangedCheckPlan(result, options = {}) {
OPENCLAW_DOCKER_ALL_DRY_RUN: "1",
OPENCLAW_DOCKER_ALL_LIVE_MODE: "only",
});
add(
"ACP bind unit tests",
["test", "src/gateway/live-agent-probes.test.ts", "src/agents/acp-spawn.test.ts"],
createChangedCheckVitestEnv(baseEnv),
);
add("ACPX extension tests", ["test:extension", "acpx"], createChangedCheckVitestEnv(baseEnv));
}
const testPlan = resolveChangedTestTargetPlan(result.paths);
const runExtensionTests = result.extensionImpactFromCore;
const testTargets = runExtensionTests
? testPlan.targets.filter((target) => target !== "extensions")
: testPlan.targets;
const runChangedTestsBroad = testPlan.mode === "broad";
return {
commands,
testTargets,
runChangedTestsBroad,
runFullTests: false,
runExtensionTests,
summary: Object.entries(lanes)
.filter(([, enabled]) => enabled)
.map(([lane]) => lane)
@@ -244,61 +182,6 @@ export async function runChangedCheck(result, options = {}) {
}
}
if (plan.runFullTests) {
const status = await runPnpm(
{ name: "tests all", args: ["test"], env: createChangedCheckVitestEnv(childEnv) },
timings,
);
if (status !== 0) {
printSummary(timings, options);
return status;
}
} else if (plan.runChangedTestsBroad) {
const testArgs = options.explicitPaths
? ["test"]
: ["test", "--changed", options.base ?? "origin/main"];
const status = await runPnpm(
{
name: options.explicitPaths ? "tests all" : "tests changed broad",
args: testArgs,
env: createChangedCheckVitestEnv(childEnv),
},
timings,
);
if (status !== 0) {
printSummary(timings, options);
return status;
}
} else if (plan.testTargets.length > 0) {
const status = await runPnpm(
{
name: "tests changed",
args: ["test", ...plan.testTargets],
env: createChangedCheckVitestEnv(childEnv),
},
timings,
);
if (status !== 0) {
printSummary(timings, options);
return status;
}
}
if (plan.runExtensionTests) {
const status = await runPnpm(
{
name: "tests extensions",
args: ["test:extensions"],
env: createChangedCheckVitestEnv(childEnv),
},
timings,
);
if (status !== 0) {
printSummary(timings, options);
return status;
}
}
printSummary(timings, options);
return 0;
} finally {
@@ -314,17 +197,11 @@ function printPlan(result, plan, options) {
const prefix = options.dryRun ? "[check:changed:dry-run]" : "[check:changed]";
console.error(`${prefix} lanes=${plan.summary || "none"}`);
if (result.extensionImpactFromCore) {
console.error(`${prefix} core contract changed; extension tests included`);
}
if (plan.runChangedTestsBroad) {
console.error(`${prefix} broad changed tests included`);
console.error(`${prefix} extension-impacting surface; extension typecheck included`);
}
for (const reason of result.reasons) {
console.error(`${prefix} ${reason}`);
}
if (plan.testTargets.length > 0) {
console.error(`${prefix} test targets=${plan.testTargets.length}`);
}
}
async function runPnpm(command, timings) {

View File

@@ -1,5 +1,6 @@
import fs from "node:fs";
import { performance } from "node:perf_hooks";
import { formatMs } from "./lib/check-timing-summary.mjs";
import { acquireLocalHeavyCheckLockSync } from "./lib/local-heavy-check-runtime.mjs";
import {
isCiLikeEnv,
@@ -271,6 +272,7 @@ async function runVitestSpecsParallel(specs, concurrency) {
}
async function main() {
const suiteStartedAt = performance.now();
const args = process.argv.slice(2);
const baseEnv = resolveLocalVitestEnv(process.env);
const { targetArgs } = parseTestProjectsArgs(args, process.cwd());
@@ -309,6 +311,7 @@ async function main() {
if (runSpecs.length === 0) {
console.error("[test] no changed test targets; skipping Vitest.");
printTestSummary("skipped", 0, performance.now() - suiteStartedAt);
return;
}
@@ -360,8 +363,11 @@ async function main() {
concurrency,
);
writeShardTimings(timings, process.cwd(), baseEnv);
console.error(
`[test] completed ${parallelSpecs.length} Vitest shards; Vitest summaries above are per-shard, not aggregate totals.`,
printTestSummary(
parallelExitCode === 0 ? "passed" : "failed",
parallelSpecs.length,
performance.now() - suiteStartedAt,
"Vitest summaries above are per-shard, not aggregate totals.",
);
releaseLockOnce();
if (parallelExitCode !== 0) {
@@ -378,18 +384,24 @@ async function main() {
if (!result) {
return;
}
if (result.timing) {
timings.push(result.timing);
}
if (result.code !== 0) {
exitCode = exitCode || result.code;
if (spec.continueOnFailure !== true) {
printTestSummary("failed", timings.length, performance.now() - suiteStartedAt);
releaseLockOnce();
process.exit(result.code);
}
}
if (result.timing) {
timings.push(result.timing);
}
}
writeShardTimings(timings, process.cwd(), baseEnv);
printTestSummary(
exitCode === 0 ? "passed" : "failed",
timings.length,
performance.now() - suiteStartedAt,
);
releaseLockOnce();
if (exitCode !== 0) {
@@ -397,6 +409,13 @@ async function main() {
}
}
function printTestSummary(status, shardCount, durationMs, detail) {
const suffix = detail ? `; ${detail}` : "";
console.error(
`[test] ${status} ${shardCount} Vitest shard${shardCount === 1 ? "" : "s"} in ${formatMs(durationMs)}${suffix}`,
);
}
main().catch((error) => {
releaseLockOnce();
console.error(error);

View File

@@ -17,7 +17,7 @@ export type VitestRunSpec = {
export type ChangedTestTargetOptions = {
cwd?: string;
env?: Record<string, string | undefined>;
focused?: boolean;
broad?: boolean;
};
export const DEFAULT_TEST_PROJECTS_VITEST_NO_OUTPUT_TIMEOUT_MS: string;

View File

@@ -1,4 +1,3 @@
import { execFileSync } from "node:child_process";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
@@ -40,7 +39,10 @@ import {
isBoundaryTestFile,
isBundledPluginDependentUnitTestFile,
} from "../test/vitest/vitest.unit-paths.mjs";
import { detectChangedLanes } from "./changed-lanes.mjs";
import {
detectChangedLanes,
listChangedPathsFromGit as listChangedPathsFromGitSource,
} from "./changed-lanes.mjs";
import { isCiLikeEnv, resolveLocalFullSuiteProfile } from "./lib/vitest-local-scheduling.mjs";
import { resolveVitestCliEntry, resolveVitestNodeArgs } from "./run-vitest.mjs";
@@ -207,7 +209,7 @@ const VITEST_CONFIG_BY_KIND = {
utils: UTILS_VITEST_CONFIG,
wizard: WIZARD_VITEST_CONFIG,
};
const BROAD_CHANGED_RERUN_PATTERNS = [
const BROAD_CHANGED_FALLBACK_PATTERNS = [
/^package\.json$/u,
/^pnpm-lock\.yaml$/u,
/^test\/setup(?:\.shared|\.extensions|-openclaw-runtime)?\.ts$/u,
@@ -305,7 +307,7 @@ const SOURCE_ROOTS_FOR_IMPORT_GRAPH = ["src", "extensions", "packages", "ui/src"
const IMPORTABLE_FILE_EXTENSIONS = [".ts", ".tsx", ".mts", ".cts"];
const IMPORT_SPECIFIER_PATTERN =
/\b(?:import|export)\s+(?:type\s+)?(?:[^'"]*?\s+from\s+)?["']([^"']+)["']|\bimport\s*\(\s*["']([^"']+)["']\s*\)/gu;
const FOCUSED_CHANGED_ENV_KEY = "OPENCLAW_TEST_CHANGED_FOCUSED";
const BROAD_CHANGED_ENV_KEY = "OPENCLAW_TEST_CHANGED_BROAD";
const VITEST_NO_OUTPUT_TIMEOUT_ENV_KEY = "OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS";
const VITEST_NO_OUTPUT_RETRY_ENV_KEY = "OPENCLAW_VITEST_NO_OUTPUT_RETRY";
export const DEFAULT_TEST_PROJECTS_VITEST_NO_OUTPUT_TIMEOUT_MS = "180000";
@@ -594,36 +596,7 @@ function resolveChannelContractTargetKind(relative) {
}
function listChangedPathsFromGit(baseRef, cwd) {
return [
...new Set([
...runGitNameOnlyDiff(cwd, [`${baseRef}...HEAD`]),
...runGitNameOnlyDiff(cwd, ["--cached", "--diff-filter=ACMR"]),
...runGitNameOnlyDiff(cwd, ["--diff-filter=ACMR"]),
...runGitLsFiles(cwd, ["--others", "--exclude-standard"]),
]),
].toSorted((left, right) => left.localeCompare(right));
}
function runGitNameOnlyDiff(cwd, extraArgs) {
return execFileSync("git", ["diff", "--name-only", ...extraArgs], {
cwd,
encoding: "utf8",
stdio: ["ignore", "pipe", "pipe"],
})
.split("\n")
.map((line) => normalizePathPattern(line.trim()))
.filter((line) => line.length > 0);
}
function runGitLsFiles(cwd, extraArgs) {
return execFileSync("git", ["ls-files", ...extraArgs], {
cwd,
encoding: "utf8",
stdio: ["ignore", "pipe", "pipe"],
})
.split("\n")
.map((line) => normalizePathPattern(line.trim()))
.filter((line) => line.length > 0);
return listChangedPathsFromGitSource({ base: baseRef, cwd });
}
function extractChangedBaseRef(args) {
@@ -665,7 +638,7 @@ function shouldKeepBroadChangedRun(changedPaths) {
return changedPaths.some((changedPath) =>
PRECISE_SOURCE_TEST_TARGETS.has(changedPath)
? false
: BROAD_CHANGED_RERUN_PATTERNS.some((pattern) => pattern.test(changedPath)),
: BROAD_CHANGED_FALLBACK_PATTERNS.some((pattern) => pattern.test(changedPath)),
);
}
@@ -685,8 +658,8 @@ function resolveToolingTestTargets(changedPath) {
return TOOLING_SOURCE_TEST_TARGETS.get(changedPath) ?? TOOLING_TEST_TARGETS.get(changedPath);
}
function shouldUseFocusedChangedTargets(env = process.env) {
const value = env[FOCUSED_CHANGED_ENV_KEY]?.trim().toLowerCase();
function shouldUseBroadChangedTargets(env = process.env) {
const value = env[BROAD_CHANGED_ENV_KEY]?.trim().toLowerCase();
return ["1", "true", "yes", "on"].includes(value ?? "");
}
@@ -741,7 +714,8 @@ export function resolveChangedTestTargetPlan(changedPaths, options = {}) {
return { mode: "targets", targets: toolingTargets };
}
const changedLanes = detectChangedLanes(changedPaths);
const focused = options.focused ?? shouldUseFocusedChangedTargets(options.env ?? {});
const env = options.env ?? {};
const useBroadFallback = options.broad ?? shouldUseBroadChangedTargets(env);
const targets = [];
for (const changedPath of changedPaths) {
const preciseTargets = resolvePreciseChangedTestTargets(changedPath, options);
@@ -749,20 +723,21 @@ export function resolveChangedTestTargetPlan(changedPaths, options = {}) {
targets.push(...preciseTargets);
continue;
}
if (focused) {
const needsBroadFallback = shouldKeepBroadChangedRun([changedPath]) || changedLanes.lanes.all;
if (needsBroadFallback) {
if (useBroadFallback) {
return { mode: "broad", targets: [] };
}
continue;
}
if (shouldKeepBroadChangedRun([changedPath]) || changedLanes.lanes.all) {
return { mode: "broad", targets: [] };
}
if (isRoutableChangedTarget(changedPath)) {
targets.push(changedPath);
}
}
if (!focused && changedLanes.lanes.all) {
if (useBroadFallback && changedLanes.lanes.all) {
return { mode: "broad", targets: [] };
}
if (!focused && changedLanes.extensionImpactFromCore) {
if (useBroadFallback && changedLanes.extensionImpactFromCore) {
targets.push("extensions");
}
return { mode: "targets", targets: [...new Set(targets)] };

View File

@@ -5,12 +5,11 @@ import { afterEach, describe, expect, it } from "vitest";
import {
detectChangedLanes,
isLiveDockerPackageScriptOnlyChange,
isPackageScriptOnlyChange,
} from "../../scripts/changed-lanes.mjs";
import {
CHANGED_CHECK_VITEST_NO_OUTPUT_TIMEOUT_MS,
createChangedCheckChildEnv,
createChangedCheckPlan,
createChangedCheckVitestEnv,
} from "../../scripts/check-changed.mjs";
import { cleanupTempDirs, makeTempRepoRoot } from "../helpers/temp-repo.js";
@@ -216,8 +215,8 @@ describe("scripts/changed-lanes", () => {
extensionTests: true,
all: false,
});
expect(plan.runExtensionTests).toBe(true);
expect(plan.testTargets).toEqual(["src/plugin-sdk/core.test.ts"]);
expect(plan.commands.map((command) => command.args[0])).toContain("tsgo:extensions");
expect(plan.commands.map((command) => command.args[0])).toContain("tsgo:extensions:test");
});
it("fails safe for root config changes", () => {
@@ -225,8 +224,8 @@ describe("scripts/changed-lanes", () => {
const plan = createChangedCheckPlan(result);
expect(result.lanes.all).toBe(true);
expect(plan.runFullTests).toBe(true);
expect(plan.commands.map((command) => command.args[0])).toContain("tsgo:all");
expect(plan.commands.map((command) => command.args[0])).not.toContain("test");
});
it("routes gitignore changes to tooling instead of all lanes", () => {
@@ -237,10 +236,9 @@ describe("scripts/changed-lanes", () => {
tooling: true,
all: false,
});
expect(plan.runFullTests).toBe(false);
expect(plan.runChangedTestsBroad).toBe(false);
expect(plan.commands.map((command) => command.args[0])).toContain("lint:scripts");
expect(plan.commands.map((command) => command.args[0])).not.toContain("tsgo:all");
expect(plan.commands.map((command) => command.args[0])).not.toContain("test");
});
it("routes live Docker ACP tooling changes through a focused gate", () => {
@@ -258,8 +256,6 @@ describe("scripts/changed-lanes", () => {
all: false,
tooling: false,
});
expect(plan.runFullTests).toBe(false);
expect(plan.runChangedTestsBroad).toBe(false);
expect(plan.commands.map((command) => command.name)).toEqual([
"conflict markers",
"typecheck core tests",
@@ -267,8 +263,6 @@ describe("scripts/changed-lanes", () => {
"lint scripts",
"live Docker shell syntax",
"live Docker scheduler dry run",
"ACP bind unit tests",
"ACPX extension tests",
]);
expect(
plan.commands.find((command) => command.name === "live Docker shell syntax"),
@@ -330,7 +324,6 @@ describe("scripts/changed-lanes", () => {
releaseMetadata: false,
all: false,
});
expect(plan.runFullTests).toBe(false);
expect(plan.commands.map((command) => command.name)).toContain("live Docker scheduler dry run");
});
@@ -400,6 +393,77 @@ describe("scripts/changed-lanes", () => {
});
});
it("classifies normal package script changes from the git diff", () => {
const dir = makeTempRepoRoot(tempDirs, "openclaw-package-scripts-");
git(dir, ["init", "-q", "--initial-branch=main"]);
writeFileSync(
path.join(dir, "package.json"),
`${JSON.stringify(
{
name: "fixture",
scripts: {
test: "node scripts/test-projects.mjs",
},
dependencies: {
leftpad: "1.0.0",
},
},
null,
2,
)}\n`,
"utf8",
);
git(dir, ["add", "package.json"]);
git(dir, [
"-c",
"user.email=test@example.com",
"-c",
"user.name=Test User",
"commit",
"-q",
"-m",
"initial",
]);
writeFileSync(
path.join(dir, "package.json"),
`${JSON.stringify(
{
name: "fixture",
scripts: {
test: "node scripts/test-projects.mjs",
"test:profile": "node scripts/profile-tests.mjs",
},
dependencies: {
leftpad: "1.0.0",
},
},
null,
2,
)}\n`,
"utf8",
);
const output = execFileSync(
process.execPath,
[path.join(repoRoot, "scripts", "changed-lanes.mjs"), "--json", "--base", "HEAD"],
{
cwd: dir,
encoding: "utf8",
env: createNestedGitEnv(),
},
);
expect(JSON.parse(output)).toMatchObject({
paths: ["package.json"],
lanes: {
tooling: true,
all: false,
liveDockerTooling: false,
},
});
});
it("keeps non-script package changes off the live Docker focused gate", () => {
const before = `${JSON.stringify(
{ name: "fixture", scripts: {}, dependencies: { leftpad: "1.0.0" } },
@@ -422,6 +486,41 @@ describe("scripts/changed-lanes", () => {
expect(isLiveDockerPackageScriptOnlyChange(before, after)).toBe(false);
});
it("routes package script-only changes through the tooling gate", () => {
const before = `${JSON.stringify(
{ name: "fixture", scripts: { test: "node test.js" }, dependencies: { leftpad: "1.0.0" } },
null,
2,
)}\n`;
const after = `${JSON.stringify(
{
name: "fixture",
scripts: {
test: "node test.js",
"test:profile": "node scripts/profile-tests.mjs",
},
dependencies: { leftpad: "1.0.0" },
},
null,
2,
)}\n`;
expect(isPackageScriptOnlyChange(before, after)).toBe(true);
const result = detectChangedLanes(["package.json"], {
packageJsonChangeKind: "tooling",
});
const plan = createChangedCheckPlan(result);
expect(result.lanes).toMatchObject({
tooling: true,
all: false,
liveDockerTooling: false,
});
expect(plan.commands.map((command) => command.args[0])).toContain("lint:scripts");
expect(plan.commands.map((command) => command.args[0])).not.toContain("tsgo:all");
});
it("keeps release metadata commits off the full changed gate", () => {
const result = detectChangedLanes([
"CHANGELOG.md",
@@ -443,7 +542,6 @@ describe("scripts/changed-lanes", () => {
core: false,
apps: false,
});
expect(plan.runFullTests).toBe(false);
expect(plan.commands.map((command) => command.args[0])).toEqual([
"check:no-conflict-markers",
"release-metadata:check",
@@ -519,26 +617,24 @@ describe("scripts/changed-lanes", () => {
tooling: true,
all: false,
});
expect(plan.testTargets).toEqual(["test/git-hooks-pre-commit.test.ts"]);
expect(plan.runFullTests).toBe(false);
expect(plan.commands.map((command) => command.args[0])).toContain("lint:scripts");
expect(plan.commands.map((command) => command.args[0])).not.toContain("test");
});
it("keeps shared Vitest wiring changes on the broad changed test path", () => {
it("keeps shared Vitest wiring changes out of check test execution", () => {
const result = detectChangedLanes(["test/vitest/vitest.shared.config.ts"]);
const plan = createChangedCheckPlan(result);
expect(plan.testTargets).toEqual([]);
expect(plan.runChangedTestsBroad).toBe(true);
expect(plan.runFullTests).toBe(false);
expect(plan.commands.map((command) => command.args[0])).toContain("lint:scripts");
expect(plan.commands.map((command) => command.args[0])).not.toContain("test");
});
it("keeps setup changes on the broad changed test path", () => {
it("keeps setup changes out of check test execution", () => {
const result = detectChangedLanes(["test/setup.ts"]);
const plan = createChangedCheckPlan(result);
expect(plan.testTargets).toEqual([]);
expect(plan.runChangedTestsBroad).toBe(true);
expect(plan.runFullTests).toBe(false);
expect(plan.commands.map((command) => command.args[0])).toContain("lint:scripts");
expect(plan.commands.map((command) => command.args[0])).not.toContain("test");
});
it("does not route generated A2UI artifacts as direct Vitest targets", () => {
@@ -548,17 +644,16 @@ describe("scripts/changed-lanes", () => {
]);
const plan = createChangedCheckPlan(result);
expect(plan.testTargets).toEqual(["test/scripts/bundle-a2ui.test.ts"]);
expect(plan.runChangedTestsBroad).toBe(false);
expect(plan.commands.map((command) => command.args[0])).toContain("tsgo:core");
expect(plan.commands.map((command) => command.args[0])).not.toContain("test");
});
it("routes changed extension Vitest configs to only their owning shard", () => {
const result = detectChangedLanes(["test/vitest/vitest.extension-discord.config.ts"]);
const plan = createChangedCheckPlan(result);
expect(plan.testTargets).toEqual(["test/vitest/vitest.extension-discord.config.ts"]);
expect(plan.runChangedTestsBroad).toBe(false);
expect(plan.runFullTests).toBe(false);
expect(plan.commands.map((command) => command.args[0])).toContain("lint:scripts");
expect(plan.commands.map((command) => command.args[0])).not.toContain("test");
});
it("keeps an empty changed path list as a no-op", () => {
@@ -580,8 +675,6 @@ describe("scripts/changed-lanes", () => {
expect(plan.commands).toEqual([
{ name: "conflict markers", args: ["check:no-conflict-markers"] },
]);
expect(plan.runChangedTestsBroad).toBe(false);
expect(plan.runFullTests).toBe(false);
});
it("keeps docs-only changes cheap", () => {
@@ -592,40 +685,5 @@ describe("scripts/changed-lanes", () => {
expect(plan.commands).toEqual([
{ name: "conflict markers", args: ["check:no-conflict-markers"] },
]);
expect(plan.runChangedTestsBroad).toBe(false);
expect(plan.runFullTests).toBe(false);
});
it("sets a ten-minute Vitest watchdog for changed checks", () => {
expect(CHANGED_CHECK_VITEST_NO_OUTPUT_TIMEOUT_MS).toBe("600000");
expect(createChangedCheckVitestEnv({ PATH: "/usr/bin" })).toMatchObject({
PATH: "/usr/bin",
OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS: CHANGED_CHECK_VITEST_NO_OUTPUT_TIMEOUT_MS,
OPENCLAW_VITEST_NO_OUTPUT_RETRY: "0",
OPENCLAW_TEST_PROJECTS_SERIAL: "1",
OPENCLAW_VITEST_MAX_WORKERS: "1",
});
expect(
createChangedCheckVitestEnv({
OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS: "45000",
OPENCLAW_VITEST_NO_OUTPUT_RETRY: "1",
}),
).toMatchObject({
OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS: "45000",
OPENCLAW_VITEST_NO_OUTPUT_RETRY: "1",
});
});
it("does not force serial changed-check tests in CI or when workers are explicit", () => {
expect(createChangedCheckVitestEnv({ CI: "true" })).not.toHaveProperty(
"OPENCLAW_VITEST_MAX_WORKERS",
);
expect(createChangedCheckVitestEnv({ OPENCLAW_VITEST_MAX_WORKERS: "4" })).toMatchObject({
OPENCLAW_VITEST_MAX_WORKERS: "4",
});
expect(
createChangedCheckVitestEnv({ OPENCLAW_TEST_PROJECTS_PARALLEL: "4" }),
).not.toHaveProperty("OPENCLAW_TEST_PROJECTS_SERIAL");
});
});

View File

@@ -25,12 +25,23 @@ describe("scripts/test-projects changed-target routing", () => {
).toEqual(["src/shared/string-normalization.test.ts", "src/utils/provider-utils.test.ts"]);
});
it("keeps the broad changed run for Vitest wiring edits", () => {
it("keeps changed mode focused by default for Vitest wiring edits", () => {
expect(
resolveChangedTargetArgs(["--changed", "origin/main"], process.cwd(), () => [
"test/vitest/vitest.shared.config.ts",
"src/utils/provider-utils.ts",
]),
).toEqual(["src/utils/provider-utils.test.ts"]);
});
it("keeps the broad changed run available for Vitest wiring edits", () => {
expect(
resolveChangedTargetArgs(
["--changed", "origin/main"],
process.cwd(),
() => ["test/vitest/vitest.shared.config.ts", "src/utils/provider-utils.ts"],
{ env: { OPENCLAW_TEST_CHANGED_BROAD: "1" } },
),
).toBeNull();
});
@@ -129,11 +140,22 @@ describe("scripts/test-projects changed-target routing", () => {
]);
});
it("keeps the broad changed run for shared test helpers", () => {
it("keeps shared test helpers cheap by default when no precise target exists", () => {
expect(
resolveChangedTargetArgs(["--changed", "origin/main"], process.cwd(), () => [
"test/helpers/channels/plugin.ts",
]),
).toEqual([]);
});
it("keeps the broad changed run available for shared test helpers", () => {
expect(
resolveChangedTargetArgs(
["--changed", "origin/main"],
process.cwd(),
() => ["test/helpers/channels/plugin.ts"],
{ env: { OPENCLAW_TEST_CHANGED_BROAD: "1" } },
),
).toBeNull();
});
@@ -174,11 +196,22 @@ describe("scripts/test-projects changed-target routing", () => {
]);
});
it("keeps the broad changed run for unknown root surfaces", () => {
it("keeps unknown root surfaces cheap by default", () => {
expect(
resolveChangedTargetArgs(["--changed", "origin/main"], process.cwd(), () => [
"unknown/file.txt",
]),
).toEqual([]);
});
it("keeps the broad changed run available for unknown root surfaces", () => {
expect(
resolveChangedTargetArgs(
["--changed", "origin/main"],
process.cwd(),
() => ["unknown/file.txt"],
{ env: { OPENCLAW_TEST_CHANGED_BROAD: "1" } },
),
).toBeNull();
});
@@ -204,11 +237,29 @@ describe("scripts/test-projects changed-target routing", () => {
).toEqual([]);
});
it("adds extension tests for public plugin SDK changes", () => {
it("keeps public plugin SDK changes focused by default", () => {
const plans = buildVitestRunPlans(["--changed", "origin/main"], process.cwd(), () => [
"src/plugin-sdk/provider-entry.ts",
]);
expect(plans).toEqual([
{
config: "test/vitest/vitest.unit-fast.config.ts",
forwardedArgs: [],
includePatterns: ["src/plugin-sdk/provider-entry.test.ts"],
watchMode: false,
},
]);
});
it("adds extension tests for public plugin SDK changes in broad changed mode", () => {
const plans = buildVitestRunPlans(
["--changed", "origin/main"],
process.cwd(),
() => ["src/plugin-sdk/provider-entry.ts"],
{ env: { OPENCLAW_TEST_CHANGED_BROAD: "1" } },
);
expect(plans).toEqual([
{
config: "test/vitest/vitest.unit-fast.config.ts",
@@ -485,11 +536,29 @@ describe("scripts/test-projects changed-target routing", () => {
]);
});
it("routes plugin-sdk source files with sibling tests narrowly plus extension tests", () => {
it("routes plugin-sdk source files with sibling tests narrowly by default", () => {
const plans = buildVitestRunPlans(["--changed", "origin/main"], process.cwd(), () => [
"src/plugin-sdk/facade-runtime.ts",
]);
expect(plans).toEqual([
{
config: "test/vitest/vitest.bundled.config.ts",
forwardedArgs: [],
includePatterns: ["src/plugin-sdk/facade-runtime.test.ts"],
watchMode: false,
},
]);
});
it("routes plugin-sdk source files with sibling tests plus extensions in broad changed mode", () => {
const plans = buildVitestRunPlans(
["--changed", "origin/main"],
process.cwd(),
() => ["src/plugin-sdk/facade-runtime.ts"],
{ env: { OPENCLAW_TEST_CHANGED_BROAD: "1" } },
);
expect(plans).toEqual([
{
config: "test/vitest/vitest.bundled.config.ts",
@@ -521,22 +590,27 @@ describe("scripts/test-projects changed-target routing", () => {
]);
});
it("keeps focused changed mode to precise targets only", () => {
expect(
resolveChangedTestTargetPlan(["package.json", "src/commands/channels.add.ts"], {
focused: true,
}),
).toEqual({
it("keeps changed mode to precise targets by default", () => {
expect(resolveChangedTestTargetPlan(["package.json", "src/commands/channels.add.ts"])).toEqual({
mode: "targets",
targets: ["src/commands/channels.add.test.ts"],
});
});
it("uses import-graph targets in focused changed mode", () => {
it("keeps broad changed fallback available through explicit env", () => {
expect(
resolveChangedTestTargetPlan(["test/helpers/plugins/plugin-registration.ts"], {
focused: true,
}).targets,
resolveChangedTestTargetPlan(["package.json", "src/commands/channels.add.ts"], {
env: { OPENCLAW_TEST_CHANGED_BROAD: "1" },
}),
).toEqual({
mode: "broad",
targets: [],
});
});
it("uses import-graph targets in default changed mode", () => {
expect(
resolveChangedTestTargetPlan(["test/helpers/plugins/plugin-registration.ts"]).targets,
).toContain("extensions/openrouter/index.test.ts");
});