diff --git a/CHANGELOG.md b/CHANGELOG.md index c5fd66d0f7c..28816334d5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ Docs: https://docs.openclaw.ai - Docs/showcase: add a scannable hero, complete section jump links, and a responsive video grid for community examples. (#48493) Thanks @jchopard69. - Agents/local models: add `agents.defaults.localModelMode: "lean"` to drop heavyweight default tools like `browser`, `cron`, and `message`, reducing prompt size for weaker local-model setups without changing the normal path. Thanks @ImLukeF. +- QA/Matrix: split Matrix live QA into a source-linked `qa-matrix` runner and keep repo-private `qa-*` surfaces out of packaged and published builds. (#66723) Thanks @gumadeiras. ### Fixes diff --git a/docs/cli/plugins.md b/docs/cli/plugins.md index 0ed8425cd3a..9f86c75422c 100644 --- a/docs/cli/plugins.md +++ b/docs/cli/plugins.md @@ -197,6 +197,12 @@ Use `--link` to avoid copying a local directory (adds to `plugins.load.paths`): openclaw plugins install -l ./my-plugin ``` +Repo QA example (source-linked dev surface; not shipped in packaged installs): + +```bash +openclaw plugins install -l ./extensions/qa-matrix +``` + `--force` is not supported with `--link` because linked installs reuse the source path instead of copying over a managed install target. diff --git a/docs/help/testing.md b/docs/help/testing.md index 26924635889..68e597dc060 100644 --- a/docs/help/testing.md +++ b/docs/help/testing.md @@ -67,9 +67,13 @@ These commands sit beside the main test suites when you need QA-lab realism: - Starts the Docker-backed QA site for operator-style QA work. - `pnpm openclaw qa matrix` - Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver. + - This QA host is repo/dev-only today. Packaged OpenClaw installs do not ship + `qa-lab`, so they do not expose `openclaw qa`. + - Repo checkouts can link the in-tree plugin directly: + `openclaw plugins install -l ./extensions/qa-matrix`. - Provisions three temporary Matrix users (`driver`, `sut`, `observer`) plus one private room, then starts a QA gateway child with the real Matrix plugin as the SUT transport. - Uses the pinned stable Tuwunel image `ghcr.io/matrix-construct/tuwunel:v1.5.1` by default. Override with `OPENCLAW_QA_MATRIX_TUWUNEL_IMAGE` when you need to test a different image. - - Matrix currently supports only `--credential-source env` because the lane provisions disposable users locally. + - Matrix does not expose shared credential-source flags because the lane provisions disposable users locally. - Writes a Matrix QA report, summary, and observed-events artifact under `.artifacts/qa-e2e/...`. - `pnpm openclaw qa telegram` - Runs the Telegram live QA lane against a real private group using the driver and SUT bot tokens from env. @@ -170,11 +174,12 @@ Adding a channel to the markdown QA system requires exactly two things: 1. A transport adapter for the channel. 2. A scenario pack that exercises the channel contract. -Do not add a channel-specific QA runner when the shared `qa-lab` runner can +Do not add a new top-level QA command root when the shared `qa-lab` host can own the flow. -`qa-lab` owns the shared mechanics: +`qa-lab` owns the shared host mechanics: +- the `openclaw qa` command root - suite startup and teardown - worker concurrency - artifact writing @@ -182,8 +187,9 @@ own the flow. - scenario execution - compatibility aliases for older `qa-channel` scenarios -The channel adapter owns the transport contract: +Runner plugins own the transport contract: +- how `openclaw qa ` is mounted beneath the shared `qa` root - how the gateway is configured for that transport - how readiness is checked - how inbound events are injected @@ -194,17 +200,20 @@ The channel adapter owns the transport contract: The minimum adoption bar for a new channel is: -1. Implement the transport adapter on the shared `qa-lab` seam. -2. Register the adapter in the transport registry. -3. Keep transport-specific mechanics inside the adapter or the channel harness. -4. Author or adapt markdown scenarios under `qa/scenarios/`. -5. Use the generic scenario helpers for new scenarios. -6. Keep existing compatibility aliases working unless the repo is doing an intentional migration. +1. Keep `qa-lab` as the owner of the shared `qa` root. +2. Implement the transport runner on the shared `qa-lab` host seam. +3. Keep transport-specific mechanics inside the runner plugin or channel harness. +4. Mount the runner as `openclaw qa ` instead of registering a competing root command. + Runner plugins should declare `qaRunners` in `openclaw.plugin.json` and export a matching `qaRunnerCliRegistrations` array from `runtime-api.ts`. + Keep `runtime-api.ts` light; lazy CLI and runner execution should stay behind separate entrypoints. +5. Author or adapt markdown scenarios under `qa/scenarios/`. +6. Use the generic scenario helpers for new scenarios. +7. Keep existing compatibility aliases working unless the repo is doing an intentional migration. The decision rule is strict: - If behavior can be expressed once in `qa-lab`, put it in `qa-lab`. -- If behavior depends on one channel transport, keep it in that adapter or plugin harness. +- If behavior depends on one channel transport, keep it in that runner plugin or plugin harness. - If a scenario needs a new capability that more than one channel can use, add a generic helper instead of a channel-specific branch in `suite.ts`. - If a behavior is only meaningful for one transport, keep the scenario transport-specific and make that explicit in the scenario contract. diff --git a/docs/plugins/manifest.md b/docs/plugins/manifest.md index 26c8fd98edc..c48fc64af79 100644 --- a/docs/plugins/manifest.md +++ b/docs/plugins/manifest.md @@ -56,6 +56,8 @@ Use it for: plugin before runtime loads - static capability ownership snapshots used for bundled compat wiring and contract coverage +- cheap QA runner metadata that the shared `openclaw qa` host can inspect + before plugin runtime loads - channel-specific config metadata that should merge into catalog and validation surfaces without loading runtime - config UI hints @@ -158,6 +160,7 @@ Those belong in your plugin code and `package.json`. | `providerAuthChoices` | No | `object[]` | Cheap auth-choice metadata for onboarding pickers, preferred-provider resolution, and simple CLI flag wiring. | | `activation` | No | `object` | Cheap activation hints for provider, command, channel, route, and capability-triggered loading. Metadata only; plugin runtime still owns actual behavior. | | `setup` | No | `object` | Cheap setup/onboarding descriptors that discovery and setup surfaces can inspect without loading plugin runtime. | +| `qaRunners` | No | `object[]` | Cheap QA runner descriptors used by the shared `openclaw qa` host before plugin runtime loads. | | `contracts` | No | `object` | Static bundled capability snapshot for speech, realtime transcription, realtime voice, media-understanding, image-generation, music-generation, video-generation, web-fetch, web search, and tool ownership. | | `channelConfigs` | No | `Record` | Manifest-owned channel config metadata merged into discovery and validation surfaces before runtime loads. | | `skills` | No | `string[]` | Skill directories to load, relative to the plugin root. | @@ -219,6 +222,29 @@ uses this metadata for diagnostics without importing plugin runtime code. Use `activation` when the plugin can cheaply declare which control-plane events should activate it later. +## qaRunners reference + +Use `qaRunners` when a plugin contributes one or more transport runners beneath +the shared `openclaw qa` root. Keep this metadata cheap and static; the plugin +runtime still owns actual CLI registration through a lightweight +`runtime-api.ts` surface that exports `qaRunnerCliRegistrations`. + +```json +{ + "qaRunners": [ + { + "commandName": "matrix", + "description": "Run the Docker-backed Matrix live QA lane against a disposable homeserver" + } + ] +} +``` + +| Field | Required | Type | What it means | +| ------------- | -------- | -------- | ------------------------------------------------------------------ | +| `commandName` | Yes | `string` | Subcommand mounted beneath `openclaw qa`, for example `matrix`. | +| `description` | No | `string` | Fallback help text used when the shared host needs a stub command. | + This block is metadata only. It does not register runtime behavior, and it does not replace `register(...)`, `setupEntry`, or other runtime/plugin entrypoints. Current consumers use it as a narrowing hint before broader plugin loading, so diff --git a/extensions/qa-lab/runtime-api.ts b/extensions/qa-lab/runtime-api.ts index 801051438fb..0d61dc0b7b9 100644 --- a/extensions/qa-lab/runtime-api.ts +++ b/extensions/qa-lab/runtime-api.ts @@ -1 +1,2 @@ export * from "./src/runtime-api.js"; +export { startQaLiveLaneGateway } from "./src/live-transports/shared/live-gateway.runtime.js"; diff --git a/extensions/qa-lab/src/cli.runtime.test.ts b/extensions/qa-lab/src/cli.runtime.test.ts index 29732d7dea3..984d6b0d401 100644 --- a/extensions/qa-lab/src/cli.runtime.test.ts +++ b/extensions/qa-lab/src/cli.runtime.test.ts @@ -8,7 +8,6 @@ const { runQaSuiteFromRuntime, runQaCharacterEval, runQaMultipass, - runMatrixQaLive, runTelegramQaLive, startQaLabServer, writeQaDockerHarnessFiles, @@ -20,7 +19,6 @@ const { runQaSuiteFromRuntime: vi.fn(), runQaCharacterEval: vi.fn(), runQaMultipass: vi.fn(), - runMatrixQaLive: vi.fn(), runTelegramQaLive: vi.fn(), startQaLabServer: vi.fn(), writeQaDockerHarnessFiles: vi.fn(), @@ -52,10 +50,6 @@ vi.mock("./multipass.runtime.js", () => ({ runQaMultipass, })); -vi.mock("./live-transports/matrix/matrix-live.runtime.js", () => ({ - runMatrixQaLive, -})); - vi.mock("./live-transports/telegram/telegram-live.runtime.js", () => ({ runTelegramQaLive, })); @@ -88,7 +82,6 @@ import { runQaParityReportCommand, runQaSuiteCommand, } from "./cli.runtime.js"; -import { runQaMatrixCommand } from "./live-transports/matrix/cli.runtime.js"; import { runQaTelegramCommand } from "./live-transports/telegram/cli.runtime.js"; describe("qa cli runtime", () => { @@ -100,7 +93,6 @@ describe("qa cli runtime", () => { runQaCharacterEval.mockReset(); runQaManualLane.mockReset(); runQaMultipass.mockReset(); - runMatrixQaLive.mockReset(); runTelegramQaLive.mockReset(); startQaLabServer.mockReset(); writeQaDockerHarnessFiles.mockReset(); @@ -139,13 +131,6 @@ describe("qa cli runtime", () => { vmName: "openclaw-qa-test", scenarioIds: ["channel-chat-baseline"], }); - runMatrixQaLive.mockResolvedValue({ - outputDir: "/tmp/matrix", - reportPath: "/tmp/matrix/report.md", - summaryPath: "/tmp/matrix/summary.json", - observedEventsPath: "/tmp/matrix/observed.json", - scenarios: [], - }); runTelegramQaLive.mockResolvedValue({ outputDir: "/tmp/telegram", reportPath: "/tmp/telegram/report.md", @@ -226,30 +211,6 @@ describe("qa cli runtime", () => { }); }); - it("resolves matrix qa repo-root-relative paths before dispatching", async () => { - await runQaMatrixCommand({ - repoRoot: "/tmp/openclaw-repo", - outputDir: ".artifacts/qa/matrix", - providerMode: "live-frontier", - primaryModel: "openai/gpt-5.4", - alternateModel: "openai/gpt-5.4", - fastMode: true, - scenarioIds: ["matrix-thread-follow-up"], - sutAccountId: "sut-live", - }); - - expect(runMatrixQaLive).toHaveBeenCalledWith({ - repoRoot: path.resolve("/tmp/openclaw-repo"), - outputDir: path.resolve("/tmp/openclaw-repo", ".artifacts/qa/matrix"), - providerMode: "live-frontier", - primaryModel: "openai/gpt-5.4", - alternateModel: "openai/gpt-5.4", - fastMode: true, - scenarioIds: ["matrix-thread-follow-up"], - sutAccountId: "sut-live", - }); - }); - it("rejects output dirs that escape the repo root", () => { expect(() => resolveRepoRelativeOutputDir("/tmp/openclaw-repo", "../outside")).toThrow( "--output-dir must stay within the repo root.", @@ -273,20 +234,6 @@ describe("qa cli runtime", () => { ); }); - it("defaults matrix qa runs onto the live provider lane", async () => { - await runQaMatrixCommand({ - repoRoot: "/tmp/openclaw-repo", - scenarioIds: ["matrix-thread-follow-up"], - }); - - expect(runMatrixQaLive).toHaveBeenCalledWith( - expect.objectContaining({ - repoRoot: path.resolve("/tmp/openclaw-repo"), - providerMode: "live-frontier", - }), - ); - }); - it("normalizes legacy live-openai suite runs onto the frontier provider mode", async () => { await runQaSuiteCommand({ repoRoot: "/tmp/openclaw-repo", diff --git a/extensions/qa-lab/src/cli.test.ts b/extensions/qa-lab/src/cli.test.ts index 5fc231ce621..8d60b8079fd 100644 --- a/extensions/qa-lab/src/cli.test.ts +++ b/extensions/qa-lab/src/cli.test.ts @@ -1,22 +1,76 @@ import { Command } from "commander"; +import type { QaRunnerCliContribution } from "openclaw/plugin-sdk/qa-runner-runtime"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +const TEST_QA_RUNNER = { + pluginId: "qa-runner-test", + commandName: "runner-test", + description: "Run the test live QA lane", + npmSpec: "@openclaw/qa-runner-test", +} as const; + +function createAvailableQaRunnerContribution() { + return { + pluginId: TEST_QA_RUNNER.pluginId, + commandName: TEST_QA_RUNNER.commandName, + status: "available" as const, + registration: { + commandName: TEST_QA_RUNNER.commandName, + register: vi.fn((qa: Command) => { + qa.command(TEST_QA_RUNNER.commandName).action(() => undefined); + }), + }, + } satisfies QaRunnerCliContribution; +} + +function createMissingQaRunnerContribution(): QaRunnerCliContribution { + return { + pluginId: TEST_QA_RUNNER.pluginId, + commandName: TEST_QA_RUNNER.commandName, + description: TEST_QA_RUNNER.description, + status: "missing", + npmSpec: TEST_QA_RUNNER.npmSpec, + }; +} + +function createBlockedQaRunnerContribution(): QaRunnerCliContribution { + return { + pluginId: TEST_QA_RUNNER.pluginId, + commandName: TEST_QA_RUNNER.commandName, + description: TEST_QA_RUNNER.description, + status: "blocked", + }; +} + +function createConflictingQaRunnerContribution(commandName: string): QaRunnerCliContribution { + return { + pluginId: TEST_QA_RUNNER.pluginId, + commandName, + description: TEST_QA_RUNNER.description, + status: "blocked", + }; +} + const { runQaCredentialsAddCommand, runQaCredentialsListCommand, runQaCredentialsRemoveCommand, - runQaMatrixCommand, runQaTelegramCommand, } = vi.hoisted(() => ({ runQaCredentialsAddCommand: vi.fn(), runQaCredentialsListCommand: vi.fn(), runQaCredentialsRemoveCommand: vi.fn(), - runQaMatrixCommand: vi.fn(), runQaTelegramCommand: vi.fn(), })); -vi.mock("./live-transports/matrix/cli.runtime.js", () => ({ - runQaMatrixCommand, +const { listQaRunnerCliContributions } = vi.hoisted(() => ({ + listQaRunnerCliContributions: vi.fn<() => QaRunnerCliContribution[]>(() => [ + createAvailableQaRunnerContribution(), + ]), +})); + +vi.mock("openclaw/plugin-sdk/qa-runner-runtime", () => ({ + listQaRunnerCliContributions, })); vi.mock("./live-transports/telegram/cli.runtime.js", () => ({ @@ -36,63 +90,71 @@ describe("qa cli registration", () => { beforeEach(() => { program = new Command(); - registerQaLabCli(program); runQaCredentialsAddCommand.mockReset(); runQaCredentialsListCommand.mockReset(); runQaCredentialsRemoveCommand.mockReset(); - runQaMatrixCommand.mockReset(); runQaTelegramCommand.mockReset(); + listQaRunnerCliContributions + .mockReset() + .mockReturnValue([createAvailableQaRunnerContribution()]); + registerQaLabCli(program); }); afterEach(() => { vi.clearAllMocks(); }); - it("registers the matrix and telegram live transport subcommands", () => { + it("registers discovered and built-in live transport subcommands", () => { const qa = program.commands.find((command) => command.name() === "qa"); expect(qa).toBeDefined(); expect(qa?.commands.map((command) => command.name())).toEqual( - expect.arrayContaining(["matrix", "telegram", "credentials"]), + expect.arrayContaining([TEST_QA_RUNNER.commandName, "telegram", "credentials"]), ); }); - it("routes matrix CLI flags into the lane runtime", async () => { - await program.parseAsync([ - "node", - "openclaw", - "qa", - "matrix", - "--repo-root", - "/tmp/openclaw-repo", - "--output-dir", - ".artifacts/qa/matrix", - "--provider-mode", - "mock-openai", - "--model", - "mock-openai/gpt-5.4", - "--alt-model", - "mock-openai/gpt-5.4-alt", - "--scenario", - "matrix-thread-follow-up", - "--scenario", - "matrix-thread-isolation", - "--fast", - "--sut-account", - "sut-live", - ]); + it("delegates discovered qa runner registration through the generic host seam", () => { + const [{ registration }] = listQaRunnerCliContributions.mock.results[0]?.value; + expect(registration.register).toHaveBeenCalledTimes(1); + }); - expect(runQaMatrixCommand).toHaveBeenCalledWith({ - repoRoot: "/tmp/openclaw-repo", - outputDir: ".artifacts/qa/matrix", - providerMode: "mock-openai", - primaryModel: "mock-openai/gpt-5.4", - alternateModel: "mock-openai/gpt-5.4-alt", - fastMode: true, - scenarioIds: ["matrix-thread-follow-up", "matrix-thread-isolation"], - sutAccountId: "sut-live", - credentialSource: undefined, - credentialRole: undefined, - }); + it("keeps Telegram credential flags on the shared host CLI", () => { + const qa = program.commands.find((command) => command.name() === "qa"); + const telegram = qa?.commands.find((command) => command.name() === "telegram"); + const optionNames = telegram?.options.map((option) => option.long) ?? []; + + expect(optionNames).toEqual( + expect.arrayContaining(["--credential-source", "--credential-role"]), + ); + }); + + it("shows an install hint when a discovered runner plugin is unavailable", async () => { + listQaRunnerCliContributions.mockReset().mockReturnValue([createMissingQaRunnerContribution()]); + const missingProgram = new Command(); + registerQaLabCli(missingProgram); + + await expect( + missingProgram.parseAsync(["node", "openclaw", "qa", TEST_QA_RUNNER.commandName]), + ).rejects.toThrow(`openclaw plugins install ${TEST_QA_RUNNER.npmSpec}`); + }); + + it("shows an enable hint when a discovered runner plugin is installed but blocked", async () => { + listQaRunnerCliContributions.mockReset().mockReturnValue([createBlockedQaRunnerContribution()]); + const blockedProgram = new Command(); + registerQaLabCli(blockedProgram); + + await expect( + blockedProgram.parseAsync(["node", "openclaw", "qa", TEST_QA_RUNNER.commandName]), + ).rejects.toThrow(`Enable or allow plugin "${TEST_QA_RUNNER.pluginId}"`); + }); + + it("rejects discovered runners that collide with built-in qa subcommands", () => { + listQaRunnerCliContributions + .mockReset() + .mockReturnValue([createConflictingQaRunnerContribution("manual")]); + + expect(() => registerQaLabCli(new Command())).toThrow( + 'QA runner command "manual" conflicts with an existing qa subcommand', + ); }); it("routes telegram CLI defaults into the lane runtime", async () => { diff --git a/extensions/qa-lab/src/cli.ts b/extensions/qa-lab/src/cli.ts index 3eb0b6f8a0f..435a08b9486 100644 --- a/extensions/qa-lab/src/cli.ts +++ b/extensions/qa-lab/src/cli.ts @@ -1,6 +1,6 @@ import type { Command } from "commander"; import { collectString } from "./cli-options.js"; -import { LIVE_TRANSPORT_QA_CLI_REGISTRATIONS } from "./live-transports/cli.js"; +import { listLiveTransportQaCliRegistrations } from "./live-transports/cli.js"; import type { QaProviderModeInput } from "./run-config.js"; import { hasQaScenarioPack } from "./scenario-catalog.js"; @@ -183,6 +183,12 @@ export function isQaLabCliAvailable(): boolean { return hasQaScenarioPack(); } +function assertNoQaSubcommandCollision(qa: Command, commandName: string) { + if (qa.commands.some((command) => command.name() === commandName)) { + throw new Error(`QA runner command "${commandName}" conflicts with an existing qa subcommand`); + } +} + export function registerQaLabCli(program: Command) { const qa = program .command("qa") @@ -284,10 +290,6 @@ export function registerQaLabCli(program: Command) { }, ); - for (const lane of LIVE_TRANSPORT_QA_CLI_REGISTRATIONS) { - lane.register(qa); - } - qa.command("character-eval") .description("Run the character QA scenario across live models and write a judged report") .option("--repo-root ", "Repository root to target when running from a neutral cwd") @@ -579,4 +581,9 @@ export function registerQaLabCli(program: Command) { .action(async (opts: { host?: string; port?: number }) => { await runQaMockOpenAi(opts); }); + + for (const lane of listLiveTransportQaCliRegistrations()) { + assertNoQaSubcommandCollision(qa, lane.commandName); + lane.register(qa); + } } diff --git a/extensions/qa-lab/src/live-transports/cli.ts b/extensions/qa-lab/src/live-transports/cli.ts index 2038ff80a8c..9d2275ae6b4 100644 --- a/extensions/qa-lab/src/live-transports/cli.ts +++ b/extensions/qa-lab/src/live-transports/cli.ts @@ -1,8 +1,78 @@ -import { matrixQaCliRegistration } from "./matrix/cli.js"; +import { listQaRunnerCliContributions } from "openclaw/plugin-sdk/qa-runner-runtime"; import type { LiveTransportQaCliRegistration } from "./shared/live-transport-cli.js"; import { telegramQaCliRegistration } from "./telegram/cli.js"; +function createMissingQaRunnerCliRegistration(params: { + commandName: string; + description: string; + npmSpec: string; +}): LiveTransportQaCliRegistration { + return { + commandName: params.commandName, + register(qa) { + qa.command(params.commandName) + .description(params.description) + .action(() => { + throw new Error( + `QA runner "${params.commandName}" not installed. Install it with "openclaw plugins install ${params.npmSpec}".`, + ); + }); + }, + }; +} + +function createBlockedQaRunnerCliRegistration(params: { + commandName: string; + description?: string; + pluginId: string; +}): LiveTransportQaCliRegistration { + return { + commandName: params.commandName, + register(qa) { + qa.command(params.commandName) + .description(params.description ?? `Run the ${params.commandName} live QA lane`) + .action(() => { + throw new Error( + `QA runner "${params.commandName}" is installed but not active. Enable or allow plugin "${params.pluginId}" in your OpenClaw config, then try again.`, + ); + }); + }, + }; +} + +function createQaRunnerCliRegistration( + runner: ReturnType[number], +): LiveTransportQaCliRegistration { + if (runner.status === "available") { + return runner.registration; + } + if (runner.status === "blocked") { + return createBlockedQaRunnerCliRegistration({ + commandName: runner.commandName, + description: runner.description, + pluginId: runner.pluginId, + }); + } + return createMissingQaRunnerCliRegistration({ + commandName: runner.commandName, + description: + runner.description ?? + `Run the ${runner.commandName} live QA lane (install ${runner.npmSpec} first)`, + npmSpec: runner.npmSpec, + }); +} + export const LIVE_TRANSPORT_QA_CLI_REGISTRATIONS: readonly LiveTransportQaCliRegistration[] = [ telegramQaCliRegistration, - matrixQaCliRegistration, ]; + +export function listLiveTransportQaCliRegistrations(): readonly LiveTransportQaCliRegistration[] { + const liveRegistrations = [...LIVE_TRANSPORT_QA_CLI_REGISTRATIONS]; + const discoveredRunners = listQaRunnerCliContributions(); + + for (const runner of discoveredRunners) { + liveRegistrations.push(createQaRunnerCliRegistration(runner)); + } + + return liveRegistrations; +} diff --git a/extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts b/extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts index fe389ebc500..65ab2cf4e18 100644 --- a/extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts +++ b/extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts @@ -33,6 +33,11 @@ export type LiveTransportQaCliRegistration = { register(qa: Command): void; }; +export type LiveTransportQaCredentialCliOptions = { + sourceDescription?: string; + roleDescription?: string; +}; + export function createLazyCliRuntimeLoader(load: () => Promise) { let promise: Promise | null = null; return async () => { @@ -61,13 +66,14 @@ export function mapLiveTransportQaCommanderOptions( export function registerLiveTransportQaCli(params: { qa: Command; commandName: string; + credentialOptions?: LiveTransportQaCredentialCliOptions; description: string; outputDirHelp: string; scenarioHelp: string; sutAccountHelp: string; run: (opts: LiveTransportQaCommandOptions) => Promise; }) { - params.qa + const command = params.qa .command(params.commandName) .description(params.description) .option("--repo-root ", "Repository root to target when running from a neutral cwd") @@ -81,22 +87,27 @@ export function registerLiveTransportQaCli(params: { .option("--alt-model ", "Alternate provider/model ref") .option("--scenario ", params.scenarioHelp, collectString, []) .option("--fast", "Enable provider fast mode where supported", false) - .option("--sut-account ", params.sutAccountHelp, "sut") - .option( + .option("--sut-account ", params.sutAccountHelp, "sut"); + + if (params.credentialOptions) { + command.option( "--credential-source ", - "Credential source for live lanes: env or convex (default: env)", - ) - .option( - "--credential-role ", - "Credential role for convex auth: maintainer or ci (default: maintainer)", - ) - .action(async (opts: LiveTransportQaCommanderOptions) => { - await params.run(mapLiveTransportQaCommanderOptions(opts)); - }); + params.credentialOptions.sourceDescription ?? + "Credential source for live lanes: env or convex (default: env)", + ); + if (params.credentialOptions.roleDescription) { + command.option("--credential-role ", params.credentialOptions.roleDescription); + } + } + + command.action(async (opts: LiveTransportQaCommanderOptions) => { + await params.run(mapLiveTransportQaCommanderOptions(opts)); + }); } export function createLiveTransportQaCliRegistration(params: { commandName: string; + credentialOptions?: LiveTransportQaCredentialCliOptions; description: string; outputDirHelp: string; scenarioHelp: string; @@ -109,6 +120,7 @@ export function createLiveTransportQaCliRegistration(params: { registerLiveTransportQaCli({ qa, commandName: params.commandName, + credentialOptions: params.credentialOptions, description: params.description, outputDirHelp: params.outputDirHelp, scenarioHelp: params.scenarioHelp, diff --git a/extensions/qa-lab/src/live-transports/telegram/cli.ts b/extensions/qa-lab/src/live-transports/telegram/cli.ts index e957f229ebb..6237476b16f 100644 --- a/extensions/qa-lab/src/live-transports/telegram/cli.ts +++ b/extensions/qa-lab/src/live-transports/telegram/cli.ts @@ -20,6 +20,10 @@ async function runQaTelegram(opts: LiveTransportQaCommandOptions) { export const telegramQaCliRegistration: LiveTransportQaCliRegistration = createLiveTransportQaCliRegistration({ commandName: "telegram", + credentialOptions: { + sourceDescription: "Credential source for Telegram QA: env or convex (default: env)", + roleDescription: "Credential role for convex auth: maintainer or ci (default: maintainer)", + }, description: "Run the manual Telegram live QA lane against a private bot-to-bot group harness", outputDirHelp: "Telegram QA artifact directory", scenarioHelp: "Run only the named Telegram QA scenario (repeatable)", diff --git a/extensions/qa-lab/src/runtime-api.ts b/extensions/qa-lab/src/runtime-api.ts index da7390ce5df..602832e034c 100644 --- a/extensions/qa-lab/src/runtime-api.ts +++ b/extensions/qa-lab/src/runtime-api.ts @@ -3,6 +3,7 @@ export type { OpenClawConfig } from "openclaw/plugin-sdk/config-runtime"; export { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry"; export { callGatewayFromCli } from "openclaw/plugin-sdk/browser-node-runtime"; export type { PluginRuntime } from "openclaw/plugin-sdk/runtime-store"; +export { defaultQaRuntimeModelForMode } from "./model-selection.runtime.js"; export { buildQaTarget, createQaBusThread, diff --git a/extensions/qa-lab/src/self-check.ts b/extensions/qa-lab/src/self-check.ts index 4c29fdf2bb3..09931b6ed99 100644 --- a/extensions/qa-lab/src/self-check.ts +++ b/extensions/qa-lab/src/self-check.ts @@ -81,7 +81,7 @@ export async function runQaSelfCheckAgainstState(params: { timeline, notes: params.notes ?? [ "Vertical slice: qa-channel + qa-lab bus + private debugger surface.", - "Docker orchestration, matrix runs, and auto-fix loops remain follow-up work.", + "Docker orchestration, additional QA runners, and auto-fix loops remain follow-up work.", ], }); diff --git a/extensions/qa-matrix/cli.runtime.ts b/extensions/qa-matrix/cli.runtime.ts new file mode 100644 index 00000000000..4959d167a37 --- /dev/null +++ b/extensions/qa-matrix/cli.runtime.ts @@ -0,0 +1 @@ +export { runQaMatrixCommand } from "./src/cli.runtime.js"; diff --git a/extensions/qa-matrix/cli.ts b/extensions/qa-matrix/cli.ts new file mode 100644 index 00000000000..8bcc6de2457 --- /dev/null +++ b/extensions/qa-matrix/cli.ts @@ -0,0 +1 @@ +export { qaRunnerCliRegistrations, registerMatrixQaCli } from "./src/cli.js"; diff --git a/extensions/qa-matrix/index.ts b/extensions/qa-matrix/index.ts new file mode 100644 index 00000000000..f584ecb7ed6 --- /dev/null +++ b/extensions/qa-matrix/index.ts @@ -0,0 +1,8 @@ +import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry"; + +export default definePluginEntry({ + id: "qa-matrix", + name: "QA Matrix", + description: "Matrix QA transport runner and substrate", + register() {}, +}); diff --git a/extensions/qa-matrix/openclaw.plugin.json b/extensions/qa-matrix/openclaw.plugin.json new file mode 100644 index 00000000000..718ad710609 --- /dev/null +++ b/extensions/qa-matrix/openclaw.plugin.json @@ -0,0 +1,16 @@ +{ + "id": "qa-matrix", + "name": "QA Matrix", + "description": "Matrix QA transport runner and substrate", + "qaRunners": [ + { + "commandName": "matrix", + "description": "Run the Docker-backed Matrix live QA lane against a disposable homeserver" + } + ], + "configSchema": { + "type": "object", + "additionalProperties": false, + "properties": {} + } +} diff --git a/extensions/qa-matrix/package.json b/extensions/qa-matrix/package.json new file mode 100644 index 00000000000..624ce3cdd59 --- /dev/null +++ b/extensions/qa-matrix/package.json @@ -0,0 +1,34 @@ +{ + "name": "@openclaw/qa-matrix", + "version": "2026.4.12", + "description": "OpenClaw Matrix QA runner plugin", + "type": "module", + "devDependencies": { + "@openclaw/plugin-sdk": "workspace:*", + "openclaw": "workspace:*" + }, + "peerDependencies": { + "openclaw": ">=2026.4.12" + }, + "peerDependenciesMeta": { + "openclaw": { + "optional": true + } + }, + "openclaw": { + "extensions": [ + "./index.ts" + ], + "install": { + "npmSpec": "@openclaw/qa-matrix", + "defaultChoice": "npm", + "minHostVersion": ">=2026.4.12" + }, + "compat": { + "pluginApi": ">=2026.4.12" + }, + "build": { + "openclawVersion": "2026.4.12" + } + } +} diff --git a/extensions/qa-matrix/runtime-api.ts b/extensions/qa-matrix/runtime-api.ts new file mode 100644 index 00000000000..127fa5b033e --- /dev/null +++ b/extensions/qa-matrix/runtime-api.ts @@ -0,0 +1 @@ +export { qaRunnerCliRegistrations } from "./cli.js"; diff --git a/extensions/qa-matrix/runtime.ts b/extensions/qa-matrix/runtime.ts new file mode 100644 index 00000000000..6e56bad1416 --- /dev/null +++ b/extensions/qa-matrix/runtime.ts @@ -0,0 +1 @@ +export { runMatrixQaLive } from "./src/runners/contract/runtime.js"; diff --git a/extensions/qa-matrix/src/cli-options.ts b/extensions/qa-matrix/src/cli-options.ts new file mode 100644 index 00000000000..1576d43291f --- /dev/null +++ b/extensions/qa-matrix/src/cli-options.ts @@ -0,0 +1,4 @@ +export function collectString(value: string, previous: string[]) { + const trimmed = value.trim(); + return trimmed ? [...previous, trimmed] : previous; +} diff --git a/extensions/qa-matrix/src/cli-paths.ts b/extensions/qa-matrix/src/cli-paths.ts new file mode 100644 index 00000000000..529527fdff6 --- /dev/null +++ b/extensions/qa-matrix/src/cli-paths.ts @@ -0,0 +1,16 @@ +import path from "node:path"; + +export function resolveRepoRelativeOutputDir(repoRoot: string, outputDir?: string) { + if (!outputDir) { + return undefined; + } + if (path.isAbsolute(outputDir)) { + throw new Error("--output-dir must be a relative path inside the repo root."); + } + const resolved = path.resolve(repoRoot, outputDir); + const relative = path.relative(repoRoot, resolved); + if (relative.startsWith("..") || path.isAbsolute(relative)) { + throw new Error("--output-dir must stay within the repo root."); + } + return resolved; +} diff --git a/extensions/qa-lab/src/live-transports/matrix/cli.runtime.test.ts b/extensions/qa-matrix/src/cli.runtime.test.ts similarity index 96% rename from extensions/qa-lab/src/live-transports/matrix/cli.runtime.test.ts rename to extensions/qa-matrix/src/cli.runtime.test.ts index a73080273e1..81a91e3ee9e 100644 --- a/extensions/qa-lab/src/live-transports/matrix/cli.runtime.test.ts +++ b/extensions/qa-matrix/src/cli.runtime.test.ts @@ -2,7 +2,7 @@ import { describe, expect, it, vi } from "vitest"; const runMatrixQaLive = vi.hoisted(() => vi.fn()); -vi.mock("./matrix-live.runtime.js", () => ({ +vi.mock("./runners/contract/runtime.js", () => ({ runMatrixQaLive, })); diff --git a/extensions/qa-lab/src/live-transports/matrix/cli.runtime.ts b/extensions/qa-matrix/src/cli.runtime.ts similarity index 77% rename from extensions/qa-lab/src/live-transports/matrix/cli.runtime.ts rename to extensions/qa-matrix/src/cli.runtime.ts index f38843aa7dc..485f0ff023d 100644 --- a/extensions/qa-lab/src/live-transports/matrix/cli.runtime.ts +++ b/extensions/qa-matrix/src/cli.runtime.ts @@ -1,9 +1,9 @@ -import type { LiveTransportQaCommandOptions } from "../shared/live-transport-cli.js"; +import { runMatrixQaLive } from "./runners/contract/runtime.js"; +import type { LiveTransportQaCommandOptions } from "./shared/live-transport-cli.js"; import { printLiveTransportQaArtifacts, resolveLiveTransportQaRunOptions, -} from "../shared/live-transport-cli.runtime.js"; -import { runMatrixQaLive } from "./matrix-live.runtime.js"; +} from "./shared/live-transport-cli.runtime.js"; export async function runQaMatrixCommand(opts: LiveTransportQaCommandOptions) { const runOptions = resolveLiveTransportQaRunOptions(opts); diff --git a/extensions/qa-matrix/src/cli.test.ts b/extensions/qa-matrix/src/cli.test.ts new file mode 100644 index 00000000000..4a5dd704f6a --- /dev/null +++ b/extensions/qa-matrix/src/cli.test.ts @@ -0,0 +1,29 @@ +import { Command } from "commander"; +import { describe, expect, it } from "vitest"; +import { matrixQaCliRegistration } from "./cli.js"; + +describe("matrix qa cli registration", () => { + it("keeps disposable Matrix lane flags focused", () => { + const qa = new Command(); + + matrixQaCliRegistration.register(qa); + + const matrix = qa.commands.find((command) => command.name() === "matrix"); + const optionNames = matrix?.options.map((option) => option.long) ?? []; + + expect(optionNames).toEqual( + expect.arrayContaining([ + "--repo-root", + "--output-dir", + "--provider-mode", + "--model", + "--alt-model", + "--scenario", + "--fast", + "--sut-account", + ]), + ); + expect(optionNames).not.toContain("--credential-source"); + expect(optionNames).not.toContain("--credential-role"); + }); +}); diff --git a/extensions/qa-lab/src/live-transports/matrix/cli.ts b/extensions/qa-matrix/src/cli.ts similarity index 90% rename from extensions/qa-lab/src/live-transports/matrix/cli.ts rename to extensions/qa-matrix/src/cli.ts index 9c9cfd88668..6a81d8a5502 100644 --- a/extensions/qa-lab/src/live-transports/matrix/cli.ts +++ b/extensions/qa-matrix/src/cli.ts @@ -4,7 +4,7 @@ import { createLiveTransportQaCliRegistration, type LiveTransportQaCliRegistration, type LiveTransportQaCommandOptions, -} from "../shared/live-transport-cli.js"; +} from "./shared/live-transport-cli.js"; type MatrixQaCliRuntime = typeof import("./cli.runtime.js"); @@ -27,6 +27,8 @@ export const matrixQaCliRegistration: LiveTransportQaCliRegistration = run: runQaMatrix, }); +export const qaRunnerCliRegistrations = [matrixQaCliRegistration] as const; + export function registerMatrixQaCli(qa: Command) { matrixQaCliRegistration.register(qa); } diff --git a/extensions/qa-matrix/src/docker-runtime.ts b/extensions/qa-matrix/src/docker-runtime.ts new file mode 100644 index 00000000000..ada9dff01bc --- /dev/null +++ b/extensions/qa-matrix/src/docker-runtime.ts @@ -0,0 +1,274 @@ +import { createServer } from "node:net"; +import { runExec } from "openclaw/plugin-sdk/process-runtime"; +import { fetchWithSsrFGuard } from "openclaw/plugin-sdk/ssrf-runtime"; + +export type RunCommand = ( + command: string, + args: string[], + cwd: string, +) => Promise<{ stdout: string; stderr: string }>; + +export type FetchLike = (input: string) => Promise<{ ok: boolean }>; + +export async function fetchHealthUrl(url: string): Promise<{ ok: boolean }> { + const { response, release } = await fetchWithSsrFGuard({ + url, + init: { + signal: AbortSignal.timeout(2_000), + }, + policy: { allowPrivateNetwork: true }, + auditContext: "qa-matrix-docker-health-check", + }); + try { + return { ok: response.ok }; + } finally { + await release(); + } +} + +export function describeError(error: unknown) { + if (error instanceof Error) { + return error.message; + } + if (typeof error === "string") { + return error; + } + return JSON.stringify(error); +} + +async function isPortFree(port: number) { + return await new Promise((resolve) => { + const server = createServer(); + server.once("error", () => resolve(false)); + server.listen(port, "127.0.0.1", () => { + server.close(() => resolve(true)); + }); + }); +} + +async function findFreePort() { + return await new Promise((resolve, reject) => { + const server = createServer(); + server.once("error", reject); + server.listen(0, () => { + const address = server.address(); + if (!address || typeof address === "string") { + server.close(); + reject(new Error("failed to find free port")); + return; + } + server.close((error) => { + if (error) { + reject(error); + return; + } + resolve(address.port); + }); + }); + }); +} + +export async function resolveHostPort(preferredPort: number, pinned: boolean) { + if (pinned || (await isPortFree(preferredPort))) { + return preferredPort; + } + return await findFreePort(); +} + +function trimCommandOutput(output: string) { + const trimmed = output.trim(); + if (!trimmed) { + return ""; + } + const lines = trimmed.split("\n"); + return lines.length <= 120 ? trimmed : lines.slice(-120).join("\n"); +} + +export async function execCommand(command: string, args: string[], cwd: string) { + try { + return await runExec(command, args, { cwd, maxBuffer: 10 * 1024 * 1024 }); + } catch (error) { + const failedProcess = error as Error & { stdout?: string; stderr?: string }; + const renderedStdout = trimCommandOutput(failedProcess.stdout ?? ""); + const renderedStderr = trimCommandOutput(failedProcess.stderr ?? ""); + throw new Error( + [ + `Command failed: ${[command, ...args].join(" ")}`, + renderedStderr ? `stderr:\n${renderedStderr}` : "", + renderedStdout ? `stdout:\n${renderedStdout}` : "", + ] + .filter(Boolean) + .join("\n\n"), + { cause: error }, + ); + } +} + +export async function waitForHealth( + url: string, + deps: { + label?: string; + composeFile?: string; + fetchImpl: FetchLike; + sleepImpl: (ms: number) => Promise; + timeoutMs?: number; + pollMs?: number; + }, +) { + const timeoutMs = deps.timeoutMs ?? 360_000; + const pollMs = deps.pollMs ?? 1_000; + const startMs = Date.now(); + const deadline = startMs + timeoutMs; + let lastError: unknown = null; + + while (Date.now() < deadline) { + try { + const response = await deps.fetchImpl(url); + if (response.ok) { + return; + } + lastError = new Error(`Health check returned non-OK for ${url}`); + } catch (error) { + lastError = error; + } + await deps.sleepImpl(pollMs); + } + + const elapsedSec = Math.round((Date.now() - startMs) / 1000); + const service = deps.label ?? url; + const lines = [ + `${service} did not become healthy within ${elapsedSec}s (limit ${Math.round(timeoutMs / 1000)}s).`, + lastError ? `Last error: ${describeError(lastError)}` : "", + `Hint: check container logs with \`docker compose -f ${deps.composeFile ?? ""} logs\` and verify the port is not already in use.`, + ]; + throw new Error(lines.filter(Boolean).join("\n")); +} + +async function isHealthy(url: string, fetchImpl: FetchLike) { + try { + const response = await fetchImpl(url); + return response.ok; + } catch { + return false; + } +} + +function normalizeDockerServiceStatus(row?: { Health?: string; State?: string }) { + const health = row?.Health?.trim(); + if (health) { + return health; + } + const state = row?.State?.trim(); + if (state) { + return state; + } + return "unknown"; +} + +function parseDockerComposePsRows(stdout: string) { + const trimmed = stdout.trim(); + if (!trimmed) { + return [] as Array<{ Health?: string; State?: string }>; + } + + try { + const parsed = JSON.parse(trimmed) as + | Array<{ Health?: string; State?: string }> + | { Health?: string; State?: string }; + if (Array.isArray(parsed)) { + return parsed; + } + return [parsed]; + } catch { + return trimmed + .split("\n") + .map((line) => line.trim()) + .filter(Boolean) + .map((line) => JSON.parse(line) as { Health?: string; State?: string }); + } +} + +export async function waitForDockerServiceHealth( + service: string, + composeFile: string, + repoRoot: string, + runCommand: RunCommand, + sleepImpl: (ms: number) => Promise, + timeoutMs = 360_000, + pollMs = 1_000, +) { + const startMs = Date.now(); + const deadline = startMs + timeoutMs; + let lastStatus = "unknown"; + + while (Date.now() < deadline) { + try { + const { stdout } = await runCommand( + "docker", + ["compose", "-f", composeFile, "ps", "--format", "json", service], + repoRoot, + ); + const rows = parseDockerComposePsRows(stdout); + const row = rows[0]; + lastStatus = normalizeDockerServiceStatus(row); + if (lastStatus === "healthy" || lastStatus === "running") { + return; + } + } catch (error) { + lastStatus = describeError(error); + } + await sleepImpl(pollMs); + } + + const elapsedSec = Math.round((Date.now() - startMs) / 1000); + throw new Error( + [ + `${service} did not become healthy within ${elapsedSec}s (limit ${Math.round(timeoutMs / 1000)}s).`, + `Last status: ${lastStatus}`, + `Hint: check container logs with \`docker compose -f ${composeFile} logs ${service}\`.`, + ].join("\n"), + ); +} + +export async function resolveComposeServiceUrl( + service: string, + port: number, + composeFile: string, + repoRoot: string, + runCommand: RunCommand, + fetchImpl?: FetchLike, +) { + const { stdout: containerStdout } = await runCommand( + "docker", + ["compose", "-f", composeFile, "ps", "-q", service], + repoRoot, + ); + const containerId = containerStdout.trim(); + if (!containerId) { + return null; + } + const { stdout: ipStdout } = await runCommand( + "docker", + [ + "inspect", + "--format", + "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}", + containerId, + ], + repoRoot, + ); + const ip = ipStdout.trim(); + if (!ip) { + return null; + } + const baseUrl = `http://${ip}:${port}/`; + if (!fetchImpl) { + return baseUrl; + } + return (await isHealthy(`${baseUrl}healthz`, fetchImpl)) ? baseUrl : null; +} + +export const __testing = { + fetchHealthUrl, + normalizeDockerServiceStatus, +}; diff --git a/extensions/qa-matrix/src/report.ts b/extensions/qa-matrix/src/report.ts new file mode 100644 index 00000000000..f0d9b8c0704 --- /dev/null +++ b/extensions/qa-matrix/src/report.ts @@ -0,0 +1,100 @@ +export type QaReportCheck = { + name: string; + status: "pass" | "fail" | "skip"; + details?: string; +}; + +export type QaReportScenario = { + name: string; + status: "pass" | "fail" | "skip"; + details?: string; + steps?: QaReportCheck[]; +}; + +function pushDetailsBlock(lines: string[], label: string, details: string, indent = "") { + if (!details.includes("\n")) { + lines.push(`${indent}- ${label}: ${details}`); + return; + } + lines.push(`${indent}- ${label}:`); + lines.push("", "```text", details, "```"); +} + +export function renderQaMarkdownReport(params: { + title: string; + startedAt: Date; + finishedAt: Date; + checks?: QaReportCheck[]; + scenarios?: QaReportScenario[]; + timeline?: string[]; + notes?: string[]; +}) { + const checks = params.checks ?? []; + const scenarios = params.scenarios ?? []; + const passCount = + checks.filter((check) => check.status === "pass").length + + scenarios.filter((scenario) => scenario.status === "pass").length; + const failCount = + checks.filter((check) => check.status === "fail").length + + scenarios.filter((scenario) => scenario.status === "fail").length; + + const lines = [ + `# ${params.title}`, + "", + `- Started: ${params.startedAt.toISOString()}`, + `- Finished: ${params.finishedAt.toISOString()}`, + `- Duration ms: ${params.finishedAt.getTime() - params.startedAt.getTime()}`, + `- Passed: ${passCount}`, + `- Failed: ${failCount}`, + "", + ]; + + if (checks.length > 0) { + lines.push("## Checks", ""); + for (const check of checks) { + lines.push(`- [${check.status === "pass" ? "x" : " "}] ${check.name}`); + if (check.details) { + pushDetailsBlock(lines, "Details", check.details, " "); + } + } + } + + if (scenarios.length > 0) { + lines.push("", "## Scenarios", ""); + for (const scenario of scenarios) { + lines.push(`### ${scenario.name}`); + lines.push(""); + lines.push(`- Status: ${scenario.status}`); + if (scenario.details) { + pushDetailsBlock(lines, "Details", scenario.details); + } + if (scenario.steps?.length) { + lines.push("- Steps:"); + for (const step of scenario.steps) { + lines.push(` - [${step.status === "pass" ? "x" : " "}] ${step.name}`); + if (step.details) { + pushDetailsBlock(lines, "Details", step.details, " "); + } + } + } + lines.push(""); + } + } + + if (params.timeline && params.timeline.length > 0) { + lines.push("## Timeline", ""); + for (const item of params.timeline) { + lines.push(`- ${item}`); + } + } + + if (params.notes && params.notes.length > 0) { + lines.push("", "## Notes", ""); + for (const note of params.notes) { + lines.push(`- ${note}`); + } + } + + lines.push(""); + return lines.join("\n"); +} diff --git a/extensions/qa-matrix/src/run-config.ts b/extensions/qa-matrix/src/run-config.ts new file mode 100644 index 00000000000..36ea7f59efc --- /dev/null +++ b/extensions/qa-matrix/src/run-config.ts @@ -0,0 +1,9 @@ +export type QaProviderMode = "mock-openai" | "live-frontier"; +export type QaProviderModeInput = QaProviderMode | "live-openai"; + +export function normalizeQaProviderMode(input: unknown): QaProviderMode { + if (input === "mock-openai") { + return "mock-openai"; + } + return "live-frontier"; +} diff --git a/extensions/qa-matrix/src/runners/contract/model-selection.test.ts b/extensions/qa-matrix/src/runners/contract/model-selection.test.ts new file mode 100644 index 00000000000..ede92b96d9c --- /dev/null +++ b/extensions/qa-matrix/src/runners/contract/model-selection.test.ts @@ -0,0 +1,51 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const loadQaLabRuntimeModule = vi.hoisted(() => vi.fn()); +const defaultQaRuntimeModelForMode = vi.hoisted(() => vi.fn()); + +vi.mock("openclaw/plugin-sdk/qa-lab-runtime", () => ({ + loadQaLabRuntimeModule, +})); + +describe("matrix qa model selection", () => { + beforeEach(() => { + defaultQaRuntimeModelForMode.mockReset().mockImplementation((mode, options) => + options?.alternate ? `${mode}:alt` : `${mode}:primary`, + ); + loadQaLabRuntimeModule.mockReset().mockReturnValue({ + defaultQaRuntimeModelForMode, + }); + }); + + it("delegates default model selection through qa-lab runtime defaults", async () => { + const { resolveMatrixQaModels } = await import("./model-selection.js"); + + expect(resolveMatrixQaModels({ providerMode: "live-openai" })).toEqual({ + providerMode: "live-frontier", + primaryModel: "live-frontier:primary", + alternateModel: "live-frontier:alt", + }); + expect(defaultQaRuntimeModelForMode).toHaveBeenNthCalledWith(1, "live-frontier"); + expect(defaultQaRuntimeModelForMode).toHaveBeenNthCalledWith(2, "live-frontier", { + alternate: true, + }); + }); + + it("preserves explicit model overrides", async () => { + const { resolveMatrixQaModels } = await import("./model-selection.js"); + + expect( + resolveMatrixQaModels({ + providerMode: "mock-openai", + primaryModel: "custom-primary", + alternateModel: "custom-alt", + }), + ).toEqual({ + providerMode: "mock-openai", + primaryModel: "custom-primary", + alternateModel: "custom-alt", + }); + expect(loadQaLabRuntimeModule).not.toHaveBeenCalled(); + expect(defaultQaRuntimeModelForMode).not.toHaveBeenCalled(); + }); +}); diff --git a/extensions/qa-matrix/src/runners/contract/model-selection.ts b/extensions/qa-matrix/src/runners/contract/model-selection.ts new file mode 100644 index 00000000000..338378d85f8 --- /dev/null +++ b/extensions/qa-matrix/src/runners/contract/model-selection.ts @@ -0,0 +1,33 @@ +import { loadQaLabRuntimeModule } from "openclaw/plugin-sdk/qa-lab-runtime"; +import { normalizeQaProviderMode, type QaProviderModeInput } from "../../run-config.js"; + +export type ResolvedMatrixQaModels = { + providerMode: ReturnType; + primaryModel: string; + alternateModel: string; +}; + +export function resolveMatrixQaModels(params: { + providerMode?: QaProviderModeInput; + primaryModel?: string; + alternateModel?: string; +}): ResolvedMatrixQaModels { + const providerMode = normalizeQaProviderMode(params.providerMode ?? "live-frontier"); + const primaryModel = params.primaryModel?.trim(); + const alternateModel = params.alternateModel?.trim(); + if (primaryModel && alternateModel) { + return { + providerMode, + primaryModel, + alternateModel, + }; + } + + const qaLabRuntime = loadQaLabRuntimeModule(); + return { + providerMode, + primaryModel: primaryModel || qaLabRuntime.defaultQaRuntimeModelForMode(providerMode), + alternateModel: + alternateModel || qaLabRuntime.defaultQaRuntimeModelForMode(providerMode, { alternate: true }), + }; +} diff --git a/extensions/qa-lab/src/live-transports/matrix/matrix-live.runtime.test.ts b/extensions/qa-matrix/src/runners/contract/runtime.test.ts similarity index 99% rename from extensions/qa-lab/src/live-transports/matrix/matrix-live.runtime.test.ts rename to extensions/qa-matrix/src/runners/contract/runtime.test.ts index 061e552f8b3..f7e68d47809 100644 --- a/extensions/qa-lab/src/live-transports/matrix/matrix-live.runtime.test.ts +++ b/extensions/qa-matrix/src/runners/contract/runtime.test.ts @@ -1,6 +1,6 @@ import type { OpenClawConfig } from "openclaw/plugin-sdk/config-runtime"; import { afterEach, describe, expect, it, vi } from "vitest"; -import { __testing as liveTesting } from "./matrix-live.runtime.js"; +import { __testing as liveTesting } from "./runtime.js"; afterEach(() => { vi.useRealTimers(); diff --git a/extensions/qa-lab/src/live-transports/matrix/matrix-live.runtime.ts b/extensions/qa-matrix/src/runners/contract/runtime.ts similarity index 89% rename from extensions/qa-lab/src/live-transports/matrix/matrix-live.runtime.ts rename to extensions/qa-matrix/src/runners/contract/runtime.ts index 0b480c6ac66..12a58f3ff28 100644 --- a/extensions/qa-lab/src/live-transports/matrix/matrix-live.runtime.ts +++ b/extensions/qa-matrix/src/runners/contract/runtime.ts @@ -4,22 +4,20 @@ import path from "node:path"; import { setTimeout as sleep } from "node:timers/promises"; import type { OpenClawConfig } from "openclaw/plugin-sdk/config-runtime"; import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime"; -import { startQaGatewayChild } from "../../gateway-child.js"; +import { loadQaLabRuntimeModule } from "openclaw/plugin-sdk/qa-lab-runtime"; import type { QaReportCheck } from "../../report.js"; import { renderQaMarkdownReport } from "../../report.js"; +import { type QaProviderModeInput } from "../../run-config.js"; import { - defaultQaModelForMode, - normalizeQaProviderMode, - type QaProviderModeInput, -} from "../../run-config.js"; -import { startQaLiveLaneGateway } from "../shared/live-gateway.runtime.js"; -import { appendLiveLaneIssue, buildLiveLaneArtifactsError } from "../shared/live-lane-helpers.js"; + appendLiveLaneIssue, + buildLiveLaneArtifactsError, +} from "../../shared/live-lane-helpers.js"; import { provisionMatrixQaRoom, type MatrixQaObservedEvent, type MatrixQaProvisionResult, -} from "./matrix-driver-client.js"; -import { startMatrixQaHarness } from "./matrix-harness.runtime.js"; +} from "../../substrate/client.js"; +import { startMatrixQaHarness } from "../../substrate/harness.runtime.js"; import { MATRIX_QA_SCENARIOS, buildMatrixReplyDetails, @@ -28,7 +26,22 @@ import { runMatrixQaScenario, type MatrixQaCanaryArtifact, type MatrixQaScenarioArtifacts, -} from "./matrix-live-scenarios.js"; +} from "./scenarios.js"; +import { resolveMatrixQaModels } from "./model-selection.js"; + +type MatrixQaGatewayChild = { + call( + method: string, + params: Record, + options?: { timeoutMs?: number }, + ): Promise; + restart(): Promise; +}; + +type MatrixQaLiveLaneGatewayHarness = { + gateway: MatrixQaGatewayChild; + stop(): Promise; +}; type MatrixQaScenarioResult = { artifacts?: MatrixQaScenarioArtifacts; @@ -214,7 +227,7 @@ function isMatrixAccountReady(entry?: { } async function waitForMatrixChannelReady( - gateway: Awaited>, + gateway: MatrixQaGatewayChild, accountId: string, opts?: { pollMs?: number; @@ -255,6 +268,27 @@ async function waitForMatrixChannelReady( throw new Error(`matrix account "${accountId}" did not become ready`); } +async function startMatrixQaLiveLaneGateway(params: { + repoRoot: string; + transport: { + requiredPluginIds: readonly string[]; + createGatewayConfig: (params: { + baseUrl: string; + }) => Pick; + }; + transportBaseUrl: string; + providerMode: "mock-openai" | "live-frontier"; + primaryModel: string; + alternateModel: string; + fastMode?: boolean; + controlUiEnabled?: boolean; + mutateConfig?: (cfg: OpenClawConfig) => OpenClawConfig; +}): Promise { + return (await loadQaLabRuntimeModule().startQaLiveLaneGateway( + params, + )) as MatrixQaLiveLaneGatewayHarness; +} + export async function runMatrixQaLive(params: { fastMode?: boolean; outputDir?: string; @@ -271,9 +305,11 @@ export async function runMatrixQaLive(params: { path.join(repoRoot, ".artifacts", "qa-e2e", `matrix-${Date.now().toString(36)}`); await fs.mkdir(outputDir, { recursive: true }); - const providerMode = normalizeQaProviderMode(params.providerMode ?? "live-frontier"); - const primaryModel = params.primaryModel?.trim() || defaultQaModelForMode(providerMode); - const alternateModel = params.alternateModel?.trim() || defaultQaModelForMode(providerMode, true); + const { providerMode, primaryModel, alternateModel } = resolveMatrixQaModels({ + providerMode: params.providerMode, + primaryModel: params.primaryModel, + alternateModel: params.alternateModel, + }); const sutAccountId = params.sutAccountId?.trim() || "sut"; const scenarios = findMatrixQaScenarios(params.scenarioIds); const observedEvents: MatrixQaObservedEvent[] = []; @@ -317,12 +353,12 @@ export async function runMatrixQaLive(params: { const scenarioResults: MatrixQaScenarioResult[] = []; const cleanupErrors: string[] = []; let canaryArtifact: MatrixQaCanaryArtifact | undefined; - let gatewayHarness: Awaited> | null = null; + let gatewayHarness: MatrixQaLiveLaneGatewayHarness | null = null; let canaryFailed = false; const syncState: { driver?: string; observer?: string } = {}; try { - gatewayHarness = await startQaLiveLaneGateway({ + gatewayHarness = await startMatrixQaLiveLaneGateway({ repoRoot, transport: { requiredPluginIds: [], @@ -555,5 +591,6 @@ export const __testing = { buildMatrixQaConfig, buildObservedEventsArtifact, isMatrixAccountReady, + resolveMatrixQaModels, waitForMatrixChannelReady, }; diff --git a/extensions/qa-lab/src/live-transports/matrix/matrix-live-scenarios.test.ts b/extensions/qa-matrix/src/runners/contract/scenarios.test.ts similarity index 97% rename from extensions/qa-lab/src/live-transports/matrix/matrix-live-scenarios.test.ts rename to extensions/qa-matrix/src/runners/contract/scenarios.test.ts index 27ae4fab677..41f670957bb 100644 --- a/extensions/qa-lab/src/live-transports/matrix/matrix-live-scenarios.test.ts +++ b/extensions/qa-matrix/src/runners/contract/scenarios.test.ts @@ -3,19 +3,19 @@ const { createMatrixQaClient } = vi.hoisted(() => ({ createMatrixQaClient: vi.fn(), })); -vi.mock("./matrix-driver-client.js", () => ({ +vi.mock("../../substrate/client.js", () => ({ createMatrixQaClient, })); import { LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS, findMissingLiveTransportStandardScenarios, -} from "../shared/live-transport-scenarios.js"; +} from "../../shared/live-transport-scenarios.js"; import { __testing as scenarioTesting, MATRIX_QA_SCENARIOS, runMatrixQaScenario, -} from "./matrix-live-scenarios.js"; +} from "./scenarios.js"; describe("matrix live qa scenarios", () => { beforeEach(() => { diff --git a/extensions/qa-lab/src/live-transports/matrix/matrix-live-scenarios.ts b/extensions/qa-matrix/src/runners/contract/scenarios.ts similarity index 99% rename from extensions/qa-lab/src/live-transports/matrix/matrix-live-scenarios.ts rename to extensions/qa-matrix/src/runners/contract/scenarios.ts index fe4979e9eeb..7c8638d845c 100644 --- a/extensions/qa-lab/src/live-transports/matrix/matrix-live-scenarios.ts +++ b/extensions/qa-matrix/src/runners/contract/scenarios.ts @@ -3,8 +3,8 @@ import { collectLiveTransportStandardScenarioCoverage, selectLiveTransportScenarios, type LiveTransportScenarioDefinition, -} from "../shared/live-transport-scenarios.js"; -import { createMatrixQaClient, type MatrixQaObservedEvent } from "./matrix-driver-client.js"; +} from "../../shared/live-transport-scenarios.js"; +import { createMatrixQaClient, type MatrixQaObservedEvent } from "../../substrate/client.js"; export type MatrixQaScenarioId = | "matrix-thread-follow-up" diff --git a/extensions/qa-matrix/src/runtime-api.test.ts b/extensions/qa-matrix/src/runtime-api.test.ts new file mode 100644 index 00000000000..87a3796b68d --- /dev/null +++ b/extensions/qa-matrix/src/runtime-api.test.ts @@ -0,0 +1,9 @@ +import { describe, expect, it } from "vitest"; + +describe("matrix qa runtime api surface", () => { + it("keeps runner discovery lightweight", async () => { + const runtimeApi = await import("../runtime-api.js"); + + expect(Object.keys(runtimeApi).toSorted()).toEqual(["qaRunnerCliRegistrations"]); + }); +}); diff --git a/extensions/qa-matrix/src/shared/live-lane-helpers.ts b/extensions/qa-matrix/src/shared/live-lane-helpers.ts new file mode 100644 index 00000000000..cf65146123e --- /dev/null +++ b/extensions/qa-matrix/src/shared/live-lane-helpers.ts @@ -0,0 +1,18 @@ +import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime"; + +export function appendLiveLaneIssue(issues: string[], label: string, error: unknown) { + issues.push(`${label}: ${formatErrorMessage(error)}`); +} + +export function buildLiveLaneArtifactsError(params: { + heading: string; + artifacts: Record; + details?: string[]; +}) { + return [ + params.heading, + ...(params.details ?? []), + "Artifacts:", + ...Object.entries(params.artifacts).map(([label, filePath]) => `- ${label}: ${filePath}`), + ].join("\n"); +} diff --git a/extensions/qa-matrix/src/shared/live-transport-cli.runtime.ts b/extensions/qa-matrix/src/shared/live-transport-cli.runtime.ts new file mode 100644 index 00000000000..b840b2d6712 --- /dev/null +++ b/extensions/qa-matrix/src/shared/live-transport-cli.runtime.ts @@ -0,0 +1,40 @@ +import path from "node:path"; +import { resolveRepoRelativeOutputDir } from "../cli-paths.js"; +import type { QaProviderMode } from "../run-config.js"; +import { normalizeQaProviderMode } from "../run-config.js"; +import type { LiveTransportQaCommandOptions } from "./live-transport-cli.js"; + +export function resolveLiveTransportQaRunOptions( + opts: LiveTransportQaCommandOptions, +): LiveTransportQaCommandOptions & { + repoRoot: string; + providerMode: QaProviderMode; +} { + return { + repoRoot: path.resolve(opts.repoRoot ?? process.cwd()), + outputDir: resolveRepoRelativeOutputDir( + path.resolve(opts.repoRoot ?? process.cwd()), + opts.outputDir, + ), + providerMode: + opts.providerMode === undefined + ? "live-frontier" + : normalizeQaProviderMode(opts.providerMode), + primaryModel: opts.primaryModel, + alternateModel: opts.alternateModel, + fastMode: opts.fastMode, + scenarioIds: opts.scenarioIds, + sutAccountId: opts.sutAccountId, + credentialSource: opts.credentialSource?.trim(), + credentialRole: opts.credentialRole?.trim(), + }; +} + +export function printLiveTransportQaArtifacts( + laneLabel: string, + artifacts: Record, +) { + for (const [label, filePath] of Object.entries(artifacts)) { + process.stdout.write(`${laneLabel} ${label}: ${filePath}\n`); + } +} diff --git a/extensions/qa-matrix/src/shared/live-transport-cli.ts b/extensions/qa-matrix/src/shared/live-transport-cli.ts new file mode 100644 index 00000000000..92f320b4eb4 --- /dev/null +++ b/extensions/qa-matrix/src/shared/live-transport-cli.ts @@ -0,0 +1,132 @@ +import type { Command } from "commander"; +import { collectString } from "../cli-options.js"; +import type { QaProviderModeInput } from "../run-config.js"; + +export type LiveTransportQaCommandOptions = { + repoRoot?: string; + outputDir?: string; + providerMode?: QaProviderModeInput; + primaryModel?: string; + alternateModel?: string; + fastMode?: boolean; + scenarioIds?: string[]; + sutAccountId?: string; + credentialSource?: string; + credentialRole?: string; +}; + +type LiveTransportQaCommanderOptions = { + repoRoot?: string; + outputDir?: string; + providerMode?: QaProviderModeInput; + model?: string; + altModel?: string; + scenario?: string[]; + fast?: boolean; + sutAccount?: string; + credentialSource?: string; + credentialRole?: string; +}; + +export type LiveTransportQaCliRegistration = { + commandName: string; + register(qa: Command): void; +}; + +export type LiveTransportQaCredentialCliOptions = { + sourceDescription?: string; + roleDescription?: string; +}; + +export function createLazyCliRuntimeLoader(load: () => Promise) { + let promise: Promise | null = null; + return async () => { + promise ??= load(); + return await promise; + }; +} + +export function mapLiveTransportQaCommanderOptions( + opts: LiveTransportQaCommanderOptions, +): LiveTransportQaCommandOptions { + return { + repoRoot: opts.repoRoot, + outputDir: opts.outputDir, + providerMode: opts.providerMode, + primaryModel: opts.model, + alternateModel: opts.altModel, + fastMode: opts.fast, + scenarioIds: opts.scenario, + sutAccountId: opts.sutAccount, + credentialSource: opts.credentialSource, + credentialRole: opts.credentialRole, + }; +} + +export function registerLiveTransportQaCli(params: { + qa: Command; + commandName: string; + credentialOptions?: LiveTransportQaCredentialCliOptions; + description: string; + outputDirHelp: string; + scenarioHelp: string; + sutAccountHelp: string; + run: (opts: LiveTransportQaCommandOptions) => Promise; +}) { + const command = params.qa + .command(params.commandName) + .description(params.description) + .option("--repo-root ", "Repository root to target when running from a neutral cwd") + .option("--output-dir ", params.outputDirHelp) + .option( + "--provider-mode ", + "Provider mode: mock-openai or live-frontier (legacy live-openai still works)", + "live-frontier", + ) + .option("--model ", "Primary provider/model ref") + .option("--alt-model ", "Alternate provider/model ref") + .option("--scenario ", params.scenarioHelp, collectString, []) + .option("--fast", "Enable provider fast mode where supported", false) + .option("--sut-account ", params.sutAccountHelp, "sut"); + + if (params.credentialOptions) { + command.option( + "--credential-source ", + params.credentialOptions.sourceDescription ?? + "Credential source for live lanes: env or convex (default: env)", + ); + if (params.credentialOptions.roleDescription) { + command.option("--credential-role ", params.credentialOptions.roleDescription); + } + } + + command.action(async (opts: LiveTransportQaCommanderOptions) => { + await params.run(mapLiveTransportQaCommanderOptions(opts)); + }); +} + +export function createLiveTransportQaCliRegistration(params: { + commandName: string; + credentialOptions?: LiveTransportQaCredentialCliOptions; + description: string; + outputDirHelp: string; + scenarioHelp: string; + sutAccountHelp: string; + run: (opts: LiveTransportQaCommandOptions) => Promise; +}): LiveTransportQaCliRegistration { + return { + commandName: params.commandName, + register(qa: Command) { + registerLiveTransportQaCli({ + qa, + commandName: params.commandName, + credentialOptions: params.credentialOptions, + description: params.description, + outputDirHelp: params.outputDirHelp, + scenarioHelp: params.scenarioHelp, + sutAccountHelp: params.sutAccountHelp, + run: params.run, + }); + }, + }; +} diff --git a/extensions/qa-matrix/src/shared/live-transport-scenarios.ts b/extensions/qa-matrix/src/shared/live-transport-scenarios.ts new file mode 100644 index 00000000000..535bcc3de53 --- /dev/null +++ b/extensions/qa-matrix/src/shared/live-transport-scenarios.ts @@ -0,0 +1,149 @@ +export type LiveTransportStandardScenarioId = + | "canary" + | "mention-gating" + | "allowlist-block" + | "top-level-reply-shape" + | "restart-resume" + | "thread-follow-up" + | "thread-isolation" + | "reaction-observation" + | "help-command"; + +export type LiveTransportScenarioDefinition = { + id: TId; + standardId?: LiveTransportStandardScenarioId; + timeoutMs: number; + title: string; +}; + +export type LiveTransportStandardScenarioDefinition = { + description: string; + id: LiveTransportStandardScenarioId; + title: string; +}; + +export const LIVE_TRANSPORT_STANDARD_SCENARIOS: readonly LiveTransportStandardScenarioDefinition[] = + [ + { + id: "canary", + title: "Transport canary", + description: "The lane can trigger one known-good reply on the real transport.", + }, + { + id: "mention-gating", + title: "Mention gating", + description: "Messages without the required mention do not trigger a reply.", + }, + { + id: "allowlist-block", + title: "Sender allowlist block", + description: "Non-allowlisted senders do not trigger a reply.", + }, + { + id: "top-level-reply-shape", + title: "Top-level reply shape", + description: "Top-level replies stay top-level when the lane is configured that way.", + }, + { + id: "restart-resume", + title: "Restart resume", + description: "The lane still responds after a gateway restart.", + }, + { + id: "thread-follow-up", + title: "Thread follow-up", + description: "Threaded prompts receive threaded replies with the expected relation metadata.", + }, + { + id: "thread-isolation", + title: "Thread isolation", + description: "Fresh top-level prompts stay out of prior threads.", + }, + { + id: "reaction-observation", + title: "Reaction observation", + description: "Reaction events are observed and normalized correctly.", + }, + { + id: "help-command", + title: "Help command", + description: "The transport-specific help command path replies successfully.", + }, + ] as const; + +export const LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS: readonly LiveTransportStandardScenarioId[] = + [ + "canary", + "mention-gating", + "allowlist-block", + "top-level-reply-shape", + "restart-resume", + ] as const; + +const LIVE_TRANSPORT_STANDARD_SCENARIO_ID_SET = new Set( + LIVE_TRANSPORT_STANDARD_SCENARIOS.map((scenario) => scenario.id), +); + +function assertKnownStandardScenarioIds(ids: readonly LiveTransportStandardScenarioId[]) { + for (const id of ids) { + if (!LIVE_TRANSPORT_STANDARD_SCENARIO_ID_SET.has(id)) { + throw new Error(`unknown live transport standard scenario id: ${id}`); + } + } +} + +export function selectLiveTransportScenarios(params: { + ids?: string[]; + laneLabel: string; + scenarios: readonly TDefinition[]; +}) { + if (!params.ids || params.ids.length === 0) { + return [...params.scenarios]; + } + const requested = new Set(params.ids); + const selected = params.scenarios.filter((scenario) => params.ids?.includes(scenario.id)); + const missingIds = [...requested].filter( + (id) => !selected.some((scenario) => scenario.id === id), + ); + if (missingIds.length > 0) { + throw new Error(`unknown ${params.laneLabel} QA scenario id(s): ${missingIds.join(", ")}`); + } + return selected; +} + +export function collectLiveTransportStandardScenarioCoverage(params: { + alwaysOnStandardScenarioIds?: readonly LiveTransportStandardScenarioId[]; + scenarios: readonly LiveTransportScenarioDefinition[]; +}) { + const coverage: LiveTransportStandardScenarioId[] = []; + const seen = new Set(); + const append = (id: LiveTransportStandardScenarioId | undefined) => { + if (!id || seen.has(id)) { + return; + } + seen.add(id); + coverage.push(id); + }; + + assertKnownStandardScenarioIds(params.alwaysOnStandardScenarioIds ?? []); + for (const id of params.alwaysOnStandardScenarioIds ?? []) { + append(id); + } + for (const scenario of params.scenarios) { + if (scenario.standardId) { + assertKnownStandardScenarioIds([scenario.standardId]); + } + append(scenario.standardId); + } + return coverage; +} + +export function findMissingLiveTransportStandardScenarios(params: { + coveredStandardScenarioIds: readonly LiveTransportStandardScenarioId[]; + expectedStandardScenarioIds: readonly LiveTransportStandardScenarioId[]; +}) { + assertKnownStandardScenarioIds(params.coveredStandardScenarioIds); + assertKnownStandardScenarioIds(params.expectedStandardScenarioIds); + const covered = new Set(params.coveredStandardScenarioIds); + return params.expectedStandardScenarioIds.filter((id) => !covered.has(id)); +} diff --git a/extensions/qa-lab/src/live-transports/matrix/matrix-driver-client.test.ts b/extensions/qa-matrix/src/substrate/client.test.ts similarity index 99% rename from extensions/qa-lab/src/live-transports/matrix/matrix-driver-client.test.ts rename to extensions/qa-matrix/src/substrate/client.test.ts index 59a3e4ae97c..b3152da8b4e 100644 --- a/extensions/qa-lab/src/live-transports/matrix/matrix-driver-client.test.ts +++ b/extensions/qa-matrix/src/substrate/client.test.ts @@ -4,7 +4,7 @@ import { createMatrixQaClient, provisionMatrixQaRoom, type MatrixQaObservedEvent, -} from "./matrix-driver-client.js"; +} from "./client.js"; function resolveRequestUrl(input: RequestInfo | URL) { if (typeof input === "string") { diff --git a/extensions/qa-lab/src/live-transports/matrix/matrix-driver-client.ts b/extensions/qa-matrix/src/substrate/client.ts similarity index 100% rename from extensions/qa-lab/src/live-transports/matrix/matrix-driver-client.ts rename to extensions/qa-matrix/src/substrate/client.ts diff --git a/extensions/qa-lab/src/live-transports/matrix/matrix-harness.runtime.test.ts b/extensions/qa-matrix/src/substrate/harness.runtime.test.ts similarity index 98% rename from extensions/qa-lab/src/live-transports/matrix/matrix-harness.runtime.test.ts rename to extensions/qa-matrix/src/substrate/harness.runtime.test.ts index e1451fe2b39..1a907f63872 100644 --- a/extensions/qa-lab/src/live-transports/matrix/matrix-harness.runtime.test.ts +++ b/extensions/qa-matrix/src/substrate/harness.runtime.test.ts @@ -2,11 +2,7 @@ import { mkdtemp, readFile, rm } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { describe, expect, it, vi } from "vitest"; -import { - __testing, - startMatrixQaHarness, - writeMatrixQaHarnessFiles, -} from "./matrix-harness.runtime.js"; +import { __testing, startMatrixQaHarness, writeMatrixQaHarnessFiles } from "./harness.runtime.js"; describe("matrix harness runtime", () => { it("writes a pinned Tuwunel compose file and redacted manifest", async () => { diff --git a/extensions/qa-lab/src/live-transports/matrix/matrix-harness.runtime.ts b/extensions/qa-matrix/src/substrate/harness.runtime.ts similarity index 99% rename from extensions/qa-lab/src/live-transports/matrix/matrix-harness.runtime.ts rename to extensions/qa-matrix/src/substrate/harness.runtime.ts index d10b6df32b4..5b21e34d823 100644 --- a/extensions/qa-lab/src/live-transports/matrix/matrix-harness.runtime.ts +++ b/extensions/qa-matrix/src/substrate/harness.runtime.ts @@ -11,7 +11,7 @@ import { waitForHealth, type FetchLike, type RunCommand, -} from "../../docker-runtime.js"; +} from "../docker-runtime.js"; const MATRIX_QA_DEFAULT_IMAGE = "ghcr.io/matrix-construct/tuwunel:v1.5.1"; const MATRIX_QA_DEFAULT_SERVER_NAME = "matrix-qa.test"; diff --git a/package.json b/package.json index 6d141d8e48b..c2c4464e846 100644 --- a/package.json +++ b/package.json @@ -30,9 +30,8 @@ "!dist/**/*.map", "!dist/plugin-sdk/.tsbuildinfo", "!dist/extensions/qa-channel/**", - "dist/extensions/qa-channel/runtime-api.js", "!dist/extensions/qa-lab/**", - "dist/extensions/qa-lab/runtime-api.js", + "!dist/extensions/qa-matrix/**", "docs/", "!docs/.generated/**", "!docs/.i18n/zh-CN.tm.jsonl", @@ -766,6 +765,14 @@ "types": "./dist/plugin-sdk/matrix-thread-bindings.d.ts", "default": "./dist/plugin-sdk/matrix-thread-bindings.js" }, + "./plugin-sdk/qa-lab-runtime": { + "types": "./dist/plugin-sdk/qa-lab-runtime.d.ts", + "default": "./dist/plugin-sdk/qa-lab-runtime.js" + }, + "./plugin-sdk/qa-runner-runtime": { + "types": "./dist/plugin-sdk/qa-runner-runtime.d.ts", + "default": "./dist/plugin-sdk/qa-runner-runtime.js" + }, "./plugin-sdk/mattermost": { "types": "./dist/plugin-sdk/mattermost.d.ts", "default": "./dist/plugin-sdk/mattermost.js" @@ -1236,6 +1243,8 @@ "proxy:install-ca": "node --import tsx scripts/proxy-install-ca.mjs", "proxy:run": "node scripts/run-node.mjs proxy run", "proxy:start": "node scripts/run-node.mjs proxy start", + "qa-runners:check": "node --import tsx scripts/generate-qa-runner-catalog.ts --check", + "qa-runners:gen": "node --import tsx scripts/generate-qa-runner-catalog.ts --write", "qa:e2e": "node --import tsx scripts/qa-e2e.ts", "qa:lab:build": "vite build --config extensions/qa-lab/web/vite.config.ts", "qa:lab:ui": "pnpm openclaw qa ui", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 433aec270cc..3dceb2db276 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -994,6 +994,15 @@ importers: specifier: workspace:* version: link:../.. + extensions/qa-matrix: + devDependencies: + '@openclaw/plugin-sdk': + specifier: workspace:* + version: link:../../packages/plugin-sdk + openclaw: + specifier: workspace:* + version: link:../.. + extensions/qianfan: devDependencies: '@openclaw/plugin-sdk': diff --git a/scripts/build-all.mjs b/scripts/build-all.mjs index 343fb7a3c8d..34ce0b76db7 100644 --- a/scripts/build-all.mjs +++ b/scripts/build-all.mjs @@ -10,11 +10,6 @@ export const BUILD_ALL_STEPS = [ { label: "canvas:a2ui:bundle", kind: "pnpm", pnpmArgs: ["canvas:a2ui:bundle"] }, { label: "tsdown", kind: "node", args: ["scripts/tsdown-build.mjs"] }, { label: "runtime-postbuild", kind: "node", args: ["scripts/runtime-postbuild.mjs"] }, - { - label: "write-npm-update-compat-sidecars", - kind: "node", - args: ["scripts/write-npm-update-compat-sidecars.mjs"], - }, { label: "build-stamp", kind: "node", args: ["scripts/build-stamp.mjs"] }, { label: "build:plugin-sdk:dts", diff --git a/scripts/generate-qa-runner-catalog.ts b/scripts/generate-qa-runner-catalog.ts new file mode 100644 index 00000000000..a49201628bc --- /dev/null +++ b/scripts/generate-qa-runner-catalog.ts @@ -0,0 +1,35 @@ +#!/usr/bin/env node +import path from "node:path"; +import { writeBundledQaRunnerCatalog } from "../src/plugins/qa-runner-catalog.js"; + +const args = new Set(process.argv.slice(2)); +const checkOnly = args.has("--check"); +const writeMode = args.has("--write"); + +if (checkOnly === writeMode) { + console.error("Use exactly one of --check or --write."); + process.exit(1); +} + +const repoRoot = process.cwd(); +const result = await writeBundledQaRunnerCatalog({ + repoRoot, + check: checkOnly, +}); + +if (checkOnly) { + if (result.changed) { + console.error( + [ + "QA runner catalog drift detected.", + `Expected current: ${path.relative(repoRoot, result.jsonPath)}`, + "If this QA runner metadata change is intentional, run `pnpm qa-runners:gen` and commit the updated baseline file.", + "If not intentional, fix the bundled plugin metadata drift first.", + ].join("\n"), + ); + process.exit(1); + } + console.log(`OK ${path.relative(repoRoot, result.jsonPath)}`); +} else { + console.log(`Wrote ${path.relative(repoRoot, result.jsonPath)}`); +} diff --git a/scripts/lib/bundled-plugin-build-entries.mjs b/scripts/lib/bundled-plugin-build-entries.mjs index ded906f4a33..836170713d4 100644 --- a/scripts/lib/bundled-plugin-build-entries.mjs +++ b/scripts/lib/bundled-plugin-build-entries.mjs @@ -8,7 +8,7 @@ import { import { shouldBuildBundledCluster } from "./optional-bundled-clusters.mjs"; const TOP_LEVEL_PUBLIC_SURFACE_EXTENSIONS = new Set([".ts", ".js", ".mts", ".cts", ".mjs", ".cjs"]); -const NON_PACKAGED_BUNDLED_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab"]); +const NON_PACKAGED_BUNDLED_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab", "qa-matrix"]); const toPosixPath = (value) => value.replaceAll("\\", "/"); function readBundledPluginPackageJson(packageJsonPath) { diff --git a/scripts/lib/npm-update-compat-sidecars.mjs b/scripts/lib/npm-update-compat-sidecars.mjs deleted file mode 100644 index 228b120dec3..00000000000 --- a/scripts/lib/npm-update-compat-sidecars.mjs +++ /dev/null @@ -1,16 +0,0 @@ -export const NPM_UPDATE_COMPAT_SIDECARS = [ - { - path: "dist/extensions/qa-channel/runtime-api.js", - content: - "// Compatibility stub for older OpenClaw updaters. The QA channel implementation is not packaged.\nexport {};\n", - }, - { - path: "dist/extensions/qa-lab/runtime-api.js", - content: - "// Compatibility stub for older OpenClaw updaters. The QA lab implementation is not packaged.\nexport {};\n", - }, -]; - -export const NPM_UPDATE_COMPAT_SIDECAR_PATHS = new Set( - NPM_UPDATE_COMPAT_SIDECARS.map((entry) => entry.path), -); diff --git a/scripts/lib/plugin-sdk-entrypoints.json b/scripts/lib/plugin-sdk-entrypoints.json index b55448ef5f2..85274c6ccc3 100644 --- a/scripts/lib/plugin-sdk-entrypoints.json +++ b/scripts/lib/plugin-sdk-entrypoints.json @@ -179,6 +179,8 @@ "matrix-runtime-surface", "matrix-surface", "matrix-thread-bindings", + "qa-lab-runtime", + "qa-runner-runtime", "mattermost", "mattermost-policy", "memory-core", diff --git a/scripts/lib/qa-runner-catalog.json b/scripts/lib/qa-runner-catalog.json new file mode 100644 index 00000000000..06864acd78b --- /dev/null +++ b/scripts/lib/qa-runner-catalog.json @@ -0,0 +1,8 @@ +[ + { + "pluginId": "qa-matrix", + "commandName": "matrix", + "description": "Run the Docker-backed Matrix live QA lane against a disposable homeserver", + "npmSpec": "@openclaw/qa-matrix" + } +] diff --git a/scripts/openclaw-npm-postpublish-verify.ts b/scripts/openclaw-npm-postpublish-verify.ts index 7725319d51b..197d2e00abe 100644 --- a/scripts/openclaw-npm-postpublish-verify.ts +++ b/scripts/openclaw-npm-postpublish-verify.ts @@ -20,7 +20,6 @@ import { collectRootDistBundledRuntimeMirrors, collectRuntimeDependencySpecs, } from "./lib/bundled-plugin-root-runtime-mirrors.mjs"; -import { NPM_UPDATE_COMPAT_SIDECAR_PATHS } from "./lib/npm-update-compat-sidecars.mjs"; import { runInstalledWorkspaceBootstrapSmoke } from "./lib/workspace-bootstrap-smoke.mjs"; import { parseReleaseVersion, resolveNpmCommandInvocation } from "./openclaw-npm-release-check.ts"; @@ -44,13 +43,6 @@ type InstalledBundledExtensionManifestRecord = { const MAX_BUNDLED_EXTENSION_MANIFEST_BYTES = 1024 * 1024; const LEGACY_CONTEXT_ENGINE_UNRESOLVED_RUNTIME_MARKER = "Failed to load legacy context engine runtime."; -const NPM_UPDATE_COMPAT_EXTENSION_DIRS = new Set( - [...NPM_UPDATE_COMPAT_SIDECAR_PATHS].map((relativePath) => { - const pathParts = relativePath.split("/"); - pathParts.pop(); - return pathParts.join("/"); - }), -); export type PublishedInstallScenario = { name: string; @@ -183,20 +175,6 @@ function collectExpectedBundledExtensionPackageIds( return ids; } -function isNpmUpdateCompatOnlyExtensionDir(params: { - extensionId: string; - packageRoot: string; -}): boolean { - const relativeExtensionDir = `dist/extensions/${params.extensionId}`; - if (!NPM_UPDATE_COMPAT_EXTENSION_DIRS.has(relativeExtensionDir)) { - return false; - } - - return [...NPM_UPDATE_COMPAT_SIDECAR_PATHS] - .filter((relativePath) => relativePath.startsWith(`${relativeExtensionDir}/`)) - .every((relativePath) => existsSync(join(params.packageRoot, relativePath))); -} - function readBundledExtensionPackageJsons(packageRoot: string): { manifests: InstalledBundledExtensionManifestRecord[]; errors: string[]; @@ -218,9 +196,6 @@ function readBundledExtensionPackageJsons(packageRoot: string): { const extensionDirPath = join(extensionsDir, entry.name); const packageJsonPath = join(extensionsDir, entry.name, "package.json"); if (!existsSync(packageJsonPath)) { - if (isNpmUpdateCompatOnlyExtensionDir({ extensionId: entry.name, packageRoot })) { - continue; - } if (expectedPackageIds === null || expectedPackageIds.has(entry.name)) { errors.push(`installed bundled extension manifest missing: ${packageJsonPath}.`); } diff --git a/scripts/openclaw-npm-release-check.ts b/scripts/openclaw-npm-release-check.ts index 49ca6dff643..eb5c40adde0 100644 --- a/scripts/openclaw-npm-release-check.ts +++ b/scripts/openclaw-npm-release-check.ts @@ -9,7 +9,6 @@ import { resolveNpmDistTagMirrorAuth as resolveNpmDistTagMirrorAuthBase, parseReleaseVersion as parseReleaseVersionBase, } from "./lib/npm-publish-plan.mjs"; -import { NPM_UPDATE_COMPAT_SIDECAR_PATHS } from "./lib/npm-update-compat-sidecars.mjs"; import { WORKSPACE_TEMPLATE_PACK_PATHS } from "./lib/workspace-bootstrap-smoke.mjs"; type PackageJson = { @@ -465,9 +464,6 @@ function collectPackedTarballErrors(): string[] { export function collectForbiddenPackedPathErrors(paths: Iterable): string[] { const errors: string[] = []; for (const packedPath of paths) { - if (NPM_UPDATE_COMPAT_SIDECAR_PATHS.has(packedPath)) { - continue; - } const matchedRule = FORBIDDEN_PACKED_PATH_RULES.find((rule) => packedPath.startsWith(rule.prefix), ); diff --git a/scripts/write-npm-update-compat-sidecars.mjs b/scripts/write-npm-update-compat-sidecars.mjs deleted file mode 100644 index 1b65463df7a..00000000000 --- a/scripts/write-npm-update-compat-sidecars.mjs +++ /dev/null @@ -1,10 +0,0 @@ -#!/usr/bin/env node - -import fs from "node:fs"; -import path from "node:path"; -import { NPM_UPDATE_COMPAT_SIDECARS } from "./lib/npm-update-compat-sidecars.mjs"; - -for (const entry of NPM_UPDATE_COMPAT_SIDECARS) { - fs.mkdirSync(path.dirname(entry.path), { recursive: true }); - fs.writeFileSync(entry.path, entry.content, "utf8"); -} diff --git a/src/plugin-sdk/qa-lab-runtime.ts b/src/plugin-sdk/qa-lab-runtime.ts new file mode 100644 index 00000000000..6b003f3e1ca --- /dev/null +++ b/src/plugin-sdk/qa-lab-runtime.ts @@ -0,0 +1,39 @@ +import { loadBundledPluginPublicSurfaceModuleSync } from "./facade-runtime.js"; + +type QaLabRuntimeSurface = { + defaultQaRuntimeModelForMode: ( + mode: string, + options?: { + alternate?: boolean; + preferredLiveModel?: string; + }, + ) => string; + startQaLiveLaneGateway: (...args: unknown[]) => Promise; +}; + +function isMissingQaLabRuntimeError(error: unknown) { + return ( + error instanceof Error && + (error.message === "Unable to resolve bundled plugin public surface qa-lab/runtime-api.js" || + error.message.startsWith("Unable to open bundled plugin public surface ")) + ); +} + +export function loadQaLabRuntimeModule(): QaLabRuntimeSurface { + return loadBundledPluginPublicSurfaceModuleSync({ + dirName: "qa-lab", + artifactBasename: "runtime-api.js", + }); +} + +export function isQaLabRuntimeAvailable(): boolean { + try { + loadQaLabRuntimeModule(); + return true; + } catch (error) { + if (isMissingQaLabRuntimeError(error)) { + return false; + } + throw error; + } +} diff --git a/src/plugin-sdk/qa-runner-runtime.integration.test.ts b/src/plugin-sdk/qa-runner-runtime.integration.test.ts new file mode 100644 index 00000000000..1ad93b1e7b5 --- /dev/null +++ b/src/plugin-sdk/qa-runner-runtime.integration.test.ts @@ -0,0 +1,143 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { clearPluginDiscoveryCache } from "../plugins/discovery.js"; +import { clearPluginManifestRegistryCache } from "../plugins/manifest-registry.js"; +import { resetFacadeRuntimeStateForTest } from "./facade-runtime.js"; + +const ORIGINAL_ENV = { + OPENCLAW_DISABLE_BUNDLED_PLUGINS: process.env.OPENCLAW_DISABLE_BUNDLED_PLUGINS, + OPENCLAW_CONFIG_PATH: process.env.OPENCLAW_CONFIG_PATH, + OPENCLAW_DISABLE_PLUGIN_DISCOVERY_CACHE: process.env.OPENCLAW_DISABLE_PLUGIN_DISCOVERY_CACHE, + OPENCLAW_DISABLE_PLUGIN_MANIFEST_CACHE: process.env.OPENCLAW_DISABLE_PLUGIN_MANIFEST_CACHE, + OPENCLAW_PLUGIN_DISCOVERY_CACHE_MS: process.env.OPENCLAW_PLUGIN_DISCOVERY_CACHE_MS, + OPENCLAW_PLUGIN_MANIFEST_CACHE_MS: process.env.OPENCLAW_PLUGIN_MANIFEST_CACHE_MS, + OPENCLAW_TEST_FAST: process.env.OPENCLAW_TEST_FAST, +} as const; + +const tempDirs: string[] = []; + +function makeTempDir(prefix: string): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix)); + tempDirs.push(dir); + return dir; +} + +function resetQaRunnerRuntimeState() { + clearPluginDiscoveryCache(); + clearPluginManifestRegistryCache(); + resetFacadeRuntimeStateForTest(); +} + +describe("plugin-sdk qa-runner-runtime linked plugin smoke", () => { + beforeEach(() => { + resetQaRunnerRuntimeState(); + process.env.OPENCLAW_DISABLE_BUNDLED_PLUGINS = "1"; + process.env.OPENCLAW_DISABLE_PLUGIN_DISCOVERY_CACHE = "1"; + process.env.OPENCLAW_DISABLE_PLUGIN_MANIFEST_CACHE = "1"; + process.env.OPENCLAW_PLUGIN_DISCOVERY_CACHE_MS = "0"; + process.env.OPENCLAW_PLUGIN_MANIFEST_CACHE_MS = "0"; + process.env.OPENCLAW_TEST_FAST = "1"; + }); + + afterEach(() => { + resetQaRunnerRuntimeState(); + for (const dir of tempDirs.splice(0)) { + fs.rmSync(dir, { recursive: true, force: true }); + } + for (const [key, value] of Object.entries(ORIGINAL_ENV)) { + if (value === undefined) { + delete process.env[key]; + } else { + process.env[key] = value; + } + } + }); + + it("loads an activated qa runner from a linked plugin path", async () => { + const stateDir = makeTempDir("openclaw-qa-runner-state-"); + const pluginDir = path.join(stateDir, "extensions", "qa-linked"); + const configPath = path.join(stateDir, "openclaw.json"); + + fs.writeFileSync( + configPath, + JSON.stringify({ + plugins: {}, + }), + "utf8", + ); + process.env.OPENCLAW_CONFIG_PATH = configPath; + + fs.mkdirSync(pluginDir, { recursive: true }); + fs.writeFileSync( + path.join(pluginDir, "openclaw.plugin.json"), + JSON.stringify({ + id: "qa-linked", + qaRunners: [ + { + commandName: "linked", + description: "Run the linked QA lane", + }, + ], + configSchema: { + type: "object", + additionalProperties: false, + properties: {}, + }, + }), + "utf8", + ); + fs.writeFileSync( + path.join(pluginDir, "package.json"), + JSON.stringify({ + name: "@openclaw/qa-linked", + type: "module", + openclaw: { + extensions: ["./index.js"], + install: { + npmSpec: "@openclaw/qa-linked", + }, + }, + }), + "utf8", + ); + fs.writeFileSync(path.join(pluginDir, "index.js"), 'export default {};\n', "utf8"); + fs.writeFileSync( + path.join(pluginDir, "runtime-api.js"), + [ + "export const qaRunnerCliRegistrations = [", + " {", + ' commandName: "linked",', + " register() {}", + " }", + "];", + ].join("\n"), + "utf8", + ); + + const module = await import("./qa-runner-runtime.js"); + + expect(module.listQaRunnerCliContributions()).toEqual( + expect.arrayContaining([ + { + pluginId: "qa-linked", + commandName: "linked", + description: "Run the linked QA lane", + status: "available", + registration: { + commandName: "linked", + register: expect.any(Function), + }, + }, + { + pluginId: "qa-matrix", + commandName: "matrix", + description: "Run the Docker-backed Matrix live QA lane against a disposable homeserver", + status: "missing", + npmSpec: "@openclaw/qa-matrix", + }, + ]), + ); + }); +}); diff --git a/src/plugin-sdk/qa-runner-runtime.test.ts b/src/plugin-sdk/qa-runner-runtime.test.ts new file mode 100644 index 00000000000..73c3e448c5a --- /dev/null +++ b/src/plugin-sdk/qa-runner-runtime.test.ts @@ -0,0 +1,183 @@ +import type { Command } from "commander"; +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const loadPluginManifestRegistry = vi.hoisted(() => vi.fn()); +const tryLoadActivatedBundledPluginPublicSurfaceModuleSync = vi.hoisted(() => vi.fn()); +const listBundledQaRunnerCatalog = vi.hoisted(() => + vi.fn< + () => Array<{ + pluginId: string; + commandName: string; + description?: string; + npmSpec: string; + }> + >(() => []), +); + +vi.mock("../plugins/manifest-registry.js", () => ({ + loadPluginManifestRegistry, +})); + +vi.mock("../plugins/qa-runner-catalog.js", () => ({ + listBundledQaRunnerCatalog, +})); + +vi.mock("./facade-runtime.js", () => ({ + tryLoadActivatedBundledPluginPublicSurfaceModuleSync, +})); + +describe("plugin-sdk qa-runner-runtime", () => { + beforeEach(() => { + loadPluginManifestRegistry.mockReset().mockReturnValue({ + plugins: [], + diagnostics: [], + }); + listBundledQaRunnerCatalog.mockReset().mockReturnValue([]); + tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReset(); + }); + + it("stays cold until runner discovery is requested", async () => { + await import("./qa-runner-runtime.js"); + + expect(loadPluginManifestRegistry).not.toHaveBeenCalled(); + expect(tryLoadActivatedBundledPluginPublicSurfaceModuleSync).not.toHaveBeenCalled(); + }); + + it("returns activated runner registrations declared in plugin manifests", async () => { + const register = vi.fn((qa: Command) => qa); + loadPluginManifestRegistry.mockReturnValue({ + plugins: [ + { + id: "qa-matrix", + qaRunners: [ + { + commandName: "matrix", + description: "Run the Matrix live QA lane", + }, + ], + rootDir: "/tmp/qa-matrix", + }, + ], + diagnostics: [], + }); + tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReturnValue({ + qaRunnerCliRegistrations: [{ commandName: "matrix", register }], + }); + + const module = await import("./qa-runner-runtime.js"); + + expect(module.listQaRunnerCliContributions()).toEqual([ + { + pluginId: "qa-matrix", + commandName: "matrix", + description: "Run the Matrix live QA lane", + status: "available", + registration: { + commandName: "matrix", + register, + }, + }, + ]); + expect(tryLoadActivatedBundledPluginPublicSurfaceModuleSync).toHaveBeenCalledWith({ + dirName: "qa-matrix", + artifactBasename: "runtime-api.js", + }); + }); + + it("reports declared runners as blocked when the plugin is present but not activated", async () => { + loadPluginManifestRegistry.mockReturnValue({ + plugins: [ + { + id: "qa-matrix", + qaRunners: [{ commandName: "matrix" }], + rootDir: "/tmp/qa-matrix", + }, + ], + diagnostics: [], + }); + tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReturnValue(null); + + const module = await import("./qa-runner-runtime.js"); + + expect(module.listQaRunnerCliContributions()).toEqual([ + { + pluginId: "qa-matrix", + commandName: "matrix", + status: "blocked", + }, + ]); + }); + + it("reports missing optional runners from the generated catalog", async () => { + listBundledQaRunnerCatalog.mockReturnValue([ + { + pluginId: "qa-matrix", + commandName: "matrix", + description: "Run the Matrix live QA lane", + npmSpec: "@openclaw/qa-matrix", + }, + ]); + + const module = await import("./qa-runner-runtime.js"); + + expect(module.listQaRunnerCliContributions()).toEqual([ + { + pluginId: "qa-matrix", + commandName: "matrix", + description: "Run the Matrix live QA lane", + status: "missing", + npmSpec: "@openclaw/qa-matrix", + }, + ]); + }); + + it("fails fast when two plugins declare the same qa runner command", async () => { + loadPluginManifestRegistry.mockReturnValue({ + plugins: [ + { + id: "alpha", + qaRunners: [{ commandName: "matrix" }], + rootDir: "/tmp/alpha", + }, + { + id: "beta", + qaRunners: [{ commandName: "matrix" }], + rootDir: "/tmp/beta", + }, + ], + diagnostics: [], + }); + tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReturnValue(null); + + const module = await import("./qa-runner-runtime.js"); + + expect(() => module.listQaRunnerCliContributions()).toThrow( + 'QA runner command "matrix" declared by both "alpha" and "beta"', + ); + }); + + it("fails when runtime registrations include an undeclared command", async () => { + loadPluginManifestRegistry.mockReturnValue({ + plugins: [ + { + id: "qa-matrix", + qaRunners: [{ commandName: "matrix" }], + rootDir: "/tmp/qa-matrix", + }, + ], + diagnostics: [], + }); + tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReturnValue({ + qaRunnerCliRegistrations: [ + { commandName: "matrix", register: vi.fn() }, + { commandName: "extra", register: vi.fn() }, + ], + }); + + const module = await import("./qa-runner-runtime.js"); + + expect(() => module.listQaRunnerCliContributions()).toThrow( + 'QA runner plugin "qa-matrix" exported "extra" from runtime-api.js but did not declare it in openclaw.plugin.json', + ); + }); +}); diff --git a/src/plugin-sdk/qa-runner-runtime.ts b/src/plugin-sdk/qa-runner-runtime.ts new file mode 100644 index 00000000000..381fa3eb27f --- /dev/null +++ b/src/plugin-sdk/qa-runner-runtime.ts @@ -0,0 +1,161 @@ +import type { Command } from "commander"; +import type { PluginManifestRecord } from "../plugins/manifest-registry.js"; +import { loadPluginManifestRegistry } from "../plugins/manifest-registry.js"; +import { listBundledQaRunnerCatalog } from "../plugins/qa-runner-catalog.js"; +import { tryLoadActivatedBundledPluginPublicSurfaceModuleSync } from "./facade-runtime.js"; + +export type QaRunnerCliRegistration = { + commandName: string; + register(qa: Command): void; +}; + +type QaRunnerRuntimeSurface = { + qaRunnerCliRegistrations?: readonly QaRunnerCliRegistration[]; +}; + +export type QaRunnerCliContribution = + | { + pluginId: string; + commandName: string; + description?: string; + status: "available"; + registration: QaRunnerCliRegistration; + } + | { + pluginId: string; + commandName: string; + description?: string; + status: "blocked"; + } + | { + pluginId: string; + commandName: string; + description?: string; + status: "missing"; + npmSpec: string; + }; + +function listDeclaredQaRunnerPlugins(): Array< + PluginManifestRecord & { + qaRunners: NonNullable; + } +> { + return loadPluginManifestRegistry({ cache: true }) + .plugins.filter( + ( + plugin, + ): plugin is PluginManifestRecord & { + qaRunners: NonNullable; + } => Array.isArray(plugin.qaRunners) && plugin.qaRunners.length > 0, + ) + .toSorted((left, right) => { + const idCompare = left.id.localeCompare(right.id); + if (idCompare !== 0) { + return idCompare; + } + return left.rootDir.localeCompare(right.rootDir); + }); +} + +function indexRuntimeRegistrations( + pluginId: string, + surface: QaRunnerRuntimeSurface, +): ReadonlyMap { + const registrations = surface.qaRunnerCliRegistrations ?? []; + const registrationByCommandName = new Map(); + for (const registration of registrations) { + if (!registration?.commandName || typeof registration.register !== "function") { + throw new Error(`QA runner plugin "${pluginId}" exported an invalid CLI registration`); + } + if (registrationByCommandName.has(registration.commandName)) { + throw new Error( + `QA runner plugin "${pluginId}" exported duplicate CLI registration "${registration.commandName}"`, + ); + } + registrationByCommandName.set(registration.commandName, registration); + } + return registrationByCommandName; +} + +function buildKnownQaRunnerCatalog(): readonly QaRunnerCliContribution[] { + const knownRunners = listBundledQaRunnerCatalog(); + const seenCommandNames = new Map(); + return knownRunners.map((runner) => { + const previousOwner = seenCommandNames.get(runner.commandName); + if (previousOwner) { + throw new Error( + `QA runner command "${runner.commandName}" declared by both "${previousOwner}" and "${runner.pluginId}"`, + ); + } + seenCommandNames.set(runner.commandName, runner.pluginId); + return { + pluginId: runner.pluginId, + commandName: runner.commandName, + ...(runner.description ? { description: runner.description } : {}), + status: "missing" as const, + npmSpec: runner.npmSpec, + }; + }); +} + +export function listQaRunnerCliContributions(): readonly QaRunnerCliContribution[] { + const contributions = new Map(); + + for (const runner of buildKnownQaRunnerCatalog()) { + contributions.set(runner.commandName, runner); + } + + for (const plugin of listDeclaredQaRunnerPlugins()) { + const runtimeSurface = + tryLoadActivatedBundledPluginPublicSurfaceModuleSync({ + dirName: plugin.id, + artifactBasename: "runtime-api.js", + }); + const runtimeRegistrationByCommandName = runtimeSurface + ? indexRuntimeRegistrations(plugin.id, runtimeSurface) + : null; + const declaredCommandNames = new Set(plugin.qaRunners.map((runner) => runner.commandName)); + + for (const runner of plugin.qaRunners) { + const previous = contributions.get(runner.commandName); + if (previous && previous.pluginId !== plugin.id) { + throw new Error( + `QA runner command "${runner.commandName}" declared by both "${previous.pluginId}" and "${plugin.id}"`, + ); + } + + const registration = runtimeRegistrationByCommandName?.get(runner.commandName); + if (!runtimeSurface) { + contributions.set(runner.commandName, { + pluginId: plugin.id, + commandName: runner.commandName, + ...(runner.description ? { description: runner.description } : {}), + status: "blocked", + }); + continue; + } + if (!registration) { + throw new Error( + `QA runner plugin "${plugin.id}" declared "${runner.commandName}" in openclaw.plugin.json but did not export a matching CLI registration`, + ); + } + contributions.set(runner.commandName, { + pluginId: plugin.id, + commandName: runner.commandName, + ...(runner.description ? { description: runner.description } : {}), + status: "available", + registration, + }); + } + + for (const commandName of runtimeRegistrationByCommandName?.keys() ?? []) { + if (!declaredCommandNames.has(commandName)) { + throw new Error( + `QA runner plugin "${plugin.id}" exported "${commandName}" from runtime-api.js but did not declare it in openclaw.plugin.json`, + ); + } + } + } + + return [...contributions.values()]; +} diff --git a/src/plugins/bundled-plugin-metadata.test.ts b/src/plugins/bundled-plugin-metadata.test.ts index 939dac4c3d1..9bbbb92693a 100644 --- a/src/plugins/bundled-plugin-metadata.test.ts +++ b/src/plugins/bundled-plugin-metadata.test.ts @@ -131,11 +131,12 @@ describe("bundled plugin metadata", () => { }, ); - it("excludes private QA sidecars from the packaged runtime sidecar baseline", () => { + it("excludes non-packaged QA sidecars from the packaged runtime sidecar baseline", () => { expect(BUNDLED_RUNTIME_SIDECAR_PATHS).not.toContain( "dist/extensions/qa-channel/runtime-api.js", ); expect(BUNDLED_RUNTIME_SIDECAR_PATHS).not.toContain("dist/extensions/qa-lab/runtime-api.js"); + expect(BUNDLED_RUNTIME_SIDECAR_PATHS).not.toContain("dist/extensions/qa-matrix/runtime-api.js"); }); it("captures setup-entry metadata for bundled channel plugins", () => { diff --git a/src/plugins/install.test.ts b/src/plugins/install.test.ts index b32d09c0f9f..e92b22372e4 100644 --- a/src/plugins/install.test.ts +++ b/src/plugins/install.test.ts @@ -1427,6 +1427,21 @@ describe("installPluginFromArchive", () => { ).toBe(true); }); + it("does not flag the real qa-matrix plugin as dangerous install code", async () => { + const pluginDir = path.resolve(process.cwd(), "extensions", "qa-matrix"); + + const scanResult = await installSecurityScan.scanPackageInstallSource({ + extensions: ["./index.ts"], + logger: { warn: vi.fn() }, + packageDir: pluginDir, + pluginId: "qa-matrix", + packageName: "@openclaw/qa-matrix", + manifestId: "qa-matrix", + }); + + expect(scanResult?.blocked).toBeUndefined(); + }); + it("keeps blocked dependency package checks active when forced unsafe install is set", async () => { const { pluginDir, extensionsDir } = setupPluginInstallDirs(); diff --git a/src/plugins/manifest-registry.test.ts b/src/plugins/manifest-registry.test.ts index a2497b94860..2b1d2c94af4 100644 --- a/src/plugins/manifest-registry.test.ts +++ b/src/plugins/manifest-registry.test.ts @@ -499,6 +499,33 @@ describe("loadPluginManifestRegistry", () => { }); }); + it("preserves qa runner descriptors from plugin manifests", () => { + const dir = makeTempDir(); + writeManifest(dir, { + id: "qa-matrix", + qaRunners: [ + { + commandName: "matrix", + description: "Run the Matrix live QA lane", + }, + ], + configSchema: { type: "object" }, + }); + + const registry = loadSingleCandidateRegistry({ + idHint: "qa-matrix", + rootDir: dir, + origin: "bundled", + }); + + expect(registry.plugins[0]?.qaRunners).toEqual([ + { + commandName: "matrix", + description: "Run the Matrix live QA lane", + }, + ]); + }); + it("preserves channel config metadata from plugin manifests", () => { const dir = makeTempDir(); writeManifest(dir, { diff --git a/src/plugins/manifest-registry.ts b/src/plugins/manifest-registry.ts index e46aa114154..c3bca9edd3c 100644 --- a/src/plugins/manifest-registry.ts +++ b/src/plugins/manifest-registry.ts @@ -34,6 +34,7 @@ import { type PluginManifestChannelConfig, type PluginManifestContracts, type PluginManifestModelSupport, + type PluginManifestQaRunner, type PluginManifestSetup, } from "./manifest.js"; import { checkMinHostVersion } from "./min-host-version.js"; @@ -92,6 +93,7 @@ export type PluginManifestRecord = { providerAuthChoices?: PluginManifest["providerAuthChoices"]; activation?: PluginManifestActivation; setup?: PluginManifestSetup; + qaRunners?: PluginManifestQaRunner[]; skills: string[]; settingsFiles?: string[]; hooks: string[]; @@ -333,6 +335,7 @@ function buildRecord(params: { providerAuthChoices: params.manifest.providerAuthChoices, activation: params.manifest.activation, setup: params.manifest.setup, + qaRunners: params.manifest.qaRunners, skills: params.manifest.skills ?? [], settingsFiles: [], hooks: [], diff --git a/src/plugins/manifest.ts b/src/plugins/manifest.ts index 3dc39b0d941..f05eb7e4e06 100644 --- a/src/plugins/manifest.ts +++ b/src/plugins/manifest.ts @@ -80,6 +80,13 @@ export type PluginManifestSetup = { requiresRuntime?: boolean; }; +export type PluginManifestQaRunner = { + /** Subcommand mounted beneath `openclaw qa`, for example `matrix`. */ + commandName: string; + /** Optional user-facing help text for fallback host stubs. */ + description?: string; +}; + export type PluginManifestConfigLiteral = string | number | boolean | null; export type PluginManifestDangerousConfigFlag = { @@ -174,6 +181,8 @@ export type PluginManifest = { activation?: PluginManifestActivation; /** Cheap setup/onboarding metadata exposed before plugin runtime loads. */ setup?: PluginManifestSetup; + /** Cheap QA runner metadata exposed before plugin runtime loads. */ + qaRunners?: PluginManifestQaRunner[]; skills?: string[]; name?: string; description?: string; @@ -484,6 +493,28 @@ function normalizeManifestSetup(value: unknown): PluginManifestSetup | undefined return Object.keys(setup).length > 0 ? setup : undefined; } +function normalizeManifestQaRunners(value: unknown): PluginManifestQaRunner[] | undefined { + if (!Array.isArray(value)) { + return undefined; + } + const normalized: PluginManifestQaRunner[] = []; + for (const entry of value) { + if (!isRecord(entry)) { + continue; + } + const commandName = normalizeOptionalString(entry.commandName) ?? ""; + if (!commandName) { + continue; + } + const description = normalizeOptionalString(entry.description) ?? ""; + normalized.push({ + commandName, + ...(description ? { description } : {}), + }); + } + return normalized.length > 0 ? normalized : undefined; +} + function normalizeProviderAuthChoices( value: unknown, ): PluginManifestProviderAuthChoice[] | undefined { @@ -673,6 +704,7 @@ export function loadPluginManifest( const providerAuthChoices = normalizeProviderAuthChoices(raw.providerAuthChoices); const activation = normalizeManifestActivation(raw.activation); const setup = normalizeManifestSetup(raw.setup); + const qaRunners = normalizeManifestQaRunners(raw.qaRunners); const skills = normalizeTrimmedStringList(raw.skills); const contracts = normalizeManifestContracts(raw.contracts); const configContracts = normalizeManifestConfigContracts(raw.configContracts); @@ -706,6 +738,7 @@ export function loadPluginManifest( providerAuthChoices, activation, setup, + qaRunners, skills, name, description, diff --git a/src/plugins/qa-runner-catalog.ts b/src/plugins/qa-runner-catalog.ts new file mode 100644 index 00000000000..dd0bba3a460 --- /dev/null +++ b/src/plugins/qa-runner-catalog.ts @@ -0,0 +1,74 @@ +import fs from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { listBundledPluginMetadata } from "./bundled-plugin-metadata.js"; + +export type QaRunnerCatalogEntry = { + pluginId: string; + commandName: string; + description?: string; + npmSpec: string; +}; + +const QA_RUNNER_CATALOG_JSON_PATH = fileURLToPath( + new URL("../../scripts/lib/qa-runner-catalog.json", import.meta.url), +); + +export function listBundledQaRunnerCatalog(): readonly QaRunnerCatalogEntry[] { + if (!fs.existsSync(QA_RUNNER_CATALOG_JSON_PATH)) { + return []; + } + return JSON.parse(fs.readFileSync(QA_RUNNER_CATALOG_JSON_PATH, "utf8")) as QaRunnerCatalogEntry[]; +} + +export function collectBundledQaRunnerCatalog(params?: { + rootDir?: string; +}): readonly QaRunnerCatalogEntry[] { + const catalog: QaRunnerCatalogEntry[] = []; + const seenCommandNames = new Map(); + + for (const entry of listBundledPluginMetadata({ + rootDir: params?.rootDir, + includeChannelConfigs: false, + })) { + const qaRunners = entry.manifest.qaRunners ?? []; + const npmSpec = entry.packageManifest?.install?.npmSpec?.trim() || entry.packageName?.trim(); + if (!npmSpec) { + continue; + } + for (const runner of qaRunners) { + const previousOwner = seenCommandNames.get(runner.commandName); + if (previousOwner) { + throw new Error( + `QA runner command "${runner.commandName}" declared by both "${previousOwner}" and "${entry.manifest.id}"`, + ); + } + seenCommandNames.set(runner.commandName, entry.manifest.id); + catalog.push({ + pluginId: entry.manifest.id, + commandName: runner.commandName, + ...(runner.description ? { description: runner.description } : {}), + npmSpec, + }); + } + } + + return catalog.toSorted((left, right) => left.commandName.localeCompare(right.commandName)); +} + +export async function writeBundledQaRunnerCatalog(params: { + repoRoot: string; + check: boolean; +}): Promise<{ changed: boolean; jsonPath: string }> { + const jsonPath = path.join(params.repoRoot, "scripts", "lib", "qa-runner-catalog.json"); + const expectedJson = `${JSON.stringify(collectBundledQaRunnerCatalog({ rootDir: params.repoRoot }), null, 2)}\n`; + const currentJson = fs.existsSync(jsonPath) ? fs.readFileSync(jsonPath, "utf8") : ""; + const changed = currentJson !== expectedJson; + + if (!params.check && changed) { + fs.mkdirSync(path.dirname(jsonPath), { recursive: true }); + fs.writeFileSync(jsonPath, expectedJson, "utf8"); + } + + return { changed, jsonPath }; +} diff --git a/src/plugins/runtime-sidecar-paths-baseline.ts b/src/plugins/runtime-sidecar-paths-baseline.ts index 57872afe90c..d1daf2233b0 100644 --- a/src/plugins/runtime-sidecar-paths-baseline.ts +++ b/src/plugins/runtime-sidecar-paths-baseline.ts @@ -2,7 +2,7 @@ import fs from "node:fs"; import path from "node:path"; import { listBundledPluginMetadata } from "./bundled-plugin-metadata.js"; -const NON_PACKAGED_RUNTIME_SIDECAR_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab"]); +const NON_PACKAGED_RUNTIME_SIDECAR_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab", "qa-matrix"]); function buildBundledDistArtifactPath(dirName: string, artifact: string): string { return ["dist", "extensions", dirName, artifact].join("/"); diff --git a/test/openclaw-npm-postpublish-verify.test.ts b/test/openclaw-npm-postpublish-verify.test.ts index 176448210bb..51522df0e9f 100644 --- a/test/openclaw-npm-postpublish-verify.test.ts +++ b/test/openclaw-npm-postpublish-verify.test.ts @@ -301,7 +301,7 @@ describe("collectInstalledMirroredRootDependencyManifestErrors", () => { } }); - it("allows npm update compatibility sidecar directories without package.json", () => { + it("rejects private qa sidecar directories that are missing package.json", () => { const packageRoot = makeInstalledPackageRoot(); try { @@ -322,7 +322,10 @@ describe("collectInstalledMirroredRootDependencyManifestErrors", () => { "utf8", ); - expect(collectInstalledMirroredRootDependencyManifestErrors(packageRoot)).toEqual([]); + expect(collectInstalledMirroredRootDependencyManifestErrors(packageRoot)).toEqual([ + `installed bundled extension manifest missing: ${join(packageRoot, "dist/extensions/qa-channel/package.json")}.`, + `installed bundled extension manifest missing: ${join(packageRoot, "dist/extensions/qa-lab/package.json")}.`, + ]); } finally { rmSync(packageRoot, { recursive: true, force: true }); } diff --git a/test/openclaw-npm-release-check.test.ts b/test/openclaw-npm-release-check.test.ts index 910b95cf7dd..b64b683c46e 100644 --- a/test/openclaw-npm-release-check.test.ts +++ b/test/openclaw-npm-release-check.test.ts @@ -325,6 +325,8 @@ describe("collectForbiddenPackedPathErrors", () => { ]), ).toEqual([ 'npm package must not include private QA channel artifact "dist/extensions/qa-channel/package.json".', + 'npm package must not include private QA channel artifact "dist/extensions/qa-channel/runtime-api.js".', + 'npm package must not include private QA lab artifact "dist/extensions/qa-lab/runtime-api.js".', 'npm package must not include private QA lab artifact "dist/extensions/qa-lab/src/cli.js".', ]); }); diff --git a/test/scripts/bundled-plugin-build-entries.test.ts b/test/scripts/bundled-plugin-build-entries.test.ts index 03b015b7862..f60bbe23ce0 100644 --- a/test/scripts/bundled-plugin-build-entries.test.ts +++ b/test/scripts/bundled-plugin-build-entries.test.ts @@ -82,6 +82,9 @@ describe("bundled plugin build entries", () => { expect(artifacts.some((artifact) => artifact.startsWith("dist/extensions/qa-lab/"))).toBe( false, ); + expect(artifacts.some((artifact) => artifact.startsWith("dist/extensions/qa-matrix/"))).toBe( + false, + ); }); it("keeps bundled channel secret contracts on packed top-level sidecars", () => {