github-mirrors/openclaw
1
0
Fork 0
mirror of https://fastgit.cc/github.com/openclaw/openclaw synced 2026-04-23 02:13:29 +08:00
Code Issues Packages Projects Releases Wiki Activity

refactor(qa): split Matrix QA into optional plugin (#66723)

Browse Source 
Merged via squash.

Prepared head SHA: 27241bd089
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com>
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com>
Reviewed-by: @gumadeiras
This commit is contained in:
Gustavo Madeira Santana
2026-04-14 16:28:57 -04:00
committed by GitHub
parent 3425823dfb
commit 82a2db71e8
69 changed files with 2026 additions and 229 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG.md
View File

@@ -8,6 +8,7 @@ Docs: https://docs.openclaw.ai
- Docs/showcase: add a scannable hero, complete section jump links, and a responsive video grid for community examples. (#48493) Thanks @jchopard69.
- Agents/local models: add `agents.defaults.localModelMode: "lean"` to drop heavyweight default tools like `browser`, `cron`, and `message`, reducing prompt size for weaker local-model setups without changing the normal path. Thanks @ImLukeF.
- QA/Matrix: split Matrix live QA into a source-linked `qa-matrix` runner and keep repo-private `qa-*` surfaces out of packaged and published builds. (#66723) Thanks @gumadeiras.
### Fixes

6
docs/cli/plugins.md
View File

@@ -197,6 +197,12 @@ Use `--link` to avoid copying a local directory (adds to `plugins.load.paths`):
openclaw plugins install -l ./my-plugin
```
Repo QA example (source-linked dev surface; not shipped in packaged installs):
```bash
openclaw plugins install -l ./extensions/qa-matrix
```
`--force` is not supported with `--link` because linked installs reuse the
source path instead of copying over a managed install target.

31
docs/help/testing.md
View File

@@ -67,9 +67,13 @@ These commands sit beside the main test suites when you need QA-lab realism:
- Starts the Docker-backed QA site for operator-style QA work.
- `pnpm openclaw qa matrix`
- Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver.
- This QA host is repo/dev-only today. Packaged OpenClaw installs do not ship
`qa-lab`, so they do not expose `openclaw qa`.
- Repo checkouts can link the in-tree plugin directly:
`openclaw plugins install -l ./extensions/qa-matrix`.
- Provisions three temporary Matrix users (`driver`, `sut`, `observer`) plus one private room, then starts a QA gateway child with the real Matrix plugin as the SUT transport.
- Uses the pinned stable Tuwunel image `ghcr.io/matrix-construct/tuwunel:v1.5.1` by default. Override with `OPENCLAW_QA_MATRIX_TUWUNEL_IMAGE` when you need to test a different image.
- Matrix currently supports only `--credential-source env` because the lane provisions disposable users locally.
- Matrix does not expose shared credential-source flags because the lane provisions disposable users locally.
- Writes a Matrix QA report, summary, and observed-events artifact under `.artifacts/qa-e2e/...`.
- `pnpm openclaw qa telegram`
- Runs the Telegram live QA lane against a real private group using the driver and SUT bot tokens from env.
@@ -170,11 +174,12 @@ Adding a channel to the markdown QA system requires exactly two things:
1. A transport adapter for the channel.
2. A scenario pack that exercises the channel contract.
Do not add a channel-specific QA runner when the shared `qa-lab` runner can
Do not add a new top-level QA command root when the shared `qa-lab` host can
own the flow.
`qa-lab` owns the shared mechanics:
`qa-lab` owns the shared host mechanics:
- the `openclaw qa` command root
- suite startup and teardown
- worker concurrency
- artifact writing
@@ -182,8 +187,9 @@ own the flow.
- scenario execution
- compatibility aliases for older `qa-channel` scenarios
The channel adapter owns the transport contract:
Runner plugins own the transport contract:
- how `openclaw qa <runner>` is mounted beneath the shared `qa` root
- how the gateway is configured for that transport
- how readiness is checked
- how inbound events are injected
@@ -194,17 +200,20 @@ The channel adapter owns the transport contract:
The minimum adoption bar for a new channel is:
1. Implement the transport adapter on the shared `qa-lab` seam.
2. Register the adapter in the transport registry.
3. Keep transport-specific mechanics inside the adapter or the channel harness.
4. Author or adapt markdown scenarios under `qa/scenarios/`.
5. Use the generic scenario helpers for new scenarios.
6. Keep existing compatibility aliases working unless the repo is doing an intentional migration.
1. Keep `qa-lab` as the owner of the shared `qa` root.
2. Implement the transport runner on the shared `qa-lab` host seam.
3. Keep transport-specific mechanics inside the runner plugin or channel harness.
4. Mount the runner as `openclaw qa <runner>` instead of registering a competing root command.
Runner plugins should declare `qaRunners` in `openclaw.plugin.json` and export a matching `qaRunnerCliRegistrations` array from `runtime-api.ts`.
Keep `runtime-api.ts` light; lazy CLI and runner execution should stay behind separate entrypoints.
5. Author or adapt markdown scenarios under `qa/scenarios/`.
6. Use the generic scenario helpers for new scenarios.
7. Keep existing compatibility aliases working unless the repo is doing an intentional migration.
The decision rule is strict:
- If behavior can be expressed once in `qa-lab`, put it in `qa-lab`.
- If behavior depends on one channel transport, keep it in that adapter or plugin harness.
- If behavior depends on one channel transport, keep it in that runner plugin or plugin harness.
- If a scenario needs a new capability that more than one channel can use, add a generic helper instead of a channel-specific branch in `suite.ts`.
- If a behavior is only meaningful for one transport, keep the scenario transport-specific and make that explicit in the scenario contract.

26
docs/plugins/manifest.md
View File

@@ -56,6 +56,8 @@ Use it for:
plugin before runtime loads
- static capability ownership snapshots used for bundled compat wiring and
contract coverage
- cheap QA runner metadata that the shared `openclaw qa` host can inspect
before plugin runtime loads
- channel-specific config metadata that should merge into catalog and validation
surfaces without loading runtime
- config UI hints
@@ -158,6 +160,7 @@ Those belong in your plugin code and `package.json`.
| `providerAuthChoices` | No | `object[]` | Cheap auth-choice metadata for onboarding pickers, preferred-provider resolution, and simple CLI flag wiring. |
| `activation` | No | `object` | Cheap activation hints for provider, command, channel, route, and capability-triggered loading. Metadata only; plugin runtime still owns actual behavior. |
| `setup` | No | `object` | Cheap setup/onboarding descriptors that discovery and setup surfaces can inspect without loading plugin runtime. |
| `qaRunners` | No | `object[]` | Cheap QA runner descriptors used by the shared `openclaw qa` host before plugin runtime loads. |
| `contracts` | No | `object` | Static bundled capability snapshot for speech, realtime transcription, realtime voice, media-understanding, image-generation, music-generation, video-generation, web-fetch, web search, and tool ownership. |
| `channelConfigs` | No | `Record<string, object>` | Manifest-owned channel config metadata merged into discovery and validation surfaces before runtime loads. |
| `skills` | No | `string[]` | Skill directories to load, relative to the plugin root. |
@@ -219,6 +222,29 @@ uses this metadata for diagnostics without importing plugin runtime code.
Use `activation` when the plugin can cheaply declare which control-plane events
should activate it later.
## qaRunners reference
Use `qaRunners` when a plugin contributes one or more transport runners beneath
the shared `openclaw qa` root. Keep this metadata cheap and static; the plugin
runtime still owns actual CLI registration through a lightweight
`runtime-api.ts` surface that exports `qaRunnerCliRegistrations`.
```json
{
"qaRunners": [
{
"commandName": "matrix",
"description": "Run the Docker-backed Matrix live QA lane against a disposable homeserver"
}
]
}
```
| Field | Required | Type | What it means |
| ------------- | -------- | -------- | ------------------------------------------------------------------ |
| `commandName` | Yes | `string` | Subcommand mounted beneath `openclaw qa`, for example `matrix`. |
| `description` | No | `string` | Fallback help text used when the shared host needs a stub command. |
This block is metadata only. It does not register runtime behavior, and it does
not replace `register(...)`, `setupEntry`, or other runtime/plugin entrypoints.
Current consumers use it as a narrowing hint before broader plugin loading, so

1
extensions/qa-lab/runtime-api.ts
View File

@@ -1 +1,2 @@
export * from "./src/runtime-api.js";
export { startQaLiveLaneGateway } from "./src/live-transports/shared/live-gateway.runtime.js";

53
extensions/qa-lab/src/cli.runtime.test.ts
View File

@@ -8,7 +8,6 @@ const {
runQaSuiteFromRuntime,
runQaCharacterEval,
runQaMultipass,
runMatrixQaLive,
runTelegramQaLive,
startQaLabServer,
writeQaDockerHarnessFiles,
@@ -20,7 +19,6 @@ const {
runQaSuiteFromRuntime: vi.fn(),
runQaCharacterEval: vi.fn(),
runQaMultipass: vi.fn(),
runMatrixQaLive: vi.fn(),
runTelegramQaLive: vi.fn(),
startQaLabServer: vi.fn(),
writeQaDockerHarnessFiles: vi.fn(),
@@ -52,10 +50,6 @@ vi.mock("./multipass.runtime.js", () => ({
runQaMultipass,
}));
vi.mock("./live-transports/matrix/matrix-live.runtime.js", () => ({
runMatrixQaLive,
}));
vi.mock("./live-transports/telegram/telegram-live.runtime.js", () => ({
runTelegramQaLive,
}));
@@ -88,7 +82,6 @@ import {
runQaParityReportCommand,
runQaSuiteCommand,
} from "./cli.runtime.js";
import { runQaMatrixCommand } from "./live-transports/matrix/cli.runtime.js";
import { runQaTelegramCommand } from "./live-transports/telegram/cli.runtime.js";
describe("qa cli runtime", () => {
@@ -100,7 +93,6 @@ describe("qa cli runtime", () => {
runQaCharacterEval.mockReset();
runQaManualLane.mockReset();
runQaMultipass.mockReset();
runMatrixQaLive.mockReset();
runTelegramQaLive.mockReset();
startQaLabServer.mockReset();
writeQaDockerHarnessFiles.mockReset();
@@ -139,13 +131,6 @@ describe("qa cli runtime", () => {
vmName: "openclaw-qa-test",
scenarioIds: ["channel-chat-baseline"],
});
runMatrixQaLive.mockResolvedValue({
outputDir: "/tmp/matrix",
reportPath: "/tmp/matrix/report.md",
summaryPath: "/tmp/matrix/summary.json",
observedEventsPath: "/tmp/matrix/observed.json",
scenarios: [],
});
runTelegramQaLive.mockResolvedValue({
outputDir: "/tmp/telegram",
reportPath: "/tmp/telegram/report.md",
@@ -226,30 +211,6 @@ describe("qa cli runtime", () => {
});
});
it("resolves matrix qa repo-root-relative paths before dispatching", async () => {
await runQaMatrixCommand({
repoRoot: "/tmp/openclaw-repo",
outputDir: ".artifacts/qa/matrix",
providerMode: "live-frontier",
primaryModel: "openai/gpt-5.4",
alternateModel: "openai/gpt-5.4",
fastMode: true,
scenarioIds: ["matrix-thread-follow-up"],
sutAccountId: "sut-live",
});
expect(runMatrixQaLive).toHaveBeenCalledWith({
repoRoot: path.resolve("/tmp/openclaw-repo"),
outputDir: path.resolve("/tmp/openclaw-repo", ".artifacts/qa/matrix"),
providerMode: "live-frontier",
primaryModel: "openai/gpt-5.4",
alternateModel: "openai/gpt-5.4",
fastMode: true,
scenarioIds: ["matrix-thread-follow-up"],
sutAccountId: "sut-live",
});
});
it("rejects output dirs that escape the repo root", () => {
expect(() => resolveRepoRelativeOutputDir("/tmp/openclaw-repo", "../outside")).toThrow(
"--output-dir must stay within the repo root.",
@@ -273,20 +234,6 @@ describe("qa cli runtime", () => {
);
});
it("defaults matrix qa runs onto the live provider lane", async () => {
await runQaMatrixCommand({
repoRoot: "/tmp/openclaw-repo",
scenarioIds: ["matrix-thread-follow-up"],
});
expect(runMatrixQaLive).toHaveBeenCalledWith(
expect.objectContaining({
repoRoot: path.resolve("/tmp/openclaw-repo"),
providerMode: "live-frontier",
}),
);
});
it("normalizes legacy live-openai suite runs onto the frontier provider mode", async () => {
await runQaSuiteCommand({
repoRoot: "/tmp/openclaw-repo",

150
extensions/qa-lab/src/cli.test.ts
View File

@@ -1,22 +1,76 @@
import { Command } from "commander";
import type { QaRunnerCliContribution } from "openclaw/plugin-sdk/qa-runner-runtime";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
const TEST_QA_RUNNER = {
pluginId: "qa-runner-test",
commandName: "runner-test",
description: "Run the test live QA lane",
npmSpec: "@openclaw/qa-runner-test",
} as const;
function createAvailableQaRunnerContribution() {
return {
pluginId: TEST_QA_RUNNER.pluginId,
commandName: TEST_QA_RUNNER.commandName,
status: "available" as const,
registration: {
commandName: TEST_QA_RUNNER.commandName,
register: vi.fn((qa: Command) => {
qa.command(TEST_QA_RUNNER.commandName).action(() => undefined);
}),
},
} satisfies QaRunnerCliContribution;
}
function createMissingQaRunnerContribution(): QaRunnerCliContribution {
return {
pluginId: TEST_QA_RUNNER.pluginId,
commandName: TEST_QA_RUNNER.commandName,
description: TEST_QA_RUNNER.description,
status: "missing",
npmSpec: TEST_QA_RUNNER.npmSpec,
};
}
function createBlockedQaRunnerContribution(): QaRunnerCliContribution {
return {
pluginId: TEST_QA_RUNNER.pluginId,
commandName: TEST_QA_RUNNER.commandName,
description: TEST_QA_RUNNER.description,
status: "blocked",
};
}
function createConflictingQaRunnerContribution(commandName: string): QaRunnerCliContribution {
return {
pluginId: TEST_QA_RUNNER.pluginId,
commandName,
description: TEST_QA_RUNNER.description,
status: "blocked",
};
}
const {
runQaCredentialsAddCommand,
runQaCredentialsListCommand,
runQaCredentialsRemoveCommand,
runQaMatrixCommand,
runQaTelegramCommand,
} = vi.hoisted(() => ({
runQaCredentialsAddCommand: vi.fn(),
runQaCredentialsListCommand: vi.fn(),
runQaCredentialsRemoveCommand: vi.fn(),
runQaMatrixCommand: vi.fn(),
runQaTelegramCommand: vi.fn(),
}));
vi.mock("./live-transports/matrix/cli.runtime.js", () => ({
runQaMatrixCommand,
const { listQaRunnerCliContributions } = vi.hoisted(() => ({
listQaRunnerCliContributions: vi.fn<() => QaRunnerCliContribution[]>(() => [
createAvailableQaRunnerContribution(),
]),
}));
vi.mock("openclaw/plugin-sdk/qa-runner-runtime", () => ({
listQaRunnerCliContributions,
}));
vi.mock("./live-transports/telegram/cli.runtime.js", () => ({
@@ -36,63 +90,71 @@ describe("qa cli registration", () => {
beforeEach(() => {
program = new Command();
registerQaLabCli(program);
runQaCredentialsAddCommand.mockReset();
runQaCredentialsListCommand.mockReset();
runQaCredentialsRemoveCommand.mockReset();
runQaMatrixCommand.mockReset();
runQaTelegramCommand.mockReset();
listQaRunnerCliContributions
.mockReset()
.mockReturnValue([createAvailableQaRunnerContribution()]);
registerQaLabCli(program);
});
afterEach(() => {
vi.clearAllMocks();
});
it("registers the matrix and telegram live transport subcommands", () => {
it("registers discovered and built-in live transport subcommands", () => {
const qa = program.commands.find((command) => command.name() === "qa");
expect(qa).toBeDefined();
expect(qa?.commands.map((command) => command.name())).toEqual(
expect.arrayContaining(["matrix", "telegram", "credentials"]),
expect.arrayContaining([TEST_QA_RUNNER.commandName, "telegram", "credentials"]),
);
});
it("routes matrix CLI flags into the lane runtime", async () => {
await program.parseAsync([
"node",
"openclaw",
"qa",
"matrix",
"--repo-root",
"/tmp/openclaw-repo",
"--output-dir",
".artifacts/qa/matrix",
"--provider-mode",
"mock-openai",
"--model",
"mock-openai/gpt-5.4",
"--alt-model",
"mock-openai/gpt-5.4-alt",
"--scenario",
"matrix-thread-follow-up",
"--scenario",
"matrix-thread-isolation",
"--fast",
"--sut-account",
"sut-live",
]);
it("delegates discovered qa runner registration through the generic host seam", () => {
const [{ registration }] = listQaRunnerCliContributions.mock.results[0]?.value;
expect(registration.register).toHaveBeenCalledTimes(1);
});
expect(runQaMatrixCommand).toHaveBeenCalledWith({
repoRoot: "/tmp/openclaw-repo",
outputDir: ".artifacts/qa/matrix",
providerMode: "mock-openai",
primaryModel: "mock-openai/gpt-5.4",
alternateModel: "mock-openai/gpt-5.4-alt",
fastMode: true,
scenarioIds: ["matrix-thread-follow-up", "matrix-thread-isolation"],
sutAccountId: "sut-live",
credentialSource: undefined,
credentialRole: undefined,
});
it("keeps Telegram credential flags on the shared host CLI", () => {
const qa = program.commands.find((command) => command.name() === "qa");
const telegram = qa?.commands.find((command) => command.name() === "telegram");
const optionNames = telegram?.options.map((option) => option.long) ?? [];
expect(optionNames).toEqual(
expect.arrayContaining(["--credential-source", "--credential-role"]),
);
});
it("shows an install hint when a discovered runner plugin is unavailable", async () => {
listQaRunnerCliContributions.mockReset().mockReturnValue([createMissingQaRunnerContribution()]);
const missingProgram = new Command();
registerQaLabCli(missingProgram);
await expect(
missingProgram.parseAsync(["node", "openclaw", "qa", TEST_QA_RUNNER.commandName]),
).rejects.toThrow(`openclaw plugins install ${TEST_QA_RUNNER.npmSpec}`);
});
it("shows an enable hint when a discovered runner plugin is installed but blocked", async () => {
listQaRunnerCliContributions.mockReset().mockReturnValue([createBlockedQaRunnerContribution()]);
const blockedProgram = new Command();
registerQaLabCli(blockedProgram);
await expect(
blockedProgram.parseAsync(["node", "openclaw", "qa", TEST_QA_RUNNER.commandName]),
).rejects.toThrow(`Enable or allow plugin "${TEST_QA_RUNNER.pluginId}"`);
});
it("rejects discovered runners that collide with built-in qa subcommands", () => {
listQaRunnerCliContributions
.mockReset()
.mockReturnValue([createConflictingQaRunnerContribution("manual")]);
expect(() => registerQaLabCli(new Command())).toThrow(
'QA runner command "manual" conflicts with an existing qa subcommand',
);
});
it("routes telegram CLI defaults into the lane runtime", async () => {

17
extensions/qa-lab/src/cli.ts
View File

@@ -1,6 +1,6 @@
import type { Command } from "commander";
import { collectString } from "./cli-options.js";
import { LIVE_TRANSPORT_QA_CLI_REGISTRATIONS } from "./live-transports/cli.js";
import { listLiveTransportQaCliRegistrations } from "./live-transports/cli.js";
import type { QaProviderModeInput } from "./run-config.js";
import { hasQaScenarioPack } from "./scenario-catalog.js";
@@ -183,6 +183,12 @@ export function isQaLabCliAvailable(): boolean {
return hasQaScenarioPack();
}
function assertNoQaSubcommandCollision(qa: Command, commandName: string) {
if (qa.commands.some((command) => command.name() === commandName)) {
throw new Error(`QA runner command "${commandName}" conflicts with an existing qa subcommand`);
}
}
export function registerQaLabCli(program: Command) {
const qa = program
.command("qa")
@@ -284,10 +290,6 @@ export function registerQaLabCli(program: Command) {
},
);
for (const lane of LIVE_TRANSPORT_QA_CLI_REGISTRATIONS) {
lane.register(qa);
}
qa.command("character-eval")
.description("Run the character QA scenario across live models and write a judged report")
.option("--repo-root <path>", "Repository root to target when running from a neutral cwd")
@@ -579,4 +581,9 @@ export function registerQaLabCli(program: Command) {
.action(async (opts: { host?: string; port?: number }) => {
await runQaMockOpenAi(opts);
});
for (const lane of listLiveTransportQaCliRegistrations()) {
assertNoQaSubcommandCollision(qa, lane.commandName);
lane.register(qa);
}
}

74
extensions/qa-lab/src/live-transports/cli.ts
View File

@@ -1,8 +1,78 @@
import { matrixQaCliRegistration } from "./matrix/cli.js";
import { listQaRunnerCliContributions } from "openclaw/plugin-sdk/qa-runner-runtime";
import type { LiveTransportQaCliRegistration } from "./shared/live-transport-cli.js";
import { telegramQaCliRegistration } from "./telegram/cli.js";
function createMissingQaRunnerCliRegistration(params: {
commandName: string;
description: string;
npmSpec: string;
}): LiveTransportQaCliRegistration {
return {
commandName: params.commandName,
register(qa) {
qa.command(params.commandName)
.description(params.description)
.action(() => {
throw new Error(
`QA runner "${params.commandName}" not installed. Install it with "openclaw plugins install ${params.npmSpec}".`,
);
});
},
};
}
function createBlockedQaRunnerCliRegistration(params: {
commandName: string;
description?: string;
pluginId: string;
}): LiveTransportQaCliRegistration {
return {
commandName: params.commandName,
register(qa) {
qa.command(params.commandName)
.description(params.description ?? `Run the ${params.commandName} live QA lane`)
.action(() => {
throw new Error(
`QA runner "${params.commandName}" is installed but not active. Enable or allow plugin "${params.pluginId}" in your OpenClaw config, then try again.`,
);
});
},
};
}
function createQaRunnerCliRegistration(
runner: ReturnType<typeof listQaRunnerCliContributions>[number],
): LiveTransportQaCliRegistration {
if (runner.status === "available") {
return runner.registration;
}
if (runner.status === "blocked") {
return createBlockedQaRunnerCliRegistration({
commandName: runner.commandName,
description: runner.description,
pluginId: runner.pluginId,
});
}
return createMissingQaRunnerCliRegistration({
commandName: runner.commandName,
description:
runner.description ??
`Run the ${runner.commandName} live QA lane (install ${runner.npmSpec} first)`,
npmSpec: runner.npmSpec,
});
}
export const LIVE_TRANSPORT_QA_CLI_REGISTRATIONS: readonly LiveTransportQaCliRegistration[] = [
telegramQaCliRegistration,
matrixQaCliRegistration,
];
export function listLiveTransportQaCliRegistrations(): readonly LiveTransportQaCliRegistration[] {
const liveRegistrations = [...LIVE_TRANSPORT_QA_CLI_REGISTRATIONS];
const discoveredRunners = listQaRunnerCliContributions();
for (const runner of discoveredRunners) {
liveRegistrations.push(createQaRunnerCliRegistration(runner));
}
return liveRegistrations;
}

36
extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts
View File

@@ -33,6 +33,11 @@ export type LiveTransportQaCliRegistration = {
register(qa: Command): void;
};
export type LiveTransportQaCredentialCliOptions = {
sourceDescription?: string;
roleDescription?: string;
};
export function createLazyCliRuntimeLoader<T>(load: () => Promise<T>) {
let promise: Promise<T> | null = null;
return async () => {
@@ -61,13 +66,14 @@ export function mapLiveTransportQaCommanderOptions(
export function registerLiveTransportQaCli(params: {
qa: Command;
commandName: string;
credentialOptions?: LiveTransportQaCredentialCliOptions;
description: string;
outputDirHelp: string;
scenarioHelp: string;
sutAccountHelp: string;
run: (opts: LiveTransportQaCommandOptions) => Promise<void>;
}) {
params.qa
const command = params.qa
.command(params.commandName)
.description(params.description)
.option("--repo-root <path>", "Repository root to target when running from a neutral cwd")
@@ -81,22 +87,27 @@ export function registerLiveTransportQaCli(params: {
.option("--alt-model <ref>", "Alternate provider/model ref")
.option("--scenario <id>", params.scenarioHelp, collectString, [])
.option("--fast", "Enable provider fast mode where supported", false)
.option("--sut-account <id>", params.sutAccountHelp, "sut")
.option(
.option("--sut-account <id>", params.sutAccountHelp, "sut");
if (params.credentialOptions) {
command.option(
"--credential-source <source>",
"Credential source for live lanes: env or convex (default: env)",
)
.option(
"--credential-role <role>",
"Credential role for convex auth: maintainer or ci (default: maintainer)",
)
.action(async (opts: LiveTransportQaCommanderOptions) => {
await params.run(mapLiveTransportQaCommanderOptions(opts));
});
params.credentialOptions.sourceDescription ??
"Credential source for live lanes: env or convex (default: env)",
);
if (params.credentialOptions.roleDescription) {
command.option("--credential-role <role>", params.credentialOptions.roleDescription);
}
}
command.action(async (opts: LiveTransportQaCommanderOptions) => {
await params.run(mapLiveTransportQaCommanderOptions(opts));
});
}
export function createLiveTransportQaCliRegistration(params: {
commandName: string;
credentialOptions?: LiveTransportQaCredentialCliOptions;
description: string;
outputDirHelp: string;
scenarioHelp: string;
@@ -109,6 +120,7 @@ export function createLiveTransportQaCliRegistration(params: {
registerLiveTransportQaCli({
qa,
commandName: params.commandName,
credentialOptions: params.credentialOptions,
description: params.description,
outputDirHelp: params.outputDirHelp,
scenarioHelp: params.scenarioHelp,

4
extensions/qa-lab/src/live-transports/telegram/cli.ts
View File

@@ -20,6 +20,10 @@ async function runQaTelegram(opts: LiveTransportQaCommandOptions) {
export const telegramQaCliRegistration: LiveTransportQaCliRegistration =
createLiveTransportQaCliRegistration({
commandName: "telegram",
credentialOptions: {
sourceDescription: "Credential source for Telegram QA: env or convex (default: env)",
roleDescription: "Credential role for convex auth: maintainer or ci (default: maintainer)",
},
description: "Run the manual Telegram live QA lane against a private bot-to-bot group harness",
outputDirHelp: "Telegram QA artifact directory",
scenarioHelp: "Run only the named Telegram QA scenario (repeatable)",

1
extensions/qa-lab/src/runtime-api.ts
View File

@@ -3,6 +3,7 @@ export type { OpenClawConfig } from "openclaw/plugin-sdk/config-runtime";
export { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
export { callGatewayFromCli } from "openclaw/plugin-sdk/browser-node-runtime";
export type { PluginRuntime } from "openclaw/plugin-sdk/runtime-store";
export { defaultQaRuntimeModelForMode } from "./model-selection.runtime.js";
export {
buildQaTarget,
createQaBusThread,

2
extensions/qa-lab/src/self-check.ts
View File

@@ -81,7 +81,7 @@ export async function runQaSelfCheckAgainstState(params: {
timeline,
notes: params.notes ?? [
"Vertical slice: qa-channel + qa-lab bus + private debugger surface.",
"Docker orchestration, matrix runs, and auto-fix loops remain follow-up work.",
"Docker orchestration, additional QA runners, and auto-fix loops remain follow-up work.",
],
});

1
extensions/qa-matrix/cli.runtime.ts Normal file
View File

@@ -0,0 +1 @@
export { runQaMatrixCommand } from "./src/cli.runtime.js";

1
extensions/qa-matrix/cli.ts Normal file
View File

@@ -0,0 +1 @@
export { qaRunnerCliRegistrations, registerMatrixQaCli } from "./src/cli.js";

8
extensions/qa-matrix/index.ts Normal file
View File

@@ -0,0 +1,8 @@
import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
export default definePluginEntry({
id: "qa-matrix",
name: "QA Matrix",
description: "Matrix QA transport runner and substrate",
register() {},
});

16
extensions/qa-matrix/openclaw.plugin.json Normal file
View File

@@ -0,0 +1,16 @@
{
"id": "qa-matrix",
"name": "QA Matrix",
"description": "Matrix QA transport runner and substrate",
"qaRunners": [
{
"commandName": "matrix",
"description": "Run the Docker-backed Matrix live QA lane against a disposable homeserver"
}
],
"configSchema": {
"type": "object",
"additionalProperties": false,
"properties": {}
}
}

34
extensions/qa-matrix/package.json Normal file
View File

@@ -0,0 +1,34 @@
{
"name": "@openclaw/qa-matrix",
"version": "2026.4.12",
"description": "OpenClaw Matrix QA runner plugin",
"type": "module",
"devDependencies": {
"@openclaw/plugin-sdk": "workspace:*",
"openclaw": "workspace:*"
},
"peerDependencies": {
"openclaw": ">=2026.4.12"
},
"peerDependenciesMeta": {
"openclaw": {
"optional": true
}
},
"openclaw": {
"extensions": [
"./index.ts"
],
"install": {
"npmSpec": "@openclaw/qa-matrix",
"defaultChoice": "npm",
"minHostVersion": ">=2026.4.12"
},
"compat": {
"pluginApi": ">=2026.4.12"
},
"build": {
"openclawVersion": "2026.4.12"
}
}
}

1
extensions/qa-matrix/runtime-api.ts Normal file
View File

@@ -0,0 +1 @@
export { qaRunnerCliRegistrations } from "./cli.js";

1
extensions/qa-matrix/runtime.ts Normal file
View File

@@ -0,0 +1 @@
export { runMatrixQaLive } from "./src/runners/contract/runtime.js";

4
extensions/qa-matrix/src/cli-options.ts Normal file
View File

@@ -0,0 +1,4 @@
export function collectString(value: string, previous: string[]) {
const trimmed = value.trim();
return trimmed ? [...previous, trimmed] : previous;
}

16
extensions/qa-matrix/src/cli-paths.ts Normal file
View File

@@ -0,0 +1,16 @@
import path from "node:path";
export function resolveRepoRelativeOutputDir(repoRoot: string, outputDir?: string) {
if (!outputDir) {
return undefined;
}
if (path.isAbsolute(outputDir)) {
throw new Error("--output-dir must be a relative path inside the repo root.");
}
const resolved = path.resolve(repoRoot, outputDir);
const relative = path.relative(repoRoot, resolved);
if (relative.startsWith("..") || path.isAbsolute(relative)) {
throw new Error("--output-dir must stay within the repo root.");
}
return resolved;
}

2
extensions/qa-lab/src/live-transports/matrix/cli.runtime.test.ts → extensions/qa-matrix/src/cli.runtime.test.ts
View File

@@ -2,7 +2,7 @@ import { describe, expect, it, vi } from "vitest";
const runMatrixQaLive = vi.hoisted(() => vi.fn());
vi.mock("./matrix-live.runtime.js", () => ({
vi.mock("./runners/contract/runtime.js", () => ({
runMatrixQaLive,
}));

6
extensions/qa-lab/src/live-transports/matrix/cli.runtime.ts → extensions/qa-matrix/src/cli.runtime.ts
View File

@@ -1,9 +1,9 @@
import type { LiveTransportQaCommandOptions } from "../shared/live-transport-cli.js";
import { runMatrixQaLive } from "./runners/contract/runtime.js";
import type { LiveTransportQaCommandOptions } from "./shared/live-transport-cli.js";
import {
printLiveTransportQaArtifacts,
resolveLiveTransportQaRunOptions,
} from "../shared/live-transport-cli.runtime.js";
import { runMatrixQaLive } from "./matrix-live.runtime.js";
} from "./shared/live-transport-cli.runtime.js";
export async function runQaMatrixCommand(opts: LiveTransportQaCommandOptions) {
const runOptions = resolveLiveTransportQaRunOptions(opts);

29
extensions/qa-matrix/src/cli.test.ts Normal file
View File

@@ -0,0 +1,29 @@
import { Command } from "commander";
import { describe, expect, it } from "vitest";
import { matrixQaCliRegistration } from "./cli.js";
describe("matrix qa cli registration", () => {
it("keeps disposable Matrix lane flags focused", () => {
const qa = new Command();
matrixQaCliRegistration.register(qa);
const matrix = qa.commands.find((command) => command.name() === "matrix");
const optionNames = matrix?.options.map((option) => option.long) ?? [];
expect(optionNames).toEqual(
expect.arrayContaining([
"--repo-root",
"--output-dir",
"--provider-mode",
"--model",
"--alt-model",
"--scenario",
"--fast",
"--sut-account",
]),
);
expect(optionNames).not.toContain("--credential-source");
expect(optionNames).not.toContain("--credential-role");
});
});

4
extensions/qa-lab/src/live-transports/matrix/cli.ts → extensions/qa-matrix/src/cli.ts
View File

@@ -4,7 +4,7 @@ import {
createLiveTransportQaCliRegistration,
type LiveTransportQaCliRegistration,
type LiveTransportQaCommandOptions,
} from "../shared/live-transport-cli.js";
} from "./shared/live-transport-cli.js";
type MatrixQaCliRuntime = typeof import("./cli.runtime.js");
@@ -27,6 +27,8 @@ export const matrixQaCliRegistration: LiveTransportQaCliRegistration =
run: runQaMatrix,
});
export const qaRunnerCliRegistrations = [matrixQaCliRegistration] as const;
export function registerMatrixQaCli(qa: Command) {
matrixQaCliRegistration.register(qa);
}

274
extensions/qa-matrix/src/docker-runtime.ts Normal file
View File

@@ -0,0 +1,274 @@
import { createServer } from "node:net";
import { runExec } from "openclaw/plugin-sdk/process-runtime";
import { fetchWithSsrFGuard } from "openclaw/plugin-sdk/ssrf-runtime";
export type RunCommand = (
command: string,
args: string[],
cwd: string,
) => Promise<{ stdout: string; stderr: string }>;
export type FetchLike = (input: string) => Promise<{ ok: boolean }>;
export async function fetchHealthUrl(url: string): Promise<{ ok: boolean }> {
const { response, release } = await fetchWithSsrFGuard({
url,
init: {
signal: AbortSignal.timeout(2_000),
},
policy: { allowPrivateNetwork: true },
auditContext: "qa-matrix-docker-health-check",
});
try {
return { ok: response.ok };
} finally {
await release();
}
}
export function describeError(error: unknown) {
if (error instanceof Error) {
return error.message;
}
if (typeof error === "string") {
return error;
}
return JSON.stringify(error);
}
async function isPortFree(port: number) {
return await new Promise<boolean>((resolve) => {
const server = createServer();
server.once("error", () => resolve(false));
server.listen(port, "127.0.0.1", () => {
server.close(() => resolve(true));
});
});
}
async function findFreePort() {
return await new Promise<number>((resolve, reject) => {
const server = createServer();
server.once("error", reject);
server.listen(0, () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close();
reject(new Error("failed to find free port"));
return;
}
server.close((error) => {
if (error) {
reject(error);
return;
}
resolve(address.port);
});
});
});
}
export async function resolveHostPort(preferredPort: number, pinned: boolean) {
if (pinned || (await isPortFree(preferredPort))) {
return preferredPort;
}
return await findFreePort();
}
function trimCommandOutput(output: string) {
const trimmed = output.trim();
if (!trimmed) {
return "";
}
const lines = trimmed.split("\n");
return lines.length <= 120 ? trimmed : lines.slice(-120).join("\n");
}
export async function execCommand(command: string, args: string[], cwd: string) {
try {
return await runExec(command, args, { cwd, maxBuffer: 10 * 1024 * 1024 });
} catch (error) {
const failedProcess = error as Error & { stdout?: string; stderr?: string };
const renderedStdout = trimCommandOutput(failedProcess.stdout ?? "");
const renderedStderr = trimCommandOutput(failedProcess.stderr ?? "");
throw new Error(
[
`Command failed: ${[command, ...args].join(" ")}`,
renderedStderr ? `stderr:\n${renderedStderr}` : "",
renderedStdout ? `stdout:\n${renderedStdout}` : "",
]
.filter(Boolean)
.join("\n\n"),
{ cause: error },
);
}
}
export async function waitForHealth(
url: string,
deps: {
label?: string;
composeFile?: string;
fetchImpl: FetchLike;
sleepImpl: (ms: number) => Promise<unknown>;
timeoutMs?: number;
pollMs?: number;
},
) {
const timeoutMs = deps.timeoutMs ?? 360_000;
const pollMs = deps.pollMs ?? 1_000;
const startMs = Date.now();
const deadline = startMs + timeoutMs;
let lastError: unknown = null;
while (Date.now() < deadline) {
try {
const response = await deps.fetchImpl(url);
if (response.ok) {
return;
}
lastError = new Error(`Health check returned non-OK for ${url}`);
} catch (error) {
lastError = error;
}
await deps.sleepImpl(pollMs);
}
const elapsedSec = Math.round((Date.now() - startMs) / 1000);
const service = deps.label ?? url;
const lines = [
`${service} did not become healthy within ${elapsedSec}s (limit ${Math.round(timeoutMs / 1000)}s).`,
lastError ? `Last error: ${describeError(lastError)}` : "",
`Hint: check container logs with \`docker compose -f ${deps.composeFile ?? "<compose-file>"} logs\` and verify the port is not already in use.`,
];
throw new Error(lines.filter(Boolean).join("\n"));
}
async function isHealthy(url: string, fetchImpl: FetchLike) {
try {
const response = await fetchImpl(url);
return response.ok;
} catch {
return false;
}
}
function normalizeDockerServiceStatus(row?: { Health?: string; State?: string }) {
const health = row?.Health?.trim();
if (health) {
return health;
}
const state = row?.State?.trim();
if (state) {
return state;
}
return "unknown";
}
function parseDockerComposePsRows(stdout: string) {
const trimmed = stdout.trim();
if (!trimmed) {
return [] as Array<{ Health?: string; State?: string }>;
}
try {
const parsed = JSON.parse(trimmed) as
| Array<{ Health?: string; State?: string }>
| { Health?: string; State?: string };
if (Array.isArray(parsed)) {
return parsed;
}
return [parsed];
} catch {
return trimmed
.split("\n")
.map((line) => line.trim())
.filter(Boolean)
.map((line) => JSON.parse(line) as { Health?: string; State?: string });
}
}
export async function waitForDockerServiceHealth(
service: string,
composeFile: string,
repoRoot: string,
runCommand: RunCommand,
sleepImpl: (ms: number) => Promise<unknown>,
timeoutMs = 360_000,
pollMs = 1_000,
) {
const startMs = Date.now();
const deadline = startMs + timeoutMs;
let lastStatus = "unknown";
while (Date.now() < deadline) {
try {
const { stdout } = await runCommand(
"docker",
["compose", "-f", composeFile, "ps", "--format", "json", service],
repoRoot,
);
const rows = parseDockerComposePsRows(stdout);
const row = rows[0];
lastStatus = normalizeDockerServiceStatus(row);
if (lastStatus === "healthy" || lastStatus === "running") {
return;
}
} catch (error) {
lastStatus = describeError(error);
}
await sleepImpl(pollMs);
}
const elapsedSec = Math.round((Date.now() - startMs) / 1000);
throw new Error(
[
`${service} did not become healthy within ${elapsedSec}s (limit ${Math.round(timeoutMs / 1000)}s).`,
`Last status: ${lastStatus}`,
`Hint: check container logs with \`docker compose -f ${composeFile} logs ${service}\`.`,
].join("\n"),
);
}
export async function resolveComposeServiceUrl(
service: string,
port: number,
composeFile: string,
repoRoot: string,
runCommand: RunCommand,
fetchImpl?: FetchLike,
) {
const { stdout: containerStdout } = await runCommand(
"docker",
["compose", "-f", composeFile, "ps", "-q", service],
repoRoot,
);
const containerId = containerStdout.trim();
if (!containerId) {
return null;
}
const { stdout: ipStdout } = await runCommand(
"docker",
[
"inspect",
"--format",
"{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}",
containerId,
],
repoRoot,
);
const ip = ipStdout.trim();
if (!ip) {
return null;
}
const baseUrl = `http://${ip}:${port}/`;
if (!fetchImpl) {
return baseUrl;
}
return (await isHealthy(`${baseUrl}healthz`, fetchImpl)) ? baseUrl : null;
}
export const __testing = {
fetchHealthUrl,
normalizeDockerServiceStatus,
};

100
extensions/qa-matrix/src/report.ts Normal file
View File

@@ -0,0 +1,100 @@
export type QaReportCheck = {
name: string;
status: "pass" | "fail" | "skip";
details?: string;
};
export type QaReportScenario = {
name: string;
status: "pass" | "fail" | "skip";
details?: string;
steps?: QaReportCheck[];
};
function pushDetailsBlock(lines: string[], label: string, details: string, indent = "") {
if (!details.includes("\n")) {
lines.push(`${indent}- ${label}: ${details}`);
return;
}
lines.push(`${indent}- ${label}:`);
lines.push("", "```text", details, "```");
}
export function renderQaMarkdownReport(params: {
title: string;
startedAt: Date;
finishedAt: Date;
checks?: QaReportCheck[];
scenarios?: QaReportScenario[];
timeline?: string[];
notes?: string[];
}) {
const checks = params.checks ?? [];
const scenarios = params.scenarios ?? [];
const passCount =
checks.filter((check) => check.status === "pass").length +
scenarios.filter((scenario) => scenario.status === "pass").length;
const failCount =
checks.filter((check) => check.status === "fail").length +
scenarios.filter((scenario) => scenario.status === "fail").length;
const lines = [
`# ${params.title}`,
"",
`- Started: ${params.startedAt.toISOString()}`,
`- Finished: ${params.finishedAt.toISOString()}`,
`- Duration ms: ${params.finishedAt.getTime() - params.startedAt.getTime()}`,
`- Passed: ${passCount}`,
`- Failed: ${failCount}`,
"",
];
if (checks.length > 0) {
lines.push("## Checks", "");
for (const check of checks) {
lines.push(`- [${check.status === "pass" ? "x" : " "}] ${check.name}`);
if (check.details) {
pushDetailsBlock(lines, "Details", check.details, " ");
}
}
}
if (scenarios.length > 0) {
lines.push("", "## Scenarios", "");
for (const scenario of scenarios) {
lines.push(`### ${scenario.name}`);
lines.push("");
lines.push(`- Status: ${scenario.status}`);
if (scenario.details) {
pushDetailsBlock(lines, "Details", scenario.details);
}
if (scenario.steps?.length) {
lines.push("- Steps:");
for (const step of scenario.steps) {
lines.push(` - [${step.status === "pass" ? "x" : " "}] ${step.name}`);
if (step.details) {
pushDetailsBlock(lines, "Details", step.details, " ");
}
}
}
lines.push("");
}
}
if (params.timeline && params.timeline.length > 0) {
lines.push("## Timeline", "");
for (const item of params.timeline) {
lines.push(`- ${item}`);
}
}
if (params.notes && params.notes.length > 0) {
lines.push("", "## Notes", "");
for (const note of params.notes) {
lines.push(`- ${note}`);
}
}
lines.push("");
return lines.join("\n");
}

9
extensions/qa-matrix/src/run-config.ts Normal file
View File

@@ -0,0 +1,9 @@
export type QaProviderMode = "mock-openai" | "live-frontier";
export type QaProviderModeInput = QaProviderMode | "live-openai";
export function normalizeQaProviderMode(input: unknown): QaProviderMode {
if (input === "mock-openai") {
return "mock-openai";
}
return "live-frontier";
}

51
extensions/qa-matrix/src/runners/contract/model-selection.test.ts Normal file
View File

@@ -0,0 +1,51 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
const loadQaLabRuntimeModule = vi.hoisted(() => vi.fn());
const defaultQaRuntimeModelForMode = vi.hoisted(() => vi.fn());
vi.mock("openclaw/plugin-sdk/qa-lab-runtime", () => ({
loadQaLabRuntimeModule,
}));
describe("matrix qa model selection", () => {
beforeEach(() => {
defaultQaRuntimeModelForMode.mockReset().mockImplementation((mode, options) =>
options?.alternate ? `${mode}:alt` : `${mode}:primary`,
);
loadQaLabRuntimeModule.mockReset().mockReturnValue({
defaultQaRuntimeModelForMode,
});
});
it("delegates default model selection through qa-lab runtime defaults", async () => {
const { resolveMatrixQaModels } = await import("./model-selection.js");
expect(resolveMatrixQaModels({ providerMode: "live-openai" })).toEqual({
providerMode: "live-frontier",
primaryModel: "live-frontier:primary",
alternateModel: "live-frontier:alt",
});
expect(defaultQaRuntimeModelForMode).toHaveBeenNthCalledWith(1, "live-frontier");
expect(defaultQaRuntimeModelForMode).toHaveBeenNthCalledWith(2, "live-frontier", {
alternate: true,
});
});
it("preserves explicit model overrides", async () => {
const { resolveMatrixQaModels } = await import("./model-selection.js");
expect(
resolveMatrixQaModels({
providerMode: "mock-openai",
primaryModel: "custom-primary",
alternateModel: "custom-alt",
}),
).toEqual({
providerMode: "mock-openai",
primaryModel: "custom-primary",
alternateModel: "custom-alt",
});
expect(loadQaLabRuntimeModule).not.toHaveBeenCalled();
expect(defaultQaRuntimeModelForMode).not.toHaveBeenCalled();
});
});

33
extensions/qa-matrix/src/runners/contract/model-selection.ts Normal file
View File

@@ -0,0 +1,33 @@
import { loadQaLabRuntimeModule } from "openclaw/plugin-sdk/qa-lab-runtime";
import { normalizeQaProviderMode, type QaProviderModeInput } from "../../run-config.js";
export type ResolvedMatrixQaModels = {
providerMode: ReturnType<typeof normalizeQaProviderMode>;
primaryModel: string;
alternateModel: string;
};
export function resolveMatrixQaModels(params: {
providerMode?: QaProviderModeInput;
primaryModel?: string;
alternateModel?: string;
}): ResolvedMatrixQaModels {
const providerMode = normalizeQaProviderMode(params.providerMode ?? "live-frontier");
const primaryModel = params.primaryModel?.trim();
const alternateModel = params.alternateModel?.trim();
if (primaryModel && alternateModel) {
return {
providerMode,
primaryModel,
alternateModel,
};
}
const qaLabRuntime = loadQaLabRuntimeModule();
return {
providerMode,
primaryModel: primaryModel || qaLabRuntime.defaultQaRuntimeModelForMode(providerMode),
alternateModel:
alternateModel || qaLabRuntime.defaultQaRuntimeModelForMode(providerMode, { alternate: true }),
};
}

2
extensions/qa-lab/src/live-transports/matrix/matrix-live.runtime.test.ts → extensions/qa-matrix/src/runners/contract/runtime.test.ts
View File

@@ -1,6 +1,6 @@
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-runtime";
import { afterEach, describe, expect, it, vi } from "vitest";
import { __testing as liveTesting } from "./matrix-live.runtime.js";
import { __testing as liveTesting } from "./runtime.js";
afterEach(() => {
vi.useRealTimers();

69
extensions/qa-lab/src/live-transports/matrix/matrix-live.runtime.ts → extensions/qa-matrix/src/runners/contract/runtime.ts
View File

@@ -4,22 +4,20 @@ import path from "node:path";
import { setTimeout as sleep } from "node:timers/promises";
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-runtime";
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
import { startQaGatewayChild } from "../../gateway-child.js";
import { loadQaLabRuntimeModule } from "openclaw/plugin-sdk/qa-lab-runtime";
import type { QaReportCheck } from "../../report.js";
import { renderQaMarkdownReport } from "../../report.js";
import { type QaProviderModeInput } from "../../run-config.js";
import {
defaultQaModelForMode,
normalizeQaProviderMode,
type QaProviderModeInput,
} from "../../run-config.js";
import { startQaLiveLaneGateway } from "../shared/live-gateway.runtime.js";
import { appendLiveLaneIssue, buildLiveLaneArtifactsError } from "../shared/live-lane-helpers.js";
appendLiveLaneIssue,
buildLiveLaneArtifactsError,
} from "../../shared/live-lane-helpers.js";
import {
provisionMatrixQaRoom,
type MatrixQaObservedEvent,
type MatrixQaProvisionResult,
} from "./matrix-driver-client.js";
import { startMatrixQaHarness } from "./matrix-harness.runtime.js";
} from "../../substrate/client.js";
import { startMatrixQaHarness } from "../../substrate/harness.runtime.js";
import {
MATRIX_QA_SCENARIOS,
buildMatrixReplyDetails,
@@ -28,7 +26,22 @@ import {
runMatrixQaScenario,
type MatrixQaCanaryArtifact,
type MatrixQaScenarioArtifacts,
} from "./matrix-live-scenarios.js";
} from "./scenarios.js";
import { resolveMatrixQaModels } from "./model-selection.js";
type MatrixQaGatewayChild = {
call(
method: string,
params: Record<string, unknown>,
options?: { timeoutMs?: number },
): Promise<unknown>;
restart(): Promise<void>;
};
type MatrixQaLiveLaneGatewayHarness = {
gateway: MatrixQaGatewayChild;
stop(): Promise<void>;
};
type MatrixQaScenarioResult = {
artifacts?: MatrixQaScenarioArtifacts;
@@ -214,7 +227,7 @@ function isMatrixAccountReady(entry?: {
}
async function waitForMatrixChannelReady(
gateway: Awaited<ReturnType<typeof startQaGatewayChild>>,
gateway: MatrixQaGatewayChild,
accountId: string,
opts?: {
pollMs?: number;
@@ -255,6 +268,27 @@ async function waitForMatrixChannelReady(
throw new Error(`matrix account "${accountId}" did not become ready`);
}
async function startMatrixQaLiveLaneGateway(params: {
repoRoot: string;
transport: {
requiredPluginIds: readonly string[];
createGatewayConfig: (params: {
baseUrl: string;
}) => Pick<OpenClawConfig, "channels" | "messages">;
};
transportBaseUrl: string;
providerMode: "mock-openai" | "live-frontier";
primaryModel: string;
alternateModel: string;
fastMode?: boolean;
controlUiEnabled?: boolean;
mutateConfig?: (cfg: OpenClawConfig) => OpenClawConfig;
}): Promise<MatrixQaLiveLaneGatewayHarness> {
return (await loadQaLabRuntimeModule().startQaLiveLaneGateway(
params,
)) as MatrixQaLiveLaneGatewayHarness;
}
export async function runMatrixQaLive(params: {
fastMode?: boolean;
outputDir?: string;
@@ -271,9 +305,11 @@ export async function runMatrixQaLive(params: {
path.join(repoRoot, ".artifacts", "qa-e2e", `matrix-${Date.now().toString(36)}`);
await fs.mkdir(outputDir, { recursive: true });
const providerMode = normalizeQaProviderMode(params.providerMode ?? "live-frontier");
const primaryModel = params.primaryModel?.trim() || defaultQaModelForMode(providerMode);
const alternateModel = params.alternateModel?.trim() || defaultQaModelForMode(providerMode, true);
const { providerMode, primaryModel, alternateModel } = resolveMatrixQaModels({
providerMode: params.providerMode,
primaryModel: params.primaryModel,
alternateModel: params.alternateModel,
});
const sutAccountId = params.sutAccountId?.trim() || "sut";
const scenarios = findMatrixQaScenarios(params.scenarioIds);
const observedEvents: MatrixQaObservedEvent[] = [];
@@ -317,12 +353,12 @@ export async function runMatrixQaLive(params: {
const scenarioResults: MatrixQaScenarioResult[] = [];
const cleanupErrors: string[] = [];
let canaryArtifact: MatrixQaCanaryArtifact | undefined;
let gatewayHarness: Awaited<ReturnType<typeof startQaLiveLaneGateway>> | null = null;
let gatewayHarness: MatrixQaLiveLaneGatewayHarness | null = null;
let canaryFailed = false;
const syncState: { driver?: string; observer?: string } = {};
try {
gatewayHarness = await startQaLiveLaneGateway({
gatewayHarness = await startMatrixQaLiveLaneGateway({
repoRoot,
transport: {
requiredPluginIds: [],
@@ -555,5 +591,6 @@ export const __testing = {
buildMatrixQaConfig,
buildObservedEventsArtifact,
isMatrixAccountReady,
resolveMatrixQaModels,
waitForMatrixChannelReady,
};

6
extensions/qa-lab/src/live-transports/matrix/matrix-live-scenarios.test.ts → extensions/qa-matrix/src/runners/contract/scenarios.test.ts
View File

@@ -3,19 +3,19 @@ const { createMatrixQaClient } = vi.hoisted(() => ({
createMatrixQaClient: vi.fn(),
}));
vi.mock("./matrix-driver-client.js", () => ({
vi.mock("../../substrate/client.js", () => ({
createMatrixQaClient,
}));
import {
LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS,
findMissingLiveTransportStandardScenarios,
} from "../shared/live-transport-scenarios.js";
} from "../../shared/live-transport-scenarios.js";
import {
__testing as scenarioTesting,
MATRIX_QA_SCENARIOS,
runMatrixQaScenario,
} from "./matrix-live-scenarios.js";
} from "./scenarios.js";
describe("matrix live qa scenarios", () => {
beforeEach(() => {

4
extensions/qa-lab/src/live-transports/matrix/matrix-live-scenarios.ts → extensions/qa-matrix/src/runners/contract/scenarios.ts
View File

@@ -3,8 +3,8 @@ import {
collectLiveTransportStandardScenarioCoverage,
selectLiveTransportScenarios,
type LiveTransportScenarioDefinition,
} from "../shared/live-transport-scenarios.js";
import { createMatrixQaClient, type MatrixQaObservedEvent } from "./matrix-driver-client.js";
} from "../../shared/live-transport-scenarios.js";
import { createMatrixQaClient, type MatrixQaObservedEvent } from "../../substrate/client.js";
export type MatrixQaScenarioId =
| "matrix-thread-follow-up"

9
extensions/qa-matrix/src/runtime-api.test.ts Normal file
View File

@@ -0,0 +1,9 @@
import { describe, expect, it } from "vitest";
describe("matrix qa runtime api surface", () => {
it("keeps runner discovery lightweight", async () => {
const runtimeApi = await import("../runtime-api.js");
expect(Object.keys(runtimeApi).toSorted()).toEqual(["qaRunnerCliRegistrations"]);
});
});

18
extensions/qa-matrix/src/shared/live-lane-helpers.ts Normal file
View File

@@ -0,0 +1,18 @@
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
export function appendLiveLaneIssue(issues: string[], label: string, error: unknown) {
issues.push(`${label}: ${formatErrorMessage(error)}`);
}
export function buildLiveLaneArtifactsError(params: {
heading: string;
artifacts: Record<string, string>;
details?: string[];
}) {
return [
params.heading,
...(params.details ?? []),
"Artifacts:",
...Object.entries(params.artifacts).map(([label, filePath]) => `- ${label}: ${filePath}`),
].join("\n");
}

40
extensions/qa-matrix/src/shared/live-transport-cli.runtime.ts Normal file
View File

@@ -0,0 +1,40 @@
import path from "node:path";
import { resolveRepoRelativeOutputDir } from "../cli-paths.js";
import type { QaProviderMode } from "../run-config.js";
import { normalizeQaProviderMode } from "../run-config.js";
import type { LiveTransportQaCommandOptions } from "./live-transport-cli.js";
export function resolveLiveTransportQaRunOptions(
opts: LiveTransportQaCommandOptions,
): LiveTransportQaCommandOptions & {
repoRoot: string;
providerMode: QaProviderMode;
} {
return {
repoRoot: path.resolve(opts.repoRoot ?? process.cwd()),
outputDir: resolveRepoRelativeOutputDir(
path.resolve(opts.repoRoot ?? process.cwd()),
opts.outputDir,
),
providerMode:
opts.providerMode === undefined
? "live-frontier"
: normalizeQaProviderMode(opts.providerMode),
primaryModel: opts.primaryModel,
alternateModel: opts.alternateModel,
fastMode: opts.fastMode,
scenarioIds: opts.scenarioIds,
sutAccountId: opts.sutAccountId,
credentialSource: opts.credentialSource?.trim(),
credentialRole: opts.credentialRole?.trim(),
};
}
export function printLiveTransportQaArtifacts(
laneLabel: string,
artifacts: Record<string, string>,
) {
for (const [label, filePath] of Object.entries(artifacts)) {
process.stdout.write(`${laneLabel} ${label}: ${filePath}\n`);
}
}

132
extensions/qa-matrix/src/shared/live-transport-cli.ts Normal file
View File

@@ -0,0 +1,132 @@
import type { Command } from "commander";
import { collectString } from "../cli-options.js";
import type { QaProviderModeInput } from "../run-config.js";
export type LiveTransportQaCommandOptions = {
repoRoot?: string;
outputDir?: string;
providerMode?: QaProviderModeInput;
primaryModel?: string;
alternateModel?: string;
fastMode?: boolean;
scenarioIds?: string[];
sutAccountId?: string;
credentialSource?: string;
credentialRole?: string;
};
type LiveTransportQaCommanderOptions = {
repoRoot?: string;
outputDir?: string;
providerMode?: QaProviderModeInput;
model?: string;
altModel?: string;
scenario?: string[];
fast?: boolean;
sutAccount?: string;
credentialSource?: string;
credentialRole?: string;
};
export type LiveTransportQaCliRegistration = {
commandName: string;
register(qa: Command): void;
};
export type LiveTransportQaCredentialCliOptions = {
sourceDescription?: string;
roleDescription?: string;
};
export function createLazyCliRuntimeLoader<T>(load: () => Promise<T>) {
let promise: Promise<T> | null = null;
return async () => {
promise ??= load();
return await promise;
};
}
export function mapLiveTransportQaCommanderOptions(
opts: LiveTransportQaCommanderOptions,
): LiveTransportQaCommandOptions {
return {
repoRoot: opts.repoRoot,
outputDir: opts.outputDir,
providerMode: opts.providerMode,
primaryModel: opts.model,
alternateModel: opts.altModel,
fastMode: opts.fast,
scenarioIds: opts.scenario,
sutAccountId: opts.sutAccount,
credentialSource: opts.credentialSource,
credentialRole: opts.credentialRole,
};
}
export function registerLiveTransportQaCli(params: {
qa: Command;
commandName: string;
credentialOptions?: LiveTransportQaCredentialCliOptions;
description: string;
outputDirHelp: string;
scenarioHelp: string;
sutAccountHelp: string;
run: (opts: LiveTransportQaCommandOptions) => Promise<void>;
}) {
const command = params.qa
.command(params.commandName)
.description(params.description)
.option("--repo-root <path>", "Repository root to target when running from a neutral cwd")
.option("--output-dir <path>", params.outputDirHelp)
.option(
"--provider-mode <mode>",
"Provider mode: mock-openai or live-frontier (legacy live-openai still works)",
"live-frontier",
)
.option("--model <ref>", "Primary provider/model ref")
.option("--alt-model <ref>", "Alternate provider/model ref")
.option("--scenario <id>", params.scenarioHelp, collectString, [])
.option("--fast", "Enable provider fast mode where supported", false)
.option("--sut-account <id>", params.sutAccountHelp, "sut");
if (params.credentialOptions) {
command.option(
"--credential-source <source>",
params.credentialOptions.sourceDescription ??
"Credential source for live lanes: env or convex (default: env)",
);
if (params.credentialOptions.roleDescription) {
command.option("--credential-role <role>", params.credentialOptions.roleDescription);
}
}
command.action(async (opts: LiveTransportQaCommanderOptions) => {
await params.run(mapLiveTransportQaCommanderOptions(opts));
});
}
export function createLiveTransportQaCliRegistration(params: {
commandName: string;
credentialOptions?: LiveTransportQaCredentialCliOptions;
description: string;
outputDirHelp: string;
scenarioHelp: string;
sutAccountHelp: string;
run: (opts: LiveTransportQaCommandOptions) => Promise<void>;
}): LiveTransportQaCliRegistration {
return {
commandName: params.commandName,
register(qa: Command) {
registerLiveTransportQaCli({
qa,
commandName: params.commandName,
credentialOptions: params.credentialOptions,
description: params.description,
outputDirHelp: params.outputDirHelp,
scenarioHelp: params.scenarioHelp,
sutAccountHelp: params.sutAccountHelp,
run: params.run,
});
},
};
}

149
extensions/qa-matrix/src/shared/live-transport-scenarios.ts Normal file
View File

@@ -0,0 +1,149 @@
export type LiveTransportStandardScenarioId =
| "canary"
| "mention-gating"
| "allowlist-block"
| "top-level-reply-shape"
| "restart-resume"
| "thread-follow-up"
| "thread-isolation"
| "reaction-observation"
| "help-command";
export type LiveTransportScenarioDefinition<TId extends string = string> = {
id: TId;
standardId?: LiveTransportStandardScenarioId;
timeoutMs: number;
title: string;
};
export type LiveTransportStandardScenarioDefinition = {
description: string;
id: LiveTransportStandardScenarioId;
title: string;
};
export const LIVE_TRANSPORT_STANDARD_SCENARIOS: readonly LiveTransportStandardScenarioDefinition[] =
[
{
id: "canary",
title: "Transport canary",
description: "The lane can trigger one known-good reply on the real transport.",
},
{
id: "mention-gating",
title: "Mention gating",
description: "Messages without the required mention do not trigger a reply.",
},
{
id: "allowlist-block",
title: "Sender allowlist block",
description: "Non-allowlisted senders do not trigger a reply.",
},
{
id: "top-level-reply-shape",
title: "Top-level reply shape",
description: "Top-level replies stay top-level when the lane is configured that way.",
},
{
id: "restart-resume",
title: "Restart resume",
description: "The lane still responds after a gateway restart.",
},
{
id: "thread-follow-up",
title: "Thread follow-up",
description: "Threaded prompts receive threaded replies with the expected relation metadata.",
},
{
id: "thread-isolation",
title: "Thread isolation",
description: "Fresh top-level prompts stay out of prior threads.",
},
{
id: "reaction-observation",
title: "Reaction observation",
description: "Reaction events are observed and normalized correctly.",
},
{
id: "help-command",
title: "Help command",
description: "The transport-specific help command path replies successfully.",
},
] as const;
export const LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS: readonly LiveTransportStandardScenarioId[] =
[
"canary",
"mention-gating",
"allowlist-block",
"top-level-reply-shape",
"restart-resume",
] as const;
const LIVE_TRANSPORT_STANDARD_SCENARIO_ID_SET = new Set(
LIVE_TRANSPORT_STANDARD_SCENARIOS.map((scenario) => scenario.id),
);
function assertKnownStandardScenarioIds(ids: readonly LiveTransportStandardScenarioId[]) {
for (const id of ids) {
if (!LIVE_TRANSPORT_STANDARD_SCENARIO_ID_SET.has(id)) {
throw new Error(`unknown live transport standard scenario id: ${id}`);
}
}
}
export function selectLiveTransportScenarios<TDefinition extends { id: string }>(params: {
ids?: string[];
laneLabel: string;
scenarios: readonly TDefinition[];
}) {
if (!params.ids || params.ids.length === 0) {
return [...params.scenarios];
}
const requested = new Set(params.ids);
const selected = params.scenarios.filter((scenario) => params.ids?.includes(scenario.id));
const missingIds = [...requested].filter(
(id) => !selected.some((scenario) => scenario.id === id),
);
if (missingIds.length > 0) {
throw new Error(`unknown ${params.laneLabel} QA scenario id(s): ${missingIds.join(", ")}`);
}
return selected;
}
export function collectLiveTransportStandardScenarioCoverage<TId extends string>(params: {
alwaysOnStandardScenarioIds?: readonly LiveTransportStandardScenarioId[];
scenarios: readonly LiveTransportScenarioDefinition<TId>[];
}) {
const coverage: LiveTransportStandardScenarioId[] = [];
const seen = new Set<LiveTransportStandardScenarioId>();
const append = (id: LiveTransportStandardScenarioId | undefined) => {
if (!id || seen.has(id)) {
return;
}
seen.add(id);
coverage.push(id);
};
assertKnownStandardScenarioIds(params.alwaysOnStandardScenarioIds ?? []);
for (const id of params.alwaysOnStandardScenarioIds ?? []) {
append(id);
}
for (const scenario of params.scenarios) {
if (scenario.standardId) {
assertKnownStandardScenarioIds([scenario.standardId]);
}
append(scenario.standardId);
}
return coverage;
}
export function findMissingLiveTransportStandardScenarios(params: {
coveredStandardScenarioIds: readonly LiveTransportStandardScenarioId[];
expectedStandardScenarioIds: readonly LiveTransportStandardScenarioId[];
}) {
assertKnownStandardScenarioIds(params.coveredStandardScenarioIds);
assertKnownStandardScenarioIds(params.expectedStandardScenarioIds);
const covered = new Set(params.coveredStandardScenarioIds);
return params.expectedStandardScenarioIds.filter((id) => !covered.has(id));
}

2
extensions/qa-lab/src/live-transports/matrix/matrix-driver-client.test.ts → extensions/qa-matrix/src/substrate/client.test.ts
View File

@@ -4,7 +4,7 @@ import {
createMatrixQaClient,
provisionMatrixQaRoom,
type MatrixQaObservedEvent,
} from "./matrix-driver-client.js";
} from "./client.js";
function resolveRequestUrl(input: RequestInfo | URL) {
if (typeof input === "string") {

0
extensions/qa-lab/src/live-transports/matrix/matrix-driver-client.ts → extensions/qa-matrix/src/substrate/client.ts
View File

6
extensions/qa-lab/src/live-transports/matrix/matrix-harness.runtime.test.ts → extensions/qa-matrix/src/substrate/harness.runtime.test.ts
View File

@@ -2,11 +2,7 @@ import { mkdtemp, readFile, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { describe, expect, it, vi } from "vitest";
import {
__testing,
startMatrixQaHarness,
writeMatrixQaHarnessFiles,
} from "./matrix-harness.runtime.js";
import { __testing, startMatrixQaHarness, writeMatrixQaHarnessFiles } from "./harness.runtime.js";
describe("matrix harness runtime", () => {
it("writes a pinned Tuwunel compose file and redacted manifest", async () => {

2
extensions/qa-lab/src/live-transports/matrix/matrix-harness.runtime.ts → extensions/qa-matrix/src/substrate/harness.runtime.ts
View File

@@ -11,7 +11,7 @@ import {
waitForHealth,
type FetchLike,
type RunCommand,
} from "../../docker-runtime.js";
} from "../docker-runtime.js";
const MATRIX_QA_DEFAULT_IMAGE = "ghcr.io/matrix-construct/tuwunel:v1.5.1";
const MATRIX_QA_DEFAULT_SERVER_NAME = "matrix-qa.test";

13
package.json
View File

@@ -30,9 +30,8 @@
"!dist/**/*.map",
"!dist/plugin-sdk/.tsbuildinfo",
"!dist/extensions/qa-channel/**",
"dist/extensions/qa-channel/runtime-api.js",
"!dist/extensions/qa-lab/**",
"dist/extensions/qa-lab/runtime-api.js",
"!dist/extensions/qa-matrix/**",
"docs/",
"!docs/.generated/**",
"!docs/.i18n/zh-CN.tm.jsonl",
@@ -766,6 +765,14 @@
"types": "./dist/plugin-sdk/matrix-thread-bindings.d.ts",
"default": "./dist/plugin-sdk/matrix-thread-bindings.js"
},
"./plugin-sdk/qa-lab-runtime": {
"types": "./dist/plugin-sdk/qa-lab-runtime.d.ts",
"default": "./dist/plugin-sdk/qa-lab-runtime.js"
},
"./plugin-sdk/qa-runner-runtime": {
"types": "./dist/plugin-sdk/qa-runner-runtime.d.ts",
"default": "./dist/plugin-sdk/qa-runner-runtime.js"
},
"./plugin-sdk/mattermost": {
"types": "./dist/plugin-sdk/mattermost.d.ts",
"default": "./dist/plugin-sdk/mattermost.js"
@@ -1236,6 +1243,8 @@
"proxy:install-ca": "node --import tsx scripts/proxy-install-ca.mjs",
"proxy:run": "node scripts/run-node.mjs proxy run",
"proxy:start": "node scripts/run-node.mjs proxy start",
"qa-runners:check": "node --import tsx scripts/generate-qa-runner-catalog.ts --check",
"qa-runners:gen": "node --import tsx scripts/generate-qa-runner-catalog.ts --write",
"qa:e2e": "node --import tsx scripts/qa-e2e.ts",
"qa:lab:build": "vite build --config extensions/qa-lab/web/vite.config.ts",
"qa:lab:ui": "pnpm openclaw qa ui",

9
pnpm-lock.yaml generated
View File

@@ -994,6 +994,15 @@ importers:
specifier: workspace:*
version: link:../..
extensions/qa-matrix:
devDependencies:
'@openclaw/plugin-sdk':
specifier: workspace:*
version: link:../../packages/plugin-sdk
openclaw:
specifier: workspace:*
version: link:../..
extensions/qianfan:
devDependencies:
'@openclaw/plugin-sdk':

5
scripts/build-all.mjs
View File

@@ -10,11 +10,6 @@ export const BUILD_ALL_STEPS = [
{ label: "canvas:a2ui:bundle", kind: "pnpm", pnpmArgs: ["canvas:a2ui:bundle"] },
{ label: "tsdown", kind: "node", args: ["scripts/tsdown-build.mjs"] },
{ label: "runtime-postbuild", kind: "node", args: ["scripts/runtime-postbuild.mjs"] },
{
label: "write-npm-update-compat-sidecars",
kind: "node",
args: ["scripts/write-npm-update-compat-sidecars.mjs"],
},
{ label: "build-stamp", kind: "node", args: ["scripts/build-stamp.mjs"] },
{
label: "build:plugin-sdk:dts",

35
scripts/generate-qa-runner-catalog.ts Normal file
View File

@@ -0,0 +1,35 @@
#!/usr/bin/env node
import path from "node:path";
import { writeBundledQaRunnerCatalog } from "../src/plugins/qa-runner-catalog.js";
const args = new Set(process.argv.slice(2));
const checkOnly = args.has("--check");
const writeMode = args.has("--write");
if (checkOnly === writeMode) {
console.error("Use exactly one of --check or --write.");
process.exit(1);
}
const repoRoot = process.cwd();
const result = await writeBundledQaRunnerCatalog({
repoRoot,
check: checkOnly,
});
if (checkOnly) {
if (result.changed) {
console.error(
[
"QA runner catalog drift detected.",
`Expected current: ${path.relative(repoRoot, result.jsonPath)}`,
"If this QA runner metadata change is intentional, run `pnpm qa-runners:gen` and commit the updated baseline file.",
"If not intentional, fix the bundled plugin metadata drift first.",
].join("\n"),
);
process.exit(1);
}
console.log(`OK ${path.relative(repoRoot, result.jsonPath)}`);
} else {
console.log(`Wrote ${path.relative(repoRoot, result.jsonPath)}`);
}

2
scripts/lib/bundled-plugin-build-entries.mjs
View File

@@ -8,7 +8,7 @@ import {
import { shouldBuildBundledCluster } from "./optional-bundled-clusters.mjs";
const TOP_LEVEL_PUBLIC_SURFACE_EXTENSIONS = new Set([".ts", ".js", ".mts", ".cts", ".mjs", ".cjs"]);
const NON_PACKAGED_BUNDLED_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab"]);
const NON_PACKAGED_BUNDLED_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab", "qa-matrix"]);
const toPosixPath = (value) => value.replaceAll("\\", "/");
function readBundledPluginPackageJson(packageJsonPath) {

16
scripts/lib/npm-update-compat-sidecars.mjs
View File

@@ -1,16 +0,0 @@
export const NPM_UPDATE_COMPAT_SIDECARS = [
{
path: "dist/extensions/qa-channel/runtime-api.js",
content:
"// Compatibility stub for older OpenClaw updaters. The QA channel implementation is not packaged.\nexport {};\n",
},
{
path: "dist/extensions/qa-lab/runtime-api.js",
content:
"// Compatibility stub for older OpenClaw updaters. The QA lab implementation is not packaged.\nexport {};\n",
},
];
export const NPM_UPDATE_COMPAT_SIDECAR_PATHS = new Set(
NPM_UPDATE_COMPAT_SIDECARS.map((entry) => entry.path),
);

2
scripts/lib/plugin-sdk-entrypoints.json
View File

@@ -179,6 +179,8 @@
"matrix-runtime-surface",
"matrix-surface",
"matrix-thread-bindings",
"qa-lab-runtime",
"qa-runner-runtime",
"mattermost",
"mattermost-policy",
"memory-core",

8
scripts/lib/qa-runner-catalog.json Normal file
View File

@@ -0,0 +1,8 @@
[
{
"pluginId": "qa-matrix",
"commandName": "matrix",
"description": "Run the Docker-backed Matrix live QA lane against a disposable homeserver",
"npmSpec": "@openclaw/qa-matrix"
}
]

25
scripts/openclaw-npm-postpublish-verify.ts
View File

@@ -20,7 +20,6 @@ import {
collectRootDistBundledRuntimeMirrors,
collectRuntimeDependencySpecs,
} from "./lib/bundled-plugin-root-runtime-mirrors.mjs";
import { NPM_UPDATE_COMPAT_SIDECAR_PATHS } from "./lib/npm-update-compat-sidecars.mjs";
import { runInstalledWorkspaceBootstrapSmoke } from "./lib/workspace-bootstrap-smoke.mjs";
import { parseReleaseVersion, resolveNpmCommandInvocation } from "./openclaw-npm-release-check.ts";
@@ -44,13 +43,6 @@ type InstalledBundledExtensionManifestRecord = {
const MAX_BUNDLED_EXTENSION_MANIFEST_BYTES = 1024 * 1024;
const LEGACY_CONTEXT_ENGINE_UNRESOLVED_RUNTIME_MARKER =
"Failed to load legacy context engine runtime.";
const NPM_UPDATE_COMPAT_EXTENSION_DIRS = new Set(
[...NPM_UPDATE_COMPAT_SIDECAR_PATHS].map((relativePath) => {
const pathParts = relativePath.split("/");
pathParts.pop();
return pathParts.join("/");
}),
);
export type PublishedInstallScenario = {
name: string;
@@ -183,20 +175,6 @@ function collectExpectedBundledExtensionPackageIds(
return ids;
}
function isNpmUpdateCompatOnlyExtensionDir(params: {
extensionId: string;
packageRoot: string;
}): boolean {
const relativeExtensionDir = `dist/extensions/${params.extensionId}`;
if (!NPM_UPDATE_COMPAT_EXTENSION_DIRS.has(relativeExtensionDir)) {
return false;
}
return [...NPM_UPDATE_COMPAT_SIDECAR_PATHS]
.filter((relativePath) => relativePath.startsWith(`${relativeExtensionDir}/`))
.every((relativePath) => existsSync(join(params.packageRoot, relativePath)));
}
function readBundledExtensionPackageJsons(packageRoot: string): {
manifests: InstalledBundledExtensionManifestRecord[];
errors: string[];
@@ -218,9 +196,6 @@ function readBundledExtensionPackageJsons(packageRoot: string): {
const extensionDirPath = join(extensionsDir, entry.name);
const packageJsonPath = join(extensionsDir, entry.name, "package.json");
if (!existsSync(packageJsonPath)) {
if (isNpmUpdateCompatOnlyExtensionDir({ extensionId: entry.name, packageRoot })) {
continue;
}
if (expectedPackageIds === null || expectedPackageIds.has(entry.name)) {
errors.push(`installed bundled extension manifest missing: ${packageJsonPath}.`);
}

4
scripts/openclaw-npm-release-check.ts
View File

@@ -9,7 +9,6 @@ import {
resolveNpmDistTagMirrorAuth as resolveNpmDistTagMirrorAuthBase,
parseReleaseVersion as parseReleaseVersionBase,
} from "./lib/npm-publish-plan.mjs";
import { NPM_UPDATE_COMPAT_SIDECAR_PATHS } from "./lib/npm-update-compat-sidecars.mjs";
import { WORKSPACE_TEMPLATE_PACK_PATHS } from "./lib/workspace-bootstrap-smoke.mjs";
type PackageJson = {
@@ -465,9 +464,6 @@ function collectPackedTarballErrors(): string[] {
export function collectForbiddenPackedPathErrors(paths: Iterable<string>): string[] {
const errors: string[] = [];
for (const packedPath of paths) {
if (NPM_UPDATE_COMPAT_SIDECAR_PATHS.has(packedPath)) {
continue;
}
const matchedRule = FORBIDDEN_PACKED_PATH_RULES.find((rule) =>
packedPath.startsWith(rule.prefix),
);

10
scripts/write-npm-update-compat-sidecars.mjs
View File

@@ -1,10 +0,0 @@
#!/usr/bin/env node
import fs from "node:fs";
import path from "node:path";
import { NPM_UPDATE_COMPAT_SIDECARS } from "./lib/npm-update-compat-sidecars.mjs";
for (const entry of NPM_UPDATE_COMPAT_SIDECARS) {
fs.mkdirSync(path.dirname(entry.path), { recursive: true });
fs.writeFileSync(entry.path, entry.content, "utf8");
}

39
src/plugin-sdk/qa-lab-runtime.ts Normal file
View File

@@ -0,0 +1,39 @@
import { loadBundledPluginPublicSurfaceModuleSync } from "./facade-runtime.js";
type QaLabRuntimeSurface = {
defaultQaRuntimeModelForMode: (
mode: string,
options?: {
alternate?: boolean;
preferredLiveModel?: string;
},
) => string;
startQaLiveLaneGateway: (...args: unknown[]) => Promise<unknown>;
};
function isMissingQaLabRuntimeError(error: unknown) {
return (
error instanceof Error &&
(error.message === "Unable to resolve bundled plugin public surface qa-lab/runtime-api.js" ||
error.message.startsWith("Unable to open bundled plugin public surface "))
);
}
export function loadQaLabRuntimeModule(): QaLabRuntimeSurface {
return loadBundledPluginPublicSurfaceModuleSync<QaLabRuntimeSurface>({
dirName: "qa-lab",
artifactBasename: "runtime-api.js",
});
}
export function isQaLabRuntimeAvailable(): boolean {
try {
loadQaLabRuntimeModule();
return true;
} catch (error) {
if (isMissingQaLabRuntimeError(error)) {
return false;
}
throw error;
}
}

143
src/plugin-sdk/qa-runner-runtime.integration.test.ts Normal file
View File

@@ -0,0 +1,143 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { clearPluginDiscoveryCache } from "../plugins/discovery.js";
import { clearPluginManifestRegistryCache } from "../plugins/manifest-registry.js";
import { resetFacadeRuntimeStateForTest } from "./facade-runtime.js";
const ORIGINAL_ENV = {
OPENCLAW_DISABLE_BUNDLED_PLUGINS: process.env.OPENCLAW_DISABLE_BUNDLED_PLUGINS,
OPENCLAW_CONFIG_PATH: process.env.OPENCLAW_CONFIG_PATH,
OPENCLAW_DISABLE_PLUGIN_DISCOVERY_CACHE: process.env.OPENCLAW_DISABLE_PLUGIN_DISCOVERY_CACHE,
OPENCLAW_DISABLE_PLUGIN_MANIFEST_CACHE: process.env.OPENCLAW_DISABLE_PLUGIN_MANIFEST_CACHE,
OPENCLAW_PLUGIN_DISCOVERY_CACHE_MS: process.env.OPENCLAW_PLUGIN_DISCOVERY_CACHE_MS,
OPENCLAW_PLUGIN_MANIFEST_CACHE_MS: process.env.OPENCLAW_PLUGIN_MANIFEST_CACHE_MS,
OPENCLAW_TEST_FAST: process.env.OPENCLAW_TEST_FAST,
} as const;
const tempDirs: string[] = [];
function makeTempDir(prefix: string): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
tempDirs.push(dir);
return dir;
}
function resetQaRunnerRuntimeState() {
clearPluginDiscoveryCache();
clearPluginManifestRegistryCache();
resetFacadeRuntimeStateForTest();
}
describe("plugin-sdk qa-runner-runtime linked plugin smoke", () => {
beforeEach(() => {
resetQaRunnerRuntimeState();
process.env.OPENCLAW_DISABLE_BUNDLED_PLUGINS = "1";
process.env.OPENCLAW_DISABLE_PLUGIN_DISCOVERY_CACHE = "1";
process.env.OPENCLAW_DISABLE_PLUGIN_MANIFEST_CACHE = "1";
process.env.OPENCLAW_PLUGIN_DISCOVERY_CACHE_MS = "0";
process.env.OPENCLAW_PLUGIN_MANIFEST_CACHE_MS = "0";
process.env.OPENCLAW_TEST_FAST = "1";
});
afterEach(() => {
resetQaRunnerRuntimeState();
for (const dir of tempDirs.splice(0)) {
fs.rmSync(dir, { recursive: true, force: true });
}
for (const [key, value] of Object.entries(ORIGINAL_ENV)) {
if (value === undefined) {
delete process.env[key];
} else {
process.env[key] = value;
}
}
});
it("loads an activated qa runner from a linked plugin path", async () => {
const stateDir = makeTempDir("openclaw-qa-runner-state-");
const pluginDir = path.join(stateDir, "extensions", "qa-linked");
const configPath = path.join(stateDir, "openclaw.json");
fs.writeFileSync(
configPath,
JSON.stringify({
plugins: {},
}),
"utf8",
);
process.env.OPENCLAW_CONFIG_PATH = configPath;
fs.mkdirSync(pluginDir, { recursive: true });
fs.writeFileSync(
path.join(pluginDir, "openclaw.plugin.json"),
JSON.stringify({
id: "qa-linked",
qaRunners: [
{
commandName: "linked",
description: "Run the linked QA lane",
},
],
configSchema: {
type: "object",
additionalProperties: false,
properties: {},
},
}),
"utf8",
);
fs.writeFileSync(
path.join(pluginDir, "package.json"),
JSON.stringify({
name: "@openclaw/qa-linked",
type: "module",
openclaw: {
extensions: ["./index.js"],
install: {
npmSpec: "@openclaw/qa-linked",
},
},
}),
"utf8",
);
fs.writeFileSync(path.join(pluginDir, "index.js"), 'export default {};\n', "utf8");
fs.writeFileSync(
path.join(pluginDir, "runtime-api.js"),
[
"export const qaRunnerCliRegistrations = [",
" {",
' commandName: "linked",',
" register() {}",
" }",
"];",
].join("\n"),
"utf8",
);
const module = await import("./qa-runner-runtime.js");
expect(module.listQaRunnerCliContributions()).toEqual(
expect.arrayContaining([
{
pluginId: "qa-linked",
commandName: "linked",
description: "Run the linked QA lane",
status: "available",
registration: {
commandName: "linked",
register: expect.any(Function),
},
},
{
pluginId: "qa-matrix",
commandName: "matrix",
description: "Run the Docker-backed Matrix live QA lane against a disposable homeserver",
status: "missing",
npmSpec: "@openclaw/qa-matrix",
},
]),
);
});
});

183
src/plugin-sdk/qa-runner-runtime.test.ts Normal file
View File

@@ -0,0 +1,183 @@
import type { Command } from "commander";
import { beforeEach, describe, expect, it, vi } from "vitest";
const loadPluginManifestRegistry = vi.hoisted(() => vi.fn());
const tryLoadActivatedBundledPluginPublicSurfaceModuleSync = vi.hoisted(() => vi.fn());
const listBundledQaRunnerCatalog = vi.hoisted(() =>
vi.fn<
() => Array<{
pluginId: string;
commandName: string;
description?: string;
npmSpec: string;
}>
>(() => []),
);
vi.mock("../plugins/manifest-registry.js", () => ({
loadPluginManifestRegistry,
}));
vi.mock("../plugins/qa-runner-catalog.js", () => ({
listBundledQaRunnerCatalog,
}));
vi.mock("./facade-runtime.js", () => ({
tryLoadActivatedBundledPluginPublicSurfaceModuleSync,
}));
describe("plugin-sdk qa-runner-runtime", () => {
beforeEach(() => {
loadPluginManifestRegistry.mockReset().mockReturnValue({
plugins: [],
diagnostics: [],
});
listBundledQaRunnerCatalog.mockReset().mockReturnValue([]);
tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReset();
});
it("stays cold until runner discovery is requested", async () => {
await import("./qa-runner-runtime.js");
expect(loadPluginManifestRegistry).not.toHaveBeenCalled();
expect(tryLoadActivatedBundledPluginPublicSurfaceModuleSync).not.toHaveBeenCalled();
});
it("returns activated runner registrations declared in plugin manifests", async () => {
const register = vi.fn((qa: Command) => qa);
loadPluginManifestRegistry.mockReturnValue({
plugins: [
{
id: "qa-matrix",
qaRunners: [
{
commandName: "matrix",
description: "Run the Matrix live QA lane",
},
],
rootDir: "/tmp/qa-matrix",
},
],
diagnostics: [],
});
tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReturnValue({
qaRunnerCliRegistrations: [{ commandName: "matrix", register }],
});
const module = await import("./qa-runner-runtime.js");
expect(module.listQaRunnerCliContributions()).toEqual([
{
pluginId: "qa-matrix",
commandName: "matrix",
description: "Run the Matrix live QA lane",
status: "available",
registration: {
commandName: "matrix",
register,
},
},
]);
expect(tryLoadActivatedBundledPluginPublicSurfaceModuleSync).toHaveBeenCalledWith({
dirName: "qa-matrix",
artifactBasename: "runtime-api.js",
});
});
it("reports declared runners as blocked when the plugin is present but not activated", async () => {
loadPluginManifestRegistry.mockReturnValue({
plugins: [
{
id: "qa-matrix",
qaRunners: [{ commandName: "matrix" }],
rootDir: "/tmp/qa-matrix",
},
],
diagnostics: [],
});
tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReturnValue(null);
const module = await import("./qa-runner-runtime.js");
expect(module.listQaRunnerCliContributions()).toEqual([
{
pluginId: "qa-matrix",
commandName: "matrix",
status: "blocked",
},
]);
});
it("reports missing optional runners from the generated catalog", async () => {
listBundledQaRunnerCatalog.mockReturnValue([
{
pluginId: "qa-matrix",
commandName: "matrix",
description: "Run the Matrix live QA lane",
npmSpec: "@openclaw/qa-matrix",
},
]);
const module = await import("./qa-runner-runtime.js");
expect(module.listQaRunnerCliContributions()).toEqual([
{
pluginId: "qa-matrix",
commandName: "matrix",
description: "Run the Matrix live QA lane",
status: "missing",
npmSpec: "@openclaw/qa-matrix",
},
]);
});
it("fails fast when two plugins declare the same qa runner command", async () => {
loadPluginManifestRegistry.mockReturnValue({
plugins: [
{
id: "alpha",
qaRunners: [{ commandName: "matrix" }],
rootDir: "/tmp/alpha",
},
{
id: "beta",
qaRunners: [{ commandName: "matrix" }],
rootDir: "/tmp/beta",
},
],
diagnostics: [],
});
tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReturnValue(null);
const module = await import("./qa-runner-runtime.js");
expect(() => module.listQaRunnerCliContributions()).toThrow(
'QA runner command "matrix" declared by both "alpha" and "beta"',
);
});
it("fails when runtime registrations include an undeclared command", async () => {
loadPluginManifestRegistry.mockReturnValue({
plugins: [
{
id: "qa-matrix",
qaRunners: [{ commandName: "matrix" }],
rootDir: "/tmp/qa-matrix",
},
],
diagnostics: [],
});
tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReturnValue({
qaRunnerCliRegistrations: [
{ commandName: "matrix", register: vi.fn() },
{ commandName: "extra", register: vi.fn() },
],
});
const module = await import("./qa-runner-runtime.js");
expect(() => module.listQaRunnerCliContributions()).toThrow(
'QA runner plugin "qa-matrix" exported "extra" from runtime-api.js but did not declare it in openclaw.plugin.json',
);
});
});

161
src/plugin-sdk/qa-runner-runtime.ts Normal file
View File

@@ -0,0 +1,161 @@
import type { Command } from "commander";
import type { PluginManifestRecord } from "../plugins/manifest-registry.js";
import { loadPluginManifestRegistry } from "../plugins/manifest-registry.js";
import { listBundledQaRunnerCatalog } from "../plugins/qa-runner-catalog.js";
import { tryLoadActivatedBundledPluginPublicSurfaceModuleSync } from "./facade-runtime.js";
export type QaRunnerCliRegistration = {
commandName: string;
register(qa: Command): void;
};
type QaRunnerRuntimeSurface = {
qaRunnerCliRegistrations?: readonly QaRunnerCliRegistration[];
};
export type QaRunnerCliContribution =
| {
pluginId: string;
commandName: string;
description?: string;
status: "available";
registration: QaRunnerCliRegistration;
}
| {
pluginId: string;
commandName: string;
description?: string;
status: "blocked";
}
| {
pluginId: string;
commandName: string;
description?: string;
status: "missing";
npmSpec: string;
};
function listDeclaredQaRunnerPlugins(): Array<
PluginManifestRecord & {
qaRunners: NonNullable<PluginManifestRecord["qaRunners"]>;
}
> {
return loadPluginManifestRegistry({ cache: true })
.plugins.filter(
(
plugin,
): plugin is PluginManifestRecord & {
qaRunners: NonNullable<PluginManifestRecord["qaRunners"]>;
} => Array.isArray(plugin.qaRunners) && plugin.qaRunners.length > 0,
)
.toSorted((left, right) => {
const idCompare = left.id.localeCompare(right.id);
if (idCompare !== 0) {
return idCompare;
}
return left.rootDir.localeCompare(right.rootDir);
});
}
function indexRuntimeRegistrations(
pluginId: string,
surface: QaRunnerRuntimeSurface,
): ReadonlyMap<string, QaRunnerCliRegistration> {
const registrations = surface.qaRunnerCliRegistrations ?? [];
const registrationByCommandName = new Map<string, QaRunnerCliRegistration>();
for (const registration of registrations) {
if (!registration?.commandName || typeof registration.register !== "function") {
throw new Error(`QA runner plugin "${pluginId}" exported an invalid CLI registration`);
}
if (registrationByCommandName.has(registration.commandName)) {
throw new Error(
`QA runner plugin "${pluginId}" exported duplicate CLI registration "${registration.commandName}"`,
);
}
registrationByCommandName.set(registration.commandName, registration);
}
return registrationByCommandName;
}
function buildKnownQaRunnerCatalog(): readonly QaRunnerCliContribution[] {
const knownRunners = listBundledQaRunnerCatalog();
const seenCommandNames = new Map<string, string>();
return knownRunners.map((runner) => {
const previousOwner = seenCommandNames.get(runner.commandName);
if (previousOwner) {
throw new Error(
`QA runner command "${runner.commandName}" declared by both "${previousOwner}" and "${runner.pluginId}"`,
);
}
seenCommandNames.set(runner.commandName, runner.pluginId);
return {
pluginId: runner.pluginId,
commandName: runner.commandName,
...(runner.description ? { description: runner.description } : {}),
status: "missing" as const,
npmSpec: runner.npmSpec,
};
});
}
export function listQaRunnerCliContributions(): readonly QaRunnerCliContribution[] {
const contributions = new Map<string, QaRunnerCliContribution>();
for (const runner of buildKnownQaRunnerCatalog()) {
contributions.set(runner.commandName, runner);
}
for (const plugin of listDeclaredQaRunnerPlugins()) {
const runtimeSurface =
tryLoadActivatedBundledPluginPublicSurfaceModuleSync<QaRunnerRuntimeSurface>({
dirName: plugin.id,
artifactBasename: "runtime-api.js",
});
const runtimeRegistrationByCommandName = runtimeSurface
? indexRuntimeRegistrations(plugin.id, runtimeSurface)
: null;
const declaredCommandNames = new Set(plugin.qaRunners.map((runner) => runner.commandName));
for (const runner of plugin.qaRunners) {
const previous = contributions.get(runner.commandName);
if (previous && previous.pluginId !== plugin.id) {
throw new Error(
`QA runner command "${runner.commandName}" declared by both "${previous.pluginId}" and "${plugin.id}"`,
);
}
const registration = runtimeRegistrationByCommandName?.get(runner.commandName);
if (!runtimeSurface) {
contributions.set(runner.commandName, {
pluginId: plugin.id,
commandName: runner.commandName,
...(runner.description ? { description: runner.description } : {}),
status: "blocked",
});
continue;
}
if (!registration) {
throw new Error(
`QA runner plugin "${plugin.id}" declared "${runner.commandName}" in openclaw.plugin.json but did not export a matching CLI registration`,
);
}
contributions.set(runner.commandName, {
pluginId: plugin.id,
commandName: runner.commandName,
...(runner.description ? { description: runner.description } : {}),
status: "available",
registration,
});
}
for (const commandName of runtimeRegistrationByCommandName?.keys() ?? []) {
if (!declaredCommandNames.has(commandName)) {
throw new Error(
`QA runner plugin "${plugin.id}" exported "${commandName}" from runtime-api.js but did not declare it in openclaw.plugin.json`,
);
}
}
}
return [...contributions.values()];
}

3
src/plugins/bundled-plugin-metadata.test.ts
View File

@@ -131,11 +131,12 @@ describe("bundled plugin metadata", () => {
},
);
it("excludes private QA sidecars from the packaged runtime sidecar baseline", () => {
it("excludes non-packaged QA sidecars from the packaged runtime sidecar baseline", () => {
expect(BUNDLED_RUNTIME_SIDECAR_PATHS).not.toContain(
"dist/extensions/qa-channel/runtime-api.js",
);
expect(BUNDLED_RUNTIME_SIDECAR_PATHS).not.toContain("dist/extensions/qa-lab/runtime-api.js");
expect(BUNDLED_RUNTIME_SIDECAR_PATHS).not.toContain("dist/extensions/qa-matrix/runtime-api.js");
});
it("captures setup-entry metadata for bundled channel plugins", () => {

15
src/plugins/install.test.ts
View File

@@ -1427,6 +1427,21 @@ describe("installPluginFromArchive", () => {
).toBe(true);
});
it("does not flag the real qa-matrix plugin as dangerous install code", async () => {
const pluginDir = path.resolve(process.cwd(), "extensions", "qa-matrix");
const scanResult = await installSecurityScan.scanPackageInstallSource({
extensions: ["./index.ts"],
logger: { warn: vi.fn() },
packageDir: pluginDir,
pluginId: "qa-matrix",
packageName: "@openclaw/qa-matrix",
manifestId: "qa-matrix",
});
expect(scanResult?.blocked).toBeUndefined();
});
it("keeps blocked dependency package checks active when forced unsafe install is set", async () => {
const { pluginDir, extensionsDir } = setupPluginInstallDirs();

27
src/plugins/manifest-registry.test.ts
View File

@@ -499,6 +499,33 @@ describe("loadPluginManifestRegistry", () => {
});
});
it("preserves qa runner descriptors from plugin manifests", () => {
const dir = makeTempDir();
writeManifest(dir, {
id: "qa-matrix",
qaRunners: [
{
commandName: "matrix",
description: "Run the Matrix live QA lane",
},
],
configSchema: { type: "object" },
});
const registry = loadSingleCandidateRegistry({
idHint: "qa-matrix",
rootDir: dir,
origin: "bundled",
});
expect(registry.plugins[0]?.qaRunners).toEqual([
{
commandName: "matrix",
description: "Run the Matrix live QA lane",
},
]);
});
it("preserves channel config metadata from plugin manifests", () => {
const dir = makeTempDir();
writeManifest(dir, {

3
src/plugins/manifest-registry.ts
View File

@@ -34,6 +34,7 @@ import {
type PluginManifestChannelConfig,
type PluginManifestContracts,
type PluginManifestModelSupport,
type PluginManifestQaRunner,
type PluginManifestSetup,
} from "./manifest.js";
import { checkMinHostVersion } from "./min-host-version.js";
@@ -92,6 +93,7 @@ export type PluginManifestRecord = {
providerAuthChoices?: PluginManifest["providerAuthChoices"];
activation?: PluginManifestActivation;
setup?: PluginManifestSetup;
qaRunners?: PluginManifestQaRunner[];
skills: string[];
settingsFiles?: string[];
hooks: string[];
@@ -333,6 +335,7 @@ function buildRecord(params: {
providerAuthChoices: params.manifest.providerAuthChoices,
activation: params.manifest.activation,
setup: params.manifest.setup,
qaRunners: params.manifest.qaRunners,
skills: params.manifest.skills ?? [],
settingsFiles: [],
hooks: [],

33
src/plugins/manifest.ts
View File

@@ -80,6 +80,13 @@ export type PluginManifestSetup = {
requiresRuntime?: boolean;
};
export type PluginManifestQaRunner = {
/** Subcommand mounted beneath `openclaw qa`, for example `matrix`. */
commandName: string;
/** Optional user-facing help text for fallback host stubs. */
description?: string;
};
export type PluginManifestConfigLiteral = string | number | boolean | null;
export type PluginManifestDangerousConfigFlag = {
@@ -174,6 +181,8 @@ export type PluginManifest = {
activation?: PluginManifestActivation;
/** Cheap setup/onboarding metadata exposed before plugin runtime loads. */
setup?: PluginManifestSetup;
/** Cheap QA runner metadata exposed before plugin runtime loads. */
qaRunners?: PluginManifestQaRunner[];
skills?: string[];
name?: string;
description?: string;
@@ -484,6 +493,28 @@ function normalizeManifestSetup(value: unknown): PluginManifestSetup | undefined
return Object.keys(setup).length > 0 ? setup : undefined;
}
function normalizeManifestQaRunners(value: unknown): PluginManifestQaRunner[] | undefined {
if (!Array.isArray(value)) {
return undefined;
}
const normalized: PluginManifestQaRunner[] = [];
for (const entry of value) {
if (!isRecord(entry)) {
continue;
}
const commandName = normalizeOptionalString(entry.commandName) ?? "";
if (!commandName) {
continue;
}
const description = normalizeOptionalString(entry.description) ?? "";
normalized.push({
commandName,
...(description ? { description } : {}),
});
}
return normalized.length > 0 ? normalized : undefined;
}
function normalizeProviderAuthChoices(
value: unknown,
): PluginManifestProviderAuthChoice[] | undefined {
@@ -673,6 +704,7 @@ export function loadPluginManifest(
const providerAuthChoices = normalizeProviderAuthChoices(raw.providerAuthChoices);
const activation = normalizeManifestActivation(raw.activation);
const setup = normalizeManifestSetup(raw.setup);
const qaRunners = normalizeManifestQaRunners(raw.qaRunners);
const skills = normalizeTrimmedStringList(raw.skills);
const contracts = normalizeManifestContracts(raw.contracts);
const configContracts = normalizeManifestConfigContracts(raw.configContracts);
@@ -706,6 +738,7 @@ export function loadPluginManifest(
providerAuthChoices,
activation,
setup,
qaRunners,
skills,
name,
description,

74
src/plugins/qa-runner-catalog.ts Normal file
View File

@@ -0,0 +1,74 @@
import fs from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { listBundledPluginMetadata } from "./bundled-plugin-metadata.js";
export type QaRunnerCatalogEntry = {
pluginId: string;
commandName: string;
description?: string;
npmSpec: string;
};
const QA_RUNNER_CATALOG_JSON_PATH = fileURLToPath(
new URL("../../scripts/lib/qa-runner-catalog.json", import.meta.url),
);
export function listBundledQaRunnerCatalog(): readonly QaRunnerCatalogEntry[] {
if (!fs.existsSync(QA_RUNNER_CATALOG_JSON_PATH)) {
return [];
}
return JSON.parse(fs.readFileSync(QA_RUNNER_CATALOG_JSON_PATH, "utf8")) as QaRunnerCatalogEntry[];
}
export function collectBundledQaRunnerCatalog(params?: {
rootDir?: string;
}): readonly QaRunnerCatalogEntry[] {
const catalog: QaRunnerCatalogEntry[] = [];
const seenCommandNames = new Map<string, string>();
for (const entry of listBundledPluginMetadata({
rootDir: params?.rootDir,
includeChannelConfigs: false,
})) {
const qaRunners = entry.manifest.qaRunners ?? [];
const npmSpec = entry.packageManifest?.install?.npmSpec?.trim() || entry.packageName?.trim();
if (!npmSpec) {
continue;
}
for (const runner of qaRunners) {
const previousOwner = seenCommandNames.get(runner.commandName);
if (previousOwner) {
throw new Error(
`QA runner command "${runner.commandName}" declared by both "${previousOwner}" and "${entry.manifest.id}"`,
);
}
seenCommandNames.set(runner.commandName, entry.manifest.id);
catalog.push({
pluginId: entry.manifest.id,
commandName: runner.commandName,
...(runner.description ? { description: runner.description } : {}),
npmSpec,
});
}
}
return catalog.toSorted((left, right) => left.commandName.localeCompare(right.commandName));
}
export async function writeBundledQaRunnerCatalog(params: {
repoRoot: string;
check: boolean;
}): Promise<{ changed: boolean; jsonPath: string }> {
const jsonPath = path.join(params.repoRoot, "scripts", "lib", "qa-runner-catalog.json");
const expectedJson = `${JSON.stringify(collectBundledQaRunnerCatalog({ rootDir: params.repoRoot }), null, 2)}\n`;
const currentJson = fs.existsSync(jsonPath) ? fs.readFileSync(jsonPath, "utf8") : "";
const changed = currentJson !== expectedJson;
if (!params.check && changed) {
fs.mkdirSync(path.dirname(jsonPath), { recursive: true });
fs.writeFileSync(jsonPath, expectedJson, "utf8");
}
return { changed, jsonPath };
}

2
src/plugins/runtime-sidecar-paths-baseline.ts
View File

@@ -2,7 +2,7 @@ import fs from "node:fs";
import path from "node:path";
import { listBundledPluginMetadata } from "./bundled-plugin-metadata.js";
const NON_PACKAGED_RUNTIME_SIDECAR_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab"]);
const NON_PACKAGED_RUNTIME_SIDECAR_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab", "qa-matrix"]);
function buildBundledDistArtifactPath(dirName: string, artifact: string): string {
return ["dist", "extensions", dirName, artifact].join("/");

7
test/openclaw-npm-postpublish-verify.test.ts
View File

@@ -301,7 +301,7 @@ describe("collectInstalledMirroredRootDependencyManifestErrors", () => {
}
});
it("allows npm update compatibility sidecar directories without package.json", () => {
it("rejects private qa sidecar directories that are missing package.json", () => {
const packageRoot = makeInstalledPackageRoot();
try {
@@ -322,7 +322,10 @@ describe("collectInstalledMirroredRootDependencyManifestErrors", () => {
"utf8",
);
expect(collectInstalledMirroredRootDependencyManifestErrors(packageRoot)).toEqual([]);
expect(collectInstalledMirroredRootDependencyManifestErrors(packageRoot)).toEqual([
`installed bundled extension manifest missing: ${join(packageRoot, "dist/extensions/qa-channel/package.json")}.`,
`installed bundled extension manifest missing: ${join(packageRoot, "dist/extensions/qa-lab/package.json")}.`,
]);
} finally {
rmSync(packageRoot, { recursive: true, force: true });
}

2
test/openclaw-npm-release-check.test.ts
View File

@@ -325,6 +325,8 @@ describe("collectForbiddenPackedPathErrors", () => {
]),
).toEqual([
'npm package must not include private QA channel artifact "dist/extensions/qa-channel/package.json".',
'npm package must not include private QA channel artifact "dist/extensions/qa-channel/runtime-api.js".',
'npm package must not include private QA lab artifact "dist/extensions/qa-lab/runtime-api.js".',
'npm package must not include private QA lab artifact "dist/extensions/qa-lab/src/cli.js".',
]);
});

3
test/scripts/bundled-plugin-build-entries.test.ts
View File

@@ -82,6 +82,9 @@ describe("bundled plugin build entries", () => {
expect(artifacts.some((artifact) => artifact.startsWith("dist/extensions/qa-lab/"))).toBe(
false,
);
expect(artifacts.some((artifact) => artifact.startsWith("dist/extensions/qa-matrix/"))).toBe(
false,
);
});
it("keeps bundled channel secret contracts on packed top-level sidecars", () => {