Gustavo Madeira Santana
82fe6f50ef
QA: organize scenarios by theme
2026-04-17 11:03:47 -04:00
Gustavo Madeira Santana
bb7e9823a8
QA Lab: add provider registry
...
Move mock and live provider behavior behind provider-owned definitions so suite, manual, Matrix, and transport lanes share defaults, auth staging, model config, and standalone server startup.
Add AIMock as a first-class local provider mode while keeping mock-openai as the scenario-aware deterministic lane.
2026-04-16 21:24:59 -04:00
Gustavo Madeira Santana
56a9fd4b34
QA Matrix: capture full runner output
2026-04-16 16:18:54 -04:00
Peter Steinberger
20cce166ef
test: isolate Docker live profile-key auth
2026-04-15 06:31:20 -07:00
Gustavo Madeira Santana
85eac42d34
QA: remove runner install fallback catalog
...
Drop the generated qa-runner catalog and the missing/install placeholder
path for repo-private QA runners. The host should discover bundled QA
commands from manifest plus runtime surface only.
Also trim stale qa-matrix install docs and package metadata so the
source-only QA policy stays consistent.
2026-04-14 17:37:18 -04:00
Gustavo Madeira Santana
82a2db71e8
refactor(qa): split Matrix QA into optional plugin ( #66723 )
...
Merged via squash.
Prepared head SHA: 27241bd0895599352+gumadeiras@users.noreply.github.com >
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com >
Reviewed-by: @gumadeiras
2026-04-14 16:28:57 -04:00
Peter Steinberger
a88c6f0fe7
fix: bound live video generation smoke
2026-04-14 14:59:01 +01:00
Josh Avant
3d07dfbb65
feat(qa-lab): add Convex credential broker and admin CLI ( #65596 )
...
* QA Lab: add Convex credential source for Telegram lane
* QA Lab: scaffold Convex credential broker
* QA Lab: add Convex credential admin CLI
* QA Lab: harden Convex credential security paths
* QA Broker: validate Telegram payloads on admin add
* fix: note QA Convex credential broker in changelog (#65596 ) (thanks @joshavant)
2026-04-12 22:03:42 -05:00
Tak Hoffman
c37e49f275
Add /trace toggle and fix Active Memory diagnostics
2026-04-12 13:20:22 -05:00
Marcus Castro
000fc7f233
refactor(qa): add shared QA channel contract and harden worker startup ( #64562 )
...
* refactor(qa): add shared transport contract and suite migration
* refactor(qa): harden worker gateway startup
* fix(qa): scope waits and sanitize shutdown artifacts
* fix(qa): confine artifacts and redact preserved logs
* fix(qa): block symlink escapes in artifact paths
* fix(gateway): clear shutdown race timers
* fix(qa): harden shutdown cleanup paths
* fix(qa): sanitize gateway logs in thrown errors
* fix(qa): harden suite startup and artifact paths
* fix(qa): stage bundled plugins from mutated config
* fix(qa): broaden gateway log bearer redaction
* fix(qa-channel): restore runtime export
* fix(qa): stop failed gateway startups as a process tree
* fix(qa-channel): load runtime hook from api surface
2026-04-12 15:02:57 -03:00
Eva
fd45ea2bf1
test(qa): add compaction retry parity scenario
2026-04-11 14:22:48 +01:00
Eva
55df6f11a4
fix: harden parity gate review findings
2026-04-11 14:22:48 +01:00
Eva
c73d005c7a
docs: clarify parity verdict interpretation
2026-04-11 14:22:48 +01:00
Eva
db09edacfc
qa-lab: gate parity on shared scenario coverage
2026-04-11 14:22:48 +01:00
Eva
67fdd3b4df
benchmarks: add agentic parity report gate
2026-04-11 14:22:48 +01:00
Eva
79f539d9ce
docs: clarify GPT-5.4 parity harness and review flow
2026-04-11 14:22:48 +01:00
Gustavo Madeira Santana
25445a9f2e
qa-lab: add Matrix live transport QA lane ( #64489 )
...
Merged via squash.
Prepared head SHA: ae9bb377515599352+gumadeiras@users.noreply.github.com >
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com >
Reviewed-by: @gumadeiras
2026-04-10 19:35:08 -04:00
Peter Steinberger
46a6746bca
docs: clarify codex harness validation
2026-04-11 00:13:08 +01:00
Peter Steinberger
6281dd7379
perf: reduce test import overhead
2026-04-10 23:09:37 +01:00
Peter Steinberger
c59fc764db
docs(codex): document harness command smoke
2026-04-10 23:07:25 +01:00
Peter Steinberger
972ed139a7
fix: make docs anchor audit use Mintlify CLI
2026-04-10 21:39:52 +01:00
Peter Steinberger
c16b1b7433
docs: document harness fallback policy
2026-04-10 21:27:26 +01:00
Peter Steinberger
bfc0889776
docs: document Codex harness plugin workflow
2026-04-10 21:22:16 +01:00
Peter Steinberger
3027efaf21
test: raise QA suite default concurrency
2026-04-10 13:45:57 +01:00
Peter Steinberger
4c14f55c62
test: parallelize QA suite scenarios
2026-04-10 13:45:57 +01:00
Peter Steinberger
0b0c062e97
fix: avoid Claude CLI subscription prompt classifier
2026-04-10 10:52:35 +01:00
Shakker
655cfb477a
docs: clarify multipass live auth support
2026-04-09 23:53:13 +01:00
Shakker
a04c331cc1
docs: document qa multipass runner
2026-04-09 23:53:13 +01:00
Mason
164287f056
docs-i18n: avoid ambiguous body-only wrapper unwrap ( #63808 )
...
* docs-i18n: avoid ambiguous body-only wrapper unwrap
* docs: clarify targeted testing tip
* changelog: include docs-i18n follow-up thanks
2026-04-10 00:01:17 +08:00
Peter Steinberger
cca9e5b914
test: cap broad live model sweeps
2026-04-09 01:37:55 +01:00
Peter Steinberger
d51f527cca
feat: add gh-read GitHub app helper
2026-04-08 00:09:07 +01:00
Bruce MacDonald
86f35a9bc0
chore(ollama): update suggested onboarding models ( #62626 )
...
Merged via squash.
Prepared head SHA: 48c083b88a5853428+BruceMacD@users.noreply.github.com >
Co-authored-by: BruceMacD <5853428+BruceMacD@users.noreply.github.com >
Reviewed-by: @BruceMacD
2026-04-07 11:42:29 -07:00
Peter Steinberger
e973275fd0
fix: harden claude-cli live switch smoke
2026-04-07 16:05:54 +01:00
Peter Steinberger
9d4b0d551d
fix: support inferrs string-only completions
2026-04-07 15:55:20 +01:00
Peter Steinberger
1cec37184c
fix: harden qa memory dreaming sweep
2026-04-07 12:57:33 +01:00
Peter Steinberger
4d4dbe8e15
test: share live probes with acp bind
2026-04-07 10:35:24 +01:00
Peter Steinberger
c2f9de3935
feat: unify live cli backend probes
2026-04-07 10:35:24 +01:00
Peter Steinberger
65f9fc397e
perf(test): split support boundary shard
2026-04-07 09:12:26 +01:00
Peter Steinberger
0af808b457
test: add cli backend live matrix metadata
2026-04-07 09:06:09 +01:00
Peter Steinberger
37dccb52ed
test: add gemini acp bind docker coverage
2026-04-07 07:59:45 +01:00
Peter Steinberger
ce1d2c1004
test: cover claude and codex acp bind docker smoke
2026-04-07 06:06:29 +01:00
Peter Steinberger
191f867ef6
perf(test): split ui and bundled full-suite shards
2026-04-07 00:39:05 +01:00
Peter Steinberger
f9c721d5bf
fix: add vydra kling live lane
2026-04-06 19:47:43 +01:00
Peter Steinberger
7a736bff90
perf(test): split reply queue seams and unit shards
2026-04-06 19:31:20 +01:00
Peter Steinberger
ec55902989
perf(test): tighten reply fast paths and split unit shards
2026-04-06 19:23:17 +01:00
Peter Steinberger
41ea5316aa
test: add shared media live harness
2026-04-06 19:15:31 +01:00
Peter Steinberger
b4e1747391
feat: add one-command qa lab docker launcher
2026-04-06 17:47:17 +01:00
Peter Steinberger
0335a8783c
perf(test): shard full vitest runs
2026-04-06 17:34:11 +01:00
Peter Steinberger
8a33a8d607
perf(test): trim runtime lookups and add changed bench
2026-04-06 16:49:28 +01:00
Peter Steinberger
0c5e6037b0
fix(openai): clarify auth routes in picker and docs
2026-04-06 16:14:51 +01:00
