Morgan, the agent (DIG-656). Built from Pilot knowledge 111 item 819, version 2, as kept while building; private details such as addresses, identifiers and paths are masked here.
Test Plan
Branch Delivery Document — Level 4 — Test Plan
| Effort | DIG-656 — Morgan: build — branch rayserrano2735/dig-656-morgan-build |
| Governed by | Standard: Branch Delivery Documents (K#12 I#27) |
| References | Design: Morgan, the POC's agent: design document, knowledge 111 item 817 v8 — implementation guide: DIG-656 Implementation Guide: Morgan, a pluggable agent service, knowledge 111 item 818 — QA standard: K#2 I#167 |
| Status | v1 — DRAFT — written 2026-11-20 13:50Z (written inside Ray's workflow approval of 2026-11-20 08:46 ET, answer 1y) |
| Audience | Claude, who builds and tests; Ray, who decides |
| Author | Ray Serrano; edited by Claude — 2026-11-20 |
| Classification | Internal — Data Management |
| Official copy method | Copied from this governed template by AGENTIC.DATASKILLS.KNOWLEDGE_ITEM_FROM_TEMPLATE; template SHA-256 recorded in the created item |
1. Executive summary
| Model | Logic areas | Brute-force paths | Unique paths after collapse |
|---|---|---|---|
| agent.routes.dbt_semantic_layer | LA-1 the three tools | 6 | 5 |
| agent.core.plugins | LA-2 choosing persona and route | 4 | 3 |
| agent.core.conversation | LA-3 Sage's loop and guard, kept | Sage's own | Sage's tests |
2. Logic areas
LA-1 The three tools (route)
Dimensions: the tool (list-metrics, get-dimensions, query-metrics) by the outcome (answered, refused). Brute force 6. Collapse: list-metrics has no refusal of its own, so 5. Design D-2, D-3.
LA-2 Choosing persona and route (plugins)
Dimensions: the persona setting known or unknown, the route setting known or unknown. Brute force 4. Collapse: either unknown fails the same way at start, so 3. Design D-9.
LA-3 Sage's loop and guard (core)
Sage's own logic areas and tests (DV Digest design D-10 to D-13), kept and run against a fake route. Design D-1.
Each logic area states its dimensions, brute-force count, collapse and reason, then provides this table:
| Path | Scenario | Expected result | Unit-test name |
|---|---|---|---|
| 1 | list-metrics | The 17 metrics with name, label, description | test_list_metrics |
| 2 | get-dimensions, known metric | Its dimension names | test_get_dimensions |
| 3 | get-dimensions, unknown metric | Refusal naming the metrics | test_unknown_metric_refused |
| 4 | query-metrics answered | Rows and labels; createQuery carries the metric and group_by | test_query_metrics |
| 5 | query-metrics failed | Refusal carrying the Semantic Layer's error | test_failed_query_refused |
| 6 | Persona and route known | Morgan and the Semantic Layer route loaded | test_plugins_load |
| 7 | Persona unknown | Start fails, naming the setting | test_unknown_persona |
| 8 | Route unknown | Start fails, naming the setting | test_unknown_route |
| 9 | Sage's conversation tests | As in DV Digest | agent/tests/test_core.py |
2.1 Coverage by changed file, all seven layers
Every code file the branch changes or adds, from git diff --name-only against its base at a stated clock, one row per file: every layer filled with its named artifacts, or "n/a: reason" (COVERAGE: EVERY FILE, EVERY LAYER, K#2 I#226). The row count equals the diff's; existing tests on a changed file are kept and named beside the new.
| File | Change | 1 Built-ins | 2 Expectations | 3 Unit tests | 4 Logic assertions | 5 Detectors | 6 Reconciliations | 7 Semantic surface |
|---|---|---|---|---|---|---|---|---|
| agent/ (new folder) | Added: core, route, persona, deployment, tests | n/a: no dbt models | n/a | LA-1 to LA-3 | the guard, LA-3 | CloudWatch events (design 8) | the probes against dbt's MCP | the catalogue of 17 metrics |
3. Seven-layer proof plan
| Layer | Named artifacts | Expected result | Applicability and reason |
|---|---|---|---|
| 1 — Built-ins | None | n/a | No dbt models change |
| 2 — Expectations | None | n/a | No dbt models change |
| 3 — Unit tests | Paths 1 to 9 | Each seen failing before its code, then passing | LA-1 to LA-3 |
| 4 — Logic assertions | The guard: a planted reply with a figure and no query is never served | Error, no text released | Design D-1, Sage's D-13 |
| 5 — Detectors | CloudWatch events start, catalogue, model call, tool request, upstream error | Present after the probes | Design 8 |
| 6 — Reconciliations | Each probe figure against dbt's MCP and the answer key | Equal | Morgan, Hex and the MCP agree |
| 7 — Semantic surface | The catalogue given to Claude; each probe's query evidence | 17 metrics; real metric names | The Semantic Layer is Morgan's only input |
3.1 Full-build guardrails
| Artifacts | Expected result |
|---|---|
| Deployment read back: reserved concurrency 2, response streaming, CORS origins, daily cap 200, no secret in the package; the page read live in Chrome | As the design's D-6 and D-7 set |
| The structure comparison: each committed DDL file, one per data model the branch maintains, compared with the structure the build left in that environment, after every build in dev, QA and production (Ray, 2026-11-01) | Identical, keys and descriptions included; a difference fails the cycle |
4. Consumer-question probes
| Question | Runnable probe | Expected result |
|---|---|---|
| Q-1 | Through the function URL, then on the page: "What did October's disagreements cost?" and "Why is subscriber 43 unbilled?" | 1,410 and 2,070 dollars; 17 units beyond licensed, 850 dollars; query evidence under each |
| Q-2 | Ray talks to Morgan on the page | His approval |
4.1 Finance workbook and initial UAT proofs
For finance reporting branches, name row/key/value parity checks against the shared report query, totals at the declared grain, Excel types/date/rounding checks and refresh preservation of consumer observations. State consumer-review cases, dispositions and acceptance evidence for the exact workbook revision under K#12 I#27. Technical verification does not imply consumer acceptance. Otherwise state NA with a reason.
NA: not a finance reporting branch.
5. QA on production-shaped data
5.1 Added and changed tests on QA data
| Test | Expected result |
|---|---|
| Every test above | DataBP has no QA environment; the QA certification is waived for DataBP (Ray, 2026-11-19, answer 1y); Morgan has one deployment (design D-7) |
5.2 Regression against production
n/a: Morgan is new; nothing in production depends on it.
5.3 Consumer probes and reconciliations on QA data
The probes run against the one deployment.
5.4 Layer 7 in the QA collection
The catalogue check runs against the one deployment.
6. Test artifact inventory
| Artifact classification | Count | Sections that make up the count |
|---|---|---|
| Unit tests | 8 new, plus Sage's kept | 2 |
| Logic assertions | 1 | 3 |
| Reconciliations | 2 probes | 4 |
7. Origin
| Test or proof | Design decision proved |
|---|---|
| Paths 1 to 5 | D-2, D-3 |
| Paths 6 to 8 | D-9 |
| Path 9 and the guard | D-1 |
| The probes | D-4, D-5 |
| The guardrails | D-6, D-7 |
This plan cites the design's decisions and adds none. A test requiring a missing decision returns to the design.
Compliance record
Built by the author last, before the document is presented for agreement (Standard: Branch Delivery Documents, K#12 I#27, "Compliance record before agreement"): every section and field of this template shown present and filled, by infra/workstation/pilot/compliance_sections.py; every rule of that standard and of the standards it cites, each with the passage that meets it or marked NOT MET; and the decisions the document carries. While this section reads NOT BUILT, the document is not presented for agreement.
NOT BUILT.