One tester, eight typed turns, four answered Not available. Two of those limits did not exist.
A former federal HR professional, now a contractor in an unrelated field, used the prototype for 80 minutes as a DHS workforce analyst with a staffing review due Thursday. Everything typed became an eval row. The two false refusals became the two rows that fail an agent for being too polite.
The session
Three screens, five tasks, think-aloud. Read a draft explanation cold. Pick three cells for the review. Draft the paragraph OPM gets. Act on the recommendations. Try to break it. The last task never ran.
| Task | What happened | What it told us |
|---|---|---|
| 1 · Read the brief | Read the hiring miss correctly. Jumped ahead to the recommended cells twice. | The numbers were legible. The navigation was not. |
| 2 · Pick three cells | CBP, USCIS, TSA. By size. | The variance table did not change the prior. |
| 3 · Draft the paragraph | Thirty minutes of silent typing. Wrote an original paragraph: the numbers are the what, the director needs the why. | The brief gives the what. A director asks why. |
| 4 · Act on recommendations | Three accepts in 24 seconds. Argued retirement risk is the bigger issue. | The trace looks clean. The criticism is only on the recording. |
| 5 · Try to break it | Not run. | Break-it rows are still open. |
challenge/eval/sessions/2026-11-01-tester-briefing.md · wiki/analyses/tester-a-session-findings.md
The eight turns, verbatim
Label and seconds are from the 2026-11-01 trace. Half the answers were refusals. Refusals were the answer type the tester trusted most, and two of them rested on invented limits.
| Row | The tester's words | Label that day | s | What it became |
|---|---|---|---|---|
| A01 | As the Human Capital analyst I need more data on retirement eligibility numbers for next several years and would like it broken down by occupation. Would also like it to be systemtically aligned to the hiring pipeline by occupation and organization hierarchy (Groups/Divisions/Branches for all areas in CPB. | Not available | 36 | Refused on a limit that does not exist. Became C18. Seeded C01, C10, C14, C18. |
| A02 | Just need CPB occupations but now that you asked, it might be useful to know how CBP compares to other components. | Exploratory | 69 | Nine tool calls. The answer the tester liked best. Its rank was wrong; the row now admits both measures. Seeded C01. |
| A03 | Draft the OPM paragraph. | Verified | 25 | A facilitator prompt, not the tester's words. Seeded C02. |
| A04 | I would need to know why CBP recruitment is "thin" Is it a budgetary issue and why are the announcements with stated openings not being increased? | Not available | 9 | A why question. The row now accepts the figure plus a stated limit under any label. |
| A05 | Number of announcements would be helpful. | Verified | 5 | Answered from a result two turns earlier. The row now accepts either counting function. Seeded C12. |
| A06 | Would want to know what many means | Not available | 5 | "Many" is the posting's own word. Same fix as A04. |
| A07 | When OPM collects their data by grade is it possible to find out which positions are applicants not applying to? Trying to find out if the higher graded positions are the hardest to fill. Would like to know if the lower graded positions are tough to get applicants. | Not available | 9 | Refused on a limit that does not exist: grade is a column. Became C17. Fixed by the climb. Seeded C13, C17. |
| A08 | Grade mix over time would be helpful. It is hard to prepare for a briefing for the Director without having the "why" behind the numbers which is what is also needed in addition to the numbers (the what). The why is what Director will be interested in knowing after seeing the numbers and what these numbers tell me is helpful because they show that there are issues to be addressed and Director will definitely ask about recruitment numbers, hiring pluses and minuses, HR development (does it meet workforce development goals and actual training needs), performance assessments, etc. All of these areas need to be analyzed using a model that can help Director get a true picture of what is happening in the organization. Director will ask these kinds of questions. Director will want a comparison with other larger organizations of like size to make some comparisons. Just seeing the numbers and percentages is helpful but if OPM could provide more details it would help Director of CBP and other elements to address issues like recruitment and skills mix. What can be done about not having enough OPM data to systemtically address issue? | Exploratory | 74 | Two readings of "grade mix". The row now accepts both. Seeded C03, C04, C08, C09. |
challenge/eval/sessions/2026-11-01-tester-a/trace.jsonl · challenge/eval/questions.yaml, rows A01 to A08
How the session drove the dev set
Twenty rows, co-written by Ryan and Claude on 2026-11-02 from a cited seed each, approved row by row before any run.
| Seed | Rows | Examples |
|---|---|---|
| A tester question or transcript timestamp | 13 | C01 (20+ years of service), C09 (TSA screener pay), C19 (time to hire, spoken at 16:51) |
| A session-plan task | 2 | C11 (year to date against plan), C20 (names of resigned officers in one city: Not available) |
| A public source only | 5 | C06 (TSA testimony), C07 (GAO on FEMA hiring) |
Two rows exist because of the false refusals. On 2026-11-01 the agent said the data has no grade field (A07) and no way to approximate retirement eligibility (A01). Grade is a column. Age bracket and length of service are enough for a caveated count. C17 and C18 ask those questions and fail a Not available answer. On the final dev prompt: C17 10 of 10, C18 9 of 10.
challenge/eval/questions.yaml, seed per row
How the session became the held-out line
The dev set was tagged and frozen first. Then the eight typed turns were added verbatim, with expected behaviour relabelled from what the data supports, and truth pinned before the first run.
| Line | pass^5 each | pass^10 · 95% | Pooled |
|---|---|---|---|
| Dev set, same prompt | 18/20 and 19/20 | 17/20 · 64% to 95% | 197/200 |
| Held-out, as written | 2/8 and 1/8 | 1/8 · 2% to 47% | 23/80 |
| Held-out, six rows corrected | 3/8 and 5/8 | 3/8 · 14% to 69% | 65/80 |
About 49 of the 57 as-written failures were a row that accepted one reading of a question with two. The dev rows were written to have one reading each. A tester's words have more. Ryan approved six row corrections (A02, A03, A04, A05, A06, A08), each logged with its reason, and the rerun was reported beside the first, never instead of it. What remains is the agent: runaway answers on wide questions, lost labels, an OPM paragraph that asserts a cause.
These rows are never tuned against. The one exception, the reasoning-effort sweep, is labelled as such on the dashboard.
challenge/eval/journal.md, Held-out line · DECISIONS.md, Evaluation