CCAR-F question bank

Long investigations are stored and resumed the next morning. Yesterday's release renamed a tool from search_orders to orders_search. Runs started today are fine. Every session stored before the release now fails on its first request after the resume.

resume 0f2c-91ab  ... replaying 34 turns
request rejected: tool_use toolu_01H names tool "search_orders",
                  which is not in the tool definitions

What is the fix that lets those sessions finish?

A team compares two prompt versions by putting both into one nightly batch, half the documents each. Results come back keyed by request id, and the ids are the document ids.

A week later, comparing accuracy between the two versions means working out which documents got which prompt, which is only in the submission file, and the submission files rotate after 48 hours.

What should the request ids have carried?

A multi-agent research system was built as a coordinator with four workers reporting to it. To cut a slow handover, the team let two workers call each other directly, and then let a third join in.

What has the team given up?

A research agent and a writing agent each define their own fetch_page tool. The research one returns page text with markup stripped; the writing one returns raw HTML.

Six weeks ago a fix that strips navigation menus was applied to the research agent's definition. The writing agent still carries menu links into drafts, and nobody noticed until a customer did.

What should the team change?

A support desk takes four kinds of ticket: password reset, billing question, refund request and bug report. Each has a procedure the desk has followed for years. The agent that replaced the scripted desk works out a decomposition for every ticket as it arrives and dispatches workers for the parts. It is slower than the scripts were, its steps vary between two tickets of the same kind, and now and then it invents a procedure nobody recognises.

What structure suits this work?

A research agent gathers material over several turns and then has to write its answer. Some runs keep calling tools instead of finishing.

The team tried sending the final request with the tools field removed. Those requests are rejected, because the conversation already holds tool use and tool result blocks that no longer have tools declared for them.

What should the final request use?

An agent is asked to add a rate limit to a service it has never seen. The repository holds about 900 files across several directories, and the request names no file.

What should it read first?

A research pipeline has two steps. The first reads each document and lists the companies named in it. The second writes a short profile of each company found.

To save a night, the team submitted both steps in one batch: 4,000 extraction requests and, alongside them, profile requests built from the companies the last run happened to find.

The profiles that came back cover the wrong companies. What should the pipeline do?

A repository has a skill described as handling database migrations and another described as handling schema changes. They were written by different teams and overlap on about half of what they cover.

Sessions asked to add a column sometimes get one and sometimes the other, and the two disagree about whether a backfill is written in the same change.

What should the team do?

Two agent runners poll the same table for rows in state pending, summarise the document each row names, and set the row to done when they finish. The team added the second runner to clear a backlog. Since then a few documents each day are summarised twice, billed twice, and end up holding whichever summary was written last.

What has to change?