How to work with AI, October 16. Four tips: a check instead of a rule, and edits by comment

Today I have four tips. You will learn how to make AI follow a rule every time, not every other time. You will also see why it is better to fix an agent's answer with a comment. I also cover how to build a test from AI mistakes and how to bring an agent into work like a new employee.

The tips come from developer Geoffrey Huntley and the people who build Codex at OpenAI. You do not need to be a programmer. For each tip, I added my opinion and an example from a dealership.

Do not write a rule for AI, set up a check that stops the mistake#

On October 13, Simon Willison shared a method from developer Geoffrey Huntley on his blog. Huntley's AI agent writes code and forgets to use the right module. He could add a rule to the agent's instructions, but such rules drift away from real work over time.

So he set up an automatic check that runs before the work is saved. If the agent skips the module, the check stops the work and tells the agent what to fix. Huntley writes that now the agent follows the rule ten times out of ten.

In simple words. AI does not always follow a rule in its instructions. A check that stops the mistake works every time.

My opinion. In a dealership, an agent answers leads, and its instructions say "always give the full price with all fees". It does this every other time. The manager writes the rule in bold, and nothing changes.

I would set up a check before sending. "Before you send, check the reply to the customer. If it has a price, compare it with the price list and make sure it includes all fees. If not, do not send it and write what to fix".

Such a check also catches mistakes by people. AI does not replace the manager. It shows who did not do the work.

Source Simon Willison's Weblog

Do not rewrite an AI answer yourself, leave a comment on what to change#

On October 15, Dan Shipper from Every told the story of how OpenAI built its Codex agent. He talked to the team. Dominik Kundel from OpenAI told him that he now makes almost all changes with comments inside the agent's work.

He does not edit by hand. He writes what to change, and the agent does it. Designer Andrew Ambrosino says you may still need to steer the agent, but you may no longer need to edit its work yourself.

In simple words. When you rewrite an AI answer yourself, the AI does not learn what was wrong. When you write a comment, the AI makes the change itself, and faster than you.

My opinion. A salesperson gets an AI draft of a reply to a customer about a trade-in. The draft is fine, but the tone is dry and there is no question about the mileage. The salesperson deletes half and writes it again. Tomorrow, the AI makes the same mistake.

I would write a comment. "Make the tone warmer, like you write to a friend. Add a question about the mileage and about any accidents. Remove the second line about the offer".

Comments that come back again and again during the week, I would move into the agent's instructions. AI prepares, a person decides and is responsible.

Source Every

Build a set of test questions from AI mistakes and run it after every change#

In the same article, OpenAI researcher Amelia Glaese says that trusting your feelings is not useful in research. Her colleague Michelle Pokrass, as the author describes, watched real people work with the model. The team turned the places where the model failed into tests they could repeat.

In simple words. "It seems better" is not a test. Write down the cases where AI made a mistake and check each new version on them.

My opinion. A manager changes the agent's instructions and the next day says "it is better now". Nobody knows why it is better. A week later, the agent mixes up trim levels again.

I would keep a table of twenty real customer questions where the agent made mistakes. For example, "How much per month is a Camry on a loan with no down payment". Or "Will you take my 2015 car", "Can I do a test drive on Sunday at nine". After each change to the instructions, I would run all twenty and count how many answers became correct.

Source Every

Work next to the agent in the first week, then give it whole tasks#

Codex product lead Alexander Embiricos, as Shipper writes, advises to work closely with an agent at first, like when a new person joins the team. Later, when you know which tasks the agent can handle, you give it those tasks fully. Shipper himself writes that he leaves hard tasks to the agent overnight once he trusts it.

In simple words. At first, watch how AI works and correct it. When you see that it can do the task, give the task to it fully.

My opinion. A dealer sets up an agent to answer leads at night and gives it everything at once. A week later, the dealer finds out that the agent booked test drives at times when the store is closed.

In the first week, I would read every agent reply in the morning and review it like the work of a new salesperson. Then I would give it the tasks it did without mistakes, for example visit confirmations. Test drive booking and price talks would stay under review for one more week.

Dealer data from different countries shows that 40% of inquiries come outside working hours. So a night agent is worth this week of checks.

Source Every

What to watch#

These are new videos from the last two days. I have not reviewed them yet, so the notes come from the title and description.

• OpenAI. GPT-Live-1.1 has new voices and better accents, an update to voice mode in ChatGPT with new voices in six languages
• OpenAI. Build more personal sites with Codex, how to make a page that updates itself on a schedule
• AI Engineer. Your AI is not the problem. Your infrastructure is, Jiachen Jiang from Coder on why a rule in a chat does not stop an agent that has too much access
• AI Engineer. How to save on tokens, Alvin Pane from OutRival on how to compare a cheaper model with the one you use now
• No Priors. The changing physics of cybersecurity and drug development, a talk with OpenAI President Greg Brockman
• OpenAI. How Oracle uses Codex to help business users get answers, how staff describe the result they want and the agent builds the report