Voice-Controlled Robotics Agent

Overview

You speak a command like "stack the red cube on top of the blue cylinder" or "arrange all objects in a line", and a Franka Panda arm carries it out in real time in a MuJoCo physics simulation.

Deepgram transcribes what you said, Claude Sonnet breaks the request into an ordered list of sub-goals, and the arm works through them one action at a time. Gravity, collisions and contact forces are all live, so things genuinely go wrong: a grip slips, a stack tips over, a cylinder rolls off. The agent reads the new scene on the next step and plans around what actually happened.

When the model says the task is done, that is treated as a proposal, not a result. A second independent call compares the live scene against the intended arrangement and can reject the claim.

Commands it handles:

• Pick and place to a named location or absolute coordinates
• Stacking, including order-specific requests like "reverse the stack"
• Spatial arrangements, lines, rows, corners, grids
• Relative instructions like "move it to the other side" or "put it next to the cylinder"
• Multi-object tasks like "move all cylinders to the left side"

Demo

The agent taking spoken commands and working through them end to end.

How it works

Five stages with a loop in the middle. The agent cycles between acting and checking until the scene matches the request, or until it decides the task cannot be done.

Claude Sonnet does all four jobs: the decomposition, the decision at each step, the goal check, and any replan. The part that matters is that it runs once per action rather than once per task. Every step it gets the live scene again, so it can read that the cube it just placed is back on the table and deal with that, instead of following a plan that stopped being true three steps ago.

Voice runs through Deepgram, Nova-3 for speech to text and Aura-2 for text to speech. Without a Deepgram key it falls back to faster-whisper and pyttsx3, both running locally.

Planning and decomposition

What the agent does at each step is decided by one system prompt, sent to Claude alongside a text description of the live scene. Each section of it has a job:

• Available actions, pick, place, move, open/close gripper, wait, report success/partial/failure, and ask the user a clarifying question
• Coordinate system, axes and reachable workspace bounds, so the agent never targets a position the arm physically cannot reach
• Table placement rules, how to interpret "left side", "the other side", "across the table"
• Pre-computed arrangements, exact safe coordinates for lines, rows, corners, and grids, so the agent looks them up instead of guessing
• Rules, the edge cases: ambiguous references, stacking limits the physics won't allow, and reporting honestly when only part of a task got done
• Strict output format, every response must be {"reasoning", "action", "parameters"} so each step is machine-parseable

Problem: "move everything to the other side" ran forever. The agent moved an object across, looked at the scene again, and from where the object now sat "the other side" was where it came from, so it moved it back, over and over until the step budget ran out.

Solution: relative wording is resolved once, at planning time, into an absolute side of the table, and the plan the agent sees each step is that resolved one. The scene description backs this up by labelling every object with a fixed side instead of a relative one. On top of that a guard counts re-grasps of objects that were already placed successfully and stops at three, while a re-grasp after a failed placement is a legitimate retry and is not counted, which an earlier version got wrong and gave up on dropped objects after one try. A three-object move went from 30 steps and failing to 7 steps and done.

That system prompt is around 10,800 characters and goes out with every per-action call, so it is cached with a five minute breakpoint. After the first call of a task it costs roughly 3 input tokens instead of 2,600.

When a plan fails

Plans fail here all the time, because the physics is real. The gripper closes a fraction too early, a stack tips, an object slides out of reach. Every failure is classified by type first, since a slipped grasp and an unreachable target need different answers.

Problem: after a failed placement the agent would move on to the next object and leave the task half done.

Solution: at first the failure went into the next prompt as a generic "that didn't work, try again", and the model ignored it. What worked was making the instruction specific and blocking, then escalating if the same object keeps failing:

• Retry, and nothing else. The placement primitive names the object that failed, and the next prompt tells the agent to grasp that one again and touch nothing else until it is placed.
• Twice failed, change the approach. Move the wobbly piece to a temporary spot, put the problem object down, then rebuild.
• Off the table, stop trying. An object that has left the table is marked unreachable, so no retries are spent on it.
• Four failures, replan. The agent replans from the current scene, and after two replans it reports failure rather than looping.

The piece I would defend hardest is the check on success. A separate call compares the live scene to the original request, and it is strict: two objects side by side are not a stack, a toppled tower is not a finished one. If it disagrees, the missing piece goes into the next prompt, up to three times, and then the agent reports what it actually managed.

Some requests are impossible, and saying so is the honest answer. A third cylinder on a two-cylinder stack falls every time: holding a 5 cm cylinder puts the fingers right at the edge of the one below, so opening them shoves it sideways. A sphere never stays on top either. Fifteen or so grip and release strategies later, both are hard limits in the planner. Ask for a four cylinder tower and the agent builds two, leaves the rest, and tells you why.

Motion and collisions

Arm motion goes through inverse kinematics: mink, backed by the daqp solver, works out the joint velocities that move the gripper toward a target point.

Problem: mink solves for the gripper pose and nothing else. It knows nothing about the arm's own links, the table, or the objects on it, so a straight line to the target will drag the arm through a stack on the way.

Solution: every motion routes through a transit arc instead of driving straight at the target.

lift straight up  →  travel at a safe height  →  descend onto target

The travel height is the lowest one that clears the tallest object on the table, clamped between TRANSIT_Z_MIN and TRANSIT_Z_MAX. Staying low is deliberate: higher up the wrist tilts, and about 22° of tilt is enough to roll a friction-held cylinder out of the fingers mid carry. Placement approaches are capped by APPROACH_CAP_Z at 0.62 m so the arm never has to climb above that band on the final leg. On the descent onto a stack the wrist orientation is locked as well, since position-only IK let it spin 10 to 15° about its own axis, which twisted a held cylinder until it was lying horizontal. A placement geometry that failed three times out of three worked three out of three after the lock.

This is routing around the problem rather than solving it. A collision-aware planner would find shorter paths; the arc is conservative and often takes a longer way round than it needs to.

The web portal

The portal is a FastAPI app with a plain JavaScript front end, using three transports for the three things it has to carry. The simulation video is an MJPEG stream at about 30 fps, state goes over a WebSocket as a JSON snapshot five times a second, and commands and scene edits are REST calls.

It started as a Streamlit app that pushed each video frame as base64 and choked at 8 to 12 fps. Splitting the video and the state onto transports built for them fixed that without touching any of the simulation, agent or voice code.

The page shows the plan as a checklist, the current action in plain language with the reasoning behind it, and every object's live coordinates. You can add and remove objects mid session, switch layouts, stop a running task between actions, and export the whole session as JSON.

Tech stack