How to use Claude to pick blocks with camera feedback
Claude Code used camera feedback to place three blocks in a taped target after calibration stalled; Rob Bercik supplied hardware changes and inspection.
Original prompt
Shared by the source author. See source notes for context.
your new goal is to place the purple, red and blue blocks within the blue marked tape drop area.
Reach people exploring AI agents
See placements →What happened
All three colored blocks reached the taped drop area in the reported experiment; arrangement and earlier force errors limit claims about reliability.
Reported by Rob Bercik — Personal blogHow the agent did it
- Codex prepared the control stack and dashboard while Bercik addressed the robot’s power-supply problem.
- Bercik redirected Claude from geometric calibration toward camera-guided gripping and target placement.
- Claude iterated on physical attempts after a forceful mistake, while the owner improved visual markers and inspected the final placement.
Where a person stepped in
Bercik assembled and modified the hardware, supplied calibration measurements, changed markers and the task, and inspected the overnight result; an early attempt almost loosened the clamp.
Rob Bercik gave coding agents access to a LeRobot SO-101 arm with overhead and wrist cameras. Codex prepared the Raspberry Pi control stack, dashboard and torque controls. Serial failures during movement limited early work until the power supply was replaced. Claude Code then continued the manipulation task.
A geometric calibration approach was frustrating. Bercik measured poses, but the arm was not actively correcting its motion from camera feedback. He redirected the work toward a concrete objective: identify a block visually, grip it and place it in a marked area.
The first gripping attempt pushed too hard against the table and nearly loosened the mounting clamp. The agent corrected its calculation. Human changes also mattered: Bercik replaced frayed white tape with red tape so the gripper edge was easier to see, then added blue and purple blocks.
By the following morning, all three blocks were inside the taped target. The account supports a completed pick-and-place experiment, with imperfect arrangement and an unsafe early attempt. Bercik supplied the hardware, measurements, task changes and physical inspection; reliable autonomous manipulation or later training goals were not established.
Prompt
Original prompt
Source: Personal blog · Rob Bercik
Frequently asked questions
What task finally worked?
Placing the purple, red and blue blocks inside the taped area.
Did the robot work safely from the start?
No. An early gripping attempt pushed too aggressively against the table.
What did the owner contribute?
Hardware setup, measurements, clearer visual markers, task direction and physical inspection.
More details
About this account
Codex prepared the control stack and dashboard while Bercik addressed the robot’s power-supply problem. Bercik redirected Claude from geometric calibration toward camera-guided gripping and target placement.
Original HTML account recovered and read through HTTP. Claude performed the final manipulation; Codex prepared the stack and later video. The original prompt excerpt names the final objective. Results are one owner-reported experiment with an unsafe early attempt. Original prompt excerpt.
Credit: Rob Bercik — Personal blog
Source published 25 Sept 2026
Source checked 14 Oct 2026
Tools and topics
Claude Code · Codex · LeRobot SO-101 · Raspberry Pi 3 · Cameras
robot manipulation · camera feedback · physical testing
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.