# Goof-E, MK I: I Built Tony Stark's Clumsiest Robot for My Desk > A low-cost LeRobot SO-101 arm driven by an AI agent and built in a few weeks By Preetam Joshi · 2026-11-03 · https://www.preetamjoshi.com/writing/goof-e/ # Intro I have been taking a class for the past six weeks: ["The Future of Robotics"](https://continuingstudies.stanford.edu/courses/professional-and-personal-development/the-future-of-robotics/20261_TECH-80) , taught by [Steve Cousins](https://profiles.stanford.edu/steve-cousins) as part of Stanford's continuing studies program. Steve talks about how sci-fi movies inspire real life innovations. For me personally, [Iron Man movies](https://en.wikipedia.org/wiki/Iron_Man_in_other_media#Film) (especially from MCU) have been a source of fascination. It would be super cool someday to push a button on your watch to have billions of gold-titanium nano-particles flow over your body to morph into a super-cool body suit that can take flight and get you to your workspace in minutes (ref: [the bleeding edge armor](https://en.wikipedia.org/wiki/Iron_Man%27s_armor#Bleeding_Edge_Armor))! I have been generally interested in Robotics, however, tinkering/hacking in this field has not been very accessible. Open Source platforms like [LeRobot from HuggingFace](https://huggingface.co/lerobot) have made it quite easy nowadays to get hack on robots. As a personal goal as part of this course, I wanted to get my hands dirty by assembling and programming a robot from scratch. I decided to start small and built a simple version of the clumsy robot [Dum-E](https://www.youtube.com/watch?v=_dsG4h1FjIo). *Remember the scene where Dum-E saves Tony when Obadiah Stane steals Tony's original arc reactor?* So here is a short video of "Goof-E, MK I" - a clumsy desk robot inspired by Dum-E (edited by the lovely and extremely talented [Manmeet Kaur](https://www.youtube.com/channel/UCUtb95WNTsAWZzOtmJEMk1w/about)):
Hope you enjoyed it! # Building Goof-E (MK I) The goal of this section is to help someone who wants to get started on building a robot on their own. I will share details on the hardware: how I sourced the parts, assembled them and calibrated the arms and the software: the workflow, models used and the agent powering the robot arm. I have also open-sourced the [git repo](https://github.com/pjoshi30/goof-e) that contains all relevant code used by Goof-E (details below). ## Assembling the Hardware I used the [SO-101 platform from LeRobot](https://huggingface.co/docs/lerobot/en/so101). The project page has a lot of resources on how to source parts. I personally went with the "DIY Kit / Unassembled" option on [HiWonder](https://www.hiwonder.com/products/lerobot-so-101?variant=42198960144471&srsltid=AYCX9vIBwrkV3vYPqxBIJUItsavVyk1_VTBt7m0Gvw8zqYygbGG8eYFa). The unassembled kit allows you to understand each part as you assemble the robot. This is what comes in the box: ![Unassembled Raw Parts of the SO-101 arm](unassembled_raw_parts.png) The package includes 2 Robot Arms: a leader and a follower arm (the arm in the video is the follower arm), motors for each arm, webcams and power adapters. It also comes with clamps but I instead went with heavy duty metal spring clamps ([Amazon link](https://www.amazon.com/dp/B0C9SQWRVT)) because they felt sturdier than the clamps shipped with this package. I also bought a C920 Logitech 1080p webcam ([Amazon link](https://www.amazon.com/dp/B006JH8T3S)) that became the "eyes" of the robot arm. To assemble the robot, the [instructions provided in this page](https://huggingface.co/docs/lerobot/so101) are fairly accurate. The parts that are missing/unclear is which motors go into which arm, the power adapter required for each arm and wiring of the motors in the arm, once built. I used Claude and ChatGPT extensively to fill in the gaps. The other tool that was super helpful in this assembly was a Husky Precision Screwdriver set ([Home Depot link](https://www.homedepot.com/pep/Precision-Driver-Kit-25-Piece-432360240/312980352)). Screwdriver with a magnetic tip is super helpful when handling tiny screws. Each arm comes with a servo driver board. These boards are essentially a translator between your computer and the motors. It lets one USB-C cable control 6 motors on each arm. The board has a USB-C port and a separate barrel jack (that is used to draw power needed by the motors because USB-C cannot meet the power needs). These servo boards mount nicely in a little slot at the base of the arm. At the end of assembly, you end up with the follower arm: ![Follower Arm](follower_arm.png) and the leader arm (mainly used for calibration and teaching poses): ![Leader Arm](leader_arm.png) This is the whole desk setup: ![Full Desk Setup](full_desk_shot.png) Notice the webcam mounted on the boom arm on the top left corner of the desk, the arms' servo driver boards are connected to a USB-C 7-port hub with a DC power adapter. This hub has multiple USB ports (5Gbps+ data transfer across all 4 ports) and a dedicated power which is needed for the webcam - I needed it since my Macbook Air only has two USB-C ports ([Best Buy link](https://www.bestbuy.com/product/j5create-usb-c-7-port-hub-5a2c-with-dc-power-adapter-space-gray/J36445R6FK/sku/6679836)). Also make sure to buy a couple of extension cords - together all components on my desk need about 8 power points. ## Calibration In order to ensure reasonable accuracy of the tool tip (typically within 4mm of the target), calibration is an important step. Below is a list of components that needed calibration: 1. Motor and Arm Calibration The video and the steps described in the [LeRobot SO-101 arm page](https://huggingface.co/docs/lerobot/main/en/so101#calibration-video) is mostly sufficient. However, since we use degree mode in LeRobot `use_degrees=true`, LeRobot does not cap commands to the calibrated joint range. The servo has a limit and it would stop but silently. I added [custom code](https://github.com/pjoshi30/goof-e/blob/main/deskbot/arm.py#L94) that clamps every target 3 degrees inside that range. 2. Camera to desk Calibration This step is needed to help the arm determine the exact position on the desk from an image taken from the webcam mounted on the boom arm. To do this, I taped a printed sheet with ArUco markers that give a pixel-to-desk-centimeter map accurate to about 0.5mm. This is what the sheet looks like: ![Sheet with ArUco markers](aruco_sheet.png) 3. Arm to desk Calibration Using the leader arm, I guided the tip of the leader arm (using teleop) to touch the known points in the sheet while the code recorded the joint readings, then it fitted where the arm sits and its small joint errors. This process resulted in an average error of 4.2mm. The picture on top that shows the whole desk setup was taken during this arm-to-desk calibration session. I had a couple of observations during this process: * The first fit looked great but it placed the arm's base about 3.8cm under the desk with a 7cm long tip (yes I used a small measuring tape). The fix was to add some obvious common sense constraints like joint offsets, base height and tip. See [this code](https://github.com/pjoshi30/goof-e/blob/main/deskbot/kinematics.py#L189) for more details. * I tried a "hover 1cm above" test, but the arm touched the desk in this case. This was because the servos sag under gravity upto 7mm low on far reaches. The fix was: after each move, re-read the joints and correct accordingly. This piece of [code is the fix](https://github.com/pjoshi30/goof-e/blob/main/deskbot/arm.py#L220). In general, the teleop of the arm with the leader arm takes a bit of practice to get used to. ## Perception The Logitech C920 webcam is mounted directly on the boom arm faces the table directly from the top. Each pixel maps cleanly on to a spot on the Desk. This helps map a command like "Where is the pen?" into flat 2D geometry. For this particular setup, we dont have a depth camera or a wrist camera on the arm. This is an example of an image (the object detection use classic vision plus a Claude model for reading the image): ![First Look at the Desk](simple_vision_ex_1.png) Notice anything strange in this picture? *Yes*, the power bank is being identified as two objects, the airpods are not being detected because of the white background. This happens because this first iteration of the algorithm, compared any given live image with a photo of the empty desk and whatever changed between those photos is considered an object. It measured position, size and grip angle precisely but touching objects were merged into a single blob. I needed a better algorithm. I ruled out SAM (Segment Anything) because it was too slow to run on a Mac. I instead chose [Gemini Robotics-ER2](https://deepmind.google/models/gemini-robotics/embodied-reasoning/) which is Google's robotics tuned model. It does not give an outline so its points seed a split of "something changed here" area, cutting touching objects apart along their edges. Their free tier is rather low, its easy to exhaust the 20 requests per day limit when iterating on robotic projects. The illustration below shows how the simple vision + Claude detection approach compares to Gemini Robotics-ER2: ![Side by side comparison simple v/s Gemini Robotics-ER2](simple_robotics_er_2.png) I still have the first simple algorithm as a fallback when Gemini Robotics-ER2 fails or is not available. Another observation here is that lighting needs to be consistent. Daylight v/s night time lighting can affect simpler algorithms. However, the Gemini model is quite robust. ## The Brain - an agent that plans, talks and emotes Goof-E's brain is an agent that listens, understands, plans and acts using tool calls. Here is the overall workflow: ![Agent Workflow](agent_workflow.png) This entire workflow was built in a few days by Claude Code (using Opus 5.5). It uses the Whisper model (MLX version that runs on a Mac with Apple Silicon: [HuggingFace Link](https://huggingface.co/mlx-community/whisper-large-v3-turbo)) to transcribe speech to text. You can see the relevant code [here](https://github.com/pjoshi30/goof-e/blob/main/deskbot/voice.py#L9). The agent decides which tool to call: look at the desk, move an object, remember a place, forget a place, express an emotion. An agent allows the arm to correct itself midtask and also interact with the user through the user interface. The user interface (as seen in the video) is a simple Jarvis like visualization with a chat panel on the right. This interface serves two purposes: a) two-way conversation between the user and the robot and b) to explain what the robot is perceiving, its plans and what actions it is taking. The agent also emotes - a happy and a sad expression. Last but not least, the sound of the motors when the arm is operating is not very pleasing to the ears. In the Stanford class, [Catie Cuan](https://catiecuan.com/) gave a guest lecture on choreorobotics - one of my favorite things was this concept of [music mode](https://youtu.be/MJhesGakX1w?si=K-jSABPZVvyWEvOp&t=663) which essentially is a software that plays music while the robot moves. I attempted to create the most basic version of that beautiful symphony by making the agent play soothing music at the same time when the arm starts moving - this helps drown out the noise from the motors. While the arm works for several "happy paths", it has many limitations and is rather rudimentary at this point. As you can see in the video, the arm picks up an object and places it at the edge of the desk and the object falls off the desk. The robot isn't able to predict that placing the object at that location would lead to the object falling off the desk. ## Lessons and What's next A few key takeaways from this exercise for me were a) Opus class off-the-shelf models aren't a great fit (at this time) for object detection for robotics, instead a robotic-tuned model made a big difference b) perceiving the physical world (size/weight of objects, smoothness of objects etc.) continues to be a hard problem, c) adding an agentic loop with a planning step helped avoid the silly failures and finally d) personality matters: the interface, music, emotions and the voice (NLG) makes people smile and relate to the robot. *So what's next?* A couple of things on my mind: First, I want to do a proper evaluation of this setup to understand the types of failure and success modes. Second, I want to explore real-to-sim (like [Homebody](https://tml.stanford.edu/homebody/)) to build a digital twin for this Desk setup. I'll end with a rather abstract thought that occured to me as I wrapped up this exercise. Its a quote from [Panchatantra](https://en.wikipedia.org/wiki/Panchatantra) which is an ancient Indian collection of interrelated animal fables in Sanskrit:

वरं बुद्धिर्न सा विद्या विद्याया बुद्धिरुत्तमा।
बुद्धिहीना विनश्यन्ति यथा ते सिंहकारकाः॥

varaṃ buddhir na sā vidyā, vidyāyā buddhir uttamā;
buddhihīnā vinaśyanti yathā te siṃhakārakāḥ

It translates to: "Good sense is better than learning; judgement surpasses knowledge. Those without judgement perish, like the lionmakers". This verse closes the story of four scholars who use their learning to bring a dead lion back to life. The one man with common sense warns them and then climbs a tree. The lion wakes up and eats the other three. Goof-E never lacked knowledge. Its perception model, Gemini ER2 could name any object quite accurately from a single glance. Every hard bug was a missing piece of judgement like checking if something fits the gripper before reaching for it or knowing when not to trust the camera. Judgement surpasses knowledge.