How to Build a LeRobot Embodied AI StarArm Series Robotic Arm and Complete a Custom Grasping Task

Product Overview

1. Open source and easy to customize This series of servo products is provided by FashionStar and offers an open-source 6+1 DOF robotic arm solution that is easy to customize and extend.
2. Supports LeRobot platform integration It is specifically designed for integration with the LeRobot platform. This platform provides PyTorch models, datasets, and tools for imitation learning on real-world robotic tasks, including data collection, simulation, training, and deployment.
3. Rich learning resources Comprehensive open-source learning resources are provided, covering environment setup, installation and configuration, as well as custom grasping examples to help users get started quickly and build robotic applications.
4. Compatible with Nvidia platforms Deployment is supported on the reComputer Mini J4012 Orin NX 16GB platform.

Key Features

• No assembly required: Start immediately and step into the AI era right out of the box.
• 6+1 DOF structural design with a 420 mm reach for flexible operation.
• Smart parallel gripper with a maximum opening of 66 mm, plus modular fingertips for precise grasping.

Initial Environment Setup

For Ubuntu x86:

• Ubuntu 22.04
• CUDA 12+
• Python 3.10
• Torch 2.6

Installation and Configuration

Install LeRobot

Install PyTorch, Torchvision, and other dependencies according to your CUDA version.

1. Install Miniconda. For Jetson:

   wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh
   chmod +x Miniconda3-latest-Linux-aarch64.sh
   ./Miniconda3-latest-Linux-aarch64.sh
   source ~/.bashrc
   

   Or, for x86 Ubuntu 22.04:

   mkdir -p ~/miniconda3
   cd miniconda3
   wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
   bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
   rm ~/miniconda3/miniconda.sh
   source ~/miniconda3/bin/activate
   conda init --all
   

2. Create and activate a new conda environment for LeRobot:

   conda create -y -n lerobot python=3.10 && conda activate lerobot
   

3. Clone the LeRobot repository:

   git clone https://github.com/servodevelop/lerobot.git
   

4. If you are using Miniconda, install ffmpeg in the environment:

   conda install ffmpeg -c conda-forge
   

   This typically installs ffmpeg 7.X built with the libsvtav1 encoder for your platform. If libsvtav1 is not supported, which you can check with ffmpeg -encoders, you can do the following:

   • For all platforms: explicitly install ffmpeg 7.X:

   conda install ffmpeg=7.1.1 -c conda-forge
   

5. Install LeRobot:

   cd ~/lerobot && pip install -e .
   

   For Jetson Jetpack devices, make sure you have installed Pytorch-gpu and Torchvision in Step 5 of this guide before running the following:

   conda install -y -c conda-forge "opencv>=4.10.0.84"  # Install OpenCV and other dependencies with conda, Jetson Jetpack 6.0+ only
   conda remove opencv   # Remove OpenCV
   pip3 install opencv-python==4.10.0.84  # Install the specified OpenCV version with pip3
   conda install -y -c conda-forge ffmpeg
   conda uninstall numpy
   pip3 install numpy==1.26.0  # This version must stay compatible with torchvision
   

6. Install the FashionStar motor dependencies:

   pip install lerobot_teleoperator_stararm102   # Install StarArm102_leader with pip
   pip install lerobot_robot_stararm102    # Install StarArm102_follower with pip
   

7. Check PyTorch and Torchvision

   Because installing the LeRobot environment through pip may uninstall the original GPU build of PyTorch and Torchvision and replace them with CPU versions, check them in Python:

   import torch
   print(torch.cuda.is_available())
   

   If the output is False, reinstall PyTorch and Torchvision according to the official guide.

Wiring

https://github.com/user-attachments/assets/56130bd9-21ee-4ae4-9cac-3817ac4d659f

Arm Port Setup

Run the following command in the terminal to locate the ports for the two robotic arms:

lerobot-find-port

For example:

1. Example output when identifying the Leader port, such as /dev/tty.usbmodem575E0031751 on macOS or possibly /dev/ttyUSB0 on Linux
2. Example output when identifying the Follower port, such as /dev/tty.usbmodem575E0032081 on macOS or possibly /dev/ttyUSB1 on Linux

Note

If you cannot detect ttyUSB0 serial port information, try the following steps.

List all USB ports:

lsusb

After detection succeeds, inspect the ttyUSB information:

sudo dmesg | grep ttyUSB

If the last line shows a disconnect, brltty is occupying that USB device number. Remove it:

sudo apt remove brltty

Finally, grant permissions:

sudo chmod 777 /dev/ttyUSB*

Calibration

If this is your first calibration, rotate each joint left and right to its physical limit.

If you are recalibrating, follow the prompt, type c, and press Enter.

The table below shows reference values. In most cases, the real limit values should stay within ±10° of these references.

Servo ID	Reference Lower Limit	Reference Upper Limit	Notes
motor_0	-180°	180°	Rotate to the physical limit
motor_1	-90°	90°	Rotate to the physical limit
motor_2	-90°	90°	Rotate to the physical limit
motor_3	-180°	180°	No hard stop, rotate to the reference upper and lower angle limits
motor_4	-90°	90°	Rotate to the physical limit
motor_5	-180°	180°	No hard stop, rotate to the reference upper and lower angle limits
motor_6	0°	100°	Rotate to the physical limit

leader

Tip

Connect the leader to /dev/ttyUSB0, or modify the command below.

lerobot-calibrate     --teleop.type=lerobot_teleoperator_stararm102 --teleop.port=/dev/ttyUSB0 --teleop.id=my_awesome_stararm102_leader_arm

follower

Tip

Connect the follower to /dev/ttyUSB1, or modify the command below.

lerobot-calibrate     --robot.type=lerobot_robot_stararm102 --robot.port=/dev/ttyUSB1 --robot.id=my_awesome_stararm102_follower_arm

Teleoperation

https://github.com/user-attachments/assets/23b3aa00-9889-48d3-ae2c-00ad50595e0a

You are now ready to teleoperate your robot without cameras. Run the following simple script:

lerobot-teleoperate \
    --robot.type=lerobot_robot_stararm102 \
    --robot.port=/dev/ttyUSB1 \
    --robot.id=my_awesome_stararm102_follower_arm \
    --teleop.type=lerobot_teleoperator_stararm102 \
    --teleop.port=/dev/ttyUSB0 \
    --teleop.id=my_awesome_stararm102_leader_arm

The teleoperation command automatically detects the following:

1. Any missing calibration and starts the calibration process.
2. The robot and teleoperator devices, then starts teleoperation.

After the program starts, the hover button still works.

Add Cameras

https://github.com/user-attachments/assets/82650b56-96be-4151-9260-2ed6ab8b133f

After connecting two USB cameras, run the following script to check the camera port numbers. Do not connect the cameras through a USB hub, because a slow hub may prevent image data from being read.

lerobot-find-cameras opencv # or realsense for Intel Realsense cameras

The terminal prints output similar to the following. On my laptop, the built-in cameras appear as Camera0 and Camera1, and index_or_path is 2 and 4 respectively.

--- Detected Cameras ---
Camera #0:
  Name: OpenCV Camera @ /dev/video2
  Type: OpenCV
  Id: /dev/video2
  Backend api: V4L2
  Default stream profile:
    Format: 0.0
    Width: 640
    Height: 480
    Fps: 30.0
--------------------
Camera #1:
  Name: OpenCV Camera @ /dev/video4
  Type: OpenCV
  Id: /dev/video4
  Backend api: V4L2
  Default stream profile:
    Format: 0.0
    Width: 640
    Height: 360
    Fps: 30.0
--------------------

Finalizing image saving...
Image capture finished. Images saved to outputs/captured_images

After confirming the external cameras, replace the cameras configuration below and you will be able to display the camera feeds on the computer during teleoperation:

lerobot-teleoperate \
    --robot.type=lerobot_robot_stararm102 \
    --robot.port=/dev/ttyUSB1 \
    --robot.id=my_awesome_stararm102_follower_arm \
    --robot.cameras="{ front: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
    --teleop.type=lerobot_teleoperator_stararm102 \
    --teleop.port=/dev/ttyUSB0 \
    --teleop.id=my_awesome_stararm102_leader_arm \
    --display_data=true
# new
# lerobot-teleoperate \
#     --robot.type=lerobot_robot_stararm102 \
#     --robot.port=/dev/ttyUSB1 \
#     --robot.id=my_awesome_stararm102_follower_arm \
#     --robot.cameras="{ front: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30, fourcc: "MJPG"}}" \
#     --teleop.type=lerobot_teleoperator_stararm102 \
#     --teleop.port=/dev/ttyUSB0 \
#     --teleop.id=my_awesome_stararm102_leader_arm \
#     --display_data=true

Dataset Creation and Collection

https://github.com/user-attachments/assets/8bb25714-783a-4f29-83dd-58b457aed80c

Tip

If you want to use Hugging Face Hub features to upload your dataset, refer to the official article below. This tutorial does not cover that part.

Imitation Learning for Robots

Once you are comfortable with teleoperation, you can start collecting your first dataset.

Record 10 episodes:

lerobot-record \
    --robot.type=lerobot_robot_stararm102 \
    --robot.port=/dev/ttyUSB1 \
    --robot.id=my_awesome_stararm102_follower_arm \
    --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30},front: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}}" \
    --teleop.type=lerobot_teleoperator_stararm102 \
    --teleop.port=/dev/ttyUSB0 \
    --teleop.id=my_awesome_stararm102_leader_arm \
    --display_data=true \
    --dataset.repo_id=starai/record-test \
    --dataset.episode_time_s=30 \
    --dataset.reset_time_s=30 \
    --dataset.num_episodes=10 \
    --dataset.push_to_hub=False \
    --dataset.single_task="Grab the black cube"

Recording Features

• record provides a set of tools for capturing and managing data during robot operation.

1. Data Storage

• The data is stored in the LeRobotDataset format and written to disk during recording.

2. Checkpoints and Resume

• Checkpoints are created automatically during recording.
• If something goes wrong, you can rerun the same command with --resume=true to resume. When resuming, --dataset.num_episodes must be set to the additional number of episodes to record, not the total target number of episodes in the dataset.
• To start over from scratch, manually delete the dataset directory.

3. Recording Parameters

Use command-line arguments to configure the recording workflow:

• --dataset.episode_time_s=60: duration of each recorded episode in seconds (default: 60).
• --dataset.reset_time_s=60: environment reset time after each episode in seconds (default: 60).
• --dataset.num_episodes=50: total number of recorded episodes (default: 50).

4. Keyboard Controls During Recording

Use keyboard shortcuts to control the recording workflow:

• Press Right Arrow (→) to stop the current episode or reset period early and move to the next one.
• Press Left Arrow (←) to cancel the current episode and record it again.
• Press ESC to stop the session immediately, encode the video, and upload the dataset.

Tip

On Linux, if the left arrow, right arrow, and escape keys do not work during recording, make sure the $DISPLAY environment variable is set. See the pynput limitations.

Once you are comfortable with data recording, you can create a larger dataset for training. A good starter task is to grab an object from different positions and place it into a bin. We recommend recording at least 50 episodes, with 10 episodes for each object position. Keep the cameras fixed and maintain consistent grasp behavior throughout the recording. Also make sure the manipulated object remains visible in the camera view. A useful rule of thumb is that you should be able to complete the task yourself using only the camera images.

Replay an Episode

Replay recorded motions to evaluate the repeatability of the robot behavior.

lerobot-replay \
    --robot.type=lerobot_robot_stararm102 \
    --robot.port=/dev/ttyUSB1 \
    --robot.id=my_awesome_stararm102_follower_arm \
    --dataset.repo_id=starai/record-test \
    --dataset.episode=1 # choose the episode you want to replay

Training

To train a policy that controls your robot, here is an example command:

lerobot-train \
  --dataset.repo_id=starai/record-test \
  --policy.type=act \
  --output_dir=outputs/train/act_stararm102_test \
  --job_name=act_stararm102_test \
  --policy.device=cuda \
  --wandb.enable=False \
  --policy.repo_id=starai/my_policy

1. We provide the dataset through dataset.repo_id=starai/record-test.
2. We set policy.type=act. This loads the configuration from configuration_act.py. Importantly, the policy automatically adapts to the motor states, motor actions, and the number of cameras stored in your dataset, such as laptop and phone.
3. We optionally use Weights & Biases to visualize training curves. If you enable it, make sure you have logged in by running wandb.enable=true and then wandb login.

To resume training from a checkpoint:

lerobot-train \
  --config_path=outputs/train/act_stararm102_test/checkpoints/last/pretrained_model/train_config.json \
  --resume=true

Run Inference and Evaluation

Run the following command to record 10 evaluation episodes:

lerobot-record  \
  --robot.type=lerobot_robot_stararm102 \
  --robot.port=/dev/ttyUSB1 \
  --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30},front: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}}" \
  --robot.id=my_awesome_stararm102_follower_arm \
  --display_data=false \
  --dataset.repo_id=starai/eval_record-test \
  --dataset.single_task="Put lego brick into the transparent box" \
  --policy.path=outputs/train/act_stararm102_test/checkpoints/last/pretrained_model
  # <- Teleop optional if you want to teleoperate in between episodes \
  # --teleop.type=lerobot_teleoperator_stararm102 \
  # --teleop.port=/dev/ttyUSB0 \
  # --teleop.id=my_awesome_stararm102_leader_arm \

FAQ

• If you follow this document, please clone the recommended GitHub repository: https://github.com/servodevelop/lerobot.git.

• If teleoperation works normally but camera-enabled teleoperation cannot display the image interface, refer to this page.

• If you encounter a libtiff issue during dataset teleoperation, update libtiff:

conda install libtiff==4.5.0  #for Ubuntu 22.04 is libtiff==4.5.1

• Installing LeRobot may automatically uninstall the GPU version of PyTorch, so you may need to reinstall a GPU-enabled Torch build manually.

• For Jetson, install Pytorch and Torchvision before running conda install -y -c conda-forge ffmpeg, otherwise you may hit an ffmpeg version mismatch when compiling torchvision.

• On a laptop with an RTX 3060 8GB, training ACT on 50 groups of data takes about 6 hours. On machines with an RTX 4090 or A100, training 50 groups usually takes about 2 to 3 hours.

• During data collection, keep the camera position, angle, and ambient lighting stable, and avoid capturing too much unstable background or pedestrian movement. Large environmental changes can cause grasping to fail during deployment.

• Make sure the num-episodes value in the data collection command is large enough. Do not stop recording manually halfway through, because the mean and variance of the dataset are computed only after collection finishes, and those statistics are necessary for training.

• If the program says it cannot read image data from a USB camera, make sure the camera is not connected through a hub. The USB camera must be connected directly to the device to guarantee enough image transfer bandwidth.

Reference Documents

Seeed Studio English Wiki: How to use the SO10xArm robotic arm in Lerobot | Seeed Studio Wiki

Hugging Face Project: Lerobot

Hugging Face: LeRobot

ACT or ALOHA: Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

VQ-BeT: VQ-BeT: Behavior Generation with Latent Actions

Diffusion Policy: Diffusion Policy

TD-MPC: TD-MPC