Skip to content

How to Build a LeRobot StarAI Robotic Arm and Complete a Custom Grasping Task

Product Introduction

  1. Open source and easy for secondary development
    This series of servo products is provided by FashionStar Technology. It is an open-source 6+1 DOF robotic arm solution designed for secondary development.
  2. Supports LeRobot platform integration
    It is designed for integration with the LeRobot platform. The platform provides PyTorch models, datasets, and tools for imitation learning on real-world robot tasks, including data collection, simulation, training, and deployment.
  3. Rich learning resources
    It provides comprehensive open-source learning resources, including environment setup, installation and configuration, and custom grasping task examples to help users get started quickly and develop robotic applications.
  4. Compatible with Nvidia platforms
    Deployment is supported through the reComputer Mini J4012 Orin NX 16GB platform.

Features

  • No assembly required: ready to use out of the box and ready for the AI era.

Initial Environment Setup

For Ubuntu X86:

  • Ubuntu 22.04
  • CUDA 12+
  • Python 3.10
  • Troch 2.6

Installation and Configuration

Install LeRobot

Install PyTorch, Torchvision, and related environments according to your CUDA version.

  1. Install Miniconda. For Jetson:

    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh
chmod +x Miniconda3-latest-Linux-aarch64.sh
./Miniconda3-latest-Linux-aarch64.sh
source ~/.bashrc

    Or, for X86 Ubuntu 22.04:

    mkdir -p ~/miniconda3
cd miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh
source ~/miniconda3/bin/activate
conda init --all

  2. Create and activate a new conda environment for LeRobot.

    conda create -y -n lerobot python=3.10 && conda activate lerobot

  3. Clone the LeRobot repository:

    git clone https://github.com/servodevelop/lerobot.git

  4. When using Miniconda, install ffmpeg in the environment:

    conda install ffmpeg -c conda-forge

    This usually installs ffmpeg 7.x built with the libsvtav1 encoder for your platform. If libsvtav1 is not supported, which you can check with ffmpeg -encoders, you can:

    • Explicitly install ffmpeg 7.x on all platforms:
    conda install ffmpeg=7.1.1 -c conda-forge

  5. Install LeRobot:

    cd ~/lerobot && pip install -e .

    For Jetson Jetpack devices, make sure Pytorch-gpu and Torchvision have been installed according to step 5 in this tutorial before running this step:

    conda install -y -c conda-forge "opencv>=4.10.0.84"  # 通过 conda 安装 OpenCV 和其他依赖，仅适用于 Jetson Jetpack 6.0+
conda remove opencv   # 卸载 OpenCV
pip3 install opencv-python==4.10.0.84  # 使用 pip3 安装指定版本 OpenCV
conda install -y -c conda-forge ffmpeg
conda uninstall numpy
pip3 install numpy==1.26.0  # 该版本需与 torchvision 兼容

  6. Install motor dependencies:

    pip install lerobot_teleoperator_pipermate    #使用 pip 安装 pipermate
pip install lerobot_robot_piper    #使用 pip 安装 piper
sudo apt update && sudo apt install can-utils ethtool

  7. Check PyTorch and Torchvision.

    Because installing the LeRobot environment through pip may uninstall the original PyTorch and Torchvision and install CPU versions, check it in Python:

    import torch
print(torch.cuda.is_available())

    If the output is False, reinstall PyTorch and Torchvision according to the official tutorial.

Wiring

┌─────────────────┐         USB          ┌─────────────────┐
│                 │◄────────────────────►│                 │
│   PiPER Mate    │                      │      计算机      │
│     机械臂       │                      │ (Ubuntu 22.04)  │
└─────────────────┘                      └────────┬────────┘
                                                  │
                                                 USB
                                                  │
┌─────────────────┐         CAN          ┌────────┴────────┐
│                 │◄────────────────────►│                 │
│      Piper      │                      │  CAN转USB适配器  │
│      机械臂      │                      │                 │
└─────────────────┘                      └─────────────────┘

Arm Port Settings

Enter the following command in the terminal to find the port numbers corresponding to the two robotic arms:

lerobot-find-port

For example:

  1. Example output when identifying the Leader port, such as /dev/tty.usbmodem575E0031751 on macOS or /dev/ttyUSB0 on Linux.

Note

If ttyUSB0 serial port information cannot be detected, try the following method. brltty may be occupying this USB device number. Remove it to solve the problem.

sudo apt remove brltty

Finally, grant permissions.

sudo chmod 777 /dev/ttyUSB*

Configure the CAN Device

Tip

Connect the follower to can0.

# 1. 查找所有CAN端口
bash find_all_can_port.sh
# 2. 激活can0接口（波特率1000000）
bash can_activate.sh can0 1000000

Expected output:

Both ethtool and can-utils are installed.
Interface can0 is connected to USB port 3-9.4:1.0
-------------------START-----------------------
Both ethtool and can-utils are installed.
Expected to configure a single CAN module, detected interface can0 with corresponding USB address 3-9.4:1.0.
Interface can0 is already activated with a bitrate of 1000000.
The interface name is already can0.
-------------------OVER------------------------

Teleoperation

You are ready to teleoperate your robot without cameras. Run the following simple script:

lerobot-teleoperate \
    --robot.type=lerobot_robot_piper \
    --robot.can_name=can0 \
    --robot.id=my_awesome_piper_arm \
    --teleop.type=lerobot_teleoperator_pipermate \
    --teleop.port=/dev/ttyUSB0 \
    --teleop.id=my_awesome_pipermate_arm

The teleoperation command automatically detects the following:

  1. Any missing calibration and starts the calibration process.
  2. The robot and teleoperation device, then starts teleoperation.

After the program starts, the hover button remains active.

Add Cameras

After plugging in two USB cameras, run the following script to check the camera port numbers. Avoid connecting cameras through a USB hub, because a slow hub may prevent image data from being read.

lerobot-find-cameras opencv # or realsense for Intel Realsense cameras

The terminal prints the following information. In my laptop example, the laptop cameras are Camera0 and Camera1, and the index_or_path values are 2 and 4.

--- Detected Cameras ---
Camera #0:
  Name: OpenCV Camera @ /dev/video2
  Type: OpenCV
  Id: /dev/video2
  Backend api: V4L2
  Default stream profile:
    Format: 0.0
    Width: 640
    Height: 480
    Fps: 30.0
--------------------
Camera #1:
  Name: OpenCV Camera @ /dev/video4
  Type: OpenCV
  Id: /dev/video4
  Backend api: V4L2
  Default stream profile:
    Format: 0.0
    Width: 640
    Height: 360
    Fps: 30.0
--------------------

Finalizing image saving...
Image capture finished. Images saved to outputs/captured_images

After confirming the external cameras, replace the camera information below with your camera information so the camera view can be displayed on the computer during teleoperation:

lerobot-teleoperate \
    --robot.type=lerobot_robot_piper \
    --robot.can_name=can0 \
    --robot.id=my_awesome_piper_arm \
    --robot.cameras="{ front: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
    --teleop.type=lerobot_teleoperator_pipermate \
    --teleop.port=/dev/ttyUSB0 \
    --teleop.id=my_awesome_pipermate_arm \
    --display_data=true

Dataset Creation and Collection

Tip

If you want to use Hugging Face Hub features to upload your dataset, refer to the following official article. This tutorial does not cover that part.

Imitation Learning for Robots

Once you are familiar with teleoperation, you can start your first dataset.

Record 10 episodes:

lerobot-record \
    --robot.type=lerobot_robot_piper \
    --robot.can_name=can0 \
    --robot.id=my_awesome_piper_arm \
    --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30},front: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}}" \
    --teleop.type=lerobot_teleoperator_pipermate \
    --teleop.port=/dev/ttyUSB0 \
    --teleop.id=my_awesome_pipermate_arm \
    --display_data=true \
    --dataset.repo_id=starai/record-test \
    --dataset.episode_time_s=30 \
    --dataset.reset_time_s=30 \
    --dataset.num_episodes=10 \
    --dataset.push_to_hub=False \
    --dataset.single_task="Grab the black cube"

Recording Features

  • record provides a set of tools for capturing and managing data during robot operation.

1. Data Storage

  • Data is stored in LeRobotDataset format and is written to disk during recording.

2. Checkpoints and Resume

  • Checkpoints are automatically created during recording.
  • If a problem occurs, you can resume by rerunning the same command. When resuming recording, set the value to the number of additional episodes to record, not the target total number of episodes in the dataset. --resume=true --dataset.num_episodes
  • To record from scratch, manually delete the dataset directory.

3. Recording Parameters

Use command-line parameters to configure the data recording flow:

--dataset.episode_time_s=60: duration of each data recording episode (default: 60 seconds).
--dataset.reset_time_s=60: environment reset duration after each episode (default: 60 seconds).
--dataset.num_episodes=50: total number of episodes to record (default: 50).

4. Keyboard Controls During Recording

Use keyboard shortcuts to control the data recording flow:

  • Press Right Arrow (→): stop the current episode or reset period early, then move to the next one.
  • Press Left Arrow (←): cancel the current episode and record it again.
  • Press ESC: stop the session immediately, encode videos, and upload the dataset.

Tip

On Linux, if the left/right arrow keys and Esc key do not work during data recording, make sure the $DISPLAY environment variable is set. See the pynput limitations.

Once you are familiar with data recording, you can create a larger dataset for training. A good starter task is to grasp an object from different positions and place it in a bin. We recommend recording at least 50 episodes, 10 for each location. Keep the camera fixed and keep grasping behavior consistent throughout recording. Also make sure the object being manipulated is visible in the camera. A good rule of thumb is that you should be able to complete the task yourself by looking only at the camera image.

Replay an Episode

Replay a recorded action to test the repeatability of robot motion.

lerobot-replay \
    --robot.type=lerobot_robot_piper \
    --robot.can_name=can0 \
    --robot.id=my_awesome_piper_arm \
    --dataset.repo_id=starai/record-test \
    --dataset.episode=1 # choose the episode you want to replay

Training

To train a policy that controls your robot, here is an example command:

lerobot-train \
  --dataset.repo_id=starai/record-test \
  --policy.type=act \
  --output_dir=outputs/train/act_piper_test \
  --job_name=act_piper_test \
  --policy.device=cuda \
  --wandb.enable=False \
  --policy.repo_id=starai/my_policy
  1. The dataset is provided as a parameter. dataset.repo_id=starai/record-test
  2. The policy configuration is loaded from configuration_act.py. This policy automatically adapts to the motor states, motor actions, and number of cameras saved in your dataset. policy.type=act laptop phone
  3. We provide Weights & Biases for visualizing training graphs. This is optional, but if you use it, make sure you have logged in by running wandb.enable=true wandb login.

To resume training from a checkpoint:

lerobot-train \
  --config_path=outputs/train/act_piper_test/checkpoints/last/pretrained_model/train_config.json \
  --resume=true

Run Inference and Evaluate

Run the following command to record 10 evaluation episodes:

lerobot-record  \
  --robot.type=lerobot_robot_piper \
  --robot.can_name=can0 \
  --robot.id=my_awesome_piper_arm \
  --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30},front: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}}" \
  --display_data=false \
  --dataset.repo_id=starai/eval_record-test \
  --dataset.single_task="Put lego brick into the transparent box" \
  --policy.path=outputs/train/act_piper_test/checkpoints/last/pretrained_model
  # <- Teleop optional if you want to teleoperate in between episodes \
  # --teleop.type=lerobot_teleoperator_violin \
  # --teleop.port=/dev/ttyUSB0 \
  # --teleop.id=my_awesome_leader_arm \

FAQ

  • If you use this tutorial, please git clone the GitHub repository recommended in this document: https://github.com/servodevelop/lerobot.git.

  • If teleoperation works normally but teleoperation with Camera cannot display the image interface, refer to this page.

  • If a libtiff issue occurs during dataset teleoperation, update the libtiff version.

conda install libtiff==4.5.0  #for Ubuntu 22.04 is libtiff==4.5.1

  • After installing LeRobot, the GPU version of PyTorch may be automatically uninstalled, so install torch-gpu manually if needed.

  • For Jetson, install Pytorch and Torchvsion before running conda install -y -c conda-forge ffmpeg, otherwise torchvision compilation may have an ffmpeg version mismatch.

  • Training 50 ACT data episodes takes about 6 hours on an 8 GB RTX 3060 laptop, and about 2-3 hours on computers with RTX 4090 or A100.

  • During data collection, keep the camera position, camera angle, and ambient lighting stable. Reduce unstable backgrounds and pedestrians in the camera view, otherwise large environment differences during deployment may prevent the robotic arm from grasping normally.

  • Make sure the num-episodes value in the data collection command is sufficient. Do not pause manually midway, because the dataset mean and variance are calculated only after data collection finishes, and these values are necessary for training.

  • If the program reports that USB camera image data cannot be read, make sure the USB camera is not connected through a hub. The USB camera must be connected directly to the device to ensure fast image transmission.

References

Seeed Studio English Wiki: How to use the SO10xArm robotic arm in Lerobot | Seeed Studio Wiki

Huggingface Project: Lerobot

Huggingface: LeRobot

ACT or ALOHA: Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

VQ-BeT: VQ-BeT: Behavior Generation with Latent Actions

Diffusion Policy: Diffusion Policy

TD-MPC: TD-MPC