Understanding Reinforcement Learning through OpenDuck

I. Project Research

1.1 International Projects

1.1.1 🇺🇸 OpenDuck Mini

1.1.2 🇺🇸 K-Scale Labs (Stompy)

1.1.3 🇺🇸 Berkeley Humanoid Lite

1.1.4 🇫🇷 Poppy Project & 🇰🇷 Robotis OP3


1.2 Domestic Projects

1.2.1 🇨🇳 Kit-Miao (Damiao Technology)

1.2.2 🇨🇳 Unitree Qmini (Yushu)

1.2.3 🇨🇳 AlexBot (Alexhuge1)

1.2.4 🇨🇳 HighTorque & FFTAI


II. OpenDuck Development Workflow

2.1 Phase 1: Model and Simulation Preparation

2.2 Phase 2: Motion Generation

2.3 Phase 3: Reinforcement Learning

2.4 Phase 4: Hardware Construction

2.5 Phase 5: Runtime Deployment

III. OpenDuck Repository Overview


IV. Raspberry Pi Zero 2W Deployment Process

4.1 Flash Image

4.2 SD Card Expansion

# 32GB SD card may only show 7GB after flashing
sudo raspi-config -> Advanced options -> Expand Filesystem

# Verify
df -h

4.3 APT Source Configuration

# Backup
sudo cp /etc/apt/sources.list.d/debian.sources /etc/apt/sources.list.d/debian.sources.bak
sudo cp /etc/apt/sources.list.d/raspi.sources /etc/apt/sources.list.d/raspi.sources.bak
Types: deb
URIs: https://mirrors.tuna.tsinghua.edu.cn/debian/
Suites: trixie trixie-updates trixie-backports
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb
URIs: https://mirrors.tuna.tsinghua.edu.cn/debian-security/
Suites: trixie-security
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: https://mirrors.tuna.tsinghua.edu.cn/raspberrypi/
Suites: trixie
Components: main
Signed-By: /usr/share/keyrings/raspberrypi-archive-keyring.gpg
# Update
sudo apt update
sudo apt upgrade -y

4.4 Reduce FTDI USB Serial Latency

# Create rule file
sudo tee /etc/udev/rules.d/99-usb-serial.rules >/dev/null <<'EOF'
SUBSYSTEM=="usb-serial", DRIVER=="ftdi_sio", ATTR{latency_timer}="1"
EOF

# Apply
sudo udevadm control --reload-rules
sudo udevadm trigger

4.5 Enable I2C

sudo raspi-config -> Interface Options -> I2C

4.6 Install System Packages

sudo apt install -y git unzip i2c-tools joystick python3-pip python3-venv

4.7 Configure pip Source

pip config set global.index-url https://mirrors.aliyun.com/pypi/simple
pip config set global.trusted-host mirrors.aliyun.com

# Verify
pip config list

4.8 Install Miniconda

# Create directory
mkdir download && cd download

# Download Miniconda (aarch64)
# https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh

chmod +x Miniconda3-latest-Linux-aarch64.sh
./Miniconda3-latest-Linux-aarch64.sh
# Follow prompts: Enter -> yes -> Enter -> yes

source ~/.bashrc
# Clean old configuration
conda config --remove-key channels 2>/dev/null || true
conda config --remove-key default_channels 2>/dev/null || true

# Set Tsinghua source
conda config --append default_channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
conda config --append default_channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
conda config --set custom_channels.conda-forge https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud

# Set channels
conda config --add channels conda-forge
conda config --add channels defaults
conda config --set show_channel_urls yes
conda create -n duck310 python=3.10 -y --repodata-fn current_repodata.json -v
conda activate duck310

4.9 Configure pip Acceleration and Install uv

pip config set global.index-url https://mirrors.aliyun.com/pypi/simple
pip config set global.trusted-host mirrors.aliyun.com

pip install -U uv

4.10 Install OpenDuckMini Dependencies

uv pip install -U pip setuptools wheel

uv pip install rustypot==0.1.0 onnxruntime==1.18.1 numpy 
    adafruit-circuitpython-bno055==5.4.13 scipy==1.15.1 
    pygame==2.6.0 openai==1.70.0 RPi.GPIO

4.11 Configure Proxy (Optional)

git config --global http.proxy http://your_proxy_address:your_proxy_port
git config --global https.proxy https://your_proxy_address:your_proxy_port
git config --global http.proxy http://192.168.1.196:6551
git config --global https.proxy https://192.168.1.196:6551

4.12 Install pypot and Open_Duck_Mini_Runtime

mkdir ~/project && cd ~/project
# Download: https://github.com/apirrone/Open_Duck_Mini_Runtime/tree/v2
unzip Open_Duck_Mini_Runtime-2.zip
cd Open_Duck_Mini_Runtime-2
uv pip install -e .
# Download: https://github.com/apirrone/pypot/tree/support-feetech-sts3215
unzip pypot-support-feetech-sts3215.zip
cd pypot-support-feetech-sts3215
uv pip install .

4.13 Calibrate IMU

sudo usermod -aG i2c $USER
i2cdetect -y 1

cd ~/project/Open_Duck_Mini_Runtime-2/scripts/
python calibrate_imu.py
cp imu_calib_data.pkl ~/project/Open_Duck_Mini_Runtime-2/mini_bdx_runtime/mini_bdx_runtime/

4.14 Adjust Servo Offsets

cd ~/project/Open_Duck_Mini_Runtime-2/scripts
python find_soft_offsets.py
openduckmini-motor-position.jpg

4.15 Modify Configuration File

cd ~/project/Open_Duck_Mini_Runtime-2/
cp example_config.json ~/duck_config.json
{
  "imu_upside_down": true
}

4.16 Initial Bent Leg Posture

cd ~/project/Open_Duck_Mini_Runtime-2/scripts
python turn_on.py

4.17 Test Walking

cd ~/project/Open_Duck_Mini_Runtime-2/scripts

python v2_rl_walk_mujoco.py 
    --duck_config_path ~/duck_config.json 
    --onnx_model_path ~/BEST_WALK_ONNX_2.onnx

V. RDK X5 Deployment Process

5.1 System Flashing

5.2 Install System Packages

5.3 Configure pip Source

5.4 Create venv (⚠️ Different from Raspberry Pi)

python3 -m venv --system-site-packages ~/duck_env
source ~/duck_env/bin/activate

# Verify GPIO module
python3 -c "import Hobot.GPIO; print('OK')"

5.5 Configure pip Acceleration and Install uv

pip config set global.index-url https://mirrors.aliyun.com/pypi/simple
pip config set global.trusted-host mirrors.aliyun.com

python3 -m pip install -U uv

5.6 Install Dependencies (⚠️ Different from Raspberry Pi)

python3 -m uv pip install -U pip setuptools wheel

# Note: RDK X5 uses smbus2 instead of RPi.GPIO
python3 -m uv pip install rustypot==0.1.0 onnxruntime==1.18.1 numpy 
    adafruit-circuitpython-bno055==5.4.13 scipy==1.15.1 
    pygame==2.6.0 openai==1.70.0 smbus2

5.7 Configure Proxy (Optional)

5.8 Install pypot and Runtime

mkdir ~/project && cd ~/project
unzip Open_Duck_Mini_Runtime-2_RDK_X5.zip
cd Open_Duck_Mini_Runtime-2_RDK_X5
uv pip install -e .
# Download: https://github.com/apirrone/pypot/tree/support-feetech-sts3215
unzip pypot-support-feetech-sts3215.zip
cd pypot-support-feetech-sts3215
uv pip install .

5.9 Calibrate IMU

5.10 Adjust Servo Offsets

5.11 Modify Configuration File

5.12 Initial Bent Leg Posture

5.13 Test Walking

cd ~/project/Open_Duck_Mini_Runtime-2_RDK_X5/scripts

python v2_rl_walk_mujoco.py 
    --duck_config_path ~/duck_config.json 
    --onnx_model_path ~/BEST_WALK_ONNX_2.onnx

VI. Frequently Asked Questions (FAQ)

6.1 Q1: When running find_soft_offsets.py, gravity shows horizontal posture

cd ~/project/Open_Duck_Mini_Runtime-2/scripts  # or corresponding RDK X5 path
nano set_servo_mid.py
from mini_bdx_runtime.rustypot_position_hwi import HWI
from mini_bdx_runtime.duck_config import DuckConfig
import argparse
import time
import traceback

def zero_motor(hwi, joint_id, tol=0.02, timeout=5.0):
    """Move motor to 0 rad and wait until reached."""
    print(f"Zeroing motor ID {joint_id} to 0 rad")

    try:
        current_pos = hwi.io.read_present_position([joint_id])[0]
        print(f"Current position: {current_pos:.3f} rad")

        hwi.io.write_goal_position([joint_id], [0.0])

        start_time = time.time()
        while True:
            pos = hwi.io.read_present_position([joint_id])[0]
            err = abs(pos)

            print(f"  pos={pos:.3f} rad, err={err:.3f}")

            if err < tol:
                print("✓ Zero position reached")
                return True

            if time.time() - start_time > timeout:
                print("✗ Timeout while zeroing motor")
                return False

            time.sleep(0.05)

    except Exception as e:
        print(f"✗ Error zeroing motor ID {joint_id}: {e}")
        print(traceback.format_exc())
        return False

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--id", type=int, required=True, help="Motor ID to zero")
    args = parser.parse_args()

    print("Initializing hardware interface...")
    try:
        duck_config = DuckConfig()
        hwi = HWI(duck_config=duck_config)
        print("Successfully connected to hardware")
    except Exception as e:
        print(f"Error initializing HWI: {e}")
        print(traceback.format_exc())
        return

    zero_motor(hwi, args.id)

    try:
        hwi.io.disable_torque([args.id])
        print(f"Torque disabled for motor ID {args.id}")
    except Exception:
        pass

if __name__ == "__main__":
    main()
python set_servo_mid.py --id 12
Initializing hardware interface...
Successfully connected to hardware
Zeroing motor ID 12 to 0 rad
Current position: -3.086 rad
  pos=-3.086 rad, err=3.086
  ...
✗ Timeout while zeroing motor
Torque disabled for motor ID 12


VII. Reinforcement Learning

7.1 Generate Reference Motions

7.1.1 Clone Repository and Install Dependencies

cd ~/project/open_duck_mini_ws
git clone https://github.com/apirrone/Open_Duck_reference_motion_generator.git
cd Open_Duck_reference_motion_generator

# Install dependencies using uv
uv sync

7.1.2 Batch Generate Motions

7.1.3 Verify Generated Motions (Optional)

# Use Meshcat for visualization
uv run open_duck_reference_motion_generator/gait_playground.py --duck open_duck_mini_v2

7.2 Process Motion Data

7.2.1 Polynomial Fitting

cd ~/project/open_duck_mini_ws/Open_Duck_reference_motion_generator

uv run scripts/fit_poly.py --ref_motion recordings/

7.2.2 View Fitting Results (Optional)

uv run scripts/plot_poly_fit.py --coefficients polynomial_coefficients.pkl

7.2.3 Copy to Training Directory

cp polynomial_coefficients.pkl 
   ~/project/open_duck_mini_ws/Open_Duck_Playground/playground/open_duck_mini_v2/data/

7.3 Reinforcement Learning Training

7.3.1 Clone Repository and Install Dependencies

cd ~/project/open_duck_mini_ws
git clone https://github.com/apirrone/Open_Duck_Playground.git
cd Open_Duck_Playground

uv sync

7.3.2 Start Training

python3 playground/open_duck_mini_v2/runner.py 
    --task flat_terrain_backlash 
    --num_timesteps 300000000

7.3.3 Monitor Training Progress

cd ~/project/open_duck_mini_ws/Open_Duck_Playground
tensorboard --logdir=checkpoints/

7.3.4 Training Parameters

7.3.5 Deploy to Real Robot

scp ONNX.onnx user@raspberry-pi:~/BEST_WALK_ONNX_2.onnx

Leave a Reply

Your email address will not be published. Required fields are marked *

Latest Posts