Lecture

Reinforcement Learning for Robotics

24 Nov 2022 • Richard Kuo

Introduction to Reinforcement Learning for Robot / Drone.

Embodied AI

Blog: Overview of Embodied Artificial Intelligence
Embodied AI is the field for solving AI problems for virtual robots that can move, see, speak, and interact in the virtual world and with other virtual robots — these simulated robot solutions are then transferred to real world robots
The simulated environments for Embodied AI training: SUNCG, Matterport3D, iGibson, Replica, Habitat, and DART

送餐路徑生成：從文件或上層發佈訊息獲取預先定義的路徑。
編輯航線路徑點：清除路徑中不合適的航線路徑點。
MFAC無模型自適應控制之航段管制：自動調整送餐路徑之導航點之間的航段長度，依序共分成路徑跟隨之依據以及MFAC無模型自適應控制之應用。
DWA之區域路徑傳遞：依照MFAC調整之結果，產出相關生成路徑，並以DWA進行區域設定。
自走車基於DDPG的室內路徑規劃

Paper: Long-Range Indoor Navigation with PRM-RL

Gym-Gazebo

Code: erlerobot/gym-gazebo

DART (Dynamic Animation and Robotics Toolkit)

Dartsim/dart

Python bindings: dartpy, pydart2 (deprecated)
OpenAI Gym with DART support: gym-dart (dartpy based), DartEnv (pydart2 based, deprecated)

PyBullet-Gym

PyBullet Quickstart Guide

code: rkuo2000/pybullet-gym

installation

pip install gym
pip install stable-baselines3
git clone https://github.com/rkuo2000/pybullet-gym
export PYTHONPATH=$PATH:/home/yourname/pybullet-gym

Train
python train.py Ant 10000000

Enjoy with trained-model
python enjoy.py Ant

Enjoy with pretrained weights
python enjoy_Ant.py
python enjoy_HumanoidFlagrunHarder.py (a copy from pybulletgym/examples/roboschool-weights/enjoy_TF_*.py)

PyBullet-Robots

env_name = “AtlasPyBulletEnv-v0”
atlas_v4_with_multisense.urdf

RoboCar Gym

Pybullet-RoboCar

Blog:
Creating OpenAI Gym Environments with PyBullet (Part 1)
Creating OpenAI Gym Environments with PyBullet (Part 2)

Quadruped Gym

Motion Imitation

Code: TF 1.15

Code: PyTorch
For Training:
python motion_imitation/run_torch.py --mode train --motion_file 'dog_pace.txt|dog_spin.txt' \ --int_save_freq 10000000 --visualize --num_envs 50 --type_name 'dog_pace'
For Testing:
python motion_imitation/run_torch.py --mode test --motion_file 'dog_pace.txt' --model_file 'file_path' \ --encoder_file 'file_path' --visualize

Rex: an open-source quadruped robot

Code: nicrusso7/rex-gym

Drones Gym

PyBullet-Gym for Drones

Installation

sudo apt install ffmpeg
pip install numpy pillow matplotlib cycler
pip install gym pybullet stable_baselines3 ray[rllib]
git clone https://github.com/rkuo2000/gym-pybullet-drones.git
cd gym-pybullet-drones

Train & Enjoy
python train.py # modify train.py for different env, algorithm and timesteps
python enjoy.py # modify enjoy.py for different env
Fly using DSLPIDControl.py:（PID飛行）
python examples/fly.py --num_drones 1
To learn take-off:（起飛）
python examples/learn.py
compare.py which replays and compare to a trace saved in files/example_trace.pkl

Experiments
cd experiments/learning

env : hover, takeoff, flythrugate, tune（旋停, 起飛, 穿越, 調整）
algo: a2c, ppo, sac, td3, ddpg

To learn hover:（旋停）
python singleagent.py --env hover --algo a2c

To visualize the best trained agent:
python test_singleagent.py --exp ./results/save-hover-a2c

For multi-agent RL, using rllib:
python multiagent.py --num_drones 3 --env hover --algo a2c --num_workers 2

Flightmare

Flightmare is a flexible modular quadrotor simulator.

Introduction
Prerequisites
Install Python Packages
Install ROS

running ROS

roslaunch flightros rotors_gazebo.launch

flighRL

cd /path/to/flightmare/flightrl
pip install .
cd examples
python3 run_drone_control.py --train 1

AirSim

Assistive Gym

Paper: Assistive Gym: A Physics Simulation Framework for Assistive Robotics

Four collaborative robots (PR2, Jaco, Baxter, Sawyer)
Support for the Stretch and PANDA robots

Code: Healthcare-Robotics/assistive-gym
Using Pybullet & Tensorflow 1.14.0

conda create -n assist
conda activate assist

git clone https://github.com/Healthcare-Robotics/assistive-gym.git
cd assistive-gym
pip3 install -e .

python3 -m assistive_gym --env "FeedingJaco-v1"

Assistive VR Gym

Paper: Assistive VR Gym: Interactions with Real People to Improve Virtual Assistive Robots
Code: Healthcare-Robotics/assistive-vr-gym

conda create -n assist-vr
conda activate assist-vr

git clone -b vr https://github.com/Zackory/bullet3.git
cd bullet3
pip install .

git clone https://github.com/Healthcare-Robotics/assistive-vr-gym.git
cd assistive-vr-gym
pip install .