安装vulkan

sudo apt-get install libvulkan1
sudo apt-get install vulkan-utils
或者
sudo apt install vulkan-tools
vulkaninfo
​

如果不存在，请尝试创建包含以下内容的文件：/usr/share/vulkan/icd.d/nvidia_icd.json

{
"file_format_version" : "1.0.0",
"ICD": {
"library_path": "libGLX_nvidia.so.0",
"api_version" : "1.2.155"
}
}

测试安装

python -m mani_skill.examples.demo_random_action

import gymnasium as gym
import mani_skill.envs

env = gym.make(
"PickCube-v1", # there are more tasks e.g. "PushCube-v1", "PegInsertionSide-v1", ...
num_envs=1,
obs_mode="state", # there is also "state_dict", "rgbd", ...
control_mode="pd_ee_delta_pose", # there is also "pd_joint_delta_pos", ...
render_mode="human"
)
print("Observation space", env.observation_space)
print("Action space", env.action_space)

obs, _ = env.reset(seed=0) # reset with a seed for determinism
done = False
while not done:
action = env.action_space.sample()
obs, reward, terminated, truncated, info = env.step(action)
done = terminated or truncated
env.render() # a display is required to render
env.close()

run headless / without a display

python -m mani_skill.examples.demo_random_action -e PickCube-v1

run with A GUI and ray tracing

python -m mani_skill.examples.demo_random_action -e PickCube-v1 --render-mode="human" --shader="rt-fast"

python -m mani_skill.examples.benchmarking.gpu_sim --num-envs=1024

并行真的超快

rendering RGB + Depth data from all cameras

python -m mani_skill.examples.benchmarking.gpu_sim --num-envs=64 --obs-mode="rgbd"

directly save 64 videos of the visual observations put into one video

python -m mani_skill.examples.benchmarking.gpu_sim --num-envs=64 --save-video

import gymnasium as gym
import mani_skill.envs

env = gym.make(
"PickCube-v1",
obs_mode="state",
control_mode="pd_joint_delta_pos",
num_envs=16,
parallel_in_single_scene=True,
viewer_camera_configs=dict(shader_pack="rt-fast"),
)
env.reset()
while True:
env.step(env.action_space.sample())
env.render_human()

python -m mani_skill.examples.demo_random_action -e PushCube-v1 -b gpu --render-mode human --seed 42

生成随机模拟

默认下载数据的路径

python -m mani_skill.utils.download_asset "ReplicaCAD"

python -m mani_skill.examples.demo_random_action -e "ReplicaCAD_SceneManipulation-v1"
--render-mode="rgb_array" --record-dir="videos"

python -m mani_skill.examples.demo_random_action -e "ReplicaCAD_SceneManipulation-v1"
--render-mode="human" # run with GUI

python -m mani_skill.examples.demo_random_action -e "ReplicaCAD_SceneManipulation-v1"
--render-mode="human" --shader="rt-fast" # faster ray-tracing option but lower quality

python -m mani_skill.examples.demo_random_action -e "ReplicaCAD_SceneManipulation-v1"
--render-mode="human" --shader="rt"

比较推荐中等分辨率

python -m mani_skill.examples.demo_random_action -e "TwoRobotStackCube-v1"
--render-mode="human"

python -m mani_skill.examples.demo_random_action -e "RotateValveLevel2-v1"
--render-mode="human"

灵巧手

python -m mani_skill.examples.demo_random_action -e "RotateSingleObjectInHandLevel3-v1"
--render-mode="human"

模拟触觉

要使用 4096 个并行任务对 PickCube-v1 任务上的 GPU 模拟进行基准测试，您可以运行

python -m mani_skill.examples.benchmarking.gpu_sim -e "PickCube-v1" -n 4096

速度非常之快

要保存代理获得的视觉观察视频（在本例中只有 rgb 和深度），您可以运行

python -m mani_skill.examples.benchmarking.gpu_sim -e "PickCube-v1" -n 64
--save-video --render-mode="sensors"

它应该运行得相当快！（4090 上 3000+ fps，您可以增加环境数量以获得更高的 FPS）。您可以更改--render-mode="rgb_array"为从更高质量的相机进行渲染。

要尝试各种并行模拟功能，您可以运行

python -m mani_skill.examples.benchmarking.gpu_sim -e "PickSingleYCB-v1" -n 64
--save-video --render-mode="sensors"

同时训练抓取不同的物体

python -m mani_skill.examples.benchmarking.gpu_sim -e "OpenCabinetDrawer-v1" -n 64
--save-video --render-mode="sensors"

显示了在每个并行环境中具有不同对象和关节的两个任务。以下是 OpenCabinetDrawer 任务的一个示例。

更多详细信息和性能基准测试结果可在本页面找到

交互控制#

单击+拖动远程操作：

简单的工具，可以单击并拖动 Panda 机械臂的末端执行器来解决各种任务。您只需单击并拖动，按“n”移动到您拖动到的位置，“g”切换打开/关闭抓取，然后重复。按“q”退出并保存结果视频。

python -m mani_skill.examples.teleoperation.interactive_panda -e "StackCube-v1"

有关如何使用此工具（用于演示和数据收集）的更多详细信息，请参阅主页。下面的视频显示了系统的运行情况。

关于这个，有一个bug，我提了discord

• python -m mani_skill.examples.teleoperation.interactive_panda -e "StackCube-v1" Traceback (most recent call last): File "/home/kewei/micromamba/envs/dl/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/kewei/micromamba/envs/dl/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/kewei/micromamba/envs/dl/lib/python3.9/site-packages/mani_skill/examples/teleoperation/interactive_panda.py", line 11, in from mani_skill.examples.motionplanning.panda_stick.motionplanner import \ ModuleNotFoundError: No module named 'mani_skill.examples.motionplanning.panda_stick' hi could anyone help me with this problem?

• [17:36]

  ok well I have solved it. Just install from github not pypi helped me out

• kewei_—_ 今天17:43

  it might be a issue in example code. the right should be

  from mani_skill.examples.motionplanning.panda.motionplanner_stick import PandaStickMotionPlanningSolver

运动规划解决方案#

我们为熊猫手臂在某些任务上提供了运动规划解决方案/演示，您现在可以尝试并使用以下内容记录演示：

python -m mani_skill.examples.motionplanning.panda.run -e "PickCube-v1" # runs headless and only saves video
python -m mani_skill.examples.motionplanning.panda.run -e "StackCube-v1" --vis # opens up the GUI
python -m mani_skill.examples.motionplanning.panda.run -h # open up a help menu and also show what tasks have solutions

(dl) kewei@sxy-gpus3090:~/Desktop$ python -m mani_skill.examples.motionplanning.panda.run -h # open up a help menu and also show what tasks have solutions
usage: run.py [-h] [-e ENV_ID] [-o OBS_MODE] [-n NUM_TRAJ]
[--only-count-success] [--reward-mode REWARD_MODE]
[-b SIM_BACKEND] [--render-mode RENDER_MODE] [--vis]
[--save-video] [--traj-name TRAJ_NAME] [--shader SHADER]
[--record-dir RECORD_DIR] [--num-procs NUM_PROCS]

optional arguments:
-h, --help show this help message and exit
-e ENV_ID, --env-id ENV_ID
Environment to run motion planning solver on.
Available options are ['DrawTriangle-v1',
'PickCube-v1', 'StackCube-v1', 'PegInsertionSide-v1',
'PlugCharger-v1', 'PushCube-v1', 'PullCubeTool-v1',
'LiftPegUpright-v1', 'PullCube-v1']
-o OBS_MODE, --obs-mode OBS_MODE
Observation mode to use. Usually this is kept as
'none' as observations are not necesary to be stored,
they can be replayed later via the
mani_skill.trajectory.replay_trajectory script.
-n NUM_TRAJ, --num-traj NUM_TRAJ
Number of trajectories to generate.
--only-count-success If true, generates trajectories until num_traj of them
are successful and only saves the successful
trajectories/videos
--reward-mode REWARD_MODE
-b SIM_BACKEND, --sim-backend SIM_BACKEND
Which simulation backend to use. Can be 'auto', 'cpu',
'gpu'
--render-mode RENDER_MODE
can be 'sensors' or 'rgb_array' which only affect what
is saved to videos
--vis whether or not to open a GUI to visualize the solution
live
--save-video whether or not to save videos locally
--traj-name TRAJ_NAME
The name of the trajectory .h5 file that will be
created.
--shader SHADER Change shader used for rendering. Default is 'default'
which is very fast. Can also be 'rt' for ray tracing
and generating photo-realistic renders. Can also be
'rt-fast' for a faster but lower quality ray-traced
renderer
--record-dir RECORD_DIR
where to save the recorded trajectories
--num-procs NUM_PROCS
Number of processes to use to help parallelize the
trajectory replay process. This uses CPU
multiprocessing and only works with the CPU simulation
backend at the moment.

Real2Sim 评估#

ManiSkill3 支持通过 GPU 模拟 + 渲染 RT-1 和 Octo 等策略进行极快的 real2sim 评估。有关支持哪些环境的详细信息，请参阅此页面。要运行 RT-1 和 Octo 的推理，请参阅SimplerEnv 项目maniskill3的分支。

可视化点云数据

您可以运行以下命令来可视化点云观测结果（需要显示器才能工作）

pip install "pyglet<2" # make sure to install this dependency
python -m mani_skill.examples.demo_vis_pcd -e "StackCube-v1"

/home/kewei/17robo/ManiSkill/mani_skill/examples/demo_vis_pcd.py
这个文件也要改

在 demo_vis_pcd.py 的第 46 行：

pcd = trimesh.points.PointCloud(xyz, colors)

你需要在 xyz 和 colors 变量上 调用 .cpu().numpy() 来确保它们是 CPU 上的 NumPy 数组，而不是 GPU Tensor。

修改代码：

pcd = trimesh.points.PointCloud(xyz.cpu().numpy(), colors.cpu().numpy())

可视化分割数据#

您可以运行以下命令来可视化分段数据

python -m mani_skill.examples.demo_vis_segmentation -e "StackCube-v1"
python -m mani_skill.examples.demo_vis_segmentation -e "StackCube-v1"
--id id_of_part # mask out everything but the selected part

这一句命令目前还有报错

Visualizing 2 RGBD cameras
ID to Actor/Link name mappings
0: Background
1: Link, name - panda_link0
2: Link, name - panda_link1
3: Link, name - panda_link2
4: Link, name - panda_link3
5: Link, name - panda_link4
6: Link, name - panda_link5
7: Link, name - panda_link6
8: Link, name - panda_link7
9: Link, name - panda_link8
10: Link, name - panda_hand
11: Link, name - panda_hand_tcp
12: Link, name - panda_leftfinger
13: Link, name - panda_rightfinger
14: Link, name - camera_base_link
15: Link, name - camera_link
16: Actor, name - table-workspace
17: Actor, name - ground
18: Actor, name - cubeA
19: Actor, name - cubeB
Traceback (most recent call last):
File "/home/kewei/micromamba/envs/dl/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/kewei/micromamba/envs/dl/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/kewei/17robo/ManiSkill/mani_skill/examples/demo_vis_segmentation.py", line 144, in
main(parse_args())
File "/home/kewei/17robo/ManiSkill/mani_skill/examples/demo_vis_segmentation.py", line 118, in main
selected_id = reverse_seg_id_map[selected_id]
KeyError: 'id_of_part'

可视化相机纹理（RGB、深度、反照率等）#

您可以运行以下命令来可视化相机生成的任意数量的纹理。请注意，默认情况下使用的着色器是“默认”着色器，它输出您可能需要的几乎所有纹理。请参阅相机和着色器页面

python -m mani_skill.examples.demo_vis_textures -e "StackCube-v1" -o rgb+depth
python -m mani_skill.examples.demo_vis_textures -e "OpenCabinetDrawer-v1" -o rgb+depth+albedo+normal

可视化重置分布#

确定强化学习和模仿学习等机器学习算法的任务难度在很大程度上取决于任务的重置分布。要查看任何任务的重置分布（重复 env.reset 调用的结果），您可以运行以下命令将视频保存到videos文件夹中

python -m mani_skill.examples.demo_reset_distribution -e "PegInsertionSide-v1" --record-dir="videos"

可视化任意机器人#

运行以下命令打开查看器，显示空场景中给定的任意机器人（只有地板）。如果要可视化任何预定义的关键帧，您还可以指定不同的关键帧。

python -m mani_skill.examples.demo_robot -r "panda"