Note

Download notebook and run it offline.

Tutorial03 - Human NeRF

Overview

🚀 This tutorial provides an example of rendering animated skeletal meshes from different points of view. The rendered images can support various research topics, including human pose and shape estimation (HPS) and novel view synthesis for human (Human NeRF). By the end of this tutorial, you will be able to:

  • Initialize XRFeitoria

  • Import a skeletal mesh with animation

  • Import another skeletal mesh without animation and setup animation for it

  • Set Actor’s location in the Level

  • Create a Sequence for rendering

  • Add multiple static cameras in the Sequence

  • Add a moving camera with transform keys in the Sequence

  • Render images and annotations

1. Initialization

Then, similar to Tutorial01, specify your engine path and initialize XRFeitoria.

[ ]:

import xrfeitoria as xf

[ ]:

# Replace with your executable path
engine_exec_path = 'C:/Program Files/Blender Foundation/Blender 3.3/blender.exe'
# engine_exec_path = 'C:/Program Files/Epic Games/UE_5.2/Engine/Binaries/Win64/UnrealEditor-Cmd.exe'

[ ]:

from pathlib import Path

exec_path_stem = Path(engine_exec_path).stem.lower()
if 'blender' in exec_path_stem:
    # Open Blender
    render_engine = 'blender'
    xf_runner = xf.init_blender(exec_path=engine_exec_path, background=False, new_process=True)
elif 'unreal' in exec_path_stem:
    # Unreal Engine requires a project to be opened
    # Here we use a sample project, which is downloaded from the following link
    # You can also use your own project
    import shutil

    from xrfeitoria.utils.downloader import download
    unreal_project_zip = download(url='https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/unreal_project/UE_Sample.zip',
                                    dst_dir="./tutorial03/assets/")
    shutil.unpack_archive(filename=unreal_project_zip, extract_dir='./tutorial03/assets/')

    # Open Unreal Engine
    render_engine = 'unreal'
    xf_runner = xf.init_unreal(exec_path=engine_exec_path,
                                background=False,
                                new_process=True,
                                project_path='./tutorial03/assets/UE_sample/UE_sample.uproject')

✨ Now you can see a new Blender/Unreal Engine process has started.

2. Import skeletal meshes to Level

Download the skeletal meshes in SynBody to local folder and import them to Level.

[ ]:

from xrfeitoria.utils.downloader import download

# Download the skeletal meshes
actor1_path = download('https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/assets/SMPL-XL/SMPL-XL-00439__Subject_75_F_12.fbx', dst_dir="./tutorial03/assets/")
actor2_path = download('https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/assets/SMPL-XL/SMPL-XL-00045.fbx', dst_dir="./tutorial03/assets/")
actor2_motion_path = download('https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/assets/SMPL-XL/walking__15_01.fbx', dst_dir="./tutorial03/assets/")

Here we import two actors. The actor1 has animation and the actor2 has no animation.

And we set different stencil value for each Actor to distinguish different Actors when rendering segmentation masks.

[ ]:

# Import the skeletal mesh
actor1 = xf_runner.Actor.import_from_file(file_path=actor1_path, stencil_value=100)
actor2 = xf_runner.Actor.import_from_file(file_path=actor2_path, stencil_value=200)

Then, we load an animation from another file and set it to the actor2

[ ]:

actor2.setup_animation(animation_path=actor2_motion_path)

We can also modify the properties of the actors, such as the location, rotation, and scale.

[ ]:

# Set the location of the two actors to make their distance to be 1.0 meter
actor1_location = actor1.location
actor2_location = actor2.location
actor2.location = (actor1_location[0] + 1.0, actor1_location[1], actor1_location[2])

Now they look like:

  • Blender image0

  • Unreal Engine image1

If you use Unreal Engine, the Level should be saved after been modified.

[ ]:

# save the level
if render_engine == 'unreal':
    xf_runner.utils.save_current_level()

3. Add a sequence for rendering

Sequence is a multifunctional class in XRFeitoria. It can be used for: - rendering - adding transform keys - grouping different objects.

Here, we use it for rendering.

[ ]:

import math

from xrfeitoria.data_structure.models import RenderPass
from xrfeitoria.data_structure.models import SequenceTransformKey as SeqTransKey

# Use `with` statement to create a sequence, and it will be automatically close the sequence after the code block is executed.
# The argument `seq_length` controls the number of frames to be rendered.
sequence_name = 'MySequence'
frame_num = 6
with xf_runner.Sequence.new(seq_name=sequence_name, seq_length=frame_num, replace=True) as seq:

    # Get the bounding boxes of the actors
    actor1_bbox = actor1.bound_box
    actor2_bbox = actor2.bound_box

    # Get the center location of the actors
    actor1_center = ((actor1_bbox[0][0] + actor1_bbox[1][0]) / 2, (actor1_bbox[0][1] + actor1_bbox[1][1]) / 2, (actor1_bbox[0][2] + actor1_bbox[1][2]) / 2)
    actor2_center = ((actor2_bbox[0][0] + actor2_bbox[1][0]) / 2, (actor2_bbox[0][1] + actor2_bbox[1][1]) / 2, (actor2_bbox[0][2] + actor2_bbox[1][2]) / 2)
    actors_center = ((actor1_center[0] + actor2_center[0]) / 2, (actor1_center[1] + actor2_center[1]) / 2, (actor1_center[2] + actor2_center[2]) / 2)

    ##########################################################################
    # Add 6 static cameras and a moving camera around the actors for rendering
    ##########################################################################
    # Set cameras' field of view to 90°
    camera_fov = 90
    # Set cameras' distance to 3.0m
    distance_to_actor = 3.0
    # Prepare the transform keys for moving camera
    transform_keys = []
    # calculate the location and rotation of the cameras
    for i in range(6):
        azimuth = 360 / 6 * i
        azimuth_radians = math.radians(azimuth)

        x = distance_to_actor * math.cos(azimuth_radians) + actors_center[0]
        y = distance_to_actor * math.sin(azimuth_radians) + actors_center[1]
        z = 0.0 + actors_center[2]
        location = (x, y, z)
        # Set camera's rotation to look at the actor's center
        rotation = xf_runner.utils.get_rotation_to_look_at(location=location, target=actors_center)

        # Add a static camera
        static_camera = seq.spawn_camera(
            camera_name=f'static_camera_{i}',
            location=location,
            rotation=rotation,
            fov=camera_fov,
        )

        # Add a transform key to the moving camera
        transform_keys.append(
            SeqTransKey(
                frame=i,
                location=location,
                rotation=rotation,
                interpolation='AUTO',
            )
        )

    # Add a moving camera rotating around the actors
    moving_camera = seq.spawn_camera_with_keys(
        camera_name=f'moving_camera',
        transform_keys=transform_keys,
        fov=camera_fov,
    )

    # Add a render job to renderer
    # In render job, you can specify the output path, resolution, render passes, etc.
    # The output path is the path to save the rendered data.
    # The resolution is the resolution of the rendered image.
    # The render passes define what kind of data you want to render, such as img, depth, normal, etc.
    # and what kind of format you want to save, such as png, exr, etc.
    seq.add_to_renderer(
        output_path=f'./tutorial03/outputs/{render_engine}/',
        resolution=(1280, 720),
        render_passes=[RenderPass('img', 'png'),
                       RenderPass('mask', 'exr'),
                       RenderPass('normal', 'exr'),
                       RenderPass('diffuse', 'exr')]
    )

4. Render

The following code renders all the render jobs and save the images to the output_path set in seq.add_to_renderer above.

[ ]:

# Render
xf_runner.render()

Check the output_path, and you can see that for the frame i, the image rendered by the moving camera is the same as the image rendered by the ith static camera. For example, the moving_camera/0002.png is the same as the static_camera_2/0002.png.

[ ]:

import matplotlib.pyplot as plt

from xrfeitoria.utils.viewer import Viewer

xf_viewer = Viewer(sequence_dir=f'./tutorial03/outputs/{render_engine}/{sequence_name}/')

moving_camera_img = xf_viewer.get_img(camera_name='moving_camera', frame=2)
static_camera_img = xf_viewer.get_img(camera_name='static_camera_2', frame=2)

plt.figure(figsize=(20, 20))

plt.subplot(1, 2, 1)
plt.imshow(moving_camera_img)
plt.axis('off')
plt.title('moving_camera/0002.png')

plt.subplot(1, 2, 2)
plt.imshow(static_camera_img)
plt.axis('off')
plt.title('static_camera_2/0002.png')

View the rendered images and annotations of the camera static_camera_2 by:

[ ]:

import matplotlib.pyplot as plt

from xrfeitoria.utils.viewer import Viewer

xf_viewer = Viewer(sequence_dir=f'./tutorial03/outputs/{render_engine}/{sequence_name}/')

camera_name = 'static_camera_2'
for i in range(frame_num):
    img = xf_viewer.get_img(camera_name=camera_name, frame=i)
    mask = xf_viewer.get_mask(camera_name=camera_name, frame=i)
    normal = xf_viewer.get_normal(camera_name=camera_name, frame=i)
    diffuse = xf_viewer.get_diffuse(camera_name=camera_name, frame=i)

    plt.figure(figsize=(20, 20))

    plt.subplot(1, 4, 1)
    plt.imshow(img)
    plt.axis('off')
    plt.title('img')

    plt.subplot(1, 4, 2)
    plt.imshow(mask)
    plt.axis('off')
    plt.title('mask')

    plt.subplot(1, 4, 3)
    plt.imshow(normal)
    plt.axis('off')
    plt.title('normal')

    plt.subplot(1, 4, 4)
    plt.imshow(diffuse)
    plt.axis('off')
    plt.title('diffuse')

Hint: When using Unreal Engine, if the images of the mask look weird, try running the notebook again.

Finally, close the engine by:

[ ]:

xf_runner.close()

Ref to api docs, you can always use with statement to ensure the engine is closed when the codes are finished.