Welcome to XRFeitoria’s documentation!

XRFeitoria is a rendering toolbox for generating synthetic data photorealistic with ground-truth annotations. It is a part of the OpenXRLab project.

---

Major Features

• Support rendering photorealistic images with ground-truth annotations.

• Support multiple engine backends, including Unreal Engine and Blender.

• Support assets/camera management, including import, export, and delete.

• Support a CLI tool to render images from a mesh file.

---

Installation

pip install xrfeitoria

# to use visualization tools
pip install xrfeitoria[vis]

Requirements

• Python >= 3.8

• (optional) Unreal Engine >= 5.1

  Windows

  Linux

• (optional) Blender >= 3.0

  Windows

  Linux

  MacOS

---

CLI

A CLI tool to render images from a mesh file.

xf-render --help

# render a mesh file
xf-render {mesh_file}

# for example
wget https://graphics.stanford.edu/~mdfisher/Data/Meshes/bunny.obj
xf-render bunny.obj

Tutorials

A quick start guide to using the XRFeitoria package. The following tutorials cover the basic operations needed to render with XRFeitoria.

Note

Download notebook and run it offline.

Tutorial01 - Getting Started

Overview

🚀 This tutorial demonstrates a minimal application of XRFeitoria. By the end of this tutorial, you will be able to:

• Open Blender/Unreal Engine by XRFeitoria

• Import a mesh

• Add a camera

• Render images and get annotations of the mesh

1. Import XRFeitoria

After installing pip install xrfeitoria[vis], you can import it as follows:

import xrfeitoria as xf

2. Choose engine

XRFeitoria supports both Blender and Unreal Engine. Choose your engine and replace the following engine_exec_path with your own engine path.

Then, initialize XRFeitoria and open the engine by xf.init_blender or by xf.init_unreal.

# Replace with your executable path
engine_exec_path = 'C:/Program Files/Blender Foundation/Blender 3.3/blender.exe'
# engine_exec_path = 'C:/Program Files/Epic Games/UE_5.2/Engine/Binaries/Win64/UnrealEditor-Cmd.exe'

from pathlib import Path

exec_path_stem = Path(engine_exec_path).stem.lower()
if 'blender' in exec_path_stem:
    # Open Blender
    render_engine = 'blender'
    xf_runner = xf.init_blender(exec_path=engine_exec_path, background=False, new_process=True)
elif 'unreal' in exec_path_stem:
    # Unreal Engine requires a project to be opened
    # Here we use a sample project, which is downloaded from the following link
    # You can also use your own project
    import shutil

    from xrfeitoria.utils.downloader import download
    unreal_project_zip = download(url='https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/unreal_project/UE_Sample.zip',
                                    dst_dir="./tutorial01/assets/")
    shutil.unpack_archive(filename=unreal_project_zip, extract_dir='./tutorial01/assets/')

    # Open Unreal Engine
    render_engine = 'unreal'
    xf_runner = xf.init_unreal(exec_path=engine_exec_path,
                                background=False,
                                new_process=True,
                                project_path='./tutorial01/assets/UE_sample/UE_sample.uproject')

✨ Now you can see a new Blender/Unreal Engine process has been started.

• Blender

• Unreal Engine

3. Import a mesh

Download the scanned Koupen Chan model by the following cell.

from xrfeitoria.utils.downloader import download

# Download the Koupen-chan model
kc_path = download(url='https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/assets/koupen_chan.fbx',
                   dst_dir="./tutorial01/assets/")

Import the .fbx file to create an Actor instance.

Actor is the container of a mesh. By using Actor, you can place a mesh in the space and set its transform data (location, rotation and scale).

# Import the Koupen-chan
actor_kc = xf_runner.Actor.import_from_file(file_path=kc_path, stencil_value=255)

Switch to the engine window, and you can see the Koupen chan has been imported. The space to place Actor is called Level, and you can add, remove, or modify Actor in the Level.

• Blender

• Unreal Engine

If you use Unreal Engine, the Level should be saved after been modified.

# save the level
if render_engine == 'unreal':
    xf_runner.utils.save_current_level()

4. Add a sequence

Sequence is a multifunctional class in XRFeitoria. It can be used for:

• rendering

• adding transform keys

• grouping different objects

Here, we use it for rendering. Firstly we will add a Camera in the Sequence by using the function spawn_camera and set its location, rotation, and focal length. Then, we will add a render job to the renderer and set the render settings by using the function add_to_renderer.

from xrfeitoria.data_structure.models import RenderPass

# Use `with` statement to create a sequence, and it will be automatically close the sequence after the code block is executed.
sequence_name = 'MySequence'
with xf_runner.Sequence.new(seq_name=sequence_name, replace=True) as seq:

    # Add a camera and make it look at the koupen-chan
    camera_location = (0.0, -0.8, 0.0)
    camera_rotation = xf_runner.utils.get_rotation_to_look_at(location=camera_location, target=actor_kc.location)
    camera = seq.spawn_camera(location=camera_location, rotation=camera_rotation, fov=90)

    # Add a render job to renderer
    # In render job, you can specify the output path, resolution, render passes, etc.
    # The ``output_path`` is the path to save the rendered data.
    # The ``resolution`` is the resolution of the rendered image.
    # The ``render_passes`` define what kind of data you want to render, such as img, mask, normal, etc.
    # and what kind of format you want to save, such as png, exr, etc.
    seq.add_to_renderer(
        output_path=f'./tutorial01/outputs/{render_engine}/',
        resolution=(1280, 720),
        render_passes=[RenderPass('img', 'png'),
                       RenderPass('mask', 'exr'),
                       RenderPass('normal', 'exr'),
                       RenderPass('diffuse', 'exr')]
    )

5. Render

Use the following cell to render and save the images to the output_path you set in seq.add_to_renderer above.

xf_runner.render()

Check the output_path, and you can see the rendered images and various annotations. The following code shows the rendered image and the corresponding annotations.

import matplotlib.pyplot as plt

from xrfeitoria.utils.viewer import Viewer

xf_viewer = Viewer(sequence_dir=f'./tutorial01/outputs/{render_engine}/{sequence_name}/')
img = xf_viewer.get_img(camera_name=camera.name, frame=0)
diffuse = xf_viewer.get_diffuse(camera_name=camera.name, frame=0)
mask = xf_viewer.get_mask(camera_name=camera.name, frame=0)
normal = xf_viewer.get_normal(camera_name=camera.name, frame=0)

plt.figure(figsize=(20, 20))

plt.subplot(1, 4, 1)
plt.imshow(img)
plt.axis('off')
plt.title('img')

plt.subplot(1, 4, 2)
plt.imshow(diffuse)
plt.axis('off')
plt.title('diffuse')

plt.subplot(1, 4, 3)
plt.imshow(mask)
plt.axis('off')
plt.title('mask')

plt.subplot(1, 4, 4)
plt.imshow(normal)
plt.axis('off')
plt.title('normal')

Hint: When using Unreal Engine, if the image of the mask looks weird, try running the notebook again.

6. Final step

🥳 This is a good start! Finally, Do remember to close the engine.

xf_runner.close()

Ref to api docs, you can always use with statement to ensure the engine is closed when the codes are finished.

7. Conclusion

In this tutorial, we imported a mesh and rendered images for it. To accomplish this procedure, essential steps should be taken:

• Initialization

• Import an Actor

• Add a Sequence

• Add a Camera

• Render

It is worth mention that while Sequence does not directly perform rendering, the creation of Sequence is necessary for adding cameras and submitting rendering jobs. And the detailed definitions of the classes Actor, Camera, and Sequence can be referred to the documentation.

Note

Download notebook and run it offline.

Tutorial02 - Randomization

Overview

🚀 By the last tutorial, you have learned the concept of Actor, Level and Sequence. In this tutorial, you will learn how to add random transform keys to an Actor in order to render multiple poses of it. By the end of this tutorial, you will be able to:

• Initialize XRFeitoria

• Import Actor and label it by stencil value

• Set the scale of Actor in the Level

• Create a Sequence for rendering and adding transform keys to Actor

• Add a camera in the Sequence

• Render images and annotations

1. Initialization

Install the following packages that will be used in this tutorial:

%pip install objaverse
%pip install scipy

Then, similar to Tutorial01, specify your engine path and initialize XRFeitoria.

import xrfeitoria as xf

# Replace with your executable path
engine_exec_path = 'C:/Program Files/Blender Foundation/Blender 3.3/blender.exe'
# engine_exec_path = 'C:/Program Files/Epic Games/UE_5.2/Engine/Binaries/Win64/UnrealEditor-Cmd.exe'

from pathlib import Path

exec_path_stem = Path(engine_exec_path).stem.lower()
if 'blender' in exec_path_stem:
    # Open Blender
    render_engine = 'blender'
    xf_runner = xf.init_blender(exec_path=engine_exec_path, background=False, new_process=True)
elif 'unreal' in exec_path_stem:
    # Unreal Engine requires a project to be opened
    # Here we use a sample project, which is downloaded from the following link
    # You can also use your own project
    import shutil

    from xrfeitoria.utils.downloader import download
    unreal_project_zip = download(url='https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/unreal_project/UE_Sample.zip',
                                    dst_dir="./tutorial02/assets/")
    shutil.unpack_archive(filename=unreal_project_zip, extract_dir='./tutorial02/assets/')

    # Open Unreal Engine
    render_engine = 'unreal'
    xf_runner = xf.init_unreal(exec_path=engine_exec_path,
                                background=False,
                                new_process=True,
                                project_path='./tutorial02/assets/UE_sample/UE_sample.uproject')

✨ Now you can see a new Blender/Unreal Engine process has started.

2. Import meshes

Download some meshes from Objaverse.

import objaverse

objects = objaverse.load_objects(
    uids=['eb0807309530496aaab9dcff67bf5c31',
          'b4065dd5ce9d46be90db3e1f3e4b9cc1',
          '0176be079c2449e7aaebfb652910a854',
          'f130ebeb60f24ed8bd3714a7ed3ba280',
          '289a2221178843a78ad433705555e16a',
          'b7f7ab9bf7244c3a8851bae3fb0bf741',
        ],
    download_processes=1
)

Import the meshes to create Actor instances in the Level.

Here we set different stencil_value for each Actor. The stencil value is used to distinguish different Actors when rendering segmentation masks.

actors = []
for idx, file_path in enumerate(objects.values()):
    actor = xf_runner.Actor.import_from_file(
        file_path=file_path,
        stencil_value=(idx+1)*10
    )
    actors.append(actor)

Switch to the engine window, and you can see the meshes has been imported.

Then, we adjust the scale of the Actors to make their sizes equal to 0.3m.

actor_size = 0.3
for actor in actors:
    _scale = actor_size / max(actor.dimensions)
    actor.scale = (_scale, _scale, _scale)

Now they look like:

• Blender

• Unreal Engine

If you use Unreal Engine, the Level should be saved after been modified.

# save the level
if render_engine == 'unreal':
    xf_runner.utils.save_current_level()

3. Add a sequence for rendering and adding transform keys

Sequence is a multifunctional class in XRFeitoria. It can be used for: - rendering - adding transform keys - grouping different objects.

Here, we use it for rendering and adding transform keys.

Transform keys record the transformation (location, rotation and scale) of an Actor or a Camera at specific frames, and the transformation between two adjacent keys will be interpolated by the specified interpolation method. By adding transform keys, you can render multiple poses of an Actor.

In XRFeitoria, transform keys are always stored in a list, and the members of the list are SequenceTransformKey object.

Firstly, we randomly generate some transform keys for each actors.

import random

from loguru import logger
from scipy.stats import qmc

from xrfeitoria.data_structure.models import SequenceTransformKey as SeqTransKey

# Set the number of frames
frame_num = 10

# Use a dictionary to store the transform keys of all actors
all_actors_transform_keys = {actor: [] for actor in actors}

# Iterate over all frames
for i in range(frame_num):

    # Generate random locations by Poisson Disk Sampling
    # The minimum distance between two actors is set to be `actor_size` we defined before
    posson_engine = qmc.PoissonDisk(d=2, radius=actor_size)
    sample_location = posson_engine.random(len(actors))

    # Set the transform keys for each actor
    for actor_idx, actor in enumerate(actors):
        actor_scale = actor.scale

        # Get the location from the samples generated by Poisson Disk Sampling
        random_location = (sample_location[actor_idx][0],
                           0.0,
                           sample_location[actor_idx][1])

        # Generate random rotations
        random_rotation = (random.random() * 360.0,
                           random.random() * 360.0,
                           random.random() * 360.0)

        # Generate random scales
        scale = random.uniform(0.5, 1.0)
        random_scale = (scale * actor_scale[0],
                        scale * actor_scale[1],
                        scale * actor_scale[2])

        # Save the transform keys
        all_actors_transform_keys[actor].append(
            SeqTransKey(
                frame=i,
                location=random_location,
                rotation=random_rotation,
                scale=random_scale,
                interpolation='AUTO',
            )
        )
    logger.info(f'Generated transform keys of frame {i}.')

Then we create a Sequence to apply the transform keys to the actors and render the images.

from xrfeitoria.data_structure.models import RenderPass

# Use the `with` statement to create a sequence, and it will be automatically close the sequence after the code block is executed.
# The argument `seq_length` controls the number of frames to be rendered.
sequence_name = 'MySequence'
with xf_runner.Sequence.new(seq_name=sequence_name, seq_length=frame_num, replace=True) as seq:
    ##############################
    ##### Add transform keys #####
    ##############################

    # The function `use_actor_with_keys` sets transform keys for the actor in the sequence.
    # The transform keys are only stored in the sequence.
    # When the sequence is closed, the actor will be restored to its original state(without transform keys).
    for actor, keys in all_actors_transform_keys.items():
        seq.use_actor_with_keys(actor=actor, transform_keys=keys)

    #####################
    ##### Rendering #####
    #####################

    # Add a camera and make it look at the specified location
    camera_location = (0.5, -3.0, 0.5)
    camera_rotation = xf_runner.utils.get_rotation_to_look_at(location=camera_location, target=(0.5, 0.0, 0.5))
    camera = seq.spawn_camera(location=camera_location, rotation=camera_rotation, fov=45)

    # Add a render job to renderer
    # In render job, you can specify the output path, resolution, render passes, etc.
    # The ``output_path`` is the path to save the rendered data.
    # The ``resolution`` is the resolution of the rendered image.
    # The ``render_passes`` define what kind of data you want to render, such as img, depth, normal, etc.
    # and what kind of format you want to save, such as png, jpg, exr, etc.
    seq.add_to_renderer(
        output_path=f'./tutorial02/outputs/{render_engine}/',
        resolution=(1280, 720),
        render_passes=[RenderPass('img', 'png'),
                       RenderPass('mask', 'exr'),
                       RenderPass('normal', 'exr'),
                       RenderPass('diffuse', 'exr')],
    )

4. Render

The following code renders all the render jobs and save the images to the output_path set in seq.add_to_renderer above.

# Render
xf_runner.render()

Check the output_path, and you can see the rendered images and their annotations. Visualize the images and annotations by the following code.

import matplotlib.pyplot as plt

from xrfeitoria.utils.viewer import Viewer

xf_viewer = Viewer(sequence_dir=f'./tutorial02/outputs/{render_engine}/{sequence_name}/')

for i in range(frame_num):
    img = xf_viewer.get_img(camera_name=camera.name, frame=i)
    mask = xf_viewer.get_mask(camera_name=camera.name, frame=i)
    normal = xf_viewer.get_normal(camera_name=camera.name, frame=i)
    diffuse = xf_viewer.get_diffuse(camera_name=camera.name, frame=i)

    plt.figure(figsize=(20, 20))

    plt.subplot(1, 4, 1)
    plt.imshow(img)
    plt.axis('off')
    plt.title('img')

    plt.subplot(1, 4, 2)
    plt.imshow(mask)
    plt.axis('off')
    plt.title('mask')

    plt.subplot(1, 4, 3)
    plt.imshow(normal)
    plt.axis('off')
    plt.title('normal')

    plt.subplot(1, 4, 4)
    plt.imshow(diffuse)
    plt.axis('off')
    plt.title('diffuse')

Hint: When using Unreal Engine, if the images of the mask look weird, try running the notebook again.

Finally, close the engine by:

# Close the engine
xf_runner.close()

Ref to api docs, you can always use with statement to ensure the engine is closed when the codes are finished.

Note

Download notebook and run it offline.

Tutorial03 - Human NeRF

Overview

🚀 This tutorial provides an example of rendering animated skeletal meshes from different points of view. The rendered images can support various research topics, including human pose and shape estimation (HPS) and novel view synthesis for human (Human NeRF). By the end of this tutorial, you will be able to:

• Initialize XRFeitoria

• Import a skeletal mesh with animation

• Import another skeletal mesh without animation and setup animation for it

• Set Actor’s location in the Level

• Create a Sequence for rendering

• Add multiple static cameras in the Sequence

• Add a moving camera with transform keys in the Sequence

• Render images and annotations

1. Initialization

Then, similar to Tutorial01, specify your engine path and initialize XRFeitoria.

import xrfeitoria as xf

# Replace with your executable path
engine_exec_path = 'C:/Program Files/Blender Foundation/Blender 3.3/blender.exe'
# engine_exec_path = 'C:/Program Files/Epic Games/UE_5.2/Engine/Binaries/Win64/UnrealEditor-Cmd.exe'

from pathlib import Path

exec_path_stem = Path(engine_exec_path).stem.lower()
if 'blender' in exec_path_stem:
    # Open Blender
    render_engine = 'blender'
    xf_runner = xf.init_blender(exec_path=engine_exec_path, background=False, new_process=True)
elif 'unreal' in exec_path_stem:
    # Unreal Engine requires a project to be opened
    # Here we use a sample project, which is downloaded from the following link
    # You can also use your own project
    import shutil

    from xrfeitoria.utils.downloader import download
    unreal_project_zip = download(url='https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/unreal_project/UE_Sample.zip',
                                    dst_dir="./tutorial03/assets/")
    shutil.unpack_archive(filename=unreal_project_zip, extract_dir='./tutorial03/assets/')

    # Open Unreal Engine
    render_engine = 'unreal'
    xf_runner = xf.init_unreal(exec_path=engine_exec_path,
                                background=False,
                                new_process=True,
                                project_path='./tutorial03/assets/UE_sample/UE_sample.uproject')

✨ Now you can see a new Blender/Unreal Engine process has started.

2. Import skeletal meshes to Level

Download the skeletal meshes in SynBody to local folder and import them to Level.

from xrfeitoria.utils.downloader import download

# Download the skeletal meshes
actor1_path = download('https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/assets/SMPL-XL/SMPL-XL-00439__Subject_75_F_12.fbx', dst_dir="./tutorial03/assets/")
actor2_path = download('https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/assets/SMPL-XL/SMPL-XL-00045.fbx', dst_dir="./tutorial03/assets/")
actor2_motion_path = download('https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/assets/SMPL-XL/walking__15_01.fbx', dst_dir="./tutorial03/assets/")

Here we import two actors. The actor1 has animation and the actor2 has no animation.

And we set different stencil value for each Actor to distinguish different Actors when rendering segmentation masks.

# Import the skeletal mesh
actor1 = xf_runner.Actor.import_from_file(file_path=actor1_path, stencil_value=100)
actor2 = xf_runner.Actor.import_from_file(file_path=actor2_path, stencil_value=200)

Then, we load an animation from another file and set it to the actor2

actor2.setup_animation(animation_path=actor2_motion_path)

We can also modify the properties of the actors, such as the location, rotation, and scale.

# Set the location of the two actors to make their distance to be 1.0 meter
actor1_location = actor1.location
actor2_location = actor2.location
actor2.location = (actor1_location[0] + 1.0, actor1_location[1], actor1_location[2])

Now they look like:

• Blender

• Unreal Engine

If you use Unreal Engine, the Level should be saved after been modified.

# save the level
if render_engine == 'unreal':
    xf_runner.utils.save_current_level()

3. Add a sequence for rendering

Sequence is a multifunctional class in XRFeitoria. It can be used for: - rendering - adding transform keys - grouping different objects.

Here, we use it for rendering.

import math

from xrfeitoria.data_structure.models import RenderPass
from xrfeitoria.data_structure.models import SequenceTransformKey as SeqTransKey

# Use `with` statement to create a sequence, and it will be automatically close the sequence after the code block is executed.
# The argument `seq_length` controls the number of frames to be rendered.
sequence_name = 'MySequence'
frame_num = 6
with xf_runner.Sequence.new(seq_name=sequence_name, seq_length=frame_num, replace=True) as seq:

    # Get the bounding boxes of the actors
    actor1_bbox = actor1.bound_box
    actor2_bbox = actor2.bound_box

    # Get the center location of the actors
    actor1_center = ((actor1_bbox[0][0] + actor1_bbox[1][0]) / 2, (actor1_bbox[0][1] + actor1_bbox[1][1]) / 2, (actor1_bbox[0][2] + actor1_bbox[1][2]) / 2)
    actor2_center = ((actor2_bbox[0][0] + actor2_bbox[1][0]) / 2, (actor2_bbox[0][1] + actor2_bbox[1][1]) / 2, (actor2_bbox[0][2] + actor2_bbox[1][2]) / 2)
    actors_center = ((actor1_center[0] + actor2_center[0]) / 2, (actor1_center[1] + actor2_center[1]) / 2, (actor1_center[2] + actor2_center[2]) / 2)

    ##########################################################################
    # Add 6 static cameras and a moving camera around the actors for rendering
    ##########################################################################
    # Set cameras' field of view to 90°
    camera_fov = 90
    # Set cameras' distance to 3.0m
    distance_to_actor = 3.0
    # Prepare the transform keys for moving camera
    transform_keys = []
    # calculate the location and rotation of the cameras
    for i in range(6):
        azimuth = 360 / 6 * i
        azimuth_radians = math.radians(azimuth)

        x = distance_to_actor * math.cos(azimuth_radians) + actors_center[0]
        y = distance_to_actor * math.sin(azimuth_radians) + actors_center[1]
        z = 0.0 + actors_center[2]
        location = (x, y, z)
        # Set camera's rotation to look at the actor's center
        rotation = xf_runner.utils.get_rotation_to_look_at(location=location, target=actors_center)

        # Add a static camera
        static_camera = seq.spawn_camera(
            camera_name=f'static_camera_{i}',
            location=location,
            rotation=rotation,
            fov=camera_fov,
        )

        # Add a transform key to the moving camera
        transform_keys.append(
            SeqTransKey(
                frame=i,
                location=location,
                rotation=rotation,
                interpolation='AUTO',
            )
        )

    # Add a moving camera rotating around the actors
    moving_camera = seq.spawn_camera_with_keys(
        camera_name=f'moving_camera',
        transform_keys=transform_keys,
        fov=camera_fov,
    )

    # Add a render job to renderer
    # In render job, you can specify the output path, resolution, render passes, etc.
    # The output path is the path to save the rendered data.
    # The resolution is the resolution of the rendered image.
    # The render passes define what kind of data you want to render, such as img, depth, normal, etc.
    # and what kind of format you want to save, such as png, exr, etc.
    seq.add_to_renderer(
        output_path=f'./tutorial03/outputs/{render_engine}/',
        resolution=(1280, 720),
        render_passes=[RenderPass('img', 'png'),
                       RenderPass('mask', 'exr'),
                       RenderPass('normal', 'exr'),
                       RenderPass('diffuse', 'exr')]
    )

4. Render

The following code renders all the render jobs and save the images to the output_path set in seq.add_to_renderer above.

# Render
xf_runner.render()

Check the output_path, and you can see that for the frame i, the image rendered by the moving camera is the same as the image rendered by the ith static camera. For example, the moving_camera/0002.png is the same as the static_camera_2/0002.png.

import matplotlib.pyplot as plt

from xrfeitoria.utils.viewer import Viewer

xf_viewer = Viewer(sequence_dir=f'./tutorial03/outputs/{render_engine}/{sequence_name}/')

moving_camera_img = xf_viewer.get_img(camera_name='moving_camera', frame=2)
static_camera_img = xf_viewer.get_img(camera_name='static_camera_2', frame=2)

plt.figure(figsize=(20, 20))

plt.subplot(1, 2, 1)
plt.imshow(moving_camera_img)
plt.axis('off')
plt.title('moving_camera/0002.png')

plt.subplot(1, 2, 2)
plt.imshow(static_camera_img)
plt.axis('off')
plt.title('static_camera_2/0002.png')

View the rendered images and annotations of the camera static_camera_2 by:

import matplotlib.pyplot as plt

from xrfeitoria.utils.viewer import Viewer

xf_viewer = Viewer(sequence_dir=f'./tutorial03/outputs/{render_engine}/{sequence_name}/')

camera_name = 'static_camera_2'
for i in range(frame_num):
    img = xf_viewer.get_img(camera_name=camera_name, frame=i)
    mask = xf_viewer.get_mask(camera_name=camera_name, frame=i)
    normal = xf_viewer.get_normal(camera_name=camera_name, frame=i)
    diffuse = xf_viewer.get_diffuse(camera_name=camera_name, frame=i)

    plt.figure(figsize=(20, 20))

    plt.subplot(1, 4, 1)
    plt.imshow(img)
    plt.axis('off')
    plt.title('img')

    plt.subplot(1, 4, 2)
    plt.imshow(mask)
    plt.axis('off')
    plt.title('mask')

    plt.subplot(1, 4, 3)
    plt.imshow(normal)
    plt.axis('off')
    plt.title('normal')

    plt.subplot(1, 4, 4)
    plt.imshow(diffuse)
    plt.axis('off')
    plt.title('diffuse')

Hint: When using Unreal Engine, if the images of the mask look weird, try running the notebook again.

Finally, close the engine by:

xf_runner.close()

Ref to api docs, you can always use with statement to ensure the engine is closed when the codes are finished.

---

Extra Samples

For more advanced examples, see the samples section.

---

xrfeitoria

Initialize firstly and run xrfeitoria.

BlenderUnreal

import xrfeitoria as xf
with xf.init_blender() as xf_runner:
    ...

xf_runner is an instance of XRFeitoriaBlender, where contains all the classes and methods to run xrfeitoria.

After initialized, use members of xf_runner.

Ref to members of XRFeitoriaBlender and XRFeitoriaUnreal.

See also

Ref to Tutorial01.

Ref to docs of xrfeitoria.init_blender and xrfeitoria.init_unreal.

xrfeitoria.factory

Inheritance diagram

---

xrfeitoria.factory.init_blender([...])	Initialize Blender with XRFeitoria, which would start Blender as RPC server, and yield a XRFeitoriaBlender object.
xrfeitoria.factory.init_unreal([exec_path, ...])	Initialize Unreal with XRFeitoria, which would start Unreal as RPC server, and yield a XRFeitoriaUnreal object.
xrfeitoria.factory.XRFeitoriaBlender([...])	Factory class contains all the classes and functions for Blender.
xrfeitoria.factory.XRFeitoriaUnreal([...])	Factory class contains all the classes and functions for Unreal.

xrfeitoria.object

Inheritance diagram

---

xrfeitoria.object.object_base.ObjectBase(name)	Base class for all objects in the world.
xrfeitoria.object.object_utils.ObjectUtilsBase()	Base class for object utils.
xrfeitoria.object.object_utils.ObjectUtilsBlender()	Object utils class for Blender.
xrfeitoria.object.object_utils.ObjectUtilsUnreal()	Object utils class for Unreal.

xrfeitoria.actor

Inheritance diagram

---

xrfeitoria.actor.actor_base.ActorBase(name)	Base class for all actors in the world.
xrfeitoria.actor.actor_blender.ActorBlender(name)	Actor class for Blender.
xrfeitoria.actor.actor_unreal.ActorUnreal(name)	Actor class for Unreal Engine.
xrfeitoria.actor.actor_blender.ShapeBlenderWrapper()	Wrapper class for shapes in Blender.
xrfeitoria.actor.actor_unreal.ShapeUnrealWrapper()	Wrapper class for shapes in Unreal Engine.

xrfeitoria.camera

Inheritance diagram

---

xrfeitoria.camera.camera_base.CameraBase(name)	Base camera class.
xrfeitoria.camera.camera_blender.CameraBlender(name)	Camera class for Blender.
xrfeitoria.camera.camera_unreal.CameraUnreal(name)	Camera class for Unreal.
xrfeitoria.camera.camera_parameter.CameraParameter(K, R, T)	Camera parameter class for pinhole camera model.

xrfeitoria.sequence

Inheritance diagram

---

xrfeitoria.sequence.sequence_base.SequenceBase()	Base sequence class.
xrfeitoria.sequence.sequence_blender.SequenceBlender()	Sequence class for Blender.
xrfeitoria.sequence.sequence_unreal.SequenceUnreal()	Sequence class for Unreal.

---

xrfeitoria.sequence.sequence_wrapper

Sequence wrapper functions.

xrfeitoria.renderer

Inheritance diagram

---

xrfeitoria.renderer.renderer_base.RendererBase()	Base renderer class.
xrfeitoria.renderer.renderer_blender.RendererBlender()	Renderer class for Blender.
xrfeitoria.renderer.renderer_unreal.RendererUnreal()	Renderer class for Unreal.

xrfeitoria.utils

Remote Functions

xrfeitoria.utils.functions.blender_functions	Remote functions for blender.
xrfeitoria.utils.functions.unreal_functions	Remote functions for unreal.

Animation utils

xrfeitoria.utils.anim.motion	Motion data structure and related functions.
xrfeitoria.utils.anim.utils	Utilities for animation data loading and dumping.

RPC runner

xrfeitoria.utils.runner

Runner for starting blender or unreal as a rpc server.

Tools

xrfeitoria.utils.tools	Utils tools for logging and progress bar.
xrfeitoria.utils.viewer	Utils for loading images and annotations.
xrfeitoria.utils.projector	Project 3D points to 2D points, and draw 3D points on image, using XRPrimer camera.
xrfeitoria.utils.validations	Validations for the arguments.

xrfeitoria.data_structure

Models

> https://docs.pydantic.dev/latest/usage/models/

xrfeitoria.data_structure.models.RenderPass	Render pass model contains render layer and image format.
xrfeitoria.data_structure.models.RenderJobBlender	Render job model for Blender.
xrfeitoria.data_structure.models.RenderJobUnreal	Render job model for Unreal.
xrfeitoria.data_structure.models.SequenceTransformKey	Transform key model for object (actor/camera) in sequence.

Enumerations

xrfeitoria.data_structure.constants.EngineEnum(value)	Render engine enum.
xrfeitoria.data_structure.constants.ImportFileFormatEnum(value)	Import file format enum.
xrfeitoria.data_structure.constants.ImageFileFormatEnum(value)	Image file format enum.
xrfeitoria.data_structure.constants.RenderOutputEnumBlender(value)	Render layer enum of Blender.
xrfeitoria.data_structure.constants.RenderOutputEnumUnreal(value)	Render layer enum of Unreal.
xrfeitoria.data_structure.constants.InterpolationEnumUnreal(value)	Keyframe interpolation enum of Unreal.
xrfeitoria.data_structure.constants.InterpolationEnumBlender(value)	Keyframe interpolation enum of Blender.
xrfeitoria.data_structure.constants.RenderEngineEnumBlender(value)	Render engine enum of Blender.
xrfeitoria.data_structure.constants.ShapeTypeEnumBlender(value)	Shape type enum of Blender.
xrfeitoria.data_structure.constants.ShapeTypeEnumUnreal(value)	Shape type enum of Unreal.

---

XRFeitoria Blender Addon

By running xf.init_blender, a XRFeitoria addon will be installed in your Blender. The source code of the addon is in src/XRFeitoriaBpy. It enables you to start RPC server and manage your sequences.

The addon panel is in the top-right corner of the 3D Viewport.

Click the button Start RPC Server to start the RPC server. The server will listen on port 9997 to receive commands from XRFeitoria and execute them in Blender.

Click the drop down box Active in the Manage sequences section to open a sequence belonging to the current level. The objects in the sequence will be loaded to the window, and the properties of the sequence such as fps, frame_start, frame_end and frame_current will be set.

Click the ‘x’ button beside the drop down box to close the sequence.

XRFeitoria Unreal Plugin

TBD

---

Frequently Asked Questions

We list some common troubles faced by many users and their corresponding solutions here. Feel free to enrich the list if you find any frequent issues and have ways to help others to solve them. If the contents here do not cover your issue, do not hesitate to create an issue!

---

API

How to use the plugin of Blender/Unreal under development

First you should clone the repo of XRFeitoria, and maybe modify the code of the plugin under src/XRFeitoriaBlender or src/XRFeitoriaUnreal. Then you can use the plugin under development by setting dev_plugin=True in init_blender or init_unreal.

You can install the plugin by:

BlenderUnreal

git clone https://github.com/openxrlab/xrfeitoria.git
cd xrfeitoria
pip install -e .
python -c "import xrfeitoria as xf; xf.init_blender(replace_plugin=True, dev_plugin=True)"

# or through the code in tests
python -m tests.blender.init --dev [-b]

Build plugins

If you want to publish plugins of your own, you can use the following command:

# install xrfeitoria first
cd xrfeitoria
pip install .

# for instance, build plugins for Blender, UE 5.1, UE 5.2, and UE 5.3 on Windows.
# using powershell where backtick(`) is the line continuation character.
python -m xrfeitoria.utils.publish_plugins `
  -u "C:/Program Files/Epic Games/UE_5.1/Engine/Binaries/Win64/UnrealEditor-Cmd.exe" `
  -u "C:/Program Files/Epic Games/UE_5.2/Engine/Binaries/Win64/UnrealEditor-Cmd.exe" `
  -u "C:/Program Files/Epic Games/UE_5.3/Engine/Binaries/Win64/UnrealEditor-Cmd.exe"

Please check the path ./src/dist for the generated plugins. XRFeitoriaBlender will be archived by default, and XRFeitoriaUnreal will only be built when you specify the Unreal editor path. Make sure you have installed the corresponding Unreal Engine and Visual Studio before building the unreal plugin.

Find out the plugin version in ./xrfeitoria/version.py. Or by:

>>> python -c "import xrfeitoria; print(xrfeitoria.__version__)"
0.6.1.dev10+gd12997e.d20240122

You can set the environment variable XRFEITORIA__DIST_ROOT and XRFEITORIA__VERSION to change the plugins used by XRFeitoria. Run your code xxx.py like:

UNIXWindows

XRFEITORIA__VERSION=$(python -c "import xrfeitoria; print(xrfeitoria.__version__)") \
XRFEITORIA__DIST_ROOT=src/dist \
    python xxx.py

What is stencil_value

stencil_value is to distinguish different actors in the sequence when rendering segmentation masks. The RGB mask color of actors will be saved in the {output_path}/actor_infos.json of the render job.

In:

• ActorBase.import_from_file

• SequenceUnreal.spawn_actor

• …

What is console_variables

console_variables is a dictionary of console variables for configure detailed rendering settings. Please refer to the official documentation for more details: Unreal-MRQ-Doc.

example:

>>> console_variables = {'r.MotionBlurQuality': 0}  # disable motion blur

In:

• RenderJobUnreal.console_variables

• …

RPC Port

The RPC port is used for communication between python and engine (blender/unreal). If the default port is occupied, or you want to use multiple engines at the same time, you can set the environment variable BLENDER_PORT or UNREAL_PORT to change it.

UNIXWindows

BLENDER_PORT=50051 python xxx.py