Note

Download notebook and run it offline.

Tutorial01 - Getting Started

Overview

🚀 This tutorial demonstrates a minimal application of XRFeitoria. By the end of this tutorial, you will be able to:

  • Open Blender/Unreal Engine by XRFeitoria

  • Import a mesh

  • Add a camera

  • Render images and get annotations of the mesh

1. Import XRFeitoria

After installing pip install xrfeitoria[vis], you can import it as follows:

[ ]:

import xrfeitoria as xf

2. Choose engine

XRFeitoria supports both Blender and Unreal Engine. Choose your engine and replace the following engine_exec_path with your own engine path.

Then, initialize XRFeitoria and open the engine by xf.init_blender or by xf.init_unreal.

[ ]:

# Replace with your executable path
engine_exec_path = 'C:/Program Files/Blender Foundation/Blender 3.3/blender.exe'
# engine_exec_path = 'C:/Program Files/Epic Games/UE_5.2/Engine/Binaries/Win64/UnrealEditor-Cmd.exe'

[ ]:

from pathlib import Path

exec_path_stem = Path(engine_exec_path).stem.lower()
if 'blender' in exec_path_stem:
    # Open Blender
    render_engine = 'blender'
    xf_runner = xf.init_blender(exec_path=engine_exec_path, background=False, new_process=True)
elif 'unreal' in exec_path_stem:
    # Unreal Engine requires a project to be opened
    # Here we use a sample project, which is downloaded from the following link
    # You can also use your own project
    import shutil

    from xrfeitoria.utils.downloader import download
    unreal_project_zip = download(url='https://github.com/openxrlab/xrfeitoria/releases/download/v0.1.0/UE--XRFeitoriaUnreal_Sample.zip',
                                    dst_dir="./tutorial01/assets/")
    shutil.unpack_archive(filename=unreal_project_zip, extract_dir='./tutorial01/assets/')

    # Open Unreal Engine
    render_engine = 'unreal'
    xf_runner = xf.init_unreal(exec_path=engine_exec_path,
                                background=False,
                                new_process=True,
                                project_path='./tutorial01/assets/UE_sample/UE_sample.uproject')

✨ Now you can see a new Blender/Unreal Engine process has been started.

  • Blender image0

  • Unreal Engine image1

3. Import a mesh

Download the scanned Koupen Chan model by the following cell.

[ ]:

from xrfeitoria.utils.downloader import download

# Download the Koupen-chan model
kc_path = download(url='https://github.com/openxrlab/xrfeitoria/releases/download/v0.1.0/assets--koupen_chan.fbx',
                   dst_dir="./tutorial01/assets/")

Import the .fbx file to create an Actor instance.

Actor is the container of a mesh. By using Actor, you can place a mesh in the space and set its transform data (location, rotation and scale).

[ ]:

# Import the Koupen-chan
actor_kc = xf_runner.Actor.import_from_file(file_path=kc_path, stencil_value=255)

Switch to the engine window, and you can see the Koupen chan has been imported. The space to place Actor is called Level, and you can add, remove, or modify Actor in the Level.

  • Blender image0

  • Unreal Engine image1

If you use Unreal Engine, the Level should be saved after been modified.

[ ]:

# save the level
if render_engine == 'unreal':
    xf_runner.utils.save_current_level()

4. Add a sequence

Sequence is a multifunctional class in XRFeitoria. It can be used for:

  • rendering

  • adding transform keys

  • grouping different objects

Here, we use it for rendering. Firstly we will add a Camera in the Sequence by using the function spawn_camera and set its location, rotation, and focal length. Then, we will add a render job to the renderer and set the render settings by using the function add_to_renderer.

[ ]:

from xrfeitoria.data_structure.models import RenderPass

# Use `with` statement to create a sequence, and it will be automatically close the sequence after the code block is executed.
sequence_name = 'MySequence'
with xf_runner.sequence(seq_name=sequence_name, replace=True) as seq:

    # Add a camera and make it look at the koupen-chan
    camera_location = (0.0, -0.8, 0.0)
    camera_rotation = xf_runner.utils.get_rotation_to_look_at(location=camera_location, target=actor_kc.location)
    camera = seq.spawn_camera(location=camera_location, rotation=camera_rotation, fov=90)

    # Add a render job to renderer
    # In render job, you can specify the output path, resolution, render passes, etc.
    # The ``output_path`` is the path to save the rendered data.
    # The ``resolution`` is the resolution of the rendered image.
    # The ``render_passes`` define what kind of data you want to render, such as img, mask, normal, etc.
    # and what kind of format you want to save, such as png, exr, etc.
    seq.add_to_renderer(
        output_path=f'./tutorial01/outputs/{render_engine}/',
        resolution=(1280, 720),
        render_passes=[RenderPass('img', 'png'),
                       RenderPass('mask', 'exr'),
                       RenderPass('normal', 'exr'),
                       RenderPass('diffuse', 'exr')]
    )

5. Render

Use the following cell to render and save the images to the output_path you set in seq.add_to_renderer above.

[ ]:

xf_runner.render()

Check the output_path, and you can see the rendered images and various annotations. The following code shows the rendered image and the corresponding annotations.

[ ]:

import matplotlib.pyplot as plt

from xrfeitoria.utils.viewer import Viewer

xf_viewer = Viewer(sequence_dir=f'./tutorial01/outputs/{render_engine}/{sequence_name}/')
img = xf_viewer.get_img(camera_name=camera.name, frame=0)
diffuse = xf_viewer.get_diffuse(camera_name=camera.name, frame=0)
mask = xf_viewer.get_mask(camera_name=camera.name, frame=0)
normal = xf_viewer.get_normal(camera_name=camera.name, frame=0)

plt.figure(figsize=(20, 20))

plt.subplot(1, 4, 1)
plt.imshow(img)
plt.axis('off')
plt.title('img')

plt.subplot(1, 4, 2)
plt.imshow(diffuse)
plt.axis('off')
plt.title('diffuse')

plt.subplot(1, 4, 3)
plt.imshow(mask)
plt.axis('off')
plt.title('mask')

plt.subplot(1, 4, 4)
plt.imshow(normal)
plt.axis('off')
plt.title('normal')

Hint: When using Unreal Engine, if the image of the mask looks weird, try running the notebook again.

6. Final step

🥳 This is a good start! Finally, Do remember to close the engine.

[ ]:

xf_runner.close()

Ref to api docs, you can always use with statement to ensure the engine is closed when the codes are finished.

7. Conclusion

In this tutorial, we imported a mesh and rendered images for it. To accomplish this procedure, essential steps should be taken:

  • Initialization

  • Import an Actor

  • Add a Sequence

  • Add a Camera

  • Render

It is worth mention that while Sequence does not directly perform rendering, the creation of Sequence is necessary for adding cameras and submitting rendering jobs. And the detailed definitions of the classes Actor, Camera, and Sequence can be referred to the documentation.