{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# Tutorial03 - Human NeRF\n",
                "\n",
                "## Overview\n",
                "\n",
                "🚀 This tutorial provides an example of rendering animated skeletal meshes from different points of view. The rendered images can support various research topics, including human pose and shape estimation (HPS) and novel view synthesis for human (Human NeRF). By the end of this tutorial, you will be able to:\n",
                "\n",
                "- Initialize XRFeitoria\n",
                "- Import a skeletal mesh with animation\n",
                "- Import another skeletal mesh without animation and setup animation for it\n",
                "- Set ``Actor``'s location in the ``Level``\n",
                "- Create a `Sequence` for rendering\n",
                "- Add multiple static cameras in the `Sequence`\n",
                "- Add a moving camera with transform keys in the `Sequence`\n",
                "- Render images and annotations"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 1. Initialization\n",
                "\n",
                "Then, similar to [Tutorial01](./01_get_started.ipynb), specify your engine path and initialize XRFeitoria."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "import xrfeitoria as xf"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "tags": []
            },
            "outputs": [],
            "source": [
                "# Replace with your executable path\n",
                "engine_exec_path = 'C:/Program Files/Blender Foundation/Blender 3.3/blender.exe'\n",
                "# engine_exec_path = 'C:/Program Files/Epic Games/UE_5.2/Engine/Binaries/Win64/UnrealEditor-Cmd.exe'"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from pathlib import Path\n",
                "\n",
                "exec_path_stem = Path(engine_exec_path).stem.lower()\n",
                "if 'blender' in exec_path_stem:\n",
                "    # Open Blender\n",
                "    render_engine = 'blender'\n",
                "    xf_runner = xf.init_blender(exec_path=engine_exec_path, background=False, new_process=True)\n",
                "elif 'unreal' in exec_path_stem:\n",
                "    # Unreal Engine requires a project to be opened\n",
                "    # Here we use a sample project, which is downloaded from the following link\n",
                "    # You can also use your own project\n",
                "    import shutil\n",
                "\n",
                "    from xrfeitoria.utils.downloader import download\n",
                "    unreal_project_zip = download(url='https://github.com/openxrlab/xrfeitoria/releases/download/v0.1.0/UE--XRFeitoriaUnreal_Sample.zip', \n",
                "                                    dst_dir=\"./tutorial03/assets/\")\n",
                "    shutil.unpack_archive(filename=unreal_project_zip, extract_dir='./tutorial03/assets/')\n",
                "\n",
                "    # Open Unreal Engine\n",
                "    render_engine = 'unreal'\n",
                "    xf_runner = xf.init_unreal(exec_path=engine_exec_path, \n",
                "                                background=False, \n",
                "                                new_process=True, \n",
                "                                project_path='./tutorial03/assets/UE_sample/UE_sample.uproject')"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "✨ Now you can see a new Blender/Unreal Engine process has started."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 2. Import skeletal meshes to Level\n",
                "\n",
                "Download the skeletal meshes in [SynBody](https://synbody.github.io) to local folder and import them to ``Level``."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from xrfeitoria.utils.downloader import download\n",
                "\n",
                "# Download the skeletal meshes\n",
                "actor1_path = download('https://github.com/openxrlab/xrfeitoria/releases/download/v0.1.0/assets--SMPL-XL--SMPL-XL-00439Subject_75_F_12.fbx', dst_dir=\"./tutorial03/assets/\")\n",
                "actor2_path = download('https://github.com/openxrlab/xrfeitoria/releases/download/v0.1.0/assets--SMPL-XL--SMPL-XL-00045.fbx', dst_dir=\"./tutorial03/assets/\")\n",
                "actor2_motion_path = download('https://github.com/openxrlab/xrfeitoria/releases/download/v0.1.0/assets--SMPL-XL--walking15_01.fbx', dst_dir=\"./tutorial03/assets/\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Here we import two actors. The ``actor1`` has animation and the ``actor2`` has no animation.\n",
                "\n",
                "And we set different [stencil value](https://xrfeitoria.readthedocs.io/en/latest/faq.html#what-is-stencil-value) for each ``Actor`` to distinguish different Actors when rendering segmentation masks."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Import the skeletal mesh\n",
                "actor1 = xf_runner.Actor.import_from_file(file_path=actor1_path, stencil_value=100)\n",
                "actor2 = xf_runner.Actor.import_from_file(file_path=actor2_path, stencil_value=200)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Then, we load an animation from another file and set it to the ``actor2``"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "actor2.setup_animation(animation_path=actor2_motion_path)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We can also modify the properties of the actors, such as the location, rotation, and scale."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Set the location of the two actors to make their distance to be 1.0 meter\n",
                "actor1_location = actor1.location\n",
                "actor2_location = actor2.location\n",
                "actor2.location = (actor1_location[0] + 1.0, actor1_location[1], actor1_location[2])"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Now they look like:\n",
                "\n",
                "- Blender\n",
                "![](https://github.com/user-attachments/assets/eba6c868-de56-478b-b36e-386d42f42e46)\n",
                "\n",
                "- Unreal Engine\n",
                "![](https://github.com/user-attachments/assets/50f32714-de0d-4f1c-8eaf-00e965c37a77)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "If you use ``Unreal Engine``, the ``Level`` should be saved after been modified."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# save the level\n",
                "if render_engine == 'unreal':\n",
                "    xf_runner.utils.save_current_level()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 3. Add a sequence for rendering\n",
                "\n",
                "``Sequence`` is a multifunctional class in XRFeitoria. It can be used for:\n",
                "- rendering\n",
                "- adding transform keys\n",
                "- grouping different objects. \n",
                "\n",
                "Here, we use it for rendering."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "import math\n",
                "\n",
                "from xrfeitoria.data_structure.models import RenderPass\n",
                "from xrfeitoria.data_structure.models import SequenceTransformKey as SeqTransKey\n",
                "\n",
                "# Use `with` statement to create a sequence, and it will be automatically close the sequence after the code block is executed.\n",
                "# The argument `seq_length` controls the number of frames to be rendered. \n",
                "sequence_name = 'MySequence'\n",
                "frame_num = 6\n",
                "with xf_runner.sequence(seq_name=sequence_name, seq_length=frame_num, replace=True) as seq:\n",
                "\n",
                "    # Get the bounding boxes of the actors\n",
                "    actor1_bbox = actor1.bound_box\n",
                "    actor2_bbox = actor2.bound_box\n",
                "\n",
                "    # Get the center location of the actors\n",
                "    actor1_center = ((actor1_bbox[0][0] + actor1_bbox[1][0]) / 2, (actor1_bbox[0][1] + actor1_bbox[1][1]) / 2, (actor1_bbox[0][2] + actor1_bbox[1][2]) / 2)\n",
                "    actor2_center = ((actor2_bbox[0][0] + actor2_bbox[1][0]) / 2, (actor2_bbox[0][1] + actor2_bbox[1][1]) / 2, (actor2_bbox[0][2] + actor2_bbox[1][2]) / 2)\n",
                "    actors_center = ((actor1_center[0] + actor2_center[0]) / 2, (actor1_center[1] + actor2_center[1]) / 2, (actor1_center[2] + actor2_center[2]) / 2)\n",
                "    \n",
                "    ##########################################################################\n",
                "    # Add 6 static cameras and a moving camera around the actors for rendering\n",
                "    ##########################################################################\n",
                "    # Set cameras' field of view to 90°\n",
                "    camera_fov = 90\n",
                "    # Set cameras' distance to 3.0m\n",
                "    distance_to_actor = 3.0\n",
                "    # Prepare the transform keys for moving camera\n",
                "    transform_keys = []\n",
                "    # calculate the location and rotation of the cameras\n",
                "    for i in range(6):\n",
                "        azimuth = 360 / 6 * i\n",
                "        azimuth_radians = math.radians(azimuth)\n",
                "\n",
                "        x = distance_to_actor * math.cos(azimuth_radians) + actors_center[0]\n",
                "        y = distance_to_actor * math.sin(azimuth_radians) + actors_center[1]\n",
                "        z = 0.0 + actors_center[2]\n",
                "        location = (x, y, z)\n",
                "        # Set camera's rotation to look at the actor's center\n",
                "        rotation = xf_runner.utils.get_rotation_to_look_at(location=location, target=actors_center)\n",
                "\n",
                "        # Add a static camera\n",
                "        static_camera = seq.spawn_camera(\n",
                "            camera_name=f'static_camera_{i}',\n",
                "            location=location,\n",
                "            rotation=rotation,\n",
                "            fov=camera_fov,\n",
                "        )\n",
                "        \n",
                "        # Add a transform key to the moving camera\n",
                "        transform_keys.append(\n",
                "            SeqTransKey(\n",
                "                frame=i,\n",
                "                location=location,\n",
                "                rotation=rotation,\n",
                "                interpolation='AUTO',\n",
                "            )\n",
                "        )  \n",
                "    \n",
                "    # Add a moving camera rotating around the actors\n",
                "    moving_camera = seq.spawn_camera_with_keys(\n",
                "        camera_name=f'moving_camera',\n",
                "        transform_keys=transform_keys,\n",
                "        fov=camera_fov,\n",
                "    )\n",
                "\n",
                "    # Add a render job to renderer\n",
                "    # In render job, you can specify the output path, resolution, render passes, etc.\n",
                "    # The output path is the path to save the rendered data.\n",
                "    # The resolution is the resolution of the rendered image.\n",
                "    # The render passes define what kind of data you want to render, such as img, depth, normal, etc.\n",
                "    # and what kind of format you want to save, such as png, exr, etc.\n",
                "    seq.add_to_renderer(\n",
                "        output_path=f'./tutorial03/outputs/{render_engine}/',\n",
                "        resolution=(1280, 720),\n",
                "        render_passes=[RenderPass('img', 'png'),\n",
                "                       RenderPass('mask', 'exr'),\n",
                "                       RenderPass('normal', 'exr'),\n",
                "                       RenderPass('diffuse', 'exr')]\n",
                "    )"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 4. Render\n",
                "\n",
                "The following code renders all the render jobs and save the images to the ``output_path`` set in ``seq.add_to_renderer`` above."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Render\n",
                "xf_runner.render()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Check the ``output_path``, and you can see that for the frame *i*, the image rendered by the moving camera is the same as the image rendered by the *i*th static camera. For example, the ``moving_camera/0002.png`` is the same as the ``static_camera_2/0002.png``."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "import matplotlib.pyplot as plt\n",
                "\n",
                "from xrfeitoria.utils.viewer import Viewer\n",
                "\n",
                "xf_viewer = Viewer(sequence_dir=f'./tutorial03/outputs/{render_engine}/{sequence_name}/')\n",
                "\n",
                "moving_camera_img = xf_viewer.get_img(camera_name='moving_camera', frame=2)\n",
                "static_camera_img = xf_viewer.get_img(camera_name='static_camera_2', frame=2)\n",
                "\n",
                "plt.figure(figsize=(20, 20))\n",
                "\n",
                "plt.subplot(1, 2, 1)\n",
                "plt.imshow(moving_camera_img)\n",
                "plt.axis('off')\n",
                "plt.title('moving_camera/0002.png')\n",
                "\n",
                "plt.subplot(1, 2, 2)\n",
                "plt.imshow(static_camera_img)\n",
                "plt.axis('off')\n",
                "plt.title('static_camera_2/0002.png')"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "View the rendered images and annotations of the camera ``static_camera_2`` by:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "import matplotlib.pyplot as plt\n",
                "\n",
                "from xrfeitoria.utils.viewer import Viewer\n",
                "\n",
                "xf_viewer = Viewer(sequence_dir=f'./tutorial03/outputs/{render_engine}/{sequence_name}/')\n",
                "\n",
                "camera_name = 'static_camera_2'\n",
                "for i in range(frame_num):\n",
                "    img = xf_viewer.get_img(camera_name=camera_name, frame=i)\n",
                "    mask = xf_viewer.get_mask(camera_name=camera_name, frame=i)\n",
                "    normal = xf_viewer.get_normal(camera_name=camera_name, frame=i)\n",
                "    diffuse = xf_viewer.get_diffuse(camera_name=camera_name, frame=i)\n",
                "\n",
                "    plt.figure(figsize=(20, 20))\n",
                "\n",
                "    plt.subplot(1, 4, 1)\n",
                "    plt.imshow(img)\n",
                "    plt.axis('off')\n",
                "    plt.title('img')\n",
                "\n",
                "    plt.subplot(1, 4, 2)\n",
                "    plt.imshow(mask)\n",
                "    plt.axis('off')\n",
                "    plt.title('mask')\n",
                "\n",
                "    plt.subplot(1, 4, 3)\n",
                "    plt.imshow(normal)\n",
                "    plt.axis('off')\n",
                "    plt.title('normal')\n",
                "\n",
                "    plt.subplot(1, 4, 4)\n",
                "    plt.imshow(diffuse)\n",
                "    plt.axis('off')\n",
                "    plt.title('diffuse')"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "> Hint: When using Unreal Engine, if the images of the mask look weird, try running the notebook again."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Finally, close the engine by:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "xf_runner.close()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Ref to [api docs](https://xrfeitoria.readthedocs.io/en/latest/apis/xrfeitoria.html), you can always use ``with`` statement to ensure the engine is closed when the codes are finished."
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "blender-pipeline",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.10.12"
        },
        "orig_nbformat": 4
    },
    "nbformat": 4,
    "nbformat_minor": 2
}