3dMD Mesh Projection Tool

A Python tool for projecting 3D meshes onto 2D camera images from 3dMD multi-camera systems.

Features

• Projects 3D meshes with multicamera support
• Supports both 3dMD ground-truth meshes and reconstructed meshes
• Interactive 3D mesh viewing
• Generates projection overlays showing mesh wireframes on camera images

This tool integrates into a broader 4D facial analysis pipeline, supporting 3dMD rig setups and my custom 3D reconstruction workflow. Ideal for validating alignment, visualizing reprojection accuracy, and debugging stereo reconstruction artifacts.

Core 3D Facial reconstruction pipeline.

Camera Setup

The system supports 3dMD 6-camera configuration

• 1A, 1B: Left stereo pair (vertical setup, grayscale)
• 2A, 2B: Right stereo pair (vertical setup, grayscale)
• 1C: Left texture camera (color)
• 2C: Right texture camera (color)

Requirements

• Python 3.11+
• OpenCV 4.8+
• Open3D 0.18+
• NumPy 1.24+
• PIL/Pillow 9.5+

# Install dependencies
# Using uv (recommended)
uv sync

# Or with pip
pip install -e .

Data Structure

Data should be organized like this:

data/
└── sequence/
    ├── calib_1A.tka
    ├── calib_1B.tka
    ├── calib_1C.tka
    ├── calib_2A.tka
    ├── calib_2B.tka
    ├── calib_2C.tka
    ├── STEREO_1A_000.bmp
    ├── STEREO_1B_000.bmp
    ├── STEREO_2A_000.bmp
    ├── STEREO_2B_000.bmp
    ├── TEXTURE_1C_000.bmp
    └── TEXTURE_2C_000.bmp
    
meshes/
├── frame_000.obj    # 3dMD ground-truth or reconstructed mesh
├── frame_001.obj
└── ...

Calibration File Format

The tool reads 3dMD .tka calibration files with the following parameters:

• %M: 3x3 rotation matrix (world to camera)
• %X, %Y, %Z: Camera position in world coordinates
• %f: Focal length (mm)
• %K, %K2: Radial distortion coefficients
• %x, %y: Pixel size (mm)
• %a, %b: Principal point (pixels)
• %is: Image size (width height)

Usage

Commands

# List available data
python main.py data/sequence --list --mesh-dir meshes/

# Project mesh onto all cameras
python main.py data/sequence --project --frame 0 --mesh-dir meshes/

# View 3D mesh interactively
python main.py data/sequence --view-mesh meshes/frame_000.obj

Project Different Mesh Types

# Project 3dMD ground-truth mesh
python main.py data/sequence --project --frame 0 --mesh-dir meshes/

# Project reconstructed mesh from different directory
python main.py data/sequence --project --frame 5 --mesh-dir reconstructed_meshes/

# Project onto specific cameras only
python main.py data/sequence --project --frame 0 --mesh-dir meshes/ --cameras 1A 1B

# Process multiple frames
python main.py data/sequence --project --frame 0 --mesh-dir meshes/
python main.py data/sequence --project --frame 1 --mesh-dir meshes/
python main.py data/sequence --project --frame 2 --mesh-dir meshes/

Mesh File Naming Patterns

The tool automatically detects meshes with these naming patterns:

• prefix_000.obj (e.g., XAN1_000.obj)
• frame_000.obj
• 000.obj
• mesh_000.obj

Output

The tool generates:

• Projection overlays: PNG images showing mesh vertices and wireframe projected onto camera images
• File naming: projection_{camera_id}_frame_{frame_number:03d}.png

Grayscale Camera Projection	Color Camera Projection
Mesh wireframe projected onto grayscale stereo camera image	Mesh wireframe projected onto color texture camera with subject's face

Acknowledgments

We gratefully acknowledge the participants who volunteered for the 3D facial data collection, whose contributions were essential to the success of this project.

The complete dataset used in this project is available at Zenodo.