e2e: yaw, pitch, roll strange behaviour

[testor2897]

Hi,

I tested the e2e function with Python 3.10.11 (Windows 10, 64 bit) and the following packages:

After some tests I put in the following code for yaw, pitch and roll (a little modified values):

      # fetch the angle values from web form in degrees
      rollAngle = float(request.form.get("RollAngle"))
      pitchAngle = float(request.form.get("PitchAngle"))
      yawAngle = float(request.form.get("YawAngle"))
      qualityValue = int(request.form.get("Quality"))

      print("Selected rollAngle, pitchAngle, yawAngle: ", rollAngle, pitchAngle, yawAngle)
...
            # use modified values for the torch rotation, pitch angle is kept original
            print("Used rollAngle, pitchAngle, yawAngle: ", rollAngle / 2, pitchAngle, -yawAngle)
            transform = transforms.ToPILImage()
            im = transform(torch_rotation(equi_image, -yawAngle, pitchAngle, rollAngle / 2))
...
@torch.jit.script
def torch_rotation(equi_image: torch.Tensor, yawAngle: float, pitchAngle: float, rollAngle: float) -> torch.Tensor:
    
    # due to memory problems with big panoramas I had to process channels separately
    indices = torch.tensor([0])
    image_R = torch.index_select(equi_image, 0, indices)
    indices = torch.tensor([1])
    image_G = torch.index_select(equi_image, 0, indices)
    indices = torch.tensor([2])
    image_B = torch.index_select(equi_image, 0, indices)
    # clean up memory
    del equi_image

    print("\nProcess red channel")
    # Rotate an equirectangular image around one more axes
    rotated_equi_R = e2e(
        image_R,                      # Equirectangular image
        h_deg=yawAngle,               # Vertical rotation/shift, Yaw
        v_deg=pitchAngle,             # Horizontal rotation/shift, Pitch
        roll=rollAngle,               # Clockwise/counter clockwise rotation, Roll
        mode='bilinear'               # Sampling interpolation
    )
    # clean up memory
    del image_R

    print("\nProcess green channel")
    # Rotate an equirectangular image around one more axes
    rotated_equi_G = e2e(
        image_G,                        # Equirectangular image
        h_deg=yawAngle,               # Vertical rotation/shift, Yaw
        v_deg=pitchAngle,             # Horizontal rotation/shift, Pitch
        roll=rollAngle,               # Clockwise/counter clockwise rotation, Roll
        mode='bilinear'               # Sampling interpolation
    )
    # clean up memory
    del image_G

    print("\nProcess blue channel")
    # Rotate an equirectangular image around one more axes
    rotated_equi_B = e2e(
        image_B,                        # Equirectangular image
        h_deg=yawAngle,               # Vertical rotation/shift, Yaw
        v_deg=pitchAngle,             # Horizontal rotation/shift, Pitch
        roll=rollAngle,               # Clockwise/counter clockwise rotation, Roll
        mode='bilinear'               # Sampling interpolation
    )
    del image_B

    return torch.vstack((rotated_equi_R,rotated_equi_G,rotated_equi_B))

The original test picture is:

WEB preview (with grid):

Test yaw=45°
For yaw=45° everything is fine.
LOG:

WEB preview (with grid):

preview = result => perfect!

Test roll=45°
For roll=45° I modify the used angle to roll/2.
LOG:

WEB preview (with grid):

preview = result with roll/2 => is this intended?

Test pitch=45°
For pitch=45° I don't understand the result..
LOG:

WEB preview (with grid):

WEB preview of result (with grid):

preview != preview of result (pitch is 45°, but there is also yaw >45°) => this is strange...

I have the following questions:

1. Is the use of half roll angle intended?
2. How do I achieve the correct pitch result?

[testor2897]

I looked into pytorch360convert.py and found:

def e2e(

...
    
    # Create rotation matrices
    Rx = rotation_matrix(
        roll_rad, torch.tensor([0.0, 0.0, 1.0], device=e_img.device, dtype=e_img.dtype)
    )
    Ry = rotation_matrix(
        pitch_rad, torch.tensor([1.0, 0.0, 1.0], device=e_img.device, dtype=e_img.dtype)
    )
    Rz = (
        rotation_matrix(
            yaw_rad,
            torch.tensor([0.0, 1.0, 0.0], device=e_img.device, dtype=e_img.dtype),
        )
        @ Rx
        @ Ry
    )

I am pretty sure it should be:

    Ry = rotation_matrix(
        pitch_rad, torch.tensor([1.0, 0.0, 0.0], device=e_img.device, dtype=e_img.dtype)
    )

So I tried it and the output is the same as roll => applying -pitch / 2 produces the correct rotation.

With the following modifications I get the correct base results (no combination yet):

            rollAngle = rollAngle / 2
            yawAngle = -yawAngle
            pitchAngle = -pitchAngle / 2

The direction of rotation from my code is oriented like it is for the Web panorama viewer. So maybe the official directions are different. I am just curios about the half angles.

Why do I need half roll and pitch angles?

[testor2897]

I suggest using a corrected version of e2e function wirh common rotation sequence R = Rz @ Ry @ Rx (# Combined rotation matrix, common sequence order). This gives more compatibiility with other panorama tools (i.e. js panorama viewer panorado ...).
Then the angle issue is fixed.

Further on I recommend some optimizations for bigger panoramas (see Discussions):

from concurrent.futures import ThreadPoolExecutor

def e2e(
    e_img: torch.Tensor,
    roll: float = 0.0,
    h_deg: float = 0.0,
    v_deg: float = 0.0,
    mode: str = "bilinear",
    channels_first: bool = True,
    chunk_size: int = 256,
    use_mixed_precision: bool = True,
    max_workers: int = 4
) -> torch.Tensor:
    """
    Optimized version for large panoramas with direct chunk processing (GPU-friendly).
    - Uses rotation matrix for roll, pitch, yaw
    - Processes image in chunks to reduce memory usage
    - Supports mixed precision for faster GPU computation
    """

    # Convert rotation angles from degrees to radians
    roll_rad = torch.tensor([roll * torch.pi / 180.0], device=e_img.device, dtype=e_img.dtype)
    yaw_rad = torch.tensor([-h_deg * torch.pi / 180.0], device=e_img.device, dtype=e_img.dtype)
    pitch_rad = torch.tensor([v_deg * torch.pi / 180.0], device=e_img.device, dtype=e_img.dtype)

    # Ensure input format is HWC for processing
    e_img = _nchw2nhwc(e_img) if channels_first else e_img
    h, w = e_img.shape[:2]

    # Change to common Euler rotation sequence
    # Prepare rotation matrix (order: Rz @ Ry @ Rx)
    Rx = rotation_matrix(roll_rad, torch.tensor([0.0, 0.0, 1.0], device=e_img.device, dtype=e_img.dtype))
    Ry = rotation_matrix(pitch_rad, torch.tensor([1.0, 0.0, 0.0], device=e_img.device, dtype=e_img.dtype))
    Rz = rotation_matrix(yaw_rad, torch.tensor([0.0, 1.0, 0.0], device=e_img.device, dtype=e_img.dtype))
    R = Rz @ Ry @ Rx  # Combined rotation matrix, common sequence order 

    # Enable mixed precision if GPU is used
    dtype = torch.float16 if use_mixed_precision and e_img.device.type == "cuda" else e_img.dtype

    # Define function to process a vertical chunk of the image
    def process_chunk(y_start: int, y_end: int):
        # Generate coordinate ranges for the chunk
        y_range = torch.arange(y_start, y_end, device=e_img.device, dtype=dtype)
        x_range = torch.arange(w, device=e_img.device, dtype=dtype)

        # Create meshgrid for pixel coordinates
        grid_y, grid_x = torch.meshgrid(y_range, x_range, indexing="ij")

        # Convert (x, y) to spherical coordinates (u, v)
        uv = coor2uv(torch.stack([grid_x, grid_y], dim=-1), h, w)

        # Convert UV to 3D unit sphere coordinates
        xyz = uv2unitxyz(uv)

        # Apply rotation matrix to 3D coordinates
        xyz_rot = xyz @ R.T

        # Convert rotated 3D coordinates back to spherical (u_rot, v_rot)
        uv_rot = xyz2uv(xyz_rot)

        # Map spherical coordinates back to image coordinates
        coor_xy = uv2coor(uv_rot, h, w)

        # Sample pixels from the original image using rotated coordinates
        return sample_equirec(e_img, coor_xy, mode=mode)

    # Process all chunks in parallel using ThreadPoolExecutor
    rotated_chunks = []
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = [
            executor.submit(process_chunk, y_start, min(y_start + chunk_size, h))
            for y_start in range(0, h, chunk_size)
        ]
        for f in futures:
            rotated_chunks.append(f.result())

    # Concatenate all processed chunks to form the final rotated image
    rotated = torch.cat(rotated_chunks, dim=0)

    # Convert back to CHW format if required
    return _nhwc2nchw(rotated) if channels_first else rotated

[ProGamerGov]

@testor2897 I think your low memory version of the e2e function could potentially be added alongside the current e2e function as e2e_large, e2e_low_mem, or something like that. As I don't think ThreadPoolExecutor is supported by torch.jit or torch.compile.

I also haven't had a chance to review your suggestions for e2e rotations yet, but you can submit them as a separate pull request from the large version of the function if you want.