JOCA: Task-Driven Joint Optimisation of Camera Hardware and Adaptive Camera Control Algorithm

Please refer to the CARLA Experiment folder for the experiment on synthetic images, and the Waymo Open Experiment folder for the experiment on real-world images.

CARLA

Simulator Setup

The CARLA Simulator version 0.9.15 is used in this experiment, please follow the official installation instructions to build the simulator from the source code using an Unreal Engine 4 editor.

Motion blur effects need to be enabled by adding an additional attribute to the RGB camera sensor. To add this attribute, the following steps need to be performed:

1. Navigate to Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor directory and modify SceneCaptureSensor.cpp and SceneCaptureSensor.h. Add a ASceneCaptureSensor::SetMotionBlurTargetFPS function and a ASceneCaptureSensor::GetMotionBlurTargetFPS function to these files. The functions follow the same format as the other sensor attribute functions.
2. Add PostProcessSettings.bOverride_MotionBlurTargetFPS = true; inside the SetCameraDefaultOverrides function in SceneCaptureSensor.cpp.
3. Navigate to Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Actor directory and modify the file ActorBlueprintFunctionLibrary.cpp. Find the MakeCameraDefinition function, under the motion blur section in this function, add the following code:

FActorVariation MBTargetFPS;
MBTargetFPS.Id = TEXT("motion_blur_target_fps");
MBTargetFPS.Type = EActorAttributeType::Float;
MBTargetFPS.RecommendedValues = { TEXT("1.0") };
MBTargetFPS.bRestrictToRecommended = false;

Then in the same function, add MBTargetFPS to the list of Definition.Variations.Append. 4. In the same file, modify the following part to the UActorBlueprintFunctionLibrary::SetCamera function with an addition for the new attribute:

Camera->SetMotionBlurIntensity(
    RetrieveActorAttributeToFloat("motion_blur_intensity", Description.Variations, 0.5f));
Camera->SetMotionBlurMaxDistortion(
    RetrieveActorAttributeToFloat("motion_blur_max_distortion", Description.Variations, 5.0f));
Camera->SetMotionBlurMinObjectScreenSize(
    RetrieveActorAttributeToFloat("motion_blur_min_object_screen_size", Description.Variations, 0.0f));
Camera->SetMotionBlurTargetFPS(
    RetrieveActorAttributeToFloat("motion_blur_target_fps", Description.Variations, 1.0f));

5. Rebuid CARLA.
6. If a package game version is preferred after this modification, export to package game following this CARLA instruction.
7. To change motion blur effects based on exposure, please refer to our camera.py file and the explaination of 'Target FPS' from the Unreal Engine Documentation.

Optimisation

To start the optimisation, first start to run the CARLA simulator, then run joint_train.py.

Testing

To evaluate the optimised camera, adaptive camera control algorithm, and the object detector, first start to run the CARLA simulator, then run joint_test.py.

Waymo Open

Dataset

Please download the Waymo Open Perception Dataset from its official website and follow the steps below to setup the data:

1. Since our implementation is based on PyTorch, please run the export_waymo_camera.py file to convert the dataset from TFRecord to images + JSON files for later use.
2. Run the add_blur.py file to first estimate and save optical flow for each image.

Optimisation

To start the optimisation, run train.py.

Testing

To evaluate the optimised camera, adaptive camera control algorithm, and the object detector, run test.py.

Citation

If you find our work useful, please cite the below paper:

@InProceedings{yan2025tacos,
    author    = {Yan, Chengyang and Bryson, Mitch and Dansereau, Donald G.},
    title     = {JOCA: Task-Driven Joint Optimisation of Camera Hardware and Adaptive Camera Control Algorithms},
    booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)},
    year      = {2026}
}