frontend: Improve scene item cropping UI

[Warchamp7]

Description

Makes a number of improvements to cropping items via a new visual UI.

While Alt is held down with a source selected, a dark overlay is drawn over the preview along with showing the cropped out regions of the source. The stretch handles also change to green to help indicate that a different action will take place when dragging one.

Dragging the source with Alt held down will now move the source within the cropped region.

Motivation and Context

Cropping can be a bit unintuitive when using alternative bounding box types. I wanted to improve that as well as give a visual aid to exactly what is outside the cropped area.

obs64_torQYPUrcX.mp4

How Has This Been Tested?

Tested cropping a source on all sides, in all bounding box types, and with all bounding box alignments.

Types of changes

• New feature (non-breaking change which adds functionality)
• Tweak (non-breaking change to improve existing functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[PatTheMav]

In my tests this doesn't work right: When cropping a source only a fixed rectangular area in the top left is overlaid over the preview, the rest of the preview is unaffected.

The implementation itself also seems wasteful to me in terms of GPU draw calls. IMO this should be done with a fragment shader that effectively renders the source with the canvas resolution.

The shader would either:

• Return the sampled RGBA values from the source texture if the fragment is within the cropped area of the source
• Return the sampled RGBA value blended with the overlay colour when the fragment is within the source area but outside the cropped area
• Return the overlay colour when the fragment is outside the source area

That should result in a single draw call for everything and because the source is rendered at the full canvas width, it would be blended directly into the render target.

Maybe @Lain-B has an idea how to make that work as it kinda runs counter to how the projection matrix is set up for sources, but it would shift the coverage calculations out of the UI code and into the shader, similar to how we made the striped line work entirely in the shader.

[Warchamp7]

In my tests this doesn't work right: When cropping a source only a fixed rectangular area in the top left is overlaid over the preview, the rest of the preview is unaffected.

Can you post an image or video of it's behavior for you?

The implementation itself also seems wasteful to me in terms of GPU draw calls. IMO this should be done with a fragment shader that effectively renders the source with the canvas resolution.

Is that concern a blocker to this PR?

similar to how we made the striped line work entirely in the shader.

I don't think we did that?

You mean the striped lines for cropping. Not the striped overlay for sources outside the canvas.

[PatTheMav]

Can you post an image or video of it's behavior for you?

Here you go:

Is that concern a blocker to this PR?

My main issue with this is that it follows down the same path that makes our current preview rendering so inefficient: Every little piece gets its own pipeline change and distinct render call, which is more or less what you're not supposed to do when interacting with the graphics APIs (and indeed the performance impact of this approach will be potentially worse in Metal, D3D12, and Vulkan) and is also the reason why selecting anything in the preview makes the CPU usage spike unreasonably.

I probably would be less concerned if preview rendering code wouldn't also be directly tied into video output rendering code - it is part of the same render loop, so any additional load generated by one impacts the other and I've already profiled cases in which the preview rendering is already heavier than output compositing.

You mean the striped lines for cropping. Not the striped overlay for sources outside the canvas.

Correct. That was a great change because it removed all that context switching.

[Warchamp7]

Converted to draft until @Lain-B or @PatTheMav has time to dig into what's going funky with the render code here, as well as investigate ways we can do this more efficiently.