Add initial support for the OpenH264 H.264 software codec

[Conan-Kudo]

Description

This allows users to leverage the OpenH264 codec from Cisco to encode H.264 video content. It is significantly reduced in capability from alternatives, but it does the job.

This also provides a framework for adding support for other H.264 software codecs provided through FFmpeg.

Motivation and Context

A number of distributions use OpenH264 distributed by Cisco to provide a fully licensed H.264 implementation (in particular, both Fedora Linux and openSUSE Linux do). However, OBS Studio cannot currently use it.

This change allows OBS Studio to use OpenH264 through FFmpeg if it's available.

How Has This Been Tested?

• Tested recording a screencast using OpenH264 + AAC with both Simple and Advanced settings modes
• Tested streaming a screencast using OpenH264 + AAC with both Simple and Advanced settings modes

This was tested on Fedora Linux 38 for x86_64 with ffmpeg 6.0.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[RytoEX]

Does this require #8015?

[Conan-Kudo]

Does this require #8015?

Not yet. This is only the wiring for the ffmpeg plugin, which I think means it only shows up in the "advanced view" right now? I literally just banged this out and am taking a bit of a break before continuing to iterate on it. I've got local builds going so I can test the functionality...

[Conan-Kudo]

@RytoEX That said, having #8015 land first would make it tremendously easier for the UI wiring I need to do...

[Conan-Kudo]

I've pulled in #8015 into this pull request to simplify things for me and to show the full context of this change.

[Conan-Kudo]

I've now tested that this works, thanks to @Gawdl3y's help! 🎉

[PatTheMav]

x264 is the built-in fallback and the app is structured around that.

The better way to solve this would be for Fedora to patch obs-x264 to use OpenH264 so that the program is no wiser.

(Also I’m unsure as to whether that should be implemented in obs-x264 itself with a specific check for Fedora, because I’m not happy about distro-specific workarounds while we have to maintain support for 3 operating systems already.)

Afaik we officially support Ubuntu and Flatpak and any distro-specific changes need to be made by distributors while packaging the app.

[Conan-Kudo]

obs-x264 is libx264 specific, an equivalent for OpenH264 would be to write a whole new plugin for libopenh264. I used ffmpeg because it already has an interface for this, we prefer OpenH264 to be accessed through ffmpeg anyway, and it would relatively straightforward to wire in, based on how AV1 was done.

[PatTheMav]

Again, that will not work because you are creating a fallback for a fallback, you are not even checking if ffmpeg supports the encoder at runtime before making that decision and with that you also make macOS and Windows fall back to OpenH264 if x264 is not found.

It creates additional code complexity and maintenance burden for an edge case on a single distribution and IMO that’s not a good deal.

[Conan-Kudo]

It creates additional code complexity and maintenance burden for an edge case on a single distribution and IMO that’s not a good deal.

It's not just a single distribution. At this point in time, it benefits four distributions:

• CentOS/RHEL
• Fedora Linux
• openSUSE Tumbleweed
• openSUSE Leap/SLE

This is also supported by the FFmpeg library included in the Freedesktop runtime for Flatpak too.

[PatTheMav]

We can add generic support for OpenH264 if it is available (as in "checked if available via FFmpeg" because we do not ship FFmpeg with OpenH264 support on Windows and macOS as we do not support that codec at all), and that availability will probably be limited to Linux only and depends on the distribution-specific variant of FFmpeg available. If that check is positive, it can be added to the list of available encoders.

But distribution-specific fixes need to be implemented/patched by distributors. Adding a global (as in: affecting all platforms) fallback on OpenH264 if the actual fallback x264 (which is commonly shipped by OBS Studio itself or expected to be available on the platform) is not a solution IMO.

Upstream distributors can then handle the added complexity of adding an OpenH264 fallback if it is available at all and abort the OBS Studio launch if it cannot be found.

Also AV1 is not a "baseline/fallback" encoder, so the comparison with its implementation is not applicable.

[derrod]

It would probably be sensible to split the encoder addition and the UI changes into separate PRs so we can move things along. The former is probably fine (with some minor code review nits).

[Conan-Kudo]

I've split out the first half of this pull request as #10135, per @derrod's request.

[DemiMarie]

But distribution-specific fixes need to be implemented/patched by distributors. Adding a global (as in: affecting all platforms) fallback on OpenH264 if the actual fallback x264 (which is commonly shipped by OBS Studio itself or expected to be available on the platform) is not a solution IMO.

On desktop Linux, OpenH264 is the worst-case fallback. Software patent restrictions mean that it is the only H.264 codec that all distros can redistribute.

[tytan652]

Translation nitpik that covers both PR.

[Conan-Kudo]

This pull request has been rebased on top of current master and has the remaining commits to merge (frontend commits).

[Conan-Kudo]

While this doesn't have a direct dependency on @AsahiLina's #12326, it does benefit from it.

[Lain-B]

Just briefly looking at it, it seems pretty trivial now that everything else has been merged. Anyone else want to give further feedback?

[Fenrirthviti]

For the fallback part of this, I would prefer if we just removed the entire fallback system in general in favor of a better error reporting that the encoder is missing. I don't want to make our already tenuous and fragile fallback that... really doesn't make sense anymore, more complex.

[Fenrirthviti]

Also, can you please explain why we're intrinsically linking OpenH264's existence to x264 in the first place?

I would prefer that if the encoder is available, it just shows up. If it's not, it doesn't. I'm not sure what value there is in NOT displaying OpenH264 if x264 is available instead of just showing them both?

[Conan-Kudo]

I was asked last time around to hide it if x264 is available. If we don't need that anymore, I can drop the commit that does that.

[Fenrirthviti]

Thanks, found the comment in one of the old reviews. That is not something we'd prefer, it can be removed, yes.

[Conan-Kudo]

I dropped it and rebased onto current master.

[Conan-Kudo]

For the fallback part of this, I would prefer if we just removed the entire fallback system in general in favor of a better error reporting that the encoder is missing. I don't want to make our already tenuous and fragile fallback that... really doesn't make sense anymore, more complex.

I would prefer we do this separately. I agree with you that the fallback logic is crufty and fragile, but purging it is currently beyond my capabilities. That said, @AsahiLina has taken the first steps to allow eliminating it with #12326.

[Fenrirthviti]

I agree that it's a separate change, yes, and outside scope of what this PR is attempting to accomplish.

We're doing some internal checking of the relevant code paths to determine if this PR is a change we're willing to accept short term, and if it is not, have a reasonable alternative proposal.

[PatTheMav]

Given that the change seems to be mostly held up by the topic of how to properly handle the fallback, I did a little dive into our UI and settings code to check how deeply integrated the fallback selection actually is, and the answer is: "Very".

Without going into too much detail here, there are many pieces in the settings code that depend on a valid encoder being set as the "current" encoder at all times, and there are further parts of OBS Studio that require those parts being available as well, etc.

---

So my current thinking (after some deliberation) is that I still don't like openh264 becoming a de-facto global fallback if x264 isn't found, because it would only ever really be a working(!) fallback on Fedora systems.

So I'd propose that instead of this (pseudocode):

if (some encoder) {
   use some encoder
} else if (some other encoder) {
   use some other encoder
} else if (x264) {
   use x264
} else {
   use openh264
}

we'd do this:

if (some encoder) {
   use some encoder
} else if (some other encoder) {
   use some other encoder
} else {
   // Time to find a fallback
   if (x264) {
      use x264
   } else if (openh264) {
      use openh264
   } else {
      gracefully abort OBS with an error message because the app is in an unworkable state
   }
}

For one it decouples x264 from being the "de-facto" fallback encoder, but instead demotes it to a "fallback option" together with open264. Either would be fine, one of them has to be present. And the code just has to ensure that some fallback encoder is provided.

This also means that the code for simple mode output cannot just add "x264" as the very first encoder, but has to check for availability just like it has to do for all other encoders.

As far as the function of the encoder drop down is concerned, I agree that it requires a fix to allow people to change a possibly broken state, but wouldn't hoist that responsibility onto this PR.