[WIP] Replace autobuild with vcpkg and CMake build system modernization#5429
[WIP] Replace autobuild with vcpkg and CMake build system modernization#5429RyeMutt wants to merge 30 commits intosecondlife:develop-linuxfrom
Conversation
Update vcpkg baseline and introduce automatic boostrap script
…o enable legacy behaviors Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
…h as networking control Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
…tspace Signed-off-by: Rye <rye@alchemyviewer.org>
…ly change and are not well suited to vcpkg Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
…ude cleanup to reduce generation times Signed-off-by: Rye <rye@alchemyviewer.org>
Fix up various build and link issues with vcpkg on linux Modernize most cmake to utilize target_sources, cleanup options and fix various vcpkg feature flags Add CMakePresets and other cmake cleanup Improve automatic selection of vcpkg target triplet based on generator and build type Fix up PCH usage when building PIE executables on linux Improve macos vcpkg build support with ninja and xcode generators Remove dead StackWalker files from llcommon Fix bugsplat integration with vcpkg build system Signed-off-by: Rye <rye@alchemyviewer.org>
Fix Debug build and ninja generator support on all platforms Clean up cmake link directives on windows and macos Modernize disabling precompiled header usage Fix Windows Debug builds being unusably slow Enable dead code and library stripping on macos for all libraries and executables Fix problems with multi-config generators on linux and single config generators on osx/windows Deduplicate newview cmake for manifest generation and clean up target dependencies Introduce OptDebug config which is Debug build against optimzied libraries Fix various source files debug build ifdefs Fix macOS deploy target not being set before first project call Fix non-xcode ibtool not following cmake deploy target Fix various small issues with arm64 support in cmake/manifest Fix gcc warning about malloc bounds in lltexturecache Rework build artifact output to reduce duplicated binary files during build process Update bugsplat-apple to 2.0.0 Update zlib-ng to 2.3.3 Fix CEF framework setup for macos to pass app validation Remove zstd dependency from minizip Enable xcode scheme generation to improve IDE experience Signed-off-by: Rye <rye@alchemyviewer.org>
…usly being stuck hidden Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
…e previously never being hit in relwithdebinfo and release builds Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
Signed-off-by: Rye <rye@alchemyviewer.org>
…32 lambdas Signed-off-by: Rye <rye@alchemyviewer.org>
…-22.04 GHA runners Signed-off-by: Rye <rye@alchemyviewer.org>
|
Is there a reason why it is not possible to do a SINGLE thing at a time per PR and NOT mix it with a bunch of other changes so we don't end up in unnecessary giant PRs? |
|
Hey @Ansariel - I know how disruptive and annoying big changes like these are, however from the LL end there was the decision to group this into a bigger package for the work we commissioned Rye to do here. That being said, it's my hope that we won't do this again. Large chunks like this is disruptive to everyone, even our engineers. So in the future I'll endeavor to ensure that our statements of work are sufficiently broken up and measurable from the start rather than what this started out as. |
|
Hi @Geenz I believe that this and other changes already in the pipe are going to prove to be the breaking point for all but the best resourced TPVs. Let's look at what's coming down:-
and that's on top of the routine work caused by LL breaking TPV changes that they're unaware of when something gets refactored. The biggest elephant in the room though is RLVa. If that ever starts appearing in the LL viewer it's going to cause major issues for every viewer that already has RLV or RLVa included. These TPVs currently have a functional RLV or RLVa and wouldn't cut across to including a LL version until its functionality meets or exceeds what they already have. Assuming the LL development is done in phases, as was previously discussed, it's going to be a merge nightmare to keep LL RLVa elements out of TPVs until it's ready for a wholesale switch over. This is a lot of disruption for minimal functional change or feature benefit in the viewer and it's putting TPVs with small or single person teams under huge strain for very little gain. I'm pretty sure many other TPVs are feeling the same. --Chorazin |
The vcpkg changes won't be out until after the Linux changes, which will come with the SLua viewer, so at least that worry of yours can be crossed-off. It will still be a lot of work of course to merge everything, but official Linux support in my opinion is extremely important and worth it.
I don't believe this to be the case at all. I have already tested building with these vcpkg changes on my local machine and it is as simple as 1 single command without any other setup, or build environment variables, or any nonsense to create the project (I tested both MSVC and with Ninja). I don't see anywhere that there is an assumption that the only way people do it is with GHA.
For the TPVs that mostly follow LL's skin I can definitely see this being a pain and not something fun to do. At least in Firestorm's case, we will likely be throwing out all of the flat UI changes because it is a gigantic merge conflict that will break way too much of our own UI changes that are in place (plans may change of course). Autobuild has been a maintenance burden for LL (they have said as much), and with Signal leaving I imagine even more of a burden now. It is more limiting and harder to work with compared to modern, widely used systems such as vcpkg. So getting it done now and getting through the struggles to move to an industry standard system that LL doesn't need to maintain, is better for everyone. |
I took a look again at #5178 and still don't see how the existing Windows packaging code is removed from the viewer with the new one-click project. TPVs already customize their installers, and I don't think many if any will choose to adopt this new approach as-is. Like the flat UI changes already addressed above me the installer will probably also just be dropped on merge by TPVs because there is no obligation to follow the lab's direction on it or really any of the things that were listed. Third-party viewers exist that are one-person operations, and can't even merge modern Linden code due to license conflicts or in some cases nearing two decades of source code drift; they've objectively been doing fine. Lastly, being nitpicky here, but with a PR prefixed "WIP do not run actions" how does one arrive at this conclusion when there aren't even finished GHA scripts to build it yet?
All this to say; I have actually played with these the changes in this PR and I find it MASSIVELY simplifies the development process. Between the whole thing being literally one command to get started, and having cmake integrations working in code editors without any fiddling to take advantage of them, this PR turns what has always been an esoteric and tribal knowledge laden nightmare into a modern batteries-included developer experience. |
|
Thanks @Hecklezz and @DarlCat for the comments. I do support the intent of this project, respect the amount of work that's gone into it and will be glad to move away from autobuild too. What troubles me though is the size of this PR and the number of merge conflicts it's likely to throw up when it first meets a TPV codebase. Knowing that this will surface after Linux does indeed make that side of things much easier. Windows packaging: I'm tempted to just exclude that too since TPVs (except Firestorm) are generally not what a new user will start out with, however it's usually the case that when old and new methods exist in parallel sooner or later there's a position where only the new is maintained and the old falls behind and eventually stops working until it's finally removed from the code base as a mercy killing. Avoiding that in a TPV keeping NSIS means an overhead of keeping an eye on changes to the new to see whether they need to be ported back to the old and dodging the bullet when old is eventually removed from upstream. 2D UI: I'm also tempted to exclude that too, however again it's not that simple. If a TPV chooses to remain 3D there's then another new maintenance overhead to identify assets created for future new features in the 2D UI style and make 3D versions. These would typically slip through in a merge (new item, so there's no merge conflict) and have to be checked for specifically whilst sorting out the merge conflicts. --Chorazin |
|
For any TPV devs who haven't ventured into merging this yet, as long as you have already merged in linux-develop, it honestly isn't that bad and the conflicts were fairly straightforward. I got this merged and all working in our internal firestorm development build for both Windows and Linux (awaiting Rye to finish Mac stuff before testing that), and everything is working and packaged almost identically how it was with autobuild. There was a few things here and there after the fact you notice and need to fix or were slightly wrong despite the viewer working, or doing a few little changes to fix debug builds, but they were all fairly easy to spot and fix. Working with vcpkg is honestly pretty nice, and although I've always used my own python helper script to avoid the annoyances of autobuild and the initial setup of it, for those who are cloning and building the viewer from scratch or don't have a helper script, they will have a far easier time and it will feel more intuitive. The only real downside of the vcpkg setup as it is right now, is if the packages aren't built and cached yet, it can take ~30+ minutes for it to do so on initial configuration before you even get to building the viewer. Ideally you only ever have to do this once (or at least rarely), so it shouldn't be an issue, but it will be one of the main differences people will notice. |
3576dd1 to
60b0bd8
Compare
…anup Fix llsechandler test failure on windows due to failing to find legacy openssl provider Fix multiply defined symbols in newview tests on some platforms Supress false positive Wfree-nonheap-object in llmutelist Fix dullahan and webrtc vcpkg ports to unbreak nuget based binary caching Fix USE_LTO option applying to debug/optdebug builds Fix python warning in llmanifest Add dependabot support for vcpkg Add USE_AVX2/USE_AVX/USE_SSE4_2 to reduce conflicts with forks that commonly provide these options Signed-off-by: Rye <rye@alchemyviewer.org>
60b0bd8 to
7aa57b3
Compare
Signed-off-by: Rye <rye@alchemyviewer.org>
…e llmeshoptimizer library by combining them into llmath
Fix CMakeLists.txt being included in cmake sources. Cmake automatically includes its list files in the IDE projects. Fix various small cmake and bash syntax errors Fix build system circularity issues with find_package(Threads) Fix BootstrapVcpkg not failing gracefully when a VCPKG_ROOT is found but not bootstrapped Fix APU library not linking to debug variant on windows Fix not finding the debug NDOF library build Fix incorrect usage of openssl provider api when staticly linked
7aa57b3 to
f4046cc
Compare
Description
This set of changes is to modernize the developer experience with vcpkg based dependency management and a cleaned up cmake build system.
Related Issues
Issue Link: #4390 #4508
Checklist
Please ensure the following before requesting review:
Additional Notes