Conversation
|
Focusing on the API design: it looks fine to me. I'm curious to hear from @panxinmiao. Are there any overlooked use cases? |
|
I still believe this design conflates two distinct concepts: the definition of transparency (a high-level API concept at the engine level) and the blending configuration (a low-level API concept at the GPU level). What we commonly refer to as “alpha blending” is just one specific configuration of the blending settings in the rendering pipeline — a particular blending mode. It corresponds to the classic alpha blending formula: As I understand it, Moreover, blending in the rendering pipeline is not exclusive to transparent objects. It's quite common for opaque objects to use blending as well (e.g., for glowing or light-accumulation effects). The engine should not restrict this capability. Additionally, if we follow this API design, how should we categorize objects during rendering — i.e., assign them to different render passes? Furthermore, the extra parameters under |
|
I'm not entirely sure if I expressed my point clearly. If my wording caused any confusion, I sincerely apologize. I would like to explain my concerns again: 1、
|
In the proposed API, what you refer to is an object with
I believe objects can be unambiguously categorized (from the pov of the renderer) based on just their
I don't agree with this distinction of high-level vs GPU-level to be honest. There is certainly a categorization, where we have four main categories, where each category has a number of options, but all it does it define how alpha values are used in rendering the object. Or are you objecting to the fact that the presets for e.g. |
No, it's important. This is exactly where the problem lies, we need clear and unambiguous categorization. Without it, it's unclear which render pass should handle the object, nd which stage of the pipeline it belongs to. Regarding object classification, we indeed didn't pay much attention to it before (prior to the simplification and refactoring of Blender), because at that time, the renderer did not yet distinguish between different rendering passes or stages — so the classification of an object didn’t matter much.
Not just that, it also determines which render pass is used and at what stage in the pipeline the object is rendered. This is not a minor technicality, it's a fundamental architectural concern in the subsequent renderer design and render pass scheduling.
I agree that As for using
Yes, that's exactly my point. |
Please help me understand, by providing an example where an object has |
For example, scene backgrounds (or objects meant to serve as scene backgrounds), decals, or helper objects and markers for debugging or visualization purposes. I once had a use case where an object had a hidden proxy geometry—only the proxy geometry wrote to the depth buffer, while the visible object itself only wrote to the color buffer but not to depth |
|
Yeah but how do you think the renderer should use the value of |
|
For example, some skybox or skydome models are modeled as a box or a sphere. See this model on Sketchfab: https://skfb.ly/oIpX9 and https://skfb.ly/oIHQL When used, these typically have the following settings: transparent=False,
render_order=-99, # to ensure they render first
depth_test=False,
depth_write=False # so they don’t participate in any depth-related operations.And they are unit-sized and have their center position follow the camera every frame. As for decals, they refer to "localized texture layers attached on top of existing geometry" to enhance visual detail—such as cracks, stains, bullet holes, or signs—without modifying the object's base geometry or main texture. In essence, a decal might be a small piece of geometry (like a plane) that is projected onto the target surface using geometric projection techniques. While these effects are usually achieved using textures with transparency (i.e., alpha blending), making them visually semi-transparent, they should not be classified as transparent objects from the rendering pipeline's perspective. Decals do not participate in the transparent object depth sorting done in the main geometry pass or any other complex rendering behavior for transparent objects (such as double-sided/double passes rendering, as well as possible future more complex processes such as depth peeling, dual depth peeling, etc.). they behave more like a screen-space overlay. Decals are typically a separate pass or a separate render group in opaque pass following opaque objects (to control the rendering order), and are directly overlaid on the result of the opaque pass. A typical decal setup maybe like this: transparent = False
depth_test = True
depth_write = False
blend_mode = "normal"Of course, the above parameter settings can be adjusted based on the actual needs of the scene. However, the key point is that low-level GPU pipeline features such as depth_write and blending method are atomic capabilities and should not be used as the basis for object classification. These features are flexible and can be freely combined by users to meet various creative needs, and we should not impose restrictions on the engine's ability to support such flexibility. We can say that a certain category of objects (e.g., "opaque" or "transparent") is typically implemented using a specific set of parameters, but we should not assume that a particular parameter configuration necessarily implies the object belongs to a certain category. |
|
Thanks for the explanation; I understand your point better now. It has everything to do with controlling the order that objects are rendered in. I'll give this some more thought ... |
|
I'm moving rather slow since I am on holiday and try to get some Pygfx done in the late evenings / early mornings. I've updated this pr and the #1144. I've not included a I went a bit back and forth with naming things but I settled on:
Further:
|
|
mmm, things still don't feel right to me. I've been reading a bit and I quite like Unity's I feel like this is a quite elegant and flexible solution. Better than ThreeJS (setting We can derive a suitable value from I'll have a stab at this soon, but figured I'd post the idea here to see what others think. |
|
I updated this pr and #1144. IMO the |
To me this looks fantastic but I also think you should just move on and accept the outcome. In the interest of other priorities. :) |
|
I believe there is one more point that needs to be clarified and documented. Specifically, which rendering passes/stages our renderer internally uses (currently, we have three passes/stages in #974: Opaque, Transparent, and Weighted), and according to our API design, what rules govern the assignment of objects to these specific rendering passes/stages. Additionally, is In other words, if an object is assigned to the Transparent stage based on the above rules, but the user sets its |
|
I guess it was a little frustrating process, but I am really impressed by the outcome. ❤️ |
|
uge |
|
Next is up is the actual implementation, which is ready in #1144 |
|
Sorry to bring up this topic again, 😓 After several days of hands-on practice and investigation, I can confirm that the renderQueue here is not the object classification mechanism we need. Essentially, it is a higher-level sorting mechanism. On the CPU side, it buckets and sorts all objects based on their renderQueue to determine the order and grouping strategy of DrawCalls.
The only difference is that Unity allows users to directly set the RenderQueue. However, Queue or RenderType itself is explicit, unambiguous, and requires user input—the engine does not infer it (and in my opinion, cannot infer it, as it is essentially a meta-attribute that reflects user intent). The Queue or RenderType values—Opaque, TransparentCutout, and Transparent—are exactly the internal phases and object classifications we need in the renderer. Additionally, while reviewing Babylon.js documentation, I noticed that Our current issue is that although I’d like to reiterate: object classification and renderer phase division are based on the engine’s own scene rendering logic and design philosophy—they exist to enable specific engine capabilities, e.g., supporting Weighted Blend OIT objects by introducing a dedicated Weighted phase (the rationale for treating it as a separate phase can be discussed elsewhere). GPU blending, on the other hand, is a rendering pipeline capability with various applications. It is not synonymous with transparency or alpha blending, nor is it exclusive to a specific object category or render phase. A simple example: Furthermore, if desired, users can achieve correct alpha-blended transparency effects even in this phase. When blending back-to-front iteratively (where When blending front-to-back (typical for the opaque phase), use: The two are mathematically equivalent. However, if users do this, the renderer still treats it as an opaque object and processes it in the opaque phase. This means all engine logic after the opaque phase (e.g., generating SSR buffers for physically based transparency, special handling for transparent-phase objects like dual-side dual-pass rendering, etc.) occurs after this object is rendered. At last, I feel the current design of concepts like Additionally, while well-intentioned to minimize user cognitive load (by reducing the number of values users need to know and set), having This is similar to how developers using pygfx must understand what Geometry represents, what Material represents, and what vertex positions in Geometry mean—while end-users of applications built on pygfx don’t need to know these details. For example, the current code defaults I think, trying to “do everything for the user” in a rendering engine often leads to bigger problems. |
|
I have a feeling the confusion comes from a need to want to classify an object as opaque or transparent. The short answer is this: matrerial.render_queue = 2000 # == material.transparent = False
material.render_queue = 3000 # == material.transparent = TrueYes, the value of
You are correct that alpha-method is a poor identifier for intent by itself. You should probably set
Same answer, set
I agree it's a rather complex system. I have struggled with this a lot, and it still frustrates me. I would very much present users with a system that Just Works. I actually think we could get close if we adopt techniques such as adaptive transparency, mlab etc. which insert output fragments into a multi-layer render target. But such techniques suffer a significant performance and memory hit. So I've somewhat accepted that we cannot really solve this problem for the user. Instead we offer more solutions that most engines do. So the complexity comes partly from the fact that we have included support for stochastic and weighted methods. This offers more ways to deal with transparency, at the cost of a more complex API 🤷
Also true. I can then recommend to explicitly set most such enums. I think it makes sense to recommend users to use explicit values in the docs. BTW: the 'auto' alpha-mode also comes from the fact that we have multiple objects (lines, points, text) that have semi-transparent fragments for aa even if the object is opaque. Such objects can still be rendered fine, especially in 2D scenario's. |
|
I believe that render_queue is not suitable as a user-facing API for expressing intent. It should be treated as an internal mechanism, since its meaning is neither clear nor intuitive to most users. In fact, the value of render_queue is usually derived from other properties, and users rarely need to set it directly. As I understand it, automatic alpha blending was introduced mainly to support anti-aliasing (AA) for points, lines, text, and similar objects. However, this actually highlights that object classification/phasing and GPU blending capabilities are orthogonal concepts, and should not be coupled together. For example, when the opacity=1, these objects should be classified as opaque, with no reason to place them in the transparent rendering stage. This applies even more strongly to Meshes: by default, they should not be treated as transparent objects (when opacity=1.0), since doing so violates user expectations, introduces potential issues, and conflicts with common usage patterns. That said, it is reasonable for these 2D objects to require GPU blending when AA is enabled. In my view, such objects should still belong to the opaque category, but enabling AA should automatically configure the GPU blending state of their rendering pipeline (or adjust the corresponding material's GPU blending parameters). Additionally, because these 2D objects essentially reside on a flat z=0 plane, they neither need depth testing nor complex sorting. A simple sequential rendering order—based on layer or order—is sufficient. |
|
There is no "opaque rendering stage" or "transparent rendering stage". And there is no way to classify an object as either opaque or transparent. There's two main things:
There is also
To address this point specifically as an example: objects that are blended but also write depth, by default end up in render_queue 2600, which sits somewhere in between the default queues for opaque (2000) and transparent (3000) objects, and objects are sorted back-to-front (so that any aa edges blend correctly). |
The |
Dividing the rendering process into stages is a necessary prerequisite for implementing some modern real-time rendering techniques. I am certain that almost all modern rendering engines at least separate the opaque and transparent stages, since many techniques require the renderer to perform specific operations between these two stages—for example, generating the SSR buffer. This was also the original motivation behind introducing object classification and stage division in #974. I will explain in detail in #974 when I have time. |
I’d also like to add a note here: as I understand it, the current "aa" for certain opaque objects (such as points, lines, and text) relies on transparent fragments, which makes the API design and implementation somewhat awkward. |
The purpose of aa is to blend the edges of objects with the things rendered behind it, to avoid jaggies. It's a very efficient way to prevent aliasing because it in the shader you can compute the exact pixel coverage. The downside is that it needs blending, so it relies on sorting to prevent artifacts. I think premultiplied alpha is a different thing. |
|
copied from #974 render stages vs render_queue I feel like they're two different ways to approach the same problem, where First the problems that I see with the currently proposed approach (in #974): The three render stages representing the generally-transparent objects, are forced in the order: transmissive, transparent, weighted. Ignoring the You added a comment for the weighted blending, saying that it should absolutely not be mixed with other transparent objects. I think there are cases where it could, as long as both groups of objects are spatially separate. And the same argument can be made for transmissive vs transparent objects! In the renderer logic in In that way, the renderer would first sorts all objects in the normal way. Then Some things that would be possible with this approach:
|
The rendering process is currently divided into three stages: opaque, transparent, and weighted blend. Transmissive and transparent objects are handled within the same rendering stage/render pass. However, I deliberately sort transmissive objects before regular transparent ones using renderQueue. This ordering brings two benefits, as transmissive objects typically do not rely on GPU blending and usually
This is exactly the issue I mentioned earlier: renderQueue is essentially a higher-level sorting mechanism, while the division of rendering stages is an inherent part of the renderer’s logic. No matter how renderQueue is adjusted, the sorting only takes effect within a given rendering stage, and cannot cross stage boundaries. Before considering how to handle Weighted Blended objects, I attempted to research the implementations of current mainstream rendering engines. However, I found that the vast majority of engines do not have built-in OIT (Order-Independent Transparency) pipelines. Even when they do, OIT is only treated as a global transparent rendering solution, separate from traditional sorting-based transparent methods. Therefore, there is a lack of readily available reference cases or best practices for using Weighted Blended objects alongside ordinary alpha blend objects. We aim to support Weighted Blended OIT and alpha blend objects simultaneously to some extent. However, the unique aspect of Weighted Blended OIT is its inherently multi-pass nature (at least requiring Pass A: rendering OIT objects to an accumulation/revealage buffer; Pass B: compositing to the main framebuffer via a full-screen pass). This differs from the standard forward rendering process. When mixing Weighted Blended OIT objects with ordinary alpha blend objects, frequent GPU rendering state switches are difficult to avoid:
If these two types of objects appear alternately, it will trigger expensive render pass switches. A reasonable idea is: since Weighted Blended OIT objects do not participate in depth sorting themselves, there is no need to handle interleaving scenarios between them and ordinary alpha blend objects. Therefore, all OIT objects can be processed uniformly at a certain stage of the rendering pipeline, thereby avoiding the complexity introduced by their multi-pass nature—that is, treating OIT as a completely independent rendering stage, decoupled from the rendering process of other objects. Based on this, categorizing Weighted Blended OIT objects as a separate type and assigning them an independent rendering stage is a natural implementation. Although some users might desire the flexibility to manually specify the rendering order to achieve interleaved rendering of OIT and ordinary transparent objects, the value of this flexibility is likely outweighed by the performance cost and implementation complexity it introduces. Frequent render pass switches are not only prone to unintentional misuse by users, leading to performance degradation, but mixing Weighted Blended objects with ordinary alpha blend objects might also cause unexpected rendering issues. Furthermore, this would impose constraints on the engine's feature expansion and maintenance. For example, adding support for two-pass rendering of double-sided transparent objects would necessitate additional handling for compatibility with Weighted Blended OIT objects, introducing unnecessary complexity. I believe this design choice is essentially an engineering trade-off. The core of real-time rendering technology lies in finding a balance between visual realism, performance cost, and implementation complexity. This is also the fundamental distinction between modern real-time rendering and offline rendering. While the basic laws of physics are inherently simple—light propagation, energy conservation, laws of reflection and refraction, all expressible precisely in mathematics—offline rendering (e.g., path tracing, Monte Carlo integration) tends to follow these physical laws directly, approximating the real world through extensive sampling and computation. In contrast, real-time rendering heavily employs approximations and simplifications (e.g., BRDF approximations, precomputed IBL) to achieve acceptable visual quality within the constraints of limited performance budgets and engineering complexity. Handling Weighted Blended OIT as a separate stage adheres to this philosophy. A similar trade-off is evident in the handling of transmissive light in #974. For transmissive objects, screen-space refraction (SSR) is needed to gather information about the scene behind them. Therefore, I tried looking at implementations in mature engines, and the final answer was straightforward. The methods adopted by the reference implementation of the glTF 2.0 Transmission extension, as well as engines like Three.js, Babylon.js, and Unity HDRP, are all very "simple and direct": after all opaque objects are rendered, a copy of the background color buffer is generated, and all transmissive objects share this buffer for sampling. Although this scheme cannot handle the superposition of multiple layers of transmissive objects, it is simple, efficient, and practical. I believe this is a simple trade-off, and one that makes sense. Firstly, in the vast majority of scenarios, there won't be overlapping and superposition of multiple transmissive objects. Furthermore, adopting this fixed strategy also inversely requires scene designers and technical artists to optimize scene design accordingly, rather than abusing the feature. Content creators inherently need to optimize asset structure and scene layout within the engine's capabilities to achieve the best visual results. All these engines divide the rendering process into multiple stages—at least an opaque stage and a transparent stage—and perform specific tasks between these stages based on logical requirements. Generating the SSR framebuffer is one such task. Some global illumination, AO, volumetric fog, and even certain post-processing effects also need to be inserted between the opaque and transparent stages. Unity even exposes injection points between these stages to let users plug in custom logic or passes. |
|
Sorry for post such a “long essay” without properly dividing it into paragraphs and section titles, 😅 . I’ll add some examples and supporting facts to certain arguments and viewpoints in the text when I have more time. |
I think you're confusing From what I understand how Unity works. It sorts all objects based on the renderQueue, then on the depth. Then per renderQueue it renders all objects in that queue, which can be a mix of objects requiring different render passes. The SSR texture is copied right when the first (transparent) object that needs it is to be rendered.
Awesome! This can be done perfectly fine with the way we iterate over objects in current Imagine the case where a user does not set
But with As for mixing the different kinds of transparent objects in a single render_queue. I don't really see a problem. The normal and translucent objects can be mixed, as long as we just copy the SSR texture beforehand. The weighted objects are also fine, because they get grouped separately from the other objects automatically because of the depth-sorting (their key is zero). So there is no need to fear these "render pass interruptions"; by default there would be just one SRR copy, and one group of weighted objects (if any). Only when the user uses |
|
In fact, the engine is deliberately designed to avoid creating multiple SSR textures for different transmissive objects. In the rendering pipeline, a full-resolution screen-sized copy is already a very heavy operation, and generating mipmaps for the SSR color buffer adds even more overhead. When multiple transmissive objects are present, this quickly becomes a performance bottleneck. I can confirm that Unity HDRP only maintains a single SSR buffer, generated at a fixed stage in the renderer. Unity also provides a BeforePreRefraction injection point, which is specifically placed before SSR generation. This is the documentation for Unity HDRP:
Blender seems to be the same: I believe the same principle applies to weighted blended objects: they should be handled in a single batch. Otherwise, the overhead of frequently interrupting the rendering pipeline and switching rendering states would far outweigh any potential benefits.
As I mentioned earlier, I believe these measures are, in fact, intentional restrictions on users creating arbitrarily complex scenes. Such complexity is usually unnecessary, and may be abused. and the engine in fact may not be able to handle them well, which could lead to potential issues. |
This documentation clearly outlines Unity’s rendering pipeline stages, including the fundamental opaque and transparent stages, and further explains the exact timing of tasks executed within or between these stages. In the transparent stage, Unity also processes refraction first and then transparents, which is consistent with our logic. As I mentioned earlier: RenderQueue is essentially a higher-level sorting mechanism, while the division of rendering stages is an inherent part of the renderer’s logic. I haven’t found detailed documentation on how Unity assigns renderable objects to different rendering stages, but I believe the division is based on whether the object’s renderQueue is greater than 2500 (opaque stage or transparent stage). Therefore, it can also be said that RenderQueue determines which rendering stage an object is assigned to. https://docs.unity3d.com/ScriptReference/Rendering.RenderQueue.html |
Sure, any added flexibility is also an opportunity for users to do something stupid 😉. I think the design would naturally allow users to create advanced use-cases. But I'm fine with restricting the SRR to being created only the first time it's needed.
Yes, you could say that. I'd even say there is not really an explicit "rendering stage". Objects are sorted by |
No, for a renderer, rendering stages are explicitly defined—this is a fundamental aspect of pipeline design. Unity follows the same principle. Tasks such as SSR generation are also executed at a specific time points. This has already been explained in the Unity document that I mentioned earlier: https://docs.unity3d.com/ScriptReference/Rendering.RenderQueue.html According to Unity’s documentation, whether an object belongs to the transparent stage is determined by whether its RenderQueue value is greater than 2500. In other words, both 2600 and 3000 are rendered in the transparent stage, with lower values rendered earlier. RenderQueue itself is not the render stage; rather, it is a higher-level sorting mechanism within a stage, intended to provide flexibility for batching optimizations. For most users, Unity recommends following the predefined RenderQueue conventions to avoid potential rendering issues. |
4 participants
Ref discussion in #1120.
I refactored the current guide, moving the content to multiple files, to make room to go into some topics deeper, like we do here for transparency.
The text does not describe how Pygfx currently work, but how I think it should work. Open for suggestions. The idea is to get (more or less) agreement on this, then implement it accordingly.