EXT_mesh_features: Features and Properties for structured data

[donmccurdy]

EXT_mesh_features defines a means of storing structured metadata associated with geometry and subcomponents of geometry within a glTF 2.0 asset.

In most realtime 3D contexts, performance requirements demand minimizing the number of nodes and meshes in an asset. These requirements compete with interactivity, as applications may wish to merge static objects while still supporting interaction or inspection on those objects. Common performance optimizations for GPU rendering — like merging geometry or GPU instancing to reduce CPU overhead — may destroy references to distinct objects, their attributes, and their behaviors.

By defining a representation of conceptual objects ("features") distinct from rendered geometry, and a means of associating structured metadata ("properties") with those features, this extension allows applications to preserve important details of 3D assets for inspection and interaction without compromising runtime performance and draw calls.

This extension will be used by embedded glTF content in 3D Tiles for loading large, tiled geospatial datasets. The extension's design is intended to be similarly useful for individual glTF 2.0 assets outside of geospatial use cases, like large AEC designs, CAD models, or photogrammetry captures.

Disambiguation: glTF has other methods of storing details that could similarly be described as metadata or properties, including KHR_xmp_json_ld, Extras, and Extensions. While those methods associate data with discrete objects in a glTF asset — nodes, materials, etc. — EXT_mesh_features is uniquely suited for properties of more granular conceptual features in subregions composed of vertices or texels.

Markdown Preview:

• README.md

[donmccurdy]

Some earlier context on the use cases this extension addresses:

• Scene complexity limits #1699

[lilleyse]

Talked with @wallabyway and others today and there were some interesting interesting takeways.

There is a strong need for global feature IDs. In some systems these are 64-bit uints; after all 32-bit uints can only represent up to 4,294,967,295 discrete features and collisions could start to occur for a variety of reasons.

The FEATURE_ID attribute can be used as a global feature ID but since it's a vertex attribute it's constrained by the allowable glTF accessor component types - 32-bit integer component types are not allowed and 64-bit integer component types do not exist. FLOAT can only provide up to 2²⁴ safe integers.

In order to support 64-bit IDs you'd need to store these in a property table with the UINT64 component type. The FEATURE_ID would be an index into the table, so there'd be one level of indirection to get the global ID.

An example of this might be worth including in the extension writeup.

[wallabyway]

Good point on the UINT24 (safe float)...

We render to a target buffer RGBA (using MRT) to get UINT32 featureID (equivalent), but that complicates implementation for others. Hmmm🤔 what to do?

[wallabyway]

RE: The section in the spec on 'Schema-definitions'

https://github.com/CesiumGS/glTF/tree/proposal-EXT_mesh_features/extensions/2.0/Vendor/EXT_mesh_features#schema-definitions

Is it possible to limit this spec to just a featureID and not define how the ID is used?

Currently, the spec has a schema requirement, which is a bit too limiting for our AEC use-cases. Having just an featureID requirement, and not a schema definition, broadens the use for this extension.

Or perhaps I mis-interpreted the "Schema-definitions" section of the spec.
If so, perhaps we could word it differently ?

[javagl]

Currently, the spec has a schema requirement, ...

The schema definition itself is not required in the top-level extension object

Is it possible to limit this spec to just a featureID and not define how the ID is used?

I think that the case that you are referring to is already covered in the "Specifying Feature IDs" section. It says

Every propertyTables index must have an associated featureIds definition, but feature IDs may be defined without a property table. [...] As a result, the length of the featureIds array must be greater than or equal to the length of the propertyTables array.

So it is valid to say

// In primitive:
"extensions": {
  "EXT_mesh_features": {
    "featureIds": [
      {"offset": 0, "repeat": 3 }
    ]
  }
}

to assign Feature IDs to the triangles of a primitive, without specifying property tables, and therefore without assigning any internal meaning to these IDs. These IDs could then be picked up and resolved externally by an application.

(Maybe I overlooked some other requirement - so if there is a reason why this wouldn't be possible based on the specification, then this should indeed be clarified).

[wallabyway]

@javagl - Ah, thank you !

I think this is worth explicitly mentioning "what you said" somewhere in the spec.

ie.
"A minimal example of connecting FeatureIDs stored in triangle vertices, that could refer to IDs in an external database of properties"

Basically "What you said" 'verbatim', along with your json example.

[zeux]

to assign Feature IDs to the triangles of a primitive, without specifying property tables, and therefore without assigning any internal meaning to these IDs. These IDs could then be picked up and resolved externally by an application.

FWIW my understanding of the spec is that this only works for un-indexed geometry.

[lilleyse]

Yeah, if you assign a unique feature ID to each triangle like with {"offset": 0, "repeat": 3 } then each vertex would be unique and it would be redundant to have an indices buffer.

But more often groups of triangles will have the same feature ID and you can still benefit from indexed geometry. Maybe the example should be {"attribute": 0} since that's the more common case I think.

[zeux]

@lilleyse Hmm I'd like to clarify this to make sure we're on the same page.

My understanding is that using offset/repeat doesn't actually make all vertices unique by itself. What I meant is that typically in indexed geometry, there's no obvious mapping between the vertices and triangles (well, short of the index buffer). When you specify offset=0 repeat=3, my understanding of the spec is that you're taking the existing vertices - whatever the number and order - and assigning feature ids to vertices, the same one to each 3 consecutive ones. If the geometry is not indexed, this in effect will result in all 3 vertices of each triangle having a unique feature id. If the geometry is indexed, you're going to get the same mesh with the same indexing, but the feature ids are not going to map to triangles.

As a positive example, imagine a tool that batches N instances of the same mesh together by transforming each instance, generating a new vertex buffer, concatenating all vertex buffers and index buffers together. E.g. gltfpack does that on request. In this case, if each mesh had V vertices and T triangles, you can specify {"offset": 0, "repeat": V}, and you'll get each copy of the mesh in the resulting merged vertex buffer to have its own feature id, essentially resulting in instance ids being specified as part of each vertex. This would only work if you are merging the same mesh (V is the same for each mesh), but this doesn't depend on T or indexing.

[lilleyse]

@zeux yes exactly, everything you wrote is my understanding too.

[ptrgags]

Just wanted to cross-link a couple relevant issues/PRs from our end for better visibility.

[ptrgags]

Note that we've had some back-and-forth about the best way to describe the different types of feature IDs in JSON schema, see CesiumGS/3d-tiles#508.

The issue is that this very flexible definition of implicit feature IDs causes problems when used in this oneOf clause, as a valid featureIdTexture is valid here as well.

[javagl]

The latest update that has been integrated here separates the definition of feature IDs for elements of a glTF asset from the actual storage and structure of metadata.

This extension now focusses on the definition of feature IDs for vertices and texels in a glTF asset.

The definition of the structure and storage of metadata is covered with the EXT_structural_metadata extension that is proposed in #2151

[emackey]

This extension is already in use by published software, right? Is this ready to be merged?

[javagl]

^{(Disclosure: I'm an independent Khronos contributor, but have been involved in the development of this extension as part of contracted (freelancer) work for Cesium. I do not speak on behalf of Khronos or Cesium here, but just try to summarize the state)}

Support for reading the extension is implemented in CesiumJS. Basic structures for the extension are part of cesium-native. An internal implementation of the extension for glTF-Transform is part of the 3d-tiles-tools. There is also an "experimental" implementation in JglTF. The actual specification seems to be "settled", insofar that there have not been significant changes for quite a while now.

I think that further reviews could be worthwhile. But as for all extension PRs, there's the question: Who of the WG could allocate the time that is necessary for that...?

[JMLX42]

Excellent idea!

If you don't mind me asking, what is the rationale against using XMP/JSON-LD like the KHR_xmp_json_ld extension does?

There are at least 2 reasons for using XMP/JSON-LD:

1. Having one single standard used to define rich metadata would be more convenient for implementation.
2. Unlike JSON Schema, the XMP/JSON-LD is not just a format/notation system. It builds on the semantic Web standards. As such, it already features a rich ecosystem that the proposed feature could leverage. Examples:
   • The ODRL standard could be leveraged to define per-submesh permissions
   • The Web Annotations standard could be leveraged to have per-submesh annotations

[javagl]

Given that there is the same comment in another PR, I think that it could make sense to address some of that with first, high level answer in one place (and split the discussion in the PRs only when necessary).

And I'll have to start with a disclaimer: I know that there are many, many, MANY standards for metadata, schemas, taxonomies, ontologies, with everything that falls under the umbrella of "semantic web" and beyond. But I don't have a clue about all that. If someone claims to be able to give an executive summary that covers everything that is relevant here, drop me a note 😉

On a high level, the main differences between this proposal (and the EXT_structural_metadata one) and things like the existing XMP extension have already been mentioned in the "Disambiguation" section of the first post here. These differences are mainly in the representation and the granularity.

Specifically: The the EXT_mesh_features and EXT_structural_metadata extension aim at supporting "identifiers" and metadata on the level of the geometry. There are some examples for these extensions in the 3d-tiles-samples/glTF repository.

For example, consider a mesh(primitive) that is stored in a glTF, and that contains four simple squares. It's a single "object" from the perspective of glTF. And XMP could be used to assign metadata to this single mesh. But with EXT_mesh_features, it is possible to assign identifiers to each "sub-geometry" within that mesh, as shown in the FeatureIdAttribute example:

Building on top of that, these IDs can be associated with metadata using the EXT_structural_metadata extension. The metadata itself is stored in binary form (which may be waaay more compact than a JSON-based representation). And again, it can be assigned to elements of what is otherwise considered to be a "single objectt" from the perspective of glTF. For example, it is possible to associate metadata with certain (surface) areas of a geometry, like this:

[JMLX42]

And I'll have to start with a disclaimer: I know that there are many, many, MANY standards for metadata, schemas, taxonomies, ontologies, with everything that falls under the umbrella of "semantic web" and beyond. But I don't have a clue about all that. If someone claims to be able to give an executive summary that covers everything that is relevant here, drop me a note 😉

It is the sole purpose of my question actually.

I'm the main author/designer of the tech behind https://smartshape.com/. Wich we are rebuilding from scratch using glTF 2.0 as the representation layer.

When I "discovered" the only way to support metadata on a per object basis was KHR_xmp_json_ld, at first as was puzzled. Then I remembered about all the things our product has to manage:

• annotations
• authorizations
• CAD metadata
• localization
• sensors
• etc...

So I started wondering: maybe the XMP/JSON-LD extension (and it's added complexity) has its purpose. And I started reviewing each of those requirements: do they have an existing W3C/ISO standard that can be leveraged by XMP/JSON-LD?

And the answer is yes. I mentioned ORDL and Web Annotations. But surely there are others. And since glTF aims at being 1. standard and 2. a transmission format, I now think those standards apply.

On a high level, the main differences between this proposal (and the EXT_structural_metadata one) and things like the existing XMP extension have already been mentioned in the "Disambiguation" section of the first post here. These differences are mainly in the representation and the granularity.

For example, consider a mesh(primitive) that is stored in a glTF, and that contains four simple squares. It's a single "object" from the perspective of glTF. And XMP could be used to assign metadata to this single mesh. But with EXT_mesh_features, it is possible to assign identifiers to each "sub-geometry" within that mesh, as shown in the FeatureIdAttribute example:

I think I understand that. That's one of the challenges when we're dealing with (for example):

• merged geometries for the sake of LoD or batching
• point cloud rendering, where each point is not an object (for obvious reasons)

So IMHO I think I understand why those extensions proposal exist. And I want them to exist. Actually I want to implement them!

Building on top of that, these IDs can be associated with metadata using the EXT_structural_metadata extension. The metadata itself is stored in binary form (which may be waaay more compact than a JSON-based representation). And again, it can be assigned to elements of what is otherwise considered to be a "single objectt" from the perspective of glTF.

My understanding is that the schema is JSON cf https://github.com/CesiumGS/glTF/tree/proposal-EXT_structural_metadata/extensions/2.0/Vendor/EXT_structural_metadata#class) but the actual data (matching that schema) is binary (in a Buffer and/or eventually through a BufferView).

So my question is: why a custom JSON format and not XMP/JSON-LD?

For example, "classes" sounds a lot like an RDF definition. XMP/JSON-LD already has namespaces to avoid class name collisions.

{
  "@context": {
    "mv": "http://example.org/movement/1.0#",
    "xmp": "http://ns.adobe.com/xap/1.0/",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  },
  "@type": "mv:PointCloudMovement",
  "mv:schema": {
    "@id": "mv:Movement",
    "@type": "rdfs:Class",
    "rdfs:label": "Movement",
    "rdfs:comment": "The movement of points in a point cloud",
    "mv:properties": [
      {
        "@id": "mv:direction",
        "@type": "rdf:Property",
        "rdfs:label": "direction",
        "rdfs:comment": "The movement direction, as a normalized 3D vector",
        "mv:datatype": "mv:Vec3",
        "mv:componentType": "xsd:float",
        "mv:required": true
      },
      {
        "@id": "mv:magnitude",
        "@type": "rdf:Property",
        "rdfs:label": "magnitude",
        "rdfs:comment": "The magnitude of the movement",
        "mv:datatype": "mv:Scalar",
        "mv:componentType": "xsd:float",
        "mv:required": true
      }
    ]
  },
  "mv:propertyMappings": {
    "@type": "mv:AttributeMapping",
    "mv:namespace": "mv",
    "mv:mappings": {
      "mv:direction": {
        "mv:attribute": "_DIRECTION"
      },
      "mv:magnitude": {
        "mv:attribute": "_MAGNITUDE"
      }
    }
  }
}

Now it sounds like a lot of efforts for just a different JSON. But really it's not.

For example JSON-LD supports composition:

{
  "@context": {
    "mv": "http://example.org/movement/1.0#",
    "schema": "http://schema.org/",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "pc": "http://example.org/pointcloud/1.0#"
  },
  "@graph": [
    {
      "@id": "pc:AnimatedPointCloud",
      "@type": ["schema:Dataset", "pc:PointCloudDataset"],
      "schema:name": "Animated Point Cloud with Movement Data",
      "schema:description": "A point cloud dataset composed of geometry and movement information",
      "schema:hasPart": [
        {"@id": "pc:GeometryComponent"},
        {"@id": "mv:MovementComponent"},
        {"@id": "pc:ColorComponent"}
      ],
      "schema:isPartOf": {
        "@id": "pc:LargerDatasetCollection"
      }
    },
    {
      "@id": "pc:GeometryComponent",
      "@type": "pc:PointCloudGeometry",
      "schema:name": "Point Geometry",
      "pc:schema": {
        "@id": "pc:Position",
        "@type": "rdfs:Class",
        "rdfs:label": "Position",
        "rdfs:comment": "3D position of points",
        "pc:properties": [
          {
            "@id": "pc:coordinates",
            "@type": "rdf:Property",
            "rdfs:label": "coordinates",
            "pc:datatype": "pc:Vec3",
            "pc:componentType": "xsd:float",
            "pc:required": true
          }
        ]
      },
      "pc:propertyMappings": {
        "@type": "pc:AttributeMapping",
        "pc:mappings": {
          "pc:coordinates": {
            "pc:attribute": "_POSITION"
          }
        }
      }
    },
    {
      "@id": "mv:MovementComponent",
      "@type": "mv:PointCloudMovement",
      "schema:name": "Movement Data",
      "mv:schema": {
        "@id": "mv:Movement",
        "@type": "rdfs:Class",
        "rdfs:label": "Movement",
        "rdfs:comment": "The movement of points in a point cloud",
        "mv:properties": [
          {
            "@id": "mv:direction",
            "@type": "rdf:Property",
            "rdfs:label": "direction",
            "rdfs:comment": "The movement direction, as a normalized 3D vector",
            "mv:datatype": "mv:Vec3",
            "mv:componentType": "xsd:float",
            "mv:required": true
          },
          {
            "@id": "mv:magnitude",
            "@type": "rdf:Property",
            "rdfs:label": "magnitude",
            "rdfs:comment": "The magnitude of the movement",
            "mv:datatype": "mv:Scalar",
            "mv:componentType": "xsd:float",
            "mv:required": true
          }
        ]
      },
      "mv:propertyMappings": {
        "@type": "mv:AttributeMapping",
        "mv:namespace": "mv",
        "mv:mappings": {
          "mv:direction": {
            "mv:attribute": "_DIRECTION"
          },
          "mv:magnitude": {
            "mv:attribute": "_MAGNITUDE"
          }
        }
      }
    },
    {
      "@id": "pc:ColorComponent",
      "@type": "pc:PointCloudColor",
      "schema:name": "Color Information",
      "pc:schema": {
        "@id": "pc:Color",
        "@type": "rdfs:Class",
        "rdfs:label": "Color",
        "rdfs:comment": "RGB color values for points",
        "pc:properties": [
          {
            "@id": "pc:rgb",
            "@type": "rdf:Property",
            "rdfs:label": "rgb",
            "rdfs:comment": "RGB color values",
            "pc:datatype": "pc:Vec3",
            "pc:componentType": "xsd:unsignedByte",
            "pc:required": true
          }
        ]
      },
      "pc:propertyMappings": {
        "@type": "pc:AttributeMapping",
        "pc:mappings": {
          "pc:rgb": {
            "pc:attribute": "_COLOR"
          }
        }
      }
    },
    {
      "@id": "pc:LargerDatasetCollection",
      "@type": "schema:DataCatalog",
      "schema:name": "Time-Series Point Cloud Collection"
    }
  ]
}

or inheritance:

{
  "@context": {
    "mv": "http://example.org/movement/1.0#",
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  },
  "@graph": [
    {
      "@id": "mv:Motion",
      "@type": "rdfs:Class",
      "rdfs:label": "Motion",
      "rdfs:comment": "Base class for any kind of motion"
    },
    {
      "@id": "mv:LinearMotion",
      "@type": "rdfs:Class",
      "rdfs:subClassOf": {"@id": "mv:Motion"},
      "rdfs:label": "Linear Motion",
      "rdfs:comment": "Motion along a straight line"
    },
    {
      "@id": "mv:RotationalMotion",
      "@type": "rdfs:Class",
      "rdfs:subClassOf": {"@id": "mv:Motion"},
      "rdfs:label": "Rotational Motion",
      "rdfs:comment": "Motion around an axis"
    },
    {
      "@id": "mv:ComplexMotion",
      "@type": "rdfs:Class",
      "rdfs:subClassOf": [
        {"@id": "mv:LinearMotion"},
        {"@id": "mv:RotationalMotion"}
      ],
      "rdfs:label": "Complex Motion",
      "rdfs:comment": "Motion combining linear and rotational components (multiple inheritance)"
    }
  ]
}

What if I want to filter a CAD file using metadata? RDF and JSON-LD already has query languages like SPARQL.

So with XMP/JSON-LD the metadata would follow the same logic as the existing per-object metadata, re-use the same definitions, leverage composition/inheritance, leverage the whole existing standard ecosystem such as querying.

[javagl]

My understanding is that the schema is JSON [...] but the actual data (matching that schema) is binary (in a Buffer and/or eventually through a BufferView).

That is correct.

So my question is: why a custom JSON format and not XMP/JSON-LD?

For example, "classes" sounds a lot like an RDF definition. XMP/JSON-LD already has namespaces to avoid class name collisions.

That's a good question, and I cannot give a definite answer right now. Some possible reasons that I could come up with, on a high level:

• Looking at the timelines, the metadata extensions seem to have been developed roughly in parallel with the XMP extension. The XMP extension is ratified in the meantime. It might be that if someone had to create these metadata extensions "from scratch" now, the option to use the XMP extension for the schema would be considered. (Although I may not understand each technical detail just by looking at them, the representations that you posted as an alternative description of the class structure at least look very reasonable for me)
• (Conversely: ) Maybe the XMP extension has some complexities that could raise the bar for entry and increase the implementation effort for the metadata extensions (although I think that the XMP extension itself is not terribly complex)
• (Related: ) Maybe XMP is "too powerful", in some way. You mentioned inheritance, for example. And one might have to investigate what that means and how that could be applied to data that is stored in binary buffers. The point is: Even if it was allowed to describe the structure of the metadata with XMP instead of the 'custom schema JSON', it would still be necessary to carefully define the valid class structures and property types that can be modeled with that

Maybe some of the developers of the XMP extension and the metadata extensions can chime in here with additional thoughts and details.

[lilleyse]

I think this extension is in a pretty good state, but there's a few things I've been thinking about:

• Should signed integer component types by allowed, so that we can assign nullFeatureId to -1?
• Should Feature ID by Texture Coordinates be split up into a dependent extension?
• Should Feature ID by Index be removed?
• Should label be removed, or perhaps renamed?
• Should featureCount be removed? CC 3d-tiles#756
• Should interpolated features be disallowed? CC 3d-tiles#764
• Should feature IDs be run-length encoded? CC 3d-tiles#366

[lilleyse]

_FEATURE_ID_n may need to be renamed to EXT_mesh_features:FEATURE_ID_n

[lilleyse]

Notes from meeting with @lexaknyazev, @abwood, and @weegeekps

• Remove SCALAR restriction so that multiple feature ID sets can be packed in the same vertex attribute. Add channels for selecting the component(s) similar to the feature ID texture approach.
• Add explicit requirement that referenced feature ID attributes must exist in the mesh
• Make attribute and texture mutually exclusive
• Clarify definition of Feature ID by Index
• Fix dependency between EXT_mesh_features and EXT_structural_metadata
  • Option 1: EXT_structural_metadata extension on feature ID set
  • Option 2: EXT_structural_metadata extension on primitive that links feature ID set with property table
• Remove alphanumeric restriction for label if it's intended to be used for UI purposes. Otherwise rename to symbol and enforce unique values.

[lilleyse]

And some next steps:

• Update EXT_mesh_features with clarifications and non-breaking changes listed above
• Merge EXT_mesh_features as a multi-vendor extension
• Start working on KHR_mesh_features with a path towards ratification about a year from now
  • Aligned with 3D Tiles 2.0 roadmap
  • May include breaking changes

[lexaknyazev]

Fix dependency between EXT_mesh_features and EXT_structural_metadata

Also Option 0: keep the schema as-is and specify that if propertyTable is present, the EXT_structural_metadata extension must also be present and it must contain the corresponding table.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 4 committers have signed the CLA.

✅ javagl
✅ lilleyse
❌ donmccurdy
❌ ptrgags
_{You have signed the CLA already but the status is still pending? Let us recheck it.}