From 5a67cf1aa7f082a1282862e82f7f231b01b96cd3 Mon Sep 17 00:00:00 2001 From: Vyacheslav Egorov Date: Thu, 4 Dec 2025 21:46:47 +0200 Subject: [PATCH 1/6] Initial proposal --- docs/buffer-metatable.md | 114 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 docs/buffer-metatable.md diff --git a/docs/buffer-metatable.md b/docs/buffer-metatable.md new file mode 100644 index 000000000..4efb8d7ed --- /dev/null +++ b/docs/buffer-metatable.md @@ -0,0 +1,114 @@ +# Extend buffer with a metatable + +## Summary + +Extend `buffer.create` with a second argument which provides a metatable for that buffer. +The metatable and some of its keys will be frozen for potential improvements in access. + +## Motivation + +Providing a buffer object with a metatable will allow creation of objects with small storage footprint which can also be matched to native structures on the side of the application using Luau. + +By having a metatable, these buffer objects can have properties, methods and other kinds of operations. + +This will also allow the behavior to be defined and extended on the Luau side, compared to userdata that is strictly defined by the host. + +In a way, this will provide an alternative to `ctypes` in luajit. + +```luau +-- A float4 in a buffer +local mt = { + __index = function(b, field) + if field == "x" then return buffer.readf32(b, 0) + elseif field == "y" then return buffer.readf32(b, 4) + elseif field == "z" then return buffer.readf32(b, 8) + elseif field == "w" then return buffer.readf32(b, 12) + else error("unknown field") end + end, + __newindex = function(b, field, value) + if field == "x" then buffer.writef32(b, 0, value) + elseif field == "y" then buffer.writef32(b, 4, value) + elseif field == "z" then buffer.writef32(b, 8, value) + elseif field == "w" then buffer.writef32(b, 12, value) + else error("unknown field") end + end +} + +local buf = buffer.create(16, mt) + +buf.x = 2 +buf.y = 4 + +assert(buf.x + buf.y + buf.z == 6) +``` + +Or alternatively: +```luau +local mt = { + __index = { + x = function(b) return buffer.readf32(b, 0) end, + y = function(b) return buffer.readf32(b, 4) end, + z = function(b) return buffer.readf32(b, 8) end, + w = function(b) return buffer.readf32(b, 12) end, + setx = function(b, value) buffer.writef32(b, 0, value) end, + sety = function(b, value) buffer.writef32(b, 4, value) end, + setz = function(b, value) buffer.writef32(b, 8, value) end, + setw = function(b, value) buffer.writef32(b, 12, value) end, + magnitude = function(b) return math.sqrt(b.x * b.x + b.y * b.y + b.z * b.z + b.w * b.w) end, + normalize = function(b) return ... end, + }, +} + +local buf = buffer.create(16, mt) + +buf:setx(2) +buf:sety(4) +buf:normalize() + +local xn = buf:x() +``` + +Or any other custom way the developer wants property access to be performed. + +## Design + +`buffer.create(size: number, metatable: table): buffer` + +Buffer construction will accept a second argument which will be a metatable for the buffer object. + +This metatable will be frozen. +In addition to that, `__index` and `__newindex` fields will also be frozen if they are a table. +Table freezing will have the same limitation as `table.freeze`, throwing an error if table metatable is locked. +Any of these tables can be frozen before the call. + +When `__index` or `__newindex` is a function, VM will be allowed to ignore changes to the environment of those function. + +The freezing is performed to provide additional guarantees for field access and other metamethod evaluation to unlock potential optimization opportunities. +This is similar to limitations placed on ctypes by luajit. +Having said that, this RFC doesn't make a promise of a particular implementation making those optimizations, so should be viewed as a buffer object usability improvement first. + +VM metatable lookups and `getmetatable` will look for the buffer object metatable first and then fall back to the global buffer metatable. +This preserves the existing buffer extensibility point for hosts. + +`setmetatable` will not be supported on buffer objects, the metatable reference is immutable. + +Equality checks in the VM will call `__eq` for buffers similar to tables and userdata. + +`__type` metatable key is ignored by `typeof`. As before, only host is allowed to define type names. + +In order for the typechecker to understand buffers with attached metatables, we propose extending the intersections to be allowed on buffers, similar to `extern` types: + +`type Obj = buffer & { x: number, y: number, z: number }` + +## Drawbacks + +This increases buffer size by 8 bytes, with the bigger impact on 0-8 byte buffers going from 16 to 24 bytes and allocated from 32 byte page. + +This RFC also introduces a special evaluation rule for metamethod functions. +While it is introduced for potential improvement in caching of operations, this might come at a surprise to users of deprecated environment modification functions. + +## Alternatives + +Instead of extending the buffer object with a limited set of functionality, we might pursue a new kind of object like Records were which can build internal field mappings for an easier optimization path. + +Another possibility is some alternative way of specifying the fields that would support building both the right __index function and internal acceleration structures. From 27a1865f0fe1a84bc77d19fed0abce97cbecb030 Mon Sep 17 00:00:00 2001 From: Vyacheslav Egorov Date: Fri, 5 Dec 2025 16:53:12 +0200 Subject: [PATCH 2/6] Address the first batch of comments on the RFC --- docs/buffer-metatable.md | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-) diff --git a/docs/buffer-metatable.md b/docs/buffer-metatable.md index 4efb8d7ed..bb6b674ea 100644 --- a/docs/buffer-metatable.md +++ b/docs/buffer-metatable.md @@ -7,13 +7,15 @@ The metatable and some of its keys will be frozen for potential improvements in ## Motivation -Providing a buffer object with a metatable will allow creation of objects with small storage footprint which can also be matched to native structures on the side of the application using Luau. +Providing a buffer object with a metatable will allow buffer objects can have properties (direct or computed), methods and other kinds of operations like operators. -By having a metatable, these buffer objects can have properties, methods and other kinds of operations. +This will allow the behavior to be defined and extended on the Luau side, compared to userdata that is strictly defined by the host. -This will also allow the behavior to be defined and extended on the Luau side, compared to userdata that is strictly defined by the host. +This extension preserves the simple small core of the Luau language while providing a flexible extension point for developers. -In a way, this will provide an alternative to `ctypes` in luajit. +Such buffers can also be made to match the structure of the host data types and to handle FFI cases. + +In a way, this will provide an alternative to luajit 'cdata' in Luau. ```luau -- A float4 in a buffer @@ -68,7 +70,7 @@ buf:normalize() local xn = buf:x() ``` -Or any other custom way the developer wants property access to be performed. +Or any other custom way the developer wants property accesses to be performed. ## Design @@ -82,19 +84,25 @@ Table freezing will have the same limitation as `table.freeze`, throwing an erro Any of these tables can be frozen before the call. When `__index` or `__newindex` is a function, VM will be allowed to ignore changes to the environment of those function. +This is similar to function inlining functionality we have where an inlined function will no longer respect its environment. The freezing is performed to provide additional guarantees for field access and other metamethod evaluation to unlock potential optimization opportunities. -This is similar to limitations placed on ctypes by luajit. +This is similar to limitations placed on 'cdata' by luajit. Having said that, this RFC doesn't make a promise of a particular implementation making those optimizations, so should be viewed as a buffer object usability improvement first. +Any chained metatables on table fields are not modified. +This provides an option to have a more dynamic behavior, but giving up on potential performance improvements of a chained indexing access. +This matches behavior of 'cdata' in luajit and will provide a familiarity for developers switching over. + VM metatable lookups and `getmetatable` will look for the buffer object metatable first and then fall back to the global buffer metatable. This preserves the existing buffer extensibility point for hosts. -`setmetatable` will not be supported on buffer objects, the metatable reference is immutable. +`setmetatable` will still not be supported on buffer objects, the metatable reference is immutable. Equality checks in the VM will call `__eq` for buffers similar to tables and userdata. -`__type` metatable key is ignored by `typeof`. As before, only host is allowed to define type names. +`__type` metatable key is ignored by `typeof`, just like it does for tables. +As before, only host is allowed to define type names. In order for the typechecker to understand buffers with attached metatables, we propose extending the intersections to be allowed on buffers, similar to `extern` types: @@ -105,10 +113,11 @@ In order for the typechecker to understand buffers with attached metatables, we This increases buffer size by 8 bytes, with the bigger impact on 0-8 byte buffers going from 16 to 24 bytes and allocated from 32 byte page. This RFC also introduces a special evaluation rule for metamethod functions. -While it is introduced for potential improvement in caching of operations, this might come at a surprise to users of deprecated environment modification functions. +It is introduced for potential improvement in caching of operations, but might come at a surprise to users of deprecated environment modification functions. +While similar to the effects of function inlining Luau performs, it is an extra item to keep in mind. ## Alternatives Instead of extending the buffer object with a limited set of functionality, we might pursue a new kind of object like Records were which can build internal field mappings for an easier optimization path. -Another possibility is some alternative way of specifying the fields that would support building both the right __index function and internal acceleration structures. +Another possibility is some alternative way of specifying the fields that would support building both the right `__index` function and internal acceleration structures. From 375a0a129bad38decd507b0c398528b9aba2cc4c Mon Sep 17 00:00:00 2001 From: Vyacheslav Egorov Date: Fri, 5 Dec 2025 16:57:59 +0200 Subject: [PATCH 3/6] Provide notes on this feature use in potential separate data section future --- docs/buffer-metatable.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/buffer-metatable.md b/docs/buffer-metatable.md index bb6b674ea..1304c5387 100644 --- a/docs/buffer-metatable.md +++ b/docs/buffer-metatable.md @@ -108,6 +108,17 @@ In order for the typechecker to understand buffers with attached metatables, we `type Obj = buffer & { x: number, y: number, z: number }` +## Forwards compatibility + +A potential evolution of the buffer type might move the data from the buffer memory block to a separate location with a redirection. + +Doing that will enable extending the buffer size without a major impact on performance (shrinking is a problem for range check eliminations). +It might also allow buffer views and slices. + +In both cases, this change is compatible. The slice will reuse a section of the data but being a buffer it will have its own metatable. + +This makes it possible to have a large memory buffer and sub-allocate buffer objects with attached custom behaviors. + ## Drawbacks This increases buffer size by 8 bytes, with the bigger impact on 0-8 byte buffers going from 16 to 24 bytes and allocated from 32 byte page. From 8fe1ebdba75e0dec47f240b9e056d500b834a846 Mon Sep 17 00:00:00 2001 From: Vyacheslav Egorov Date: Fri, 12 Dec 2025 14:13:44 +0200 Subject: [PATCH 4/6] Extend motivation to note how buffers already have a metatable, note that other metamethods work, the alignment advantage and more details on size --- docs/buffer-metatable.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/docs/buffer-metatable.md b/docs/buffer-metatable.md index 1304c5387..e43727a37 100644 --- a/docs/buffer-metatable.md +++ b/docs/buffer-metatable.md @@ -7,15 +7,20 @@ The metatable and some of its keys will be frozen for potential improvements in ## Motivation -Providing a buffer object with a metatable will allow buffer objects can have properties (direct or computed), methods and other kinds of operations like operators. +Buffer type today has a single metatable provided by the embedder and shared by all buffer instances. -This will allow the behavior to be defined and extended on the Luau side, compared to userdata that is strictly defined by the host. +This limits the flexibility as all buffers share the same behavior and those behaviors cannot be defined by user in Luau. +Even for the embedder, defining the methods in Luau is challenging and host language definitions are used instead. + +Providing a buffer object with an individual metatable will allow buffer objects to have properties (direct or computed), methods and operators, specific to certain instances. + +This will allow the behavior to be defined and extended on the Luau side, compared to userdata that is strictly defined by the embedder. This extension preserves the simple small core of the Luau language while providing a flexible extension point for developers. Such buffers can also be made to match the structure of the host data types and to handle FFI cases. -In a way, this will provide an alternative to luajit 'cdata' in Luau. +In a way, this will provide an alternative to luajit 'cdata' in Luau: ```luau -- A float4 in a buffer @@ -70,7 +75,7 @@ buf:normalize() local xn = buf:x() ``` -Or any other custom way the developer wants property accesses to be performed. +Having all the flexibility of a metatable, developer is free to define whatever properties and behaviors they want and interpret their buffer data in a form suitable for them. ## Design @@ -103,6 +108,12 @@ Equality checks in the VM will call `__eq` for buffers similar to tables and use `__type` metatable key is ignored by `typeof`, just like it does for tables. As before, only host is allowed to define type names. +This does not change how it behaved with a global host-provided metatable. + +Other metamethods (`__add`/`__len`/...) work as expected and do not change from how they work today for buffers with global host-provided metatable. + +Buffer structure size increase by the metatable aligns the data to a 16 byte boundary making it possible to store data which requires 16 byte alignment by the embedder. +This will make buffer data match the alignment guarantee of Luau userdata objects. In order for the typechecker to understand buffers with attached metatables, we propose extending the intersections to be allowed on buffers, similar to `extern` types: @@ -121,7 +132,9 @@ This makes it possible to have a large memory buffer and sub-allocate buffer obj ## Drawbacks -This increases buffer size by 8 bytes, with the bigger impact on 0-8 byte buffers going from 16 to 24 bytes and allocated from 32 byte page. +This increases buffer size by 8 bytes, with the bigger impact on 0-8 byte buffers going from 16 to 24 bytes, with an overhead decreasing with larger sizes as before. +Starting from 64 bytes the increase is often free based on the current Luau allocator design. +In particular, sizes like 64/96/128/256 do not increase in internal allocation class. This RFC also introduces a special evaluation rule for metamethod functions. It is introduced for potential improvement in caching of operations, but might come at a surprise to users of deprecated environment modification functions. From 81580ddb6adfdc265b0ac0b343d4c827b1dfe79c Mon Sep 17 00:00:00 2001 From: vegorov-rbx <75688451+vegorov-rbx@users.noreply.github.com> Date: Tue, 16 Dec 2025 17:48:33 +0200 Subject: [PATCH 5/6] Update docs/buffer-metatable.md Co-authored-by: Alexander McCord <11488393+alexmccord@users.noreply.github.com> --- docs/buffer-metatable.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/buffer-metatable.md b/docs/buffer-metatable.md index e43727a37..c12ecbfaf 100644 --- a/docs/buffer-metatable.md +++ b/docs/buffer-metatable.md @@ -79,7 +79,7 @@ Having all the flexibility of a metatable, developer is free to define whatever ## Design -`buffer.create(size: number, metatable: table): buffer` +`buffer.create(size: number, metatable: table?): buffer` Buffer construction will accept a second argument which will be a metatable for the buffer object. From 42a5765cbbc9eadd7bb300648c4f7d820e8f7646 Mon Sep 17 00:00:00 2001 From: vegorov-rbx <75688451+vegorov-rbx@users.noreply.github.com> Date: Tue, 16 Dec 2025 23:20:46 +0200 Subject: [PATCH 6/6] Use setmetatable as the proposed typing solution --- docs/buffer-metatable.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/buffer-metatable.md b/docs/buffer-metatable.md index c12ecbfaf..28472da25 100644 --- a/docs/buffer-metatable.md +++ b/docs/buffer-metatable.md @@ -115,9 +115,11 @@ Other metamethods (`__add`/`__len`/...) work as expected and do not change from Buffer structure size increase by the metatable aligns the data to a 16 byte boundary making it possible to store data which requires 16 byte alignment by the embedder. This will make buffer data match the alignment guarantee of Luau userdata objects. -In order for the typechecker to understand buffers with attached metatables, we propose extending the intersections to be allowed on buffers, similar to `extern` types: +In order for the typechecker to understand buffers with attached metatables, we propose using the `setmetatable` type function: -`type Obj = buffer & { x: number, y: number, z: number }` +`type Obj = setmetatable` + +If there is already a metatable on a buffer type, `setmetatable` will fail. ## Forwards compatibility