[SPEC | CORE] : Allow table level override for scan planning

[singhpk234]

About the change

Scan Planning Modes

Single enum ScanPlanningMode with 2 values:

• client - MUST use client-side planning
• server - MUST use server-side planning

ML : https://lists.apache.org/thread/z1g4y8b4ogdrn0jjtjlgg7yjgxdbzpvg

[nastra]

while the changes make sense to me, we may actually want to discuss this in the broader community to decide whether we want to override server-side scan planning at the table level

[singhpk234]

ML thread : https://lists.apache.org/thread/z1g4y8b4ogdrn0jjtjlgg7yjgxdbzpvg

[geruh]

Thanks for raising this @singhpk234! I like the direction here. As a user today, there are two modes either use scan planning or not. Which begs the question, when should I use one versus the other? And right now, there is no clear insight or story from the user's perspective.

Now from a catalog's perspective, the modes make sense. For instance, If the catalog is using planning to enforce governance, the required mode would signify intent. But I do have a question on the semantics. In the current implementation IIUC, optional seems to be similar to required. Rendering the combination of mode=None and Catalog support of planning, as optional. So I'd say if the intent is truly to express optionality to the client, should there be a client side override that wins. Or is the expectation that the catalog avoids sending the mode.

[singhpk234]

Thanks for the feedback @geruh !

optional seems to be similar to required.

Optional in this context is that the catalog really doesn't have an opinion on what the client decides, it can choose local and remote, the way i was thinking is lets say you are running a lot of concurrent queries in your spark cluster and your driver is slim, even though, we are using spark, we may prefer spark. That being said yes in this impl what i did if the catalog supports plan endpoint and the catalog doesn't have any opinion on this, in java impl we always do scan planning, yes being able to toggle this based on client side config would be ideal may be when the server sends optional from the server side, and from the client side we have configured required we should not allow overwritting the key to optionals and reuse the optional ? WDYT

I see @RussellSpitzer has similar feedback in ML thread too, let me take a deeper look on their feedback and respond there as well

[singhpk234]

Decision matrix : scan planning mode (required | optional | none) :

Client Config	Server Config	Supports REST?	Planning Outcome
Required	Required	Yes	Server Planning (Server overrides and fulfills requirement)
Required	Optional	Yes	Server Planning (Client mandates, Server is flexible)
Required	None	Yes	Server Planning (Client mandate takes priority for Server)
Optional	Required	Yes	Server Planning (Server Required overrides Client Optional)
Optional	Optional	Yes	Server Planning (Defaulted for REST efficiency)
Optional	None	Yes	Local Planning (Server cannot, Client is flexible)
None	Required	Yes	Server Planning (Server Required overrides Client None)
None	Optional	Yes	Local Planning (Client mandate takes priority)
None	None	Yes	Client Planning (Both agree on Client side)
Not configured	Required	Yes	Server Planning (Server Required overrides Client None)
Not configured	Optional	Yes	Local Planning
Not configured	None	Yes	Local Planning

---

Decision matrix : scan planning mode(client only | client preferred | catalog preferred | catalog only)

Client Config	Server Config	Supports REST?	Planning Outcome
Client Only	Client Only	Yes	Local Planning: Both agree on client-side implementation.
Client Only	Client Preferred	Yes	Local Planning: Both Server and Client agree.
Client Only	Catalog Preferred	Yes	Local Planning: Client ignores server optimization capabilities and chooses client only impl.
Client Only	Catalog Only	Yes	FAIL
Client Preferred	Client Only	Yes	Local Planning: Client preference aligns with server's strict client-only requirement.
Client Preferred	Client Preferred	Yes	Local Planning: Both parties prefer local execution; negotiation defaults to Client.
Client Preferred	Catalog Preferred	Yes	Local Planning: Both are flexible; override client side.
Client Preferred	Catalog Only	Yes	Server Planning: Server MUST plan; client yields its preference.
Catalog Preferred	Client Only	Yes	Local Planning: Server MUST plan locally; client yields its preference.
Catalog Preferred	Client Preferred	Yes	Server Planning: Negotiation favors client-side to minimize server-side overhead.
Catalog Preferred	Catalog Preferred	Yes	Server Planning: Both parties prefer server-side; Catalog is chosen.
Catalog Preferred	Catalog Only	Yes	Server Planning: Server MUST plan; client preference aligns.
Catalog Only	Client Only	Yes	FAIL:
Catalog Only	Client Preferred	Yes	Server Planning: Server is flexible; Catalog MUST plan.
Catalog Only	Catalog Preferred	Yes	Server Planning: Client MUST use Catalog; server agrees with preference.
Catalog Only	Catalog Only	Yes	Server Planning: Strict agreement on server-side implementation.
Not Configured	Client Only	Yes	Local Planning: Client MUST adhere to Catalog;
Not Configured	Client Preferred	Yes	Local Planning:
Not Configured	Catalog Preferred	Yes	Server Planning:
Not Configured	Catalog Only	Yes	Server Planning: Strict agreement on server-side implementation.

[RussellSpitzer]

I wasn't thinking about it quite that way. I was assuming the client is configured independently of the catalog. The client can either have a user preference or none. If none, it does whatever the catalog feeds back to it if the client supports that mode. Otherwise it does what is manually specified.

So

Client (None) -> Follow Catalog Config (Client Only, Client Preferred, CatalogPreferred or Catalog Only), Fail if the client doesn't support config
Client (ClientPlanning) -> Fail if Catalog says Catalog Only, Client planning for all other cases
Client (CatalogPlanning) -> Fail if Catalog says Client Only, Catalog planning for all other cases

A user without a preference or who wants the Catalog to make the determination just leaves this unset on the client. Or a Client which wants to override the catalog can set a specific mode and fail fast if the Catalog doesn't support it.

[pvary]

I have a concern about some catalogs starting to make every table CATALOG_ONLY, which would essentially lock users to the catalog without providing a way to migrate the data to another catalog.
Maybe we add a sentence in the spec to enforce, that there should be some users where the catalog MUST provide access to the metadata files.

WDYT?

[singhpk234]

the catalog without providing a way to migrate the data to another catalog

we would still have a way to migrate, mostly because in the loadTable we give back the metadata.json pointer (which is self describing the table state), and its the catalog ADMIN would be able to use that pointer and register table to another REST or Metastore backed catalog. In the model where storage is decoupled from compute its the administrator of the catalog who has given access the catalog to vend storage creds and it can very well take it back.

This feature is mostly like i want to read the table, can you help me with the data | delete files that corresponds to the table. Nevertheless i believe CATALOG_ONLY we think to be used primarily for gov cases also for things like scanning huge tables where planning can cause a lot of pressure on JVM (trino coordinar unstablity | spark requiring distributed planning) where catalog can do some efficient indexing (stuff like Redis) etc to help these engine.

All in all IMHO i believe vendor lock in and not being able to migrate would not be possible by exposing this option, please let me know if i am missing something.

[pvary]

The specification states:

The corresponding file location of table metadata should be returned in the `metadata-location` field

However, it does not specify that this location must be readable by any user. (Perhaps this is something we should clarify going forward.)

Before the introduction of CATALOG_ONLY tables, users were required to have direct read access to the metadata files in order to plan queries on the table. That implied an access requirement, even though it was never explicitly documented. With the introduction of CATALOG_ONLY, this implicit requirement no longer applies, and we currently do not have an explicit requirement defined in the specification either.

[amogh-jahagirdar]

We have a choice here. We can either hard fail on conflicting configs like is done below or we can just let one override the other.

I think the least surprising thing from a client perspective is to just override using the client config in case of conflicts. On average I don't expect people to be fiddling with this config. In the chance they do, they're probably intentionally trying to set it and we should attempt to respect that. Even if there's some FGAC and creds and all aren't returned and it fails later on, that's better to me.

I guess I could go either way here (as long as we log there's a conflict and we're overriding using some mechanism), but I think any approach seems better than just hard failing since that seems unneccessary.

[singhpk234]

I think the least surprising thing from a client perspective is to just override using the client config in case of conflicts

It can cause an issue in a sense that let say i am low resource client i can't run scan planning at all, so i say use catalog and the catalog says do client ? i believe if we let catalog judgement win it may lead to failure ?
But i think other way to look at it we any way override the client config with the server one if server says for some configs we can do the same then here ?
I am also open to both !