[next gen] TiFlash crash for unsupported admin command type TransferLeader

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

[2026/03/06 06:02:16.864 +08:00] [FATAL] [Exception.cpp:98] ["Code: 49, e.displayText() = DB::Exception: unsupported admin command type TransferLeader, e.what() = DB::Exception, Stack trace:\n\n\n  0x55c6840a1f33\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) [tiflash+37682995]\n                \tdbms/src/Common/StackTrace.cpp:25\n  0x55c68bc1638f\tDB::RegionRaftCommandDelegate::handleAdminRaftCmd(raft_cmdpb::AdminRequest const&, raft_cmdpb::AdminResponse const&, unsigned long, unsigned long, DB::KVStore const&, DB::RegionTable&, DB::RaftCommandResult&) [tiflash+167134095]\n                \tdbms/src/Storages/KVStore/MultiRaft/RaftCommands.cpp:280\n  0x55c68bc1edbb\tDB::KVStore::handleAdminRaftCmd(raft_cmdpb::AdminRequest&&, raft_cmdpb::AdminResponse&&, unsigned long, unsigned long, unsigned long, DB::TMTContext&) [tiflash+167169467]\n                \tdbms/src/Storages/KVStore/MultiRaft/RaftCommandsKVS.cpp:221\n  0x55c68bba8a69\tHandleAdminRaftCmd [tiflash+166685289]\n                \tdbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:123\n  0x7fc5efd43791\tproxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::handle_admin_raft_cmd::h9c07ce60e70cf0c9 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+26273681]\n  0x7fc5efe5e8ae\t_$LT$engine_store_ffi..observer..TiFlashObserver$LT$T$C$ER$GT$$u20$as$u20$raftstore..coprocessor..AdminObserver$GT$::post_exec_admin::h3efae9ab297f22e1 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+27433134]\n  0x7fc5f0c1c740\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::hccf4673cefb05378 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+41842496]\n  0x7fc5f0c24123\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::h99122f5e14aa60c4 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+41873699]\n  0x7fc5f0bfd139\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::h5ebdbdc48ab0c9d5 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+41713977]\n  0x7fc5f0c01993\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::h297e23ede1b5b1f0 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+41732499]\n  0x7fc5eff6cbe5\t_$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::h94499c390d84dbe7 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28539877]\n  0x7fc5efedfa25\tbatch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::h3cfc6addfa4de40c [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+27961893]\n  0x7fc5effa4945\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h20f7ac87f05251c9 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28768581]\n  0x7fc5efff6e89\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h28a824920b627230 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+29105801]\n  0x7fc5f1560246\tstd::sys::unix::thread::Thread::new::thread_start::hd01330a228cdd35d [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+51556934]\n                \t/rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys/unix/[thread.rs:108](http://thread.rs:108/)\n  0x7fc5ee0f02ea\tstart_thread [libc.so.6+570090]\n  0x7fc5ee1753c0\t__GI___clone3 [libc.so.6+1115072]"] [source="EngineStoreApplyRes DB::HandleAdminRaftCmd(const EngineStoreServerWrap *, BaseBuffView, BaseBuffView, RaftCmdHeader)"] [thread_id=67]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

Welcome to TiFlash"] [thread_id=1]
TiFlash build info: TiFlash\nRelease Version: v9.0.0-beta.2.pre-126-g661e9b588\nEdition: Enterprise\nGit Commit Hash: 661e9b5\nG
p"] [source=Application]

> select type,version,git_hash from information_schema.cluster_info group by type,version,git_hash;
+-----------------+---------------------------------+------------------------------------------+
| type            | version                         | git_hash                                 |
+-----------------+---------------------------------+------------------------------------------+
| tidb            | 9.0.0-beta.2.pre-1302-g4d4043c  | 4d4043c799c996eee06f8a2e8ee99d67937add12 |
| tso             | 9.0.0-beta.2.pre-308-g79149a2   | 79149a25abce3a8d41fa3e59a600afab58b3ebb4 |
| tikv            | 8.5.4+branch-HEAD               | cc9576a432ae60f23bed70dcb3bbca0aaa0f9671 |
| pd              | 9.0.0-beta.2.pre-308-g79149a2   | 79149a25abce3a8d41fa3e59a600afab58b3ebb4 |
| tiflash_write   | 9.0.0-beta.2.pre-126-g661e9b588 | 661e9b5887144636797a3e6e583d5a383ebfd17e |
| tiflash_compute | 9.0.0-beta.2.pre-126-g661e9b588 | 661e9b5887144636797a3e6e583d5a383ebfd17e |
| scheduling      | 9.0.0-beta.2.pre-308-g79149a2   | 79149a25abce3a8d41fa3e59a600afab58b3ebb4 |
+-----------------+---------------------------------+------------------------------------------+

[JaySon-Huang]

TiFlash Crash Investigation: TransferLeader Admin Command

Scope

This note investigates a TiFlash fatal error during leader transfer, and compares how
contrib/tiflash-proxy (classic) vs contrib/tiflash-proxy-next-gen route
TransferLeader through HandleAdminRaftCmd.

Executive Summary

• The crash is reproducible when TiFlash receives AdminCmdType::TransferLeader in C++.
• In TiFlash C++ (RegionRaftCommandDelegate::handleAdminRaftCmd), TransferLeader is not
  handled in the switch and falls into default, throwing:
  unsupported admin command type TransferLeader.
• pre_exec_admin == true means "skip raftstore built-in exec", not "skip post hooks".
• Classic and next-gen differ in post_exec routing:
  • classic routes by cmd.response.has_admin_response()
  • next-gen routes by cmd.request.has_admin_request()
• Because of that, next-gen can still enter post_exec_admin after pre-exec filtering,
  then call handle_admin_raft_cmd, which triggers the TiFlash C++ unsupported-type exception.

What Happened (Observed Logs)

TiFlash

• Fatal at 2026-03-06 00:01:14 +08:00
• Error: unsupported admin command type TransferLeader
• Stack includes:
  • DB::RegionRaftCommandDelegate::handleAdminRaftCmd
  • DB::KVStore::handleAdminRaftCmd
  • HandleAdminRaftCmd (FFI)

TiFlash Proxy

• transfer leader won't exec
• followed by observe admin command for the same TransferLeader request

This pair is expected under next-gen routing behavior (details below).

Behavior Comparison: Classic vs Next-Gen

1) pre_exec_admin behavior for TransferLeader

Both classic and next-gen do the same in engine-store observer forwarding logic:

• log transfer leader won't exec
• return true to filter local raftstore execution

2) Why post_exec still runs after pre_exec_admin=true

In raftstore apply_raft_cmd, even when pre_exec(...) returns true:

• it still constructs Cmd
• then still calls post_exec(...)

So pre-exec filtering and post-exec callback are not mutually exclusive.

3) Real routing difference (critical)

• classic (contrib/tiflash-proxy/.../coprocessor/dispatcher.rs)
  • post_exec dispatch uses cmd.response.has_admin_response()
  • filtered admin commands often carry empty response => may route to post_exec_query
• next-gen (contrib/tiflash-proxy-next-gen/.../coprocessor/dispatcher.rs)
  • post_exec dispatch uses cmd.request.has_admin_request()
  • filtered admin command is still an admin request => routes to post_exec_admin

This explains why the logs can show both:

• transfer leader won't exec
• observe admin command

Root Cause

Direct root cause is in TiFlash C++ admin handling:

• dbms/src/Storages/KVStore/MultiRaft/RaftCommands.cpp
• RegionRaftCommandDelegate::handleAdminRaftCmd
• switch does not include AdminCmdType::TransferLeader
• falls to default and throws unsupported admin command type

Is Classic Routing Self-Consistent?

Yes, but with weaker semantics:

• It is operationally self-consistent because classic post_exec_query can tolerate the
  filtered-admin/empty-response path.
• It is less type-explicit than next-gen because routing is based on response shape,
  not original request category.

Why Next-Gen Uses request.has_admin_request

This is semantically stricter and also safer for next-gen internals:

• next-gen post_exec_query expects custom-log query form and does
  rlog::get_custom_log(&cmd.request).unwrap()
• misrouting filtered admin into query path could panic
• request-based dispatch avoids that class of mismatch

Could Other Command Types Trigger Similar TiFlash Exception?

High risk (already observed)

• TransferLeader

Confirmed unsupported in TiFlash C++ (usually abnormal/test input)

• InvalidAdmin (covered by TiFlash unit test expecting unsupported-type exception)

Conditional/low risk under current paths

• UpdateGcPeer
  • In raftstore v1 apply path it is generally error/skip semantics.
  • In next-gen, error response usually goes through handle_error_apply instead of
    handle_admin_raft_cmd.
  • Could become a similar risk only if future path turns into successful admin forwarding.

Not in the same risk class (already handled as "useless admin" in TiFlash)

• CompactLog
• ComputeHash
• VerifyHash
• PrepareFlashback
• FinishFlashback
• BatchSwitchWitness

Key File References

• contrib/tiflash-proxy/proxy_components/engine_store_ffi/src/core/forward_raft/command.rs
• contrib/tiflash-proxy-next-gen/proxy_components/engine_store_ffi/src/core/forward_raft/command.rs
• contrib/tiflash-proxy/components/raftstore/src/coprocessor/dispatcher.rs
• contrib/tiflash-proxy-next-gen/components/raftstore/src/coprocessor/dispatcher.rs
• contrib/tiflash-proxy/components/raftstore/src/store/fsm/apply.rs
• contrib/tiflash-proxy-next-gen/components/raftstore/src/store/fsm/apply.rs
• dbms/src/Storages/KVStore/MultiRaft/RaftCommands.cpp
• dbms/src/Storages/KVStore/MultiRaft/RaftCommandsKVS.cpp
• dbms/src/Storages/KVStore/tests/gtest_new_kvstore.cpp

Evidence Snapshot (Log Excerpts)

TiFlash

[2026/03/06 00:01:14.996 +08:00] [FATAL] ... unsupported admin command type TransferLeader

TiFlash Proxy

[2026/03/06 00:01:10.973 +08:00] [ERROR] ... transfer leader won't exec
[2026/03/06 00:01:10.973 +08:00] [INFO]  ... observe admin command

PD

[2026/03/06 00:01:10.970 +08:00] send schedule command ... transfer leader from store 1148 to store 1213
[2026/03/06 00:01:10.978 +08:00] leader changed ... from=1148 to=1213

[JaySon-Huang]

Related to the feature "In-memory pessimistic lock" in next-gen https://github.com/tidbcloud/cloud-storage-engine/issues/4052

tikv toml in the testing plan

[pessimistic-txn]
in-memory = true