TON indexer: intermediate shard blocks are silently skipped — transactions can be missed

[anhthii]

Summary

The TON indexer scans all shards returned by GetBlockShardsInfo for each masterchain block, but it only scans the latest shard block per shard, missing any intermediate shard blocks produced between consecutive masterchain blocks. This means transactions landing in skipped shard blocks are silently dropped.

For a deposit/payment detection system, this is a critical gap — monitored address transactions can be permanently missed.

---

Root Cause

1. No intermediate shard block traversal

File: internal/indexer/ton.go:143-163

case master.SeqNo > t.lastMasterSeqno:
    // Fast path for wallet-detection throughput: scan only current shard blocks.
    // This skips parent-chain deep backfill and avoids long spikes when lite-servers are slow.
    shardsToScan = append(shardsToScan, currentShards...)
    for _, shard := range currentShards {
        t.shardLastSeqno[getShardID(shard)] = shard.SeqNo
    }
    t.lastMasterSeqno = master.SeqNo

Between masterchain block N and N+1, a shard can produce multiple blocks (e.g., seqno 100→101→102). GetBlockShardsInfo at masterchain N+1 only returns seqno 102. Blocks 100 and 101 are never scanned.

The shardLastSeqno map is maintained (lines 148, 157) but never read to drive backfill — it's dead state.

2. No shard split/merge handling

File: internal/indexer/ton.go:714-716

func getShardID(shard *tonlib.BlockIDExt) string {
    return fmt.Sprintf("%d|%d", shard.Workchain, shard.Shard)
}

When a shard splits (e.g., 0x8000000000000000 → 0x4000... + 0xC000...), the old shard ID becomes stale. Since shardLastSeqno isn't used for backfill anyway, this is currently moot but would need to be addressed alongside a fix.

3. TonAPI interface lacks shard block lookup

File: internal/rpc/ton/api.go:12-39

The TonAPI interface has no method to look up a shard block by workchain+shard+seqno (only masterchain lookup via LookupMasterchainBlock). A fix will require adding a LookupBlock(ctx, workchain, shard, seqno) method to traverse the shard chain.

---

Impact

• Missed deposits: Any transaction in an intermediate shard block is silently dropped. The indexer reports success and moves on.
• Worsens under load: Higher basechain throughput = more shard blocks per masterchain block = higher miss rate. This fails exactly when reliability matters most.
• No observability: There is no metric or log indicating blocks were skipped.

---

Expected Fix

For each shard in currentShards, walk backwards from the current shard seqno to the last-seen seqno (stored in shardLastSeqno) and scan all intermediate blocks. Handle shard splits by detecting new shard IDs and walking their parent chains. Add a LookupBlock method to TonAPI to support this traversal.

This is the standard approach used by TON indexers (e.g., tonkeeper, ton-indexer).