Enhance indexer transaction data analysis and storage

[will-bitlightlabs]

Blockchain Indexer Optimization: Enhancing Transaction Data Analysis and Storage

Overview

This PR implements important optimizations for the BP Node indexer, focusing on improving transaction data analysis and blockchain storage mechanisms. Following Dr. Orlovsky's feedback, we've adopted a single-chain design, integrated with the BloomFilter32 transaction tracking, and enhanced orphan block handling and chain reorganization functionalities.

Main Improvements

Single-Chain Design Implementation

• Removed automatic network detection logic in favor of writing user-configured network type during initialization
• Added network validation check at startup to ensure the configured network matches the database
• Simplified design: each BP-Node instance now processes only a single blockchain network
• Removed main usage of TABLE_CHAIN, now relying solely on block height tables to track chain state

Transaction Tracking Mechanism

• Properly integrated with the BloomFilter32 type for efficient transaction matching

Block Height and Block ID Handling

• Optimized calculate_block_height method to handle different block types:
  • Genesis block: Detects all-zero prev_hash and returns height 0
  • Normal blocks: Looks up parent block height+1 and checks for conflicts
  • Fork blocks: Identifies potential fork blocks when parent has blockid but no height
  • Orphan blocks: Recognizes and handles specially when parent block is not found in database
• Maintained block height and internal block ID as separate concepts for improved code clarity and flexibility
• Enhanced potential fork detection logic to distinguish different types of fork scenarios

Database Structure Optimization

We've optimized the database table structure, categorizing tables by functionality:

• Base Metadata Tables: Store global counters, network type, and other core information
• Block and Transaction Core Tables: Store block headers, transaction data, and their mapping relationships
• UTXO and Transaction Relationship Tables: Track UTXO creation and spending, record relationships between transactions
• Height Mapping Tables: Maintain bidirectional mapping between block heights and block IDs
• Orphan Tables: Store orphan blocks and their parent block dependency relationships
• Fork Management Tables: Record and manage fork chain information and block data

Orphan Block Handling Mechanism

• Implemented delayed orphan removal strategy, ensuring orphans are only removed from the pool after successful processing
• Optimized orphan dependency lookup and processing logic
• Enhanced error handling and logging during orphan block processing

Chain Reorganization Handling

• Implemented process_potential_fork method for handling blocks that may cause forks
• Added fork detection, recording, and management functionalities
• Implemented chain reorganization execution logic, including complete support for block rollback and application
• Enhanced UTXO set management to ensure consistency during reorganization

Error Handling and User Experience

• Added more specific error types to represent different exceptional scenarios
• Enhanced logging during initialization and startup processes
• Updated code comments to clearly explain design principles

Testing Progress

• Created TestContext structure providing a complete isolated testing environment
• Implemented basic block processing validation, orphan block handling tests, and blockchain reorganization tests
• Verified correct processing of orphan blocks after parent blocks arrive
• Tested fork detection, fork chain growth, and main chain switching functionality
• Verified transaction and UTXO state maintenance during reorganization

Special note: To avoid PR code bloat, this submission does not include the testing framework code. We will flexibly adjust the testing framework based on review suggestions and submit the complete testing framework through a separate PR once the code stabilizes.

[dr-orlovsky]

BTW, each BP Node instance will work with a single chain; and if multiple chains are needed each one will take an independent instance with data being stored in different directories. Thus, we can keep Network in the MAIN table and check on launch - and skip a table mapping chains to blocks (heights should be enough)

[dr-orlovsky]

Did an initial review of everything except main logic in blocks.

LGTM; just one question

[dr-orlovsky]

I also have pushed few improvements and fixes on top

[will-bitlightlabs]

I also have pushed few improvements and fixes on top

Thank you very much. I also have some small optimization ideas to apply simultaneously, which won't affect the overall main trunk

[will-bitlightlabs]

I also have pushed few improvements and fixes on top

Thank you very much. I also have some small optimization ideas to apply simultaneously, which won't affect the overall main trunk

DONE Merge transaction processing logic

[dr-orlovsky]

ACK ec2e560

[dr-orlovsky]

Merged. Thank you very much for the excellent and huge work!

[will-bitlightlabs]

Merged. Thank you very much for the excellent and huge work!

It's a great honor. I look forward to having more opportunities to contribute in the future :)