Skip to content

Speed up metabase resync process #3453

@roman-khimov

Description

@roman-khimov

Is your feature request related to a problem? Please describe.

I'm always frustrated when metabase resync takes too much time. Currently it iterates over all objects and does metabase.Put() for all objects as well as has some special handling for locks and tombstones. Lock and TS handling will be simplified with nspcc-dev/neofs-api#312 and nspcc-dev/neofs-api#311, but doing Put() for every object is problematic on its own because it's a BoltDB Batch() for every object and in absence of concurrent modifications this means max_batch_delay for every object. If it's 10ms per object this means ~3h for 1M objects not even counting reads and real writing time.

Describe the solution you'd like

Rework transaction handling for the resync case, open a transaction, process something like 1K objects, flush the transaction, rinse, repeat.

Describe alternatives you've considered

Keep wasting time for nothing.

Metadata

Metadata

Assignees

Labels

I3Minimal impactS4RoutineU3RegularenhancementImproving existing functionalityneofs-storageStorage node application issuesperformanceMore of something per second

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions