Support --checksum-data flag, on-the-fly checksum verification

[shlomi-noach]

resubmission of openark#4 from downstream

This PR introduces --checksum-data, an opt-in checksum verification that runs throughout the migration.

With --checksum-data enabled, each rowcopy (a range of rows copied from the original table to the ghost table) is followed by a checksum on the two tables for that range.

Checksums are executed concurrently to rowcopy and are the exception to the single thread model for gh-ost.

A checksum may well fail while the migration is running: since gh-ost works in async design, where binlog entries are applied at some point in time after they're generated, it's quite possible that ongoing traffic will make some checksums fail.

A failed range's checksum is retried and retried until successful.

When --checksum-data is enabled, cut-over does not complete if failed checksums are found. While tables are locked in preparation for cut-over, a grace period is given so that the checksum evaluation can run to completion.

This is experimental.

Risk assessment: risky!

With flag disabled (as is the default case), behavior does not change and risk is low. With flag enabled, the following happen (or can happen):

• More reads directly on master server: these are the checksum tests; they take place on both original table and ghost table. It's worth noting that the row-copy operation runs a full scan on the original table anyhow, and so the extra reads do not (should not) bring into memory data pages not already brought into memory by row-copy.

• Slower migration time due to extra reads

• Risk at time of cut-over. At this time I have no access to a busy production server so I have not verified. The following scenario is possible:

  • migration is ready for cut-over
  • there's many checksums not fully verified yet (that's because production traffic was busy and changed data even while checksums were calculated)
  • gh-ost begins cut-over, thus locks table for writes
  • Table data is now static, so theoretically all checksums should be good.
  • But there's so many checksums to evaluate that we get timeout, thus rolling back migration.
  • repeat.

  To clarify that I haven't seen this, but I predict this might show up in prod.

I'm presenting this PR upstream for visibility. It's an important change that further validates (or invalidates!) the correctness of migrated data so it may be of interest. I'd suggest massive experimentation.