ld-export-decode-metadata tool to export SQLite metadata

[Stefan-Olt]

This is the ld-export-decode-metadata tool, that allows to export the metadata that is now stored in the SQLite database into a JSON format, that is effectively the old internal metadata format.

This allows many external tools to continue working without any changes required (mostly used in vhs-decode workflows, like audio synchronization tools, custom stackers and some statistics tools)

It is heavily based on ld-json-converter, but it uses the tbc library to read the SQLite database. It has a copy of the previous JSON metadata code to write the JSON, with the class renamed to avoid naming conflicts. All fields are copied individually by value as they are provided by the API to maintain compatibility when internal metadata structure changes.

As I couldn't find any tests for ld-json-converter I did not write any for this tool either. If I just missed them, please point me to them I will add tests for this as well.

It is separate from ld-export-metadata, as it does not export metadata that was part of the original signal, but metadata generated in the decode process.

[simoninns]

Hi, you are correct in your observation that the ld-json-converter doesn't provide testing - but this is because it's not a "real tool" - it will only be in the repo until ld-decode.py outputs SQLite natively (at which point it will be removed).

If you do not include testing with this tool there will be no indication from CI/CD if the tool breaks due to a SQLite schema change (or any other change) - so you should at least add a test for PAL and NTSC using the test data images provided (I recently modified the test data to sub-module, so you can clone it locally and run the tests locally before PRing to main).

I would suggest that, for now, your tests ld-decode something, ld-json-convert the JSON and then run the result though your tool and verify the output is as expected. Then, once the SQLite work is done in ld-decode.py we can remove the middle step (and as soon as ld-decode.py outputs SQLite the CI/CD using the converter will fail - so the need to fix testing will be obvious).

I'd appreciate it if you could make those changes before the PR is accepted.

[Stefan-Olt]

Hi, I had a look at the test infrastructure. It seems there are only 3 test scripts and most tools are tested using the test-decode script. I think it would make the most sense to add the test there as def run_ld_export_decode_metadata(args) and compare it's output to the SQLite database (like run_ld_process_vbi does). The SQLite database is currently created by the converter, so when ld-decode starts outputting SQLite natively that step is removed for all tools to be tested. Any objections?

[simoninns]

No issue for me... until ld-decode.py is outputting SQLite the CI/CD will have to rely on ld-json-converter. Once @happycube is ready, the ld-json-converter steps will have to be removed from the test cases otherwise CI/CD will fail due to the ld-decode changes... so it all makes sense to me.

[Stefan-Olt]

Added a test to the test-decode script. It reads the SQLite database and compares it to the json. So the test would need to be updated in case the database layout changes (just like other tests in there that compare to the database)
I tested the test by intentionally messing things up.

[simoninns]

ok, I gave it a quick review - looks like @happycube is rapidly including SQLite support too, so let's merge.