SCC file output does not include parity bits

[caiser01]

Checklist

• I have searched the issues page for any duplicate issues open or closed and confirmed that this bug has not been reported before.
• I have tested the issue with the current build.
• I have attached log files, uploaded sample data, and commands used so that the issue can be easily reproduced by the developers.

Bug Description

The combination of ld-process-vbi and ld-export-metadata are not producing fully valid SCC files for Line 21 data. Specifically, they do not save the parity bit transmitted with each data byte into the SCC file. According to all the references I can find, the parity bit of each data byte should be present, and other software I have used to make SCC files from decoded video include the parity bits.

Steps to Reproduce

Sample files available here: https://www.dropbox.com/scl/fo/1n1rskldcfsa08ijzw34x/AI-hiBBa1UioxGD0HP8WMNE?rlkey=02zwwwvbs87dgidov1ebi7mk9&dl=0

1. Run ld-process-vbi line21-sample.tbc using the sample data linked above
2. Run ld-export-metadata --closed-captions line21-sample-ld-tools.scc line21-sample.tbc.json
3. Observe the lack of parity bits in the resultant SCC file
4. Download sccyou https://github.com/amiaopensource/sccyou
5. Run sccyou -l 2 line21-sample.mkv using the sample video linked above
6. Observe parity bits are present in the resultant SCC file

Expected Behaviour

Parity bits should be included in the Line 21 data bytes in the SCC file output from ld-export-metadata.

Example:

00:00:18:04 9470 9470 4946 2057 4520 CBCE 4F57 204F D552 20D3 49C7 CEC1 4CD3 942C 942C 942F 942F

Actual Behaviour

Parity bits are omitted from the Line 21 data bytes in the SCC file output from ld-export-metadata.

Example:
00:00:18:04 1470 1470 4946 2057 4520 4b4e 4f57 204f 5552 2053 4947 4e41 4c53 142c 142c 142f 142f

Environment

• Decode version: vhs_decode / Commit: 4247b4f
• Operating System: Windows 11, Ubuntu 24.04
• Hardware Used: Multiple

Additional Information

SCC file format documentation: http://www.theneitherworld.com/mcpoodle/SCC_TOOLS/DOCS/SCC_FORMAT.HTML

Sample SCC file from PBS program supplier documentation: https://docs.pbs.org/space/MM/5344210/Closed+Captioning#Sample-files

Line 21 Data Services specification (free):
https://shop.cta.tech/products/cta-608

Is it related to tbc-video-export?

No

[atsampson]

That should be easy enough to fix. In ld-process-vbi's ClosedCaption, make it read all 8 bits rather than reading the parity bit separately, then in ld-export-metadata's writeClosedCaptions, mask off the parity bit in sanityCheckData and write the original values to the output.

Making this change shouldn't affect ttconv (which is normally what I use for processing CC), since that always masks off the parity bit when reading SCC files.

[caiser01]

That should be easy enough to fix. In ld-process-vbi's ClosedCaption, make it read all 8 bits rather than reading the parity bit separately, then in ld-export-metadata's writeClosedCaptions, mask off the parity bit in sanityCheckData and write the original values to the output.

I played around with code myself before posting this issue. Getting ld-process-vbi to save the parity bits to the json file is easy.

Getting ld-export-metadata to then correctly save the data0 and data1 byte pairs into the SCC file is where I ran into trouble. Back in ld-process-vbi, the data0 and data1 fields are initialized to -1 as an indicator of "invalid data". In ld-export-metadata, there are multiple checks on data0 and data1 to determine if they contain valid values

ld-decode/tools/ld-export-metadata/closedcaptions.cpp

Line 138 in 4f42190

if (data0 == -1 || data1 == -1) {

ld-decode/tools/ld-export-metadata/closedcaptions.cpp

Line 142 in 4f42190

if (data0 > 0 || data1 > 0) {

that involves checking for negative values (or zero). If the parity bit happens to be 1, the byte value is interpreted as being negative, therefore invalid, and it is not written out to the SCC file.

I wasn't certain how to best handle flagging invalid data and correctly passing the parity bits, which is why I filed the issue.

There is an additional issue with ld-export-metadata that it assumes Line 21 data will only be valid and present for CC1

ld-decode/tools/ld-export-metadata/closedcaptions.cpp

Line 131 in 4f42190

if (data0 != 0x14) {

This may be true for most scenarios of laserdisc captioning but does not consider the full range of data available on Line 21 per the CTA-608 specifications (CC1-CC4 and T1-T4). This becomes a problem downstream in vhs-decode where it is not uncommon to encounter tape recorded TV broadcasts that contain more Line 21 data than just CC1.

Making this change shouldn't affect ttconv (which is normally what I use for processing CC), since that always masks off the parity bit when reading SCC files.

Correcting the SCC file output to include parity bits per the SCC file format (http://www.theneitherworld.com/mcpoodle/SCC_TOOLS/DOCS/SCC_FORMAT.HTML) should not have a negative impact on any external applications that consume SCC. If having a correctly formatted SCC file breaks another application, the developers of that application need to fix it.

[caiser01]

I've tinkered with the source code for this a bit more but still can't figure out the best way to make ld-process-vbi and ld-export-metadata save the parity bits into the SCC file as called for in the SCC specification.

Still hoping someone with a deeper knowledge of the code and the tbc json format can look into this issue.

While there are other software applications that can generate fully valid SCC files from decoded video, the methods those applications use sometimes result in garbled data. Using ld-process-vbi and ld-export-metadata is still the preferred way since they seem to decode Line 21 data more accurately than other tools.