Populate provenance nodes for spanner ingestion

[vish-cs]

Use import_metadata_mcf.mcf file in GCS to populate provenance nodes
Also, split latestVersion into latestVersion and graphDataPath fields. Convert to array to support multiple paths
Update spanner schema

[gemini-code-assist]

Summary of Changes

Hello @vish-cs, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly upgrades the Spanner ingestion pipeline by enabling the population of provenance nodes and edges, which is crucial for data lineage and traceability. It refactors the data processing mechanism to accommodate multiple graph data sources and different file formats, enhancing the pipeline's flexibility and robustness. These functional changes are complemented by necessary updates to the Spanner schema and the Dataflow job configuration script to ensure seamless operation and improved resource management.

Highlights

• Provenance Node Population: Introduced new logic within the ImportGroupPipeline to read and ingest provenance metadata from import_metadata_mcf.mcf files, converting them into Spanner nodes and edges. This ensures better tracking of data origins.
• Flexible Graph Data Path Handling: The ingestion pipeline now supports processing multiple graph data paths for each import, specified via a new graphDataPaths array in the input JSON. It also intelligently handles different graph data formats, including tfrecord.
• Spanner Schema Enhancements: The ImportStatus table has been updated to include a GraphDataPaths array column, and a new ImportVersionHistory table has been added to track different versions of imports with details like ImportName, Version, UpdateTimestamp, and Comment.
• Dataflow Template Configuration Updates: The template.sh script has been modified to reflect the new importList JSON structure, including graphDataPaths. It also now specifies spannerDatabaseId, sets num-workers, and defines worker-machine-type for Dataflow jobs.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates the Spanner ingestion pipeline to populate provenance nodes by processing a list of graph data paths for each import. The changes are logical, but I've identified a critical issue that could lead to a pipeline crash, along with a couple of medium-severity concerns regarding code maintainability and robustness. My main feedback points include fixing a potential NullPointerException during JSON parsing, improving the file type detection logic, and questioning the addition of an apparently unused database table.