make `--upstream_sync` also sync the gap analysis graph progressively

[northdpole]

Issue

What is the issue?

Current gap analysis graph is 9Gb which makes it very hard to download reliably when running locally (also you shouldn't need 9Gb of data to run the project)

Expected Behaviour

Upstream sync should either download the data necessary in order to run(graph structure).
For the Gap Analysis graph it should either grab it from a place suitable for this sort of data (e.g. google public dataset) or it should load it from upstream opportunistically and then save it in the local database

[kart-u]

hello @northdpole is this issue still open for work , I would love to contribute in it!
I like issues involving optimizations :-)

[zenowinged]

@northdpole i'd like to work on this issue. could you please assign it to me?

[sd2604]

Hey if this issue is still available, could you pls assign this to me, I'd realy like to work on this issue...

[IND-Anshuman]

Hi! @northdpole I'd like to work on this issue. I've analyzed the codebase and have a clear plan to solve the problem.

The Problem I identified: Currently, --upstream_sync only downloads the CRE graph structure but ignores the gap analysis data, which is ~9GB and impractical to download reliably for local development.

My Solution: First, I'll extend the donwload_graph_from_upstream() function in application/cmd/cre_main.py to progressively sync gap analysis data after downloading the CRE structure. It will get the list of standards from upstream, go through standard pairs, and download their gap analysis results incrementally while skipping already-cached data. I can also implement lazy loading by modifying the schedule() function in application/utils/gap_analysis.py to get the gap analysis from upstream on-demand before falling back to expensive local calculations. This means the system will first try to grab pre-calculated results from upstream , only calculating locally if upstream is unavailable or doesn't have the data.

Could you please assign this issue to me? Thanks!

[ashutosh-engineer]

@northdpole While exploring and fixing the Issue of #469 I am also exploring the Solution for this issue.
Will Share the report soon.