Add archive data auto-loading on database initialization#16
Merged
Conversation
- Create archive generation script using real NetCDF data with synthetic timestamps (7 days, 1.5M rows) - Add init script to auto-load gzip-compressed archive data on first database startup (~3 seconds) - Include archive CSV files in repo (7.6 MB total, small enough for version control) - Update database Dockerfile for proper init script execution order (01-init-schema.sql, 99-load-archive.sh) - Configure docker-compose to mount archive data directory and init script
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #16 +/- ##
==========================================
+ Coverage 64.64% 68.72% +4.08%
==========================================
Files 19 22 +3
Lines 1547 1848 +301
==========================================
+ Hits 1000 1270 +270
- Misses 547 578 +31
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- Add parse_netcdf_archive.py for direct NetCDF-to-DB loading with PostgreSQL COPY - Support configurable time window via ARCHIVE_MAX_DAYS env var (default: 7 days) - Auto-download 3-month NetCDF dataset (~209 MB) on first run - Achieve ~155K rows/sec throughput with batched processing and timestamp shifting - Update README with dual archive loading methods (CSV default vs NetCDF high-resolution)
- Add Interval (Auto/1min/5min/15min/1h/6h/1d) and Aggregation (Mean/Raw/Min/Max/Median/StdDev) dropdown variables - Refactor RSL and TSL queries to UNION ALL pattern separating raw and aggregated paths - Support auto interval via $__interval_ms with safe ::interval casting outside CASE expression
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #8
Summary of changes in this PR: