A robust, high-performance downloader for HYCOM (Hybrid Coordinate Ocean Model) ocean data with both web interface and command-line capabilities. Built with Flask and enhanced with h5netcdf-first processing for optimal performance and reliability.
- ✅ h5netcdf-first processing - Optimized NetCDF handling with automatic fallbacks
- ✅ Robust engine selection - Tries h5netcdf → autodetect → scipy for maximum compatibility
- ✅ Compressed NetCDF writing - Automatic compression with h5netcdf, fallback to scipy
- ✅ Cheap validation - Fast file validation without full array reads
- ✅ Month-end capping - Respects date boundaries properly
- ✅ Enhanced error handling - Comprehensive retry mechanisms and cleanup
- ✅ User-friendly Flask interface - Modern, responsive web UI
- ✅ Real-time monitoring - Live progress tracking with detailed status
- ✅ Configuration management - Easy parameter adjustment through web forms
- ✅ File management - Download, view, and delete files through the interface
- ✅ Keyboard shortcuts - Power user features for efficiency
- ✅ Automatic file combination - Merges NetCDF files by month with proper time handling
- ✅ Smart retry mechanism - Intelligent retry logic for failed downloads
- ✅ Progress tracking - Detailed progress bars and status updates
- ✅ Comprehensive logging - Full audit trail of all operations
- ✅ Local storage - No external dependencies, stores data locally
- Python 3.7 or higher
- Internet connection for downloading HYCOM data
- Clone or download this repository
- Install dependencies:
pip install -r requirements.txt
The application requires these Python packages:
flask- Web frameworkrequests- HTTP client for downloadsxarray- NetCDF data handlingpandas- Date/time processingtqdm- Progress barsh5netcdf- Optimized NetCDF engine (optional, with fallback)netCDF4- NetCDF support (fallback)
Access the web interface at http://localhost:5000 and configure:
- Geographic bounds (longitude/latitude)
- Date range for data download
- Variables to download (water_u, water_v, etc.)
- Download settings (retry count, timeout)
Edit the Config class in oceanos_hycom_download.py:
class Config:
# Geographic bounds (Indonesian waters example)
WEST_LON = 116.5 # Western longitude
EAST_LON = 119.0 # Eastern longitude
SOUTH_LAT = -2.0 # Southern latitude
NORTH_LAT = 0.5 # Northern latitude
# Date range
DATE_START = '2022-12-01'
DATE_END = '2022-12-31'
# Variables to download
VARIABLES = ['water_u', 'water_v']
# Download settings
MAX_RETRIES = 3 # Maximum retry attempts
TIMEOUT = 60 # Request timeout in seconds
CHUNK_SIZE = 8192 # Download chunk size-
Start the application:
python run_app.py
Or use the Windows batch file:
start_web_interface.bat
-
Open your browser and navigate to:
http://localhost:5000 -
Configure your download:
- Set geographic bounds for your area of interest
- Choose date range for data download
- Select ocean variables (water_u, water_v, etc.)
- Adjust download settings as needed
-
Start downloading:
- Click "Start Download" to begin
- Monitor real-time progress
- Use "Stop Download" if needed
-
Manage files:
- View downloaded files in the Files panel
- Download files to your local machine
- Delete files you no longer need
Run the downloader directly:
python oceanos_hycom_download.pyThis will use the configuration in the Config class and process data month by month.
- Interactive form for all download parameters
- Real-time validation of input values
- Save/load configuration presets
- Live progress bar with percentage completion
- Current file being downloaded
- Download speed and ETA
- Success/failure counts
- Start/stop downloads with confirmation
- Pause and resume functionality
- Emergency stop with cleanup
- Browse all downloaded files
- View file details (size, date, variables)
- Download files to local machine
- Delete unwanted files
- Sort by date, size, or name
Ctrl+S- Save configurationCtrl+D- Start downloadCtrl+Shift+D- Stop downloadCtrl+F- Show files panelCtrl+R- Refresh status
Files are saved in ./hycom_data/ with the format:
HYCOM_data_[YYYYMM].zip
Where:
YYYYMM- Year and month of the data (e.g., 202212 for December 2022)
Each ZIP file contains:
HYCOM_combined_[YYYYMM].nc- Combined NetCDF file with all variables and time steps- Compressed using ZIP_DEFLATED for efficient storage
The NetCDF files contain:
- Variables: Ocean current components (water_u, water_v)
- Coordinates: Longitude, latitude, time, depth
- Attributes: Creation timestamp, source information, variable list
- Compression: Automatic compression for efficient storage
hycom-data-dl/
├── app.py # Flask web application
├── run_app.py # Application launcher
├── oceanos_hycom_download.py # Core downloader (CLI + API)
├── start_web_interface.bat # Windows launcher
├── requirements.txt # Python dependencies
├── README.md # This documentation
├── LICENSE # License information
├── templates/ # HTML templates
│ ├── base.html # Base template with navigation
│ └── index.html # Main application page
├── static/ # Static web assets
│ ├── css/
│ │ └── style.css # Custom styling
│ └── js/
│ └── app.js # Frontend JavaScript
├── hycom_data/ # Output directory (auto-created)
├── temp_download/ # Temporary files (auto-created)
└── hycom_download.log # Application logs (auto-created)
- URL Generation - Creates optimized HYCOM NCSS URLs with NetCDF4 format
- Download with Retry - Robust download with exponential backoff
- Validation - Fast validation using safe engine selection
- File Combination - Intelligent merging by variable and time
- Compression - Automatic compression with fallback options
- Cleanup - Proper resource cleanup and temporary file removal
The application uses a smart engine selection strategy:
- h5netcdf - Preferred for performance and compression
- autodetect - Fallback for compatibility
- scipy - Final fallback for maximum compatibility
- Network errors - Automatic retry with exponential backoff
- File corruption - Validation and re-download
- Memory issues - Proper resource cleanup
- Disk space - Temporary file management
pip install xarray netCDF4 pandas requests flask tqdm- Ensure the application has write permissions in the current directory
- Run as administrator if necessary (Windows)
- Check your internet connection
- Reduce
TIMEOUTif connection is stable - Verify HYCOM server status
- Consider downloading smaller date ranges
- The application automatically manages memory
- Temporary files are cleaned up after processing
- Use smaller date ranges for very large datasets
# Kill existing processes
netstat -ano | findstr :5000
taskkill /F /PID <PID_NUMBER>- Download data in smaller monthly chunks
- Use SSD storage for better I/O performance
- Ensure sufficient RAM (8GB+ recommended)
- Increase
TIMEOUTvalue - Reduce
CHUNK_SIZEfor more stable downloads - Use retry mechanism (already enabled)
- h5netcdf Integration - Optimized NetCDF processing with automatic fallbacks
- Enhanced Error Handling - Comprehensive retry mechanisms and cleanup
- Improved Validation - Fast file validation without full array reads
- Better Compression - Automatic compression with multiple engine support
- Cleaner Codebase - Removed obsolete test files and improved structure
- Enhanced Documentation - Comprehensive README with troubleshooting guide
- Faster file processing with h5netcdf engine
- Better memory management with proper resource cleanup
- Improved reliability with robust error handling
- Enhanced user experience with better progress tracking
This project is licensed under the MIT License - see the LICENSE file for details.
Contributions are welcome! Please feel free to submit issues, feature requests, or pull requests.
For support and questions:
- Check the troubleshooting section above
- Review the application logs in
hycom_download.log - Open an issue on the project repository
Happy downloading! 🌊