NodeFlow is a high-performance, node-based visual programming interface designed to simplify complex Data Extraction, Transformation, and Loading (ETL) tasks. Built with a focus on stream-processing and developer transparency, it allows users to build robust data pipelines without writing a single line of boilerplate code.
- Visual Logic Canvas: Drag-and-drop nodes to define data flows using a high-performance interactive canvas.
- Stream-Based Engine: Leverages Node.js Streams to process multi-gigabyte files with a minimal memory footprint.
- Live Data Peek: Inspect data transformations in real-time at any point in the pipeline.
- Developer-First Design: Export workflows as version-control-friendly JSON schemas.
- Modular Node Library:
- Inputs: CSV, JSON, and API Endpoints.
- Transformations: Filter, Map, Column Selection, Data Cleaning (Regex/Formatting).
- Outputs: Updated files (CSV/JSON) and interactive visualizations (Charts/Graphs).
- Frontend: React 19, React Flow (Canvas Engine), Tailwind CSS, Zustand (State).
- Backend: Node.js, Web Streams API (Processing), Worker Threads (Parallelism).
- Data Visualization: Recharts.
- Security: Isolated-vm for sandboxed transformation logic.
NodeFlow separates the Visual Layer from the Execution Layer. The canvas generates a declarative JSON graph, which is then parsed by the Node.js backend to create a pipelined execution sequence:
[Source Node] -> [Transform Node] -> [Transform Node] -> [Output Node]
| | | |
(Read Stream) -> (Pipe & Map) -> (Filter/Clean) -> (Write/Graph)
Follow these steps to get your local development environment running.
Ensure you have the following installed:
- Node.js: v20.x or higher
- npm: v9.x or higher
git clone https://github.com/swayam9705/ETL-Workflow-Creator-using-Node-Based-Visual-Programming.git
cd client
npm install
npm run dev