Refactor data module into a collection of data modules

[oubiwann]

ur.core.data API Redesign

This directory contains the redesigned data layer for underack, replacing the ad-hoc ur.core.data module with a Redis-inspired API split across three focused modules.

Directory Structure

src/ur/core/data/
├── store.lfe      ;; KV operations (get, set, del, sadd, smembers, etc.)
├── table.lfe      ;; Table lifecycle (new, drop, export, import)
└── match.lfe      ;; Pattern queries (scan, filter, fold)

Module Overview

ur.core.data.store

Core key-value operations, modeled after Redis commands.

Basic ops (for set tables):

• (get table key) → value or undefined
• (set table key value) → ok
• (del table key) → ok
• (exists? table key) → boolean

Bulk ops:

• (keys table) → list of all keys
• (vals table) → list of all values
• (getall table) → list of {key, value} tuples
• (count table) → integer

Set ops (for bag tables like cables):

• (sadd table key member) → ok (add to set)
• (srem table key member) → ok (remove from set)
• (smembers table key) → list of members
• (smember? table key member) → boolean
• (scard table key) → count of members

ur.core.data.table

Table lifecycle management.

Lifecycle:

• (new name opts) → creates ETS table
• (drop name) → deletes table
• (ensure name opts) → import if exists, else create

Introspection:

• (info name) → map of table properties
• (exists? name) → boolean

Persistence:

• (export name) → save to timestamped file
• (export name filename) → save to specific file
• (import name) → load newest export
• (import name filename) → load specific file
• (exports name) → list all export files

ur.core.data.match

Pattern matching and query operations.

Pattern queries:

• (scan table) → all entries
• (scan table match-spec) → entries matching ETS match spec
• (match-keys table pattern) → keys matching Erlang pattern
• (match-vals table key-pattern) → values for matching keys

Filtering:

• (filter table pred) → entries where (pred entry) is true
• (filter-keys table pred) → keys where (pred key) is true
• (filter-vals table pred) → values where (pred val) is true

Aggregation:

• (fold table fun acc) → fold with (fun key val acc)
• (reduce table fun acc) → alias for fold

Migration from ur.core.data

Old Function	New Function	Module
kv-all	getall	store
kv-all-ks	keys	store
kv-all-vs	vals	store
kv-vs	smembers (for bags) or get (for sets)	store
del-row	srem	store
del-key-rows	del	store
table-info	info	table
export	export	table
import	import	table
re-import	import (now handles this)	table
newest-table-file	newest-file	table
import-or-new	ensure	table

Usage Examples

Cables (bag table)

;; Add a connection
(ur.core.data.store:sadd 'cables 'clock 'quantizer)

;; List all subscribers to clock
(ur.core.data.store:smembers 'cables 'clock)
;; => (quantizer noise-mod)

;; Check if connection exists
(ur.core.data.store:smember? 'cables 'clock 'quantizer)
;; => true

;; Remove a connection
(ur.core.data.store:srem 'cables 'clock 'quantizer)

Modules (set table)

;; Store module config
(ur.core.data.store:set 'modules 'clock #m(bpm 120 running true))

;; Get module config
(ur.core.data.store:get 'modules 'clock)
;; => #m(bpm 120 running true)

;; List all module names
(ur.core.data.store:keys 'modules)
;; => (clock noise lfo)

Table Management

;; Create or restore a table
(ur.core.data.table:ensure 'cables '(bag named_table public))

;; Export current state
(ur.core.data.table:export 'cables)
;; => #m(file "/home/user/.local/share/underack/data/cables-20251214.153022.ets" 
;;       table cables)

;; List all exports
(ur.core.data.table:exports 'cables)
;; => ("/home/user/.local/share/underack/data/cables-20251214.153022.ets"
;;     "/home/user/.local/share/underack/data/cables-20251213.091544.ets")

Queries

;; Find all entries where value is not 'undefined
(ur.core.data.match:filter-vals 'cables 
  (lambda (v) (=/= v 'undefined)))

;; Count connections per output
(ur.core.data.match:fold 'cables
  (lambda (k v acc)
    (let ((current (maps:get k acc 0)))
      (maps:put k (+ current 1) acc)))
  #m())
;; => #m(clock 3 noise 1 lfo 2)

Design Notes

1. Redis naming: Familiar to most developers, well-documented semantics
2. Set vs Bag: Redis sets map naturally to ETS bags (one key, multiple values)
3. Explicit modules: Clear separation of concerns vs one monolithic module
4. Predicate naming: exists?, smember? use ? suffix for boolean returns (Lisp convention)
5. No export all: Explicit exports define the public API clearly