Shape n’ Swarm: Hands-on, Shape-aware Generative Authoring with Swarm UI and LLMs

Paper Abstract

This paper introduces a novel authoring method for swarm user interfaces that combines hands-on shape manipulation and speech to convey intent for generative motion and interaction. We refer to this authoring method as shape-aware generative authoring, which is generalizable to actuated tangible user interfaces. The proof-of-concept Shape n’ Swarm tool allows users to create diverse animations and interactions with tabletop robots by hand-arranging the robots and providing spoken instructions. The system employs multiple script-generating LLM agents that work together to handle user inputs for three major generative tasks: (1) thematically interpreting the shapes created by users; (2) creating animations for the manipulated shape and (3) flexibly building interaction by mapping I/O. In a user study (n = 11), participants could easily create diverse physical animations and interactions without coding. To lead this novel research space, we also share limitations, research challenges, and design recommendations.

Repo Setup:

1. Create virtual environment
2. pip install requirements.txt
3. Make sure necessary processing packages are installed (oscP5).

Run Code:

1. Connect to Robots

1. Open another new terminal
2. cd rust_osc
3. cargo run -- -a #,#,#,... The #s are each toio number

2. Start Processing

1. Open new terminal
2. processing-java --sketch=$PWD/toio_processing --run

3. Start Frontend

1. Open another new terminal
2. cd frontend
3. npm start

4. Run Main Python Program

1. Open another terminal
2. cd UIST2025
3. python3 app.py

optional arguments: python3 app.py --terminal Type in commands instead of spacebar for speech python3 app.py --simulated. Interact purely with the GUI (allows you to skip robot connection)