Sim Structure
| The Fundamentals → Sim Structure |
|---|
Written by Connor Jakubik
Space Teams PRO (STPRO, STP, or ST) is a product that SimDynamX LLC and Texas A&M University develop and publish as a set of software and online services. The software is an aerospace-focused simulation environment and the tools for a user to create, run, and analyze their own simulations. The online services are mainly backend functionality supporting multiuser simulations and online collaboration / file sharing. This page will focus on the simulation software that gets installed and ran by users.
The Space Teams Platform source code is a large, complex project, but the portions of the software that the user is meant to interact with are built to follow intuitive design patterns. Space Teams simulations use an architecture somewhat similar to the Entity-Component-System (ECS) pattern.
Space Teams Entities are unique objects representing something that exists in the simulation.
Space Teams Parameters are similar to ECS components. Parameters hold all the state of the simulation, in key-value ParamMap data structures. Here is an example of key-value data:
From: https://redis.io/nosql/key-value-databases/
Parameters can hold values of all variable types, including numbers, arrays, nested ParamMaps, references to Entities, and more. Each Entity has parameters that describe its properties such as mass, location, and graphics representation. Space Teams programs constantly read and affect these parameters to compute and show the simulation.
Space Teams Systems are pieces of code that hold all of the behavior of the simulation. Systems are applied to Entities, and during simulation execution they affect the Parameters of the entities to evolve the simulation over time. Systems typically have zero or very few dependencies on other Systems, meaning they can be specialized to a certain environmental effect (e.g. gravity), physical process (e.g. heat transfer), analysis computation (e.g. logging telemetry), or autonomous logic (e.g. path-planning) which can be applied on any Entity.
The Space Teams Platform simulation environment is structured as a distributed system of programs, where the simulation data is replicated on all programs, and modifications to simulation state are transmitted to other programs to be eventually consistent. The Compute_Server is where all C++-defined simulation behaviors (C++ Systems) are executed. SpaceCRAFT, the Unreal Engine client program, is where any 3D graphics and sim interaction happens, including flat-screen and VR. Additional programs are started for things such as Python Systems. In all programs in the Platform, a copy of all of the Entities / Parameters exists. The general assumption for programs is that they can treat their copy of the simulation Entities and Parameters as the up-to-date current state of the simulation, as the change-replication happens in the background.
Any number of programs can join a simulation. All programs are treated as equal contributors to the simulation. This means all Space Teams simulations are multiuser simulations with zero modifications required to scale to many additional users. The network configuration of one Unreal client and a Compute_Server is basically the same as an arbitrary number of Unreal clients being connected.
Space Teams provides web services to authenticate users, manage Orgs, and provide easy methods for hosting multiuser simulations. Org access controls permissions for uploaded files, multiuser session joining, and other online details.
The user can create and edit simulations, import and configure sim assets such as 3D models, and host/browse/join multiuser sim sessions all from the SpaceCRAFT unreal client program. The simulation editing tools are similar to those used to create video games, but they are specialized for spaceflight simulation needs such as orbit adjustment, snapping objects to a planet's surface at a latitude/longitude, setting up reference frames, navigating to objects extremely far from one another, etc. The user can work with local files as well as using Space Teams' git tools for collaborative work.
To write simulation behavior in C++, a user duplicates our template System subclass through a tool to establish all the boilerplate code, then writes code inside the load(), init(), update() and other System member functions that will be called by the Compute_Server in a thread specific to that System during the simulation. The build system for Systems is CMake, and we compile to the C++20 standard. The user is free to link external libraries that are already present in Space Teams or that they include inside their self-contained packages.
For writing simulation behavior in Python instead of C++, Space Teams includes a Python API library. This library encapsulates most of the functionality of the Compute_Server. The Python scripts that use it are 100% "normal" Python, as opposed to "embedded" python scripting where you can't install pip packages. To write simulation behavior in C++, a user includes a small section at the top of a Python script which sets up the connection to the simulation and initializes global Space Teams objects. When running a sim with Python Systems, a separate python process is started per Python System instance from a STP-specific virtual environment. Those python processes load the sim then connect to the Compute_Server instance that started them to join the sim.
Details: We include the installer for a particular version of Python alongside our application because we can't easily build Python API versions for more than that one version. Our install/setup scripts create a python Virtual Environment in your
AppData/Local/SimDynamX/STPro/Versions/.../pyfolder, which is set up with our baseline set of python packages likenumpy.Python System script processes are spun off from the Compute_Server process using
.\sc_python.bat(for Windows) with special configuration options in order to connect to the running sim. The user can install more pip packages through.\sc_python_pip.bat install ____.
Main wiki page: Timekeeping
The Space Teams Platform has special requirements for timekeeping. During a multiuser simulation, it must be possible to:
- Change timescale to any value including zero and negative numbers.
- Do a time-jump to any other time point.
- Be able to represent time for any point in the relevant future and past.
- At least 1,000 years to cover past and future highly eccentric solar system object orbits.
- Provide threadsafe async access to read the time and write timescale and time-jump clock modifications.
- Synchronize clock modifications across a network to for a multiuser sim.
To fulfill these requirements, we implemented an eventual-consistency system which accumulates clock modifications over real-world "wall" time from all clients in the network, then recomputes the sim time from those modifications every time the clock is read. This means, with the assumption that all clients have a synchronized real-world "wall" time, they will have a synchronized sim-time (after clock modification messages have been received through the network). This functionality is available through the SimClock functions in SimGlobals.
Space Teams differs from other simulation platforms in that it has no "main loop", "tick numbers", or "synchronization points". All programs and threads within those programs are meant to operate with a completely independent timing scheme. To interact with the running sim, any thread establishes a "context sim-time" by reading the sim clock, which is then used as the timestamp on any state reads or writes that happen on that thread. For reading time-sensitive state such as location and orientation of an Entity, the thread uses time-based extrapolation to compensate for the difference between the context sim-time and the timestamp of the existing state values. This timestamping and extrapolation approach results in time-corrected and smooth state values being available anywhere in the simulation platform, despite underlying network and compute time-instability.
This approach maximizes the benefits of a distributed-systems architecture by allowing programs and threads to independently contribute to the simulation, and never wait on other programs. This also represents reality well by closely matching a real-world complex scenario with independent physical processes, device microcontrollers, and control loops. Want to run a System in a realtime sim at 100,000 Hz? It's possible in Space Teams.
The price of this parallelism is it requires threadsafe and asynchronously-replicated data structures to accomplish, which are much harder to develop than basic object-oriented data structures. Space Teams has a very well-matured solution to this in the GenParamMap data structure and related backend netcode. All parameter changes on Entitys in a running simulation are network-replicated to other programs asynchronously.
There are nondeterministic results in the typical async simulation, where Systems are relying on the operating system thread scheduler and sleep-times to trigger their updates. Typically, results differ within reasonable error from sim to sim. In some cases, results can be corrupted by a lapse in true update frequency due to either high sim timescale or high/irregular computation time per update cycle. To address this for cases where results must be perfectly replicated for every run, Space Teams has a deterministic mode, which uses the same exact simulation code but replaces the system-clock timing with a deterministic scheduled-ahead sequence of updates. In deterministic mode, the simulation is run as fast as possible on a single thread, jumping directly from one system-update to the next, with the context sim-time jumping accordingly. This does not yet (v0.41.1) have a mechanism to work with sim behavior written in Python Systems as those are external processes not operating in lock-step with the Compute_Server, but this feature is in development. Read-only Python Systems already function as expected.
SimGlobals is a class with a set of functions that access simulation-wide state. SimGlobals are accessible from anywhere in the simulation. The SimEntity is a singleton Entity in SimGlobals, which has parameters like any other entity in the simulation. SimEntity does not allow Systems to be applied to it, as all Systems already can do SimGlobals::GetSimEntity(). SimGlobals also provides functions for accessing the SimClock, Adding/Deleting Entities, and more.
Main wiki page: Making Sim Configs
To make a simulation in Space Teams, one makes a Simulation Configuration (sim config) file. This can be done manually by duplicating a file, through SpaceCRAFT with the New Sim Wizard menu, by saving from a Sim Editing session, or by saving the state from a running simulation. The Sim Config holds declarations of all the Entities that exist when the simulation starts, their initial conditions, System Instances that will be run, and tags for applying System Instances to certain Entities. On sim loading, programs in the Space Teams Platform independently parse the sim config to initialize their local copy of the simulation state.
Main wiki page: Logging & Error Handling
On the Compute_Server or any Python System, console output as well as logging to a file are all handled by our custom-made Atomic Logger feature. This is a threadsafe logging framework that has configurable console and file outputs, does colored text output with ANSI escape sequences, and allows many separately-configured Logger objects to write to overlapping sets of output files / pipes with proper mutual-exclusion locking.
When an unhandled exception or other internal error occurs in either a C++ System or a Python System, unless a particular option is changed, the exception is caught and printed by the Fatal log function.
In the SpaceCRAFT Unreal Engine client, fatal errors are typically handled by popping up a modal window with the error message and asking if the user wants to quit or return to the menu. SpaceCRAFT itself doesn't have logging enabled in the User Release at this time (v0.28.0); Unreal logging is limited to development builds.