From 2bf67d7f06924ed7559594d550841f8f67b9db4e Mon Sep 17 00:00:00 2001 From: Alice Date: Mon, 15 Nov 2021 03:49:53 -0500 Subject: [PATCH 01/11] Initial draft of implementation strategy --- rfcs/43-many-worlds.md | 181 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 rfcs/43-many-worlds.md diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md new file mode 100644 index 00000000..e5dc951f --- /dev/null +++ b/rfcs/43-many-worlds.md @@ -0,0 +1,181 @@ +# Feature Name: `many-worlds` + +## Summary + +Bevy apps: now with more worlds. And more schedules to match! +Transfer entities between worlds, clone schedules and spin up and down worlds with the `AppCommands`. +New `GlobalRes` resource type, which is accessible across worlds. + +## Motivation + +The ability to have multiple worlds (and unique groups of systems that run on them) is a very useful, reusable building block for separating parts of the app that interact very weakly, if at all. +Some of the more immediate use cases include: + +1. A more unified design for pipelined rendering. +2. Entity staging grounds, where complete or nearly-complete entities can be stored for quick later use (see [bevy #1446](https://github.com/bevyengine/bevy/issues/1446)). +3. Entity purgatories, where despawned entities can be kept around while their components are inspected (see [bevy #1655](https://github.com/bevyengine/bevy/issues/1655)). +4. A trivially correct structure for parallel world simulation for scientific simulation use cases. +5. Look-ahead simulation for AI, where an agent simulates a copy of the world using the systems without actually affecting game state. +6. Isolation of map chunks. + +Currently, this flavor of design requires abandoning `bevy_app` completely, carefully limiting *every* system used (including base / external systems) or possibly some very advanced shenanigans with exclusive systems. + +A more limited "sub-world" design (see #16 for further discussion) is more feasible, by storing a `World` struct inside a resource, but this design is harder to reason about, unnecessarily hierarchical and inherently limited. + +## User-facing explanation + +TODO: write me. + +Explain the proposal as if it was already included in the engine and you were teaching it to another Bevy user. That generally means: + +- Introducing new named concepts. +- Explaining the feature, ideally through simple examples of solutions to concrete problems. +- Explaining how Bevy users should *think* about the feature, and how it should impact the way they use Bevy. It should explain the impact as concretely as possible. +- If applicable, provide sample error messages, deprecation warnings, or migration guidance. +- If applicable, explain how this feature compares to similar existing features, and in what situations the user would use each one. + +## Implementation strategy + +`App`'s new struct under this proposal would be: + +```rust +pub struct App { + // WorldId is not very readable; we can steal labelling tech from systems + pub worlds: HashMap, + // This is important to preserve the current ergonomics in the simple case + pub default_world: WorldLabel, + // Each Schedule must have a `World`, but not all worlds need schedules + pub schedules: HashMap, + // Same as before + pub runner: Box, + // Designed for storing GlobalRes + // Read-only access unless you have &mut App + // Also stores an `AppCommandQueue` in some fashion + pub global_world: World +} +``` + +The standard main loop would have the following basic structure: + +1. Create a `Global`, which all worlds can access when they need threads to run their systems. +2. For each `WorldLabel` in `app.world.keys()`, initialize the corresponding schedule with the correct label. +3. For each world, run the corresponding schedule if it exists. + 1. Each world is fully independent, and each scheduler can request more threads as needed in a greedy fashion. +4. At the end of each loop, put the entire `App` back together and run any `AppCommands`. + 1. Be sure to apply standard `Commands` to each world. +5. Go to 2. + +This should be programmed using the default (and minimal) runners. +Custom schedule and synchronization behavior can be added with a custom runner. + +### Cloneable schedules + +Schedules must be cloneable in order to reasonably spin up new worlds. + +When schedules are cloned, they must become uninitialized again. + +### AppCommands + +`AppCommands` directly parallel `Commands` API and design where possible, likely abstracting out a trait. + +They operate on `&mut App`. + +The initial set of `AppCommands` needed for this RFC are: + +- `spawn_world(label: impl WorldLabel)` +- `despawn_world(label: impl WorldLabel)` +- `new_schedule(label: impl WorldLabel, schedule: Schedule)` +- `clone_schedule(old_label: impl WorldLabel, new_label: impl WorldLabel)` +- `reassign_schedule(old_label: impl WorldLabel, new_label: impl WorldLabel)` +- `remove_schedule(label: impl WorldLabel)` +- `move_entities(entities: HashSet, origin_world: impl WorldLabel, destination_world: impl WorldLabel)` +- `move_query::(origin_world: impl WorldLabel, destination_world: impl WorldLabel)` + - moves every matching entity to the destination world +- `move_resource::(origin_world: impl WorldLabel, destination_world: impl WorldLabel)` +- `other_world_events::(events: Events, destination_world: impl WorldLabel)` +- `other_world_commands(commands: Commands, destination_world: impl WorldLabel)` +- `insert_global(res: impl Resource)` + - Also used for mutating global resources +- `init_global::()` +- `remove_global::()` + +Schedule modifying commands are obviously a natural extension, but are well outside of the scope of this RFC. + +### Global resources + +In order to avoid contention issues and scheduler entanglement, global resources are read-only to ordinary worlds. +Use `AppCommands::insert_global_resource` to modify them. +They are accessible via `GlobalRes` system parameters in standard systems. + +Under the hood, each world gets a `&` to these. +This simplifies system parameter dispatch, and allows them to be accessed in exclusive systems using `World::get_global_resource::`. + +## Drawbacks + +- added complexity to the core design of `App` +- this feature will be ignored by many games with less complex data flows +- makes the need for plugin configurability even more severe +- sensible API is blocked on [system builder syntax](https://github.com/bevyengine/bevy/pull/2736) + - until then, we *could* make methods for `add_system_set_to_stage_to_world`... +- heavier use of custom runners increases the need for better reusability of the `DefaultPlugins` `winit` runner; this is a nuisance to hack on for the end user and custom runners for "classical games" are currently painful + +## Rationale and alternatives + +### Why not sub-worlds? + +Ownership semantics are substantially harder to code and reason about in a hierarchical structure than in a flat one. + +They are significantly less elegant for several use cases (e.g. scientific simulation) where no clear hierarchy of worlds exists. + +Things will also get [very weird](https://gatherer.wizards.com/pages/card/details.aspx?name=Shahrazad) if your sub-worlds are creating sub-weorlds (as is plausible in an AI design), compared to a flat structure. + +### Why are schedules associated with worlds? + +Each system must be initialized from the world it belongs to, in a reasonably expensive fashion. +Free-floating schedules, + +### Why not store a unified `Vec<(World, Schedule)>`? + +This makes our central, very visible API substantially less clear for beginners, and is a pain to work with. +By moving to a label system, we can promote explicit, extensible designs with type-safe, human readable code. + +In addition, this design prevents us from storing schedules which are not associated with any current world, which is annoying for some architectures which aggressively spawn and despawn worlds. + +### Why can't we have multiple schedules per world? + +This makes the data access / storage model slightly more complex. +No pressing use cases have emerged for this, especially since schedules can be nested. +If one arises, swapping to a multimap should be easy. + +The ability to clone schedules should give us almost everything you might want to do with this. + +## Prior art + +[#16](https://github.com/bevyengine/rfcs/pull/16) covers a related but mutually incompatible sub-world proposal. + +[Unity](https://docs.unity3d.com/Packages/com.unity.entities@0.0/manual/ecs_in_detail.html#world) has multiple worlds: + +- possible uses cases are discussed [here](https://forum.unity.com/threads/what-should-be-the-motivation-for-creating-separate-ecs-worlds.526277/) +- docs are thin +- usability seems very poor + +## Unresolved questions + +1. Can we get `App`-level storage of `AppCommandQueue` working properly? Very much the same problem as [bevy #3096](https://github.com/bevyengine/bevy/issues/3096) experienced with `Commands`. +2. Is the ability to clone entities between worlds critical / essential enough to solve [bevy #1515](https://github.com/bevyengine/bevy/issues/1515) as part of this feature (or before attempting it)? + +## Future possibilities + +1. Dedicated threads per world. + 1. Could be very useful for both audio and input. + 2. Needs profiling and design. + 3. May break on web. + 4. May behave poorly with `NonSend` resources in complex ways. +2. Built-in support for some of the more common use cases. + 1. Feature needs time to bake first, and immediate uses are appealing to end users. + 2. Entity purgatories would be interesting to have at a component level. +3. `NonSendGlobal` resources. +4. Ultra-exclusive systems could be added to allow for a system-style API to perform work across worlds or directly modify schedules. + 1. Some trickery would be required to avoid self-referential schedule modification. +5. More fair, responsive or customizable task pool strategies to better predict and balance work between worlds. +6. Use of app commands to modify schedules, as explored in [bevy #2507](https://github.com/bevyengine/bevy/pull/2507). From 03b67ec79516c0fb08c97f78f6dcb8856ca96a57 Mon Sep 17 00:00:00 2001 From: Alice Date: Mon, 15 Nov 2021 11:55:18 -0500 Subject: [PATCH 02/11] Refined `AppCommands` API slightly --- rfcs/43-many-worlds.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md index e5dc951f..9d8128c5 100644 --- a/rfcs/43-many-worlds.md +++ b/rfcs/43-many-worlds.md @@ -99,7 +99,9 @@ The initial set of `AppCommands` needed for this RFC are: - `init_global::()` - `remove_global::()` -Schedule modifying commands are obviously a natural extension, but are well outside of the scope of this RFC. +All of these methods should have an equivalent on `App`, which should be the canonical form called by these commands. + +Schedule modifying commands are a natural extension, but are well outside of the scope of this RFC. ### Global resources @@ -163,6 +165,7 @@ The ability to clone schedules should give us almost everything you might want t 1. Can we get `App`-level storage of `AppCommandQueue` working properly? Very much the same problem as [bevy #3096](https://github.com/bevyengine/bevy/issues/3096) experienced with `Commands`. 2. Is the ability to clone entities between worlds critical / essential enough to solve [bevy #1515](https://github.com/bevyengine/bevy/issues/1515) as part of this feature (or before attempting it)? +3. Should `AppCommands` take a `&mut App`, or something better scoped to reduce borrow-check complexity? ## Future possibilities @@ -179,3 +182,5 @@ The ability to clone schedules should give us almost everything you might want t 1. Some trickery would be required to avoid self-referential schedule modification. 5. More fair, responsive or customizable task pool strategies to better predict and balance work between worlds. 6. Use of app commands to modify schedules, as explored in [bevy #2507](https://github.com/bevyengine/bevy/pull/2507). + 1. We could *maybe* even pause the app, and then modify the runner. +7. Fallible `AppCommands` using the technology in [bevy #2241](https://github.com/bevyengine/bevy/pull/2241). From 99d2625db020ee7df0a0a28493e7bda51aba822d Mon Sep 17 00:00:00 2001 From: Alice Date: Mon, 15 Nov 2021 12:05:17 -0500 Subject: [PATCH 03/11] Considered standalone bevy_ecs users --- rfcs/43-many-worlds.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md index 9d8128c5..46059ef2 100644 --- a/rfcs/43-many-worlds.md +++ b/rfcs/43-many-worlds.md @@ -151,6 +151,14 @@ If one arises, swapping to a multimap should be easy. The ability to clone schedules should give us almost everything you might want to do with this. +## Why is this design at the `bevy_app` level, not `bevy_ecs`? + +Fundamentally, we need to store schedules and worlds together, and have a way to execute logic on them in a sensible, controllable way. +`bevy_ecs` has no such abstractions for this, in favor of allowing end users to roll their own wrapping data structures as desired. + +For the most part, this is trivial to replicate externally: `App` is still very simple, the hard part is in getting the design right. +`AppCommands` are the only technically-challenging bit, but those are easy to replicate in a fixed fashion using an external control flow in the event that they are needed, or they can be stolen from `bevy_app` as needed. + ## Prior art [#16](https://github.com/bevyengine/rfcs/pull/16) covers a related but mutually incompatible sub-world proposal. @@ -166,6 +174,8 @@ The ability to clone schedules should give us almost everything you might want t 1. Can we get `App`-level storage of `AppCommandQueue` working properly? Very much the same problem as [bevy #3096](https://github.com/bevyengine/bevy/issues/3096) experienced with `Commands`. 2. Is the ability to clone entities between worlds critical / essential enough to solve [bevy #1515](https://github.com/bevyengine/bevy/issues/1515) as part of this feature (or before attempting it)? 3. Should `AppCommands` take a `&mut App`, or something better scoped to reduce borrow-check complexity? +4. How do we ensure that we can move entities and resources from one `World` to another outside of `bevy_app` without cloning? + 1. This is important to ensure that stand-alone `bevy_ecs` users can use this feature set smoothly. ## Future possibilities From de737c44aa271895e4620db32bedc4d9d49d7dbb Mon Sep 17 00:00:00 2001 From: Alice Date: Mon, 15 Nov 2021 12:47:54 -0500 Subject: [PATCH 04/11] Initial user-facing explanation --- rfcs/43-many-worlds.md | 88 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 72 insertions(+), 16 deletions(-) diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md index 46059ef2..e8616077 100644 --- a/rfcs/43-many-worlds.md +++ b/rfcs/43-many-worlds.md @@ -24,15 +24,57 @@ A more limited "sub-world" design (see #16 for further discussion) is more feasi ## User-facing explanation -TODO: write me. +When you need to fully isolate entities from each other (or make sure logic doesn't actually run on some chunk of them), you can create **multiple worlds** in your Bevy app. -Explain the proposal as if it was already included in the engine and you were teaching it to another Bevy user. That generally means: +At this point, you should be familiar with the basics of both `Worlds` and `Schedules`. +Your `App` can store any number of these, each corresponding to a particular `WorldLabel`. -- Introducing new named concepts. -- Explaining the feature, ideally through simple examples of solutions to concrete problems. -- Explaining how Bevy users should *think* about the feature, and how it should impact the way they use Bevy. It should explain the impact as concretely as possible. -- If applicable, provide sample error messages, deprecation warnings, or migration guidance. -- If applicable, explain how this feature compares to similar existing features, and in what situations the user would use each one. +During each pass of the game loop, each schedule runs in parallel, modifying the world it is assigned to. +These worlds are fully isolated: they cannot view or modify any other world, and stages progress independently across schedules. +Then, the **global world** (which stores common, read-only data) is processed, and any systems in its schedule are run. +Finally, at the end of each game loop, all of the worlds wait for a synchronization step, and apply any `AppCommands` that were sent. + +`AppCommands` allow you to communicate between worlds in various fashions. +You might: + +- `spawn_world` or `despawn_world`, to create and manage new worlds to shard your game or simulate hypothetical futures +- `add_schedule`, `clone_schedule`, `reassign_schedule` or `remove_schedule` to apply logic to these new worlds +- `move_resource`, `send_other_world_command` or `send_other_world_event` to communicate between worlds +- transfer entities between worlds with `move_entities` or `move_query` +- update the value of global resources that are shared between worlds using `insert_global` + +### WorldLabels + +Each world has its own label, which must implement the `WorldLabel` trait in the same fashion as the system labels you're already familiar with. + +By default, two worlds are added to your app: `CoreWorld::Global` and `CoreWorld::Main`. +By convention, the main world should contain the vast majority of your logic, while the global world should only contain global resources (and other read-only data) that need to be accessed across worlds. + +### Global resources + +Each app has its own collection of global resources, which can be viewed (but not modified) in each of the worlds. +This allows for convenient, non-blocking sharing of things like asset collections, settings and input events. + +If needed, you can get a read-only view into the entire global world using the `GlobalWorld` system parameter. +This is non-blocking, as the global world cannot be modified in any fashion while other schedules are active. + +### Example: parallel simulations + +TODO: write an example + +### Example: world-model AI + +TODO: write an example + +### Example: entity prefab staging world + +TODO: write an example + +### Custom strategies for multiple worlds + +You can modify the default strategy described above by setting up your own custom runner, which can be used to control the execution of schedules in arbitrary ways. + +Note that schedules are initialized on particular worlds, and will not run on a world with a mismatched `WorldLabel`. ## Implementation strategy @@ -42,16 +84,14 @@ Explain the proposal as if it was already included in the engine and you were te pub struct App { // WorldId is not very readable; we can steal labelling tech from systems pub worlds: HashMap, + // Each Schedule must have a `World`, but not all worlds need schedules + pub schedules: HashMap, // This is important to preserve the current ergonomics in the simple case pub default_world: WorldLabel, - // Each Schedule must have a `World`, but not all worlds need schedules - pub schedules: HashMap, + // This is needed in order to correctly hand out read-only access + pub global_world: WorldLabel, // Same as before pub runner: Box, - // Designed for storing GlobalRes - // Read-only access unless you have &mut App - // Also stores an `AppCommandQueue` in some fashion - pub global_world: World } ``` @@ -61,9 +101,10 @@ The standard main loop would have the following basic structure: 2. For each `WorldLabel` in `app.world.keys()`, initialize the corresponding schedule with the correct label. 3. For each world, run the corresponding schedule if it exists. 1. Each world is fully independent, and each scheduler can request more threads as needed in a greedy fashion. -4. At the end of each loop, put the entire `App` back together and run any `AppCommands`. +4. Run the global schedule. +5. At the end of each loop, put the entire `App` back together and run any `AppCommands`. 1. Be sure to apply standard `Commands` to each world. -5. Go to 2. +6. Go to 2. This should be programmed using the default (and minimal) runners. Custom schedule and synchronization behavior can be added with a custom runner. @@ -109,9 +150,24 @@ In order to avoid contention issues and scheduler entanglement, global resources Use `AppCommands::insert_global_resource` to modify them. They are accessible via `GlobalRes` system parameters in standard systems. -Under the hood, each world gets a `&` to these. +Under the hood, each other world gets a `&` to these. This simplifies system parameter dispatch, and allows them to be accessed in exclusive systems using `World::get_global_resource::`. +Global resources are stored in their own world, with a corresponding schedule and the `CoreWorld::Global` label (the default world gets `CoreWorld::Main`). +The default runner special-cases the execution order of our worlds, in order to safely hand out non-blocking read-only access to this world. + +### WorldLabel + +Steal implementation from system and stage labels. +This should probably be abstracted out into its own macro at this point. + +Remove `WorldId`. + +### Plugins + +The best stop-gap strategy for dealing with third-party dependencies in this design is going to be to `App::add_plugin_to_world(plugin: impl Plugin, label: impl WorldLabel)`. +This will change the default world for that plugin to the specified label. + ## Drawbacks - added complexity to the core design of `App` From d336d4e8df223580f3bc6f0d551efffe7655a9ab Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Mon, 15 Nov 2021 16:12:59 -0500 Subject: [PATCH 05/11] Basic examples --- rfcs/43-many-worlds.md | 192 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 176 insertions(+), 16 deletions(-) diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md index e8616077..6e54bfb4 100644 --- a/rfcs/43-many-worlds.md +++ b/rfcs/43-many-worlds.md @@ -17,6 +17,7 @@ Some of the more immediate use cases include: 4. A trivially correct structure for parallel world simulation for scientific simulation use cases. 5. Look-ahead simulation for AI, where an agent simulates a copy of the world using the systems without actually affecting game state. 6. Isolation of map chunks. +7. Roll-back networking. Currently, this flavor of design requires abandoning `bevy_app` completely, carefully limiting *every* system used (including base / external systems) or possibly some very advanced shenanigans with exclusive systems. @@ -37,7 +38,7 @@ Finally, at the end of each game loop, all of the worlds wait for a synchronizat `AppCommands` allow you to communicate between worlds in various fashions. You might: -- `spawn_world` or `despawn_world`, to create and manage new worlds to shard your game or simulate hypothetical futures +- `spawn_world` or `despawn_world`, to create and manage new worlds - `add_schedule`, `clone_schedule`, `reassign_schedule` or `remove_schedule` to apply logic to these new worlds - `move_resource`, `send_other_world_command` or `send_other_world_event` to communicate between worlds - transfer entities between worlds with `move_entities` or `move_query` @@ -60,17 +61,176 @@ This is non-blocking, as the global world cannot be modified in any fashion whil ### Example: parallel simulations -TODO: write an example +Suppose you're attempting to run a complex scientific simulation in parallel, and want to slightly vary the starting conditions or rules. +Because these simulations don't need to interact in any meaningful way, this is an ideal situation in which to use multiple worlds. -### Example: world-model AI +```rust +use bevy::prelude::*; + +#[derive(WorldLabel)] +struct SimulationWorld(usize); + + +// These systems will be run in each of ours imulation worlds +struct SimulationLogicPlugin; +impl Plugin for SimulationLogicPlugin{ + fn build(self, app: &mut App){ + app + .add_system(births_system) + .add_system(deaths_system) + .add_system(migration_system) + // This system will send events to CoreWorld::Global + // so we can collect the simulation state for reporting + .add_system(report_status_system); + } +} + +fn main(){ + + const N_WORLDS = 10; + let birth_rates: Vec = BirthRate::random(10); + let death_rates: Vec = DeathRate::random(10); + + let mut app = App::new(); + // These system is added to the global world's schedule, + // which will run after all of our simulation world + app.add_plugins(DefaultPlugins.to_world(CoreWorld::Global)) + // This system will collect information send by events from each + // report_status_system in our parallel worlds, + // allowing us to watch the simulation progress over time in our window + app.add_system(plot_simulations_system.to_world(CoreWorld::Global)); + + for i in 0..N_WORLDS { + // Create a new world + app.spawn_world(SimulationWorld(i)) + // Sets the currently active world for the App, + // which controls the schedule to which systems are added + // and where resources are inserted + .set_world(SimulationWorld(i)) + // This ensures basic functionality like `Time` functions properly + // for each of our worlds + .add_plugins(MinimalPlugins) + // Add the simulation configuration as resources + // We could have also passed this into our plugin + // using a method which returns a SimulationLogicPlugin + .insert_resource(birth_rates[i]) + .insert_resource(death_rates[i]) + // Contains all of our logic, + // and will run on each simulation world in parallel + .add_plugin(SimulationLogicPlugin); + } + + // The Main world isn't being used, so we may as well remove it + app.remove_world(CoreWorld::Main); + app.remove_schedule(CoreWorld::Main); + + // Now that everything is constructed, run the app! + app.run(); +} +``` + +### Example: entity prefab staging + +In this example, we're storing ready-made entities in our global world, and rapidly pulling them into our default world as needed. +This allows us to store complex, procedurally-generated entities in a reusable fashion, +without worrying that they will be accidentally broken by the game's logic. + +```rust +fn main(){ + App::new() + .add_plugins(DefaultPlugins) + // Events to spawn prefabs are generated by systems in here, + // and sent across worlds using AppCommands + .add_plugin(GamePlugin) + // This plugin will contain a number of startup systems + // which generate entities in complex ways + .add_plugin(PrefabInitializationPlugin.to_world(CoreWorld::Global)) + // This system will read prefab-spawning events sent to it from the main world + .add_system(spawn_prefabs_system.exclusive_system().to_world(CoreWorld::Global)) + .run(); +} + +struct SpawnPrefabEvent { + // You can store whatever information you'd like in these events + // in order to make looking up the right prefab easy + identifier: Identifier +} + +// An example system which uses the prefab system +fn spawn_marine(mut app_commands: AppCommands){ + let spawn_events = Events::::new(); + + app_commands.send_events(spawn_events, CoreWorld::Global); +} + +// This system allows us to read main-world SpawnPrefabEvents +// and send over the appropriate entities +fn spawn_prefabs_system(global_world: &mut World){ + // These were sent from the main world + let spawn_events = global_world.get_resource::>(); + // This handmade resource allows users to quickly look up entities by their identifier + let prefab_index = global_world.get_resource::(); + + // This convenience method is blocked by bevy #3096 + // but that problem is bascially orthogonal to this RFC + let app_command_queue = world.app_commands(); + + // Please pretend borrow-check is happy for the sake of discussion + // Fighting with it without a compiler is no fun :( + for spawn_event in spawn_events.iter(){ + let prefab_entity = prefab_index.get(event.identifier); + // Blocked by bevy #1515 but specific to this use case + let cloned_entity = global_world.clone_entity(prefab_entity).id(); + // Sends a copy of the prefab to the main world as requested + app_commands.move_entity(cloned_entity, CoreWorld::Global, CoreWorld::Main); + } +} +``` + +This particular set-up will result in a one frame delay for spawning prefabs. +On the first frame, the spawn event is sent from the main world to the global world. +On the second frame, the entity is cloned and sent back. + +If this is a deal-breaker, you could write a custom runner that has an additional sync point each frame. + +### Advanced example: world-model AI -TODO: write an example +In this example, we're using a multiple worlds approach in order to allow complex agents to simulate the world in order to plan ahead. +By taking this strategy, we can ensure that the agents always have an accurate model of the rules of the game without code duplication. -### Example: entity prefab staging world +```rust +#[derive(WorldLabel)] +struct GameRulesWorld; + +#[derive(WorldLabel)] +struct SimulationWorld(Uuid); + +fn main(){ + App::new() + // By storing all of our logic in its own world, we can quickly duplicate it + .spawn_world(GameRulesWorld) + .set_world(GameRulesWorld) + .add_plugin(GameLogicPlugin) + // We don't need or want all of the player-facing systems in our simulations + .set_world(CoreWorld::Main) + .add_plugins(DefaultPlugins) + .add_plugin(UiPlugin) + .add_plugin(RenderingPlugin) + .add_plugin(InputPlugin) + .run(); +} -TODO: write an example +// This system is contained within `GameLogicPlugin` +fn simulate_action(action: AiActions, app_commands: AppCommands){ + app_commands + .spawn_world(SimulationWorld::new()) -### Custom strategies for multiple worlds + // FIXME: figure out the crazy structure needed to make this actually work, + // or swap to a simpler example +} +``` + +### Advanced: custom strategies for multiple worlds You can modify the default strategy described above by setting up your own custom runner, which can be used to control the execution of schedules in arbitrary ways. @@ -87,10 +247,11 @@ pub struct App { // Each Schedule must have a `World`, but not all worlds need schedules pub schedules: HashMap, // This is important to preserve the current ergonomics in the simple case - pub default_world: WorldLabel, + // This is initialized to `CoreWorld::Main` + pub current_world: WorldLabel, // This is needed in order to correctly hand out read-only access pub global_world: WorldLabel, - // Same as before + // Exactly the same as before pub runner: Box, } ``` @@ -125,16 +286,20 @@ The initial set of `AppCommands` needed for this RFC are: - `spawn_world(label: impl WorldLabel)` - `despawn_world(label: impl WorldLabel)` +- `set_world(label: impl WorldLabel)` +- `clone_world(old_label: impl WorldLabel, new_label: impl WorldLabel)` - `new_schedule(label: impl WorldLabel, schedule: Schedule)` + - if systems are added to a world without an existing schedule, make one - `clone_schedule(old_label: impl WorldLabel, new_label: impl WorldLabel)` - `reassign_schedule(old_label: impl WorldLabel, new_label: impl WorldLabel)` - `remove_schedule(label: impl WorldLabel)` +- `move_entity(entities: Entity, origin_world: impl WorldLabel, destination_world: impl WorldLabel)` - `move_entities(entities: HashSet, origin_world: impl WorldLabel, destination_world: impl WorldLabel)` - `move_query::(origin_world: impl WorldLabel, destination_world: impl WorldLabel)` - moves every matching entity to the destination world - `move_resource::(origin_world: impl WorldLabel, destination_world: impl WorldLabel)` -- `other_world_events::(events: Events, destination_world: impl WorldLabel)` -- `other_world_commands(commands: Commands, destination_world: impl WorldLabel)` +- `send_events::(events: Events, destination_world: impl WorldLabel)` +- `send_commands(commands: Commands, destination_world: impl WorldLabel)` - `insert_global(res: impl Resource)` - Also used for mutating global resources - `init_global::()` @@ -163,11 +328,6 @@ This should probably be abstracted out into its own macro at this point. Remove `WorldId`. -### Plugins - -The best stop-gap strategy for dealing with third-party dependencies in this design is going to be to `App::add_plugin_to_world(plugin: impl Plugin, label: impl WorldLabel)`. -This will change the default world for that plugin to the specified label. - ## Drawbacks - added complexity to the core design of `App` From f6b3a3bd457512327f7aac7b3fcdec13259cda47 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Mon, 15 Nov 2021 16:59:59 -0500 Subject: [PATCH 06/11] Finished AI example --- rfcs/43-many-worlds.md | 64 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 59 insertions(+), 5 deletions(-) diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md index 6e54bfb4..4f68a932 100644 --- a/rfcs/43-many-worlds.md +++ b/rfcs/43-many-worlds.md @@ -217,16 +217,69 @@ fn main(){ .add_plugin(UiPlugin) .add_plugin(RenderingPlugin) .add_plugin(InputPlugin) + // We're sticking to a one-deep search for sanity here + .add_system(simulate_action_system) + .add_system(choose_action_system) .run(); } // This system is contained within `GameLogicPlugin` -fn simulate_action(action: AiActions, app_commands: AppCommands){ - app_commands - .spawn_world(SimulationWorld::new()) +fn simulate_action_system(potential_actions: Res, mut app_commands: AppCommands, sim_query: Query<((), With)>){ + // Create a new world for each potential action + for proposed_action in potential_actions.iter() { + // These identifiers are uniquely generated and ephemeral + let sim_world = SimulationWorld::new(); + + // Making a simulation world that's identical to the main world in all relevant ways + app_commands + .spawn_world(sim_world) + .clone_schedule(GameRulesWorld, sim_world) + // As above, blocked by bevy #1515 + .clone_entities_to_world(sim_query.entities(), sim_world) + // This system will evaluate the world state, send it back with an event reporting the values + .add_system(evaluate_action_system.to_world(sim_world)) + .send_event(proposed_action, sim_world); + } +} + +struct EvaluateAction { + proposed_action: Action, + score: Score, +} + +// This system somehow evaluates the goodness of the world state +// and then sends that information back to the main world +fn evaluate_action_system(mut proposed_actions: EventReader, + app_commands: AppCommands, score: Res, + current_world_label: CurrentWorld){ + + // Exactly one proposed action should be sent per world + let action: Action = proposed_actions.iter().next().unwrap().into(); + + let evaluation = EvaluateAction{ + action, + score: *score, + } + + // Report back to the main world + app_commands.send_event(evaluation, main_world); + // We're only looking ahead one step, so we're done with the world now + app_commands.despawn_world(current_world_label); +} + +// Run in the main world, once we've evaluated our actions +fn choose_action_system(mut eval_events: EventReader, mut action_events: EventWriter){ + + let mut best_action = Action::Pass; + let mut best_score = Score(0); + + for evaluation in eval_events.iter(){ + if evaluation.score >= best_score { + best_action = evaluation.action; + } + } - // FIXME: figure out the crazy structure needed to make this actually work, - // or swap to a simpler example + action_events.send(best_action); } ``` @@ -298,6 +351,7 @@ The initial set of `AppCommands` needed for this RFC are: - `move_query::(origin_world: impl WorldLabel, destination_world: impl WorldLabel)` - moves every matching entity to the destination world - `move_resource::(origin_world: impl WorldLabel, destination_world: impl WorldLabel)` +- `send_events::(event: E, destination_world: impl WorldLabel)` - `send_events::(events: Events, destination_world: impl WorldLabel)` - `send_commands(commands: Commands, destination_world: impl WorldLabel)` - `insert_global(res: impl Resource)` From 1c2308d60f606d151623cc6fe377d0e8e5daf866 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Tue, 16 Nov 2021 00:47:05 -0500 Subject: [PATCH 07/11] Swapped to planet-transport example, notes on pipelining --- rfcs/43-many-worlds.md | 142 ++++++++++++++++++++++++++--------------- 1 file changed, 90 insertions(+), 52 deletions(-) diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md index 4f68a932..c4d7b076 100644 --- a/rfcs/43-many-worlds.md +++ b/rfcs/43-many-worlds.md @@ -48,7 +48,7 @@ You might: Each world has its own label, which must implement the `WorldLabel` trait in the same fashion as the system labels you're already familiar with. -By default, two worlds are added to your app: `CoreWorld::Global` and `CoreWorld::Main`. +By default, three worlds are added to your app: `CoreWorld::Global`, `CoreWorld::Main` and `CoreWorld::Rendering`. By convention, the main world should contain the vast majority of your logic, while the global world should only contain global resources (and other read-only data) that need to be accessed across worlds. ### Global resources @@ -129,76 +129,104 @@ fn main(){ } ``` -### Example: entity prefab staging +### Example: chunking game state -In this example, we're storing ready-made entities in our global world, and rapidly pulling them into our default world as needed. -This allows us to store complex, procedurally-generated entities in a reusable fashion, -without worrying that they will be accidentally broken by the game's logic. +In this example, we're demonstrating the structure of a simple game that contains several planets, each modelled as their own `World`. +Planets cannot interact, but we can send entities between these worlds using `AppCommands`. ```rust +use bevy::prelude::*; + +// This is a dummy world label +// used to store the schedule that controls our logic +// which is replicated across our planets +#[derive(WorldId)] +struct GameLogic; + +// In this App, we're using the main world to run our menu UI +// and then running game logic on each of our planet worlds independently. +// The active world is set with the ActiveWorld global resource +// to correspond to fn main(){ App::new() - .add_plugins(DefaultPlugins) - // Events to spawn prefabs are generated by systems in here, - // and sent across worlds using AppCommands - .add_plugin(GamePlugin) - // This plugin will contain a number of startup systems - // which generate entities in complex ways - .add_plugin(PrefabInitializationPlugin.to_world(CoreWorld::Global)) - // This system will read prefab-spawning events sent to it from the main world - .add_system(spawn_prefabs_system.exclusive_system().to_world(CoreWorld::Global)) - .run(); + // Added to CoreWorld::Main by default + .add_plugin(MenuPlugin) + .set_world(CoreWorld::Global) + // We want a consistent unit of time across all of our worlds, so it's stored as a Global + .add_plugins(MinimalPlugins.to_world(CoreWorld::Global)) + // We need an initial world for the player to spawn in + .add_startup_system(generate_initial_world_system) + // Respond to NewWorld events + .add_system(new_world_system.to_world(Core)) + // Storing the game logic in a consistent place + .add_schedule(GameLogic) + .set_world(GameLogic) + .add_plugin(LogicPlugin) + .add_plugin(InputPlugin) + .add_plugin(AudioPlugin) + .run(); +} + +// Events that trigger our app to generate a new world +struct NewWorld; + +fn generate_initial_world_system(mut app_commands: AppCommands){ + // We need to send this event to CoreWorld::Global, + // since that's where new_world_system runs. + // We could use a custom AppCommands instead of this indirection with events + // but that is a bit more involved and harder to intercept in other systems + app_commands.send_event(NewWorld, CoreWorld::Global); } -struct SpawnPrefabEvent { - // You can store whatever information you'd like in these events - // in order to make looking up the right prefab easy - identifier: Identifier +// More NewWorld events are generated when we explore the galaxy +fn new_world_system(events: EventReader, mut app_commands: AppCommands){ + for event in events.iter(){ + app_commands + .spawn_world(event.world_label) + // Each world needs its own copy of the logic + // so then it can keep running when the player isn't looking + .clone_schedule(GameLogic, event.world_label); + } } -// An example system which uses the prefab system -fn spawn_marine(mut app_commands: AppCommands){ - let spawn_events = Events::::new(); +// This event is sent whenever a system +struct BeamMeUp{ + // The entities to move between worlds + entities: HashSet, + // This type should have a blanket impl for the underlying label, + // purely for convenience + target_world: Box, +} - app_commands.send_events(spawn_events, CoreWorld::Global); +/// Move entities between worlds +// This system lives in LogicPlugin, and is replicated across each planet's worlds +fn transport_between_planets_system(events: EventReader, app_commands: AppCommands, current_world: WorldLabel){ + for event in events.iter(){ + app_commands.move_entities(event.entities, current_world, event.target_world); + } } -// This system allows us to read main-world SpawnPrefabEvents -// and send over the appropriate entities -fn spawn_prefabs_system(global_world: &mut World){ - // These were sent from the main world - let spawn_events = global_world.get_resource::>(); - // This handmade resource allows users to quickly look up entities by their identifier - let prefab_index = global_world.get_resource::(); - - // This convenience method is blocked by bevy #3096 - // but that problem is bascially orthogonal to this RFC - let app_command_queue = world.app_commands(); - - // Please pretend borrow-check is happy for the sake of discussion - // Fighting with it without a compiler is no fun :( - for spawn_event in spawn_events.iter(){ - let prefab_entity = prefab_index.get(event.identifier); - // Blocked by bevy #1515 but specific to this use case - let cloned_entity = global_world.clone_entity(prefab_entity).id(); - // Sends a copy of the prefab to the main world as requested - app_commands.move_entity(cloned_entity, CoreWorld::Global, CoreWorld::Main); +/// Change which world entities are being fetched from for pipelined rendering and audio +// This system lives in LogicPlugin, and is replicated across each planet's worlds +// and is triggered when the player changes which world they want to watch via the UI +fn swap_planet_view(events: Events, mut app_commands: AppCommands, current_world: WorldLabel){ + if events.iter().next().is_some(){ + // Demonstrates a hypothetical integration with pipelined-rendering + // (and eventually audio), where we can swap which world the entities + // to visualize and "audioize" are coming from + app_commands.set_source_world(current_world); } } ``` -This particular set-up will result in a one frame delay for spawning prefabs. -On the first frame, the spawn event is sent from the main world to the global world. -On the second frame, the entity is cloned and sent back. - -If this is a deal-breaker, you could write a custom runner that has an additional sync point each frame. - ### Advanced example: world-model AI In this example, we're using a multiple worlds approach in order to allow complex agents to simulate the world in order to plan ahead. By taking this strategy, we can ensure that the agents always have an accurate model of the rules of the game without code duplication. ```rust +use bevy::prelude::*; + #[derive(WorldLabel)] struct GameRulesWorld; @@ -299,11 +327,17 @@ pub struct App { pub worlds: HashMap, // Each Schedule must have a `World`, but not all worlds need schedules pub schedules: HashMap, + // Controls where app // This is important to preserve the current ergonomics in the simple case - // This is initialized to `CoreWorld::Main` - pub current_world: WorldLabel, + // Defaults to `CoreWorld::Main` + pub current_world: Box, // This is needed in order to correctly hand out read-only access - pub global_world: WorldLabel, + pub global_world: Box, + // Which world the entities to be used for rendering and sound come from + // The relevant components will be automatically copied and moved + // to the appropriate worlds in a pipelined fashion + // Defaults to `CoreWorld::Main` + pub source_world: Box, // Exactly the same as before pub runner: Box, } @@ -340,6 +374,9 @@ The initial set of `AppCommands` needed for this RFC are: - `spawn_world(label: impl WorldLabel)` - `despawn_world(label: impl WorldLabel)` - `set_world(label: impl WorldLabel)` + - sets `App::current_world` +- `set_global_world` +- `set_source_world` - `clone_world(old_label: impl WorldLabel, new_label: impl WorldLabel)` - `new_schedule(label: impl WorldLabel, schedule: Schedule)` - if systems are added to a world without an existing schedule, make one @@ -464,3 +501,4 @@ For the most part, this is trivial to replicate externally: `App` is still very 6. Use of app commands to modify schedules, as explored in [bevy #2507](https://github.com/bevyengine/bevy/pull/2507). 1. We could *maybe* even pause the app, and then modify the runner. 7. Fallible `AppCommands` using the technology in [bevy #2241](https://github.com/bevyengine/bevy/pull/2241). +8. Worlds as a staging ground for scenes in a prefab workflow. From fc35cfe4462ae482392ddd222d89bd960dbb2788 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Tue, 16 Nov 2021 00:49:14 -0500 Subject: [PATCH 08/11] More unresolved questions --- rfcs/43-many-worlds.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md index c4d7b076..29d95910 100644 --- a/rfcs/43-many-worlds.md +++ b/rfcs/43-many-worlds.md @@ -483,6 +483,10 @@ For the most part, this is trivial to replicate externally: `App` is still very 3. Should `AppCommands` take a `&mut App`, or something better scoped to reduce borrow-check complexity? 4. How do we ensure that we can move entities and resources from one `World` to another outside of `bevy_app` without cloning? 1. This is important to ensure that stand-alone `bevy_ecs` users can use this feature set smoothly. +5. How can we improve the end user experience when working with custom runners? + 1. This problem already exists: modifying the `winit` strategy is very painful. + 2. Split between "schedule logic" and "interfacing logic"? +6. When writing runners, how precisely do we specify world sync points? ## Future possibilities From ab3fba5ee34130d31ab6f14f69bf67ad5fa630b3 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Tue, 16 Nov 2021 11:43:03 -0500 Subject: [PATCH 09/11] Typo fix Co-authored-by: Andreas Weibye <13300393+Weibye@users.noreply.github.com> --- rfcs/43-many-worlds.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md index 29d95910..d3b81c4b 100644 --- a/rfcs/43-many-worlds.md +++ b/rfcs/43-many-worlds.md @@ -436,7 +436,7 @@ Ownership semantics are substantially harder to code and reason about in a hiera They are significantly less elegant for several use cases (e.g. scientific simulation) where no clear hierarchy of worlds exists. -Things will also get [very weird](https://gatherer.wizards.com/pages/card/details.aspx?name=Shahrazad) if your sub-worlds are creating sub-weorlds (as is plausible in an AI design), compared to a flat structure. +Things will also get [very weird](https://gatherer.wizards.com/pages/card/details.aspx?name=Shahrazad) if your sub-worlds are creating sub-worlds (as is plausible in an AI design), compared to a flat structure. ### Why are schedules associated with worlds? From 93a103f3d3dcf67df50882126d874fb925f80978 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Tue, 16 Nov 2021 12:13:04 -0500 Subject: [PATCH 10/11] Remove sentence fragment --- rfcs/43-many-worlds.md | 1 - 1 file changed, 1 deletion(-) diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md index d3b81c4b..8bf6cf0f 100644 --- a/rfcs/43-many-worlds.md +++ b/rfcs/43-many-worlds.md @@ -441,7 +441,6 @@ Things will also get [very weird](https://gatherer.wizards.com/pages/card/detail ### Why are schedules associated with worlds? Each system must be initialized from the world it belongs to, in a reasonably expensive fashion. -Free-floating schedules, ### Why not store a unified `Vec<(World, Schedule)>`? From 556a95ed2d5b5edabf08a6ad385fe063556928a9 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Tue, 16 Nov 2021 13:19:08 -0500 Subject: [PATCH 11/11] Assorted polish --- rfcs/43-many-worlds.md | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/rfcs/43-many-worlds.md b/rfcs/43-many-worlds.md index 8bf6cf0f..ac8d0ab0 100644 --- a/rfcs/43-many-worlds.md +++ b/rfcs/43-many-worlds.md @@ -17,7 +17,7 @@ Some of the more immediate use cases include: 4. A trivially correct structure for parallel world simulation for scientific simulation use cases. 5. Look-ahead simulation for AI, where an agent simulates a copy of the world using the systems without actually affecting game state. 6. Isolation of map chunks. -7. Roll-back networking. +7. Temporarily disabling entities in a system-agnostic fashion. Currently, this flavor of design requires abandoning `bevy_app` completely, carefully limiting *every* system used (including base / external systems) or possibly some very advanced shenanigans with exclusive systems. @@ -417,7 +417,7 @@ The default runner special-cases the execution order of our worlds, in order to Steal implementation from system and stage labels. This should probably be abstracted out into its own macro at this point. -Remove `WorldId`. +Remove `WorldId` in favor of an internal `WorldLabel`, which is exposed as a system parameter so systems can tell which world they're running on. ## Drawbacks @@ -426,6 +426,10 @@ Remove `WorldId`. - makes the need for plugin configurability even more severe - sensible API is blocked on [system builder syntax](https://github.com/bevyengine/bevy/pull/2736) - until then, we *could* make methods for `add_system_set_to_stage_to_world`... +- `set_world` makes the `App`'s [order-dependence](https://github.com/bevyengine/bevy/issues/1255) stronger + - this is mitigated by the standard plugin architecture + - the API created by this is very easy to read and reason about + - this surface-level API should be easy to redesign if we decide to coherently tackle that problem - heavier use of custom runners increases the need for better reusability of the `DefaultPlugins` `winit` runner; this is a nuisance to hack on for the end user and custom runners for "classical games" are currently painful ## Rationale and alternatives @@ -435,27 +439,37 @@ Remove `WorldId`. Ownership semantics are substantially harder to code and reason about in a hierarchical structure than in a flat one. They are significantly less elegant for several use cases (e.g. scientific simulation) where no clear hierarchy of worlds exists. +In cases where this hierarchy does exist, the presence of a global world and schedule should handle the use cases well. Things will also get [very weird](https://gatherer.wizards.com/pages/card/details.aspx?name=Shahrazad) if your sub-worlds are creating sub-worlds (as is plausible in an AI design), compared to a flat structure. +Finally, the ergonomics, discoverability and potential for optimization of a custom-built API for this very expressive feature is much better than `Res`-based designs. + ### Why are schedules associated with worlds? Each system must be initialized from the world it belongs to, in a reasonably expensive fashion. +We should cache this, which means persistently associating the schedules which store our systems with worlds. ### Why not store a unified `Vec<(World, Schedule)>`? This makes our central, very visible API substantially less clear for beginners, and is a pain to work with. By moving to a label system, we can promote explicit, extensible designs with type-safe, human readable code. -In addition, this design prevents us from storing schedules which are not associated with any current world, which is annoying for some architectures which aggressively spawn and despawn worlds. +In addition, the tightly-coupled design makes the following relatively common tasks much more frustrating: + +- storing schedules that are not associated with any meaningful world, to later be cloned +- storing data in worlds that have no meaningful associated schedule, to be operated on using `AppCommands` +- swapping schedules between worlds ### Why can't we have multiple schedules per world? This makes the data access / storage model slightly more complex. -No pressing use cases have emerged for this, especially since schedules can be nested. -If one arises, swapping to a multimap should be easy. +More importantly, it raises the question of "how do we run multiple schedules on the world". +We could invent an entire DSL to describe all of the different strategies one might use, or force users to write custom runners that are schedule-aware. -The ability to clone schedules should give us almost everything you might want to do with this. +But, we already have that! This is the entire reasoning behind the ability to nest and compose schedules. +Those tools may be somewhat lacking (e.g. the ability to properly loop schedules or elegantly describe turn-based games), +but that problem should be addressed properly, not papered over. ## Why is this design at the `bevy_app` level, not `bevy_ecs`? @@ -486,6 +500,7 @@ For the most part, this is trivial to replicate externally: `App` is still very 1. This problem already exists: modifying the `winit` strategy is very painful. 2. Split between "schedule logic" and "interfacing logic"? 6. When writing runners, how precisely do we specify world sync points? +7. How do we preserve the validity of cross-entity references (relations) when transferring resources and components between worlds? ## Future possibilities