Edit: Confirmed when using the callback that you wrote for removing a colony (hitting "remove" in GUI), it will sometimes crash if trying to remove a colony with a population of 0.
In the update() function in Simulation, I have a check to see if any colonies have a population of 0 ants. If there are 0 ants, removeColony() is called and the colony is removed from the world, as well as being removed under the hood, from the colonies vectors in Simulation, world, and renderer.
Strangely, it also only appears to happen when using colonies with somewhat large starting populations (>50?). If you try testing with small colony sizes, it may not occur. Additionally, I believe I have only one instance of crashing when using the actual "remove" button in the GUI (compared to crashes happening every single time otherwise).
It seems like it is either a problem with the index_vector.hpp class, or maybe the target field in Ant ...? I have been investigating this problem all day with no luck. I'll make a comment below that goes into greater detail.