feat: update inch options and url to support influxdb 3 core and improve batch production#47
feat: update inch options and url to support influxdb 3 core and improve batch production#47
Conversation
96f61ef to
d599cb2
Compare
|
|
||
| // generateBatches returns a channel for streaming batches. | ||
| func (s *Simulator) generateBatches() <-chan []byte { | ||
| ch := make(chan []byte, 10) |
There was a problem hiding this comment.
a channel of size 10 means 10 batches get buffered (naturally) but if the concurrency is higher, they might drain unsmoothly creating delays for the concurrent writers because a batch isn't available yet: the writing might stutter on inch's data generation size.
If the channel for buffering batches is slightly bigger than concurrency, the individual writeres should see a smoother pattern where the next batch is always available when it is finished with the last and ready to write the next batch (without waiting on generation).
This pr increases cpu usage but increasing the concurrent work batch generation and writing to the db under test. Also increases memory usage if concurrency is high, but really high concurrency might be better simulated with multiple inch instances running.
Adds an option to use the v3 native write endpoint so advanced control of write behavior can be used such as the "no-sync" option. Additionally, modifies the channel for buffered write batches to reflect concurrency setting, fixes an issue with retrying a write that was needlessly recompressing or copying the write batch, reduced allocations and copies by reusing buffers and letting bytes Buffer's take ownership of existing buffers. Updates golang to version 1.24 * closes #46
d599cb2 to
59d5235
Compare
Adds an option to use the v3 native write endpoint so advanced control of write behavior can be used such as the "no-sync" option.
Additionally, modifies the channel for buffered write batches to reflect concurrency setting, fixes an issue with retrying a write that was needlessly recompressing or copying the write batch, reduced allocations and copies by reusing buffers and letting bytes Buffer's take ownership of existing buffers.
Updates golang to version 1.24