[StreamManager] [KinesisConfig] upload acts as default batch_size is 500

[bpawlik]

When uploading to Kinesis Data Streams with StreamManager, uploading happens in batches of 500 instead of immediately, even though batch_size value in ExportDefinition:KinesisConfig is not specified. According to documentation:

batch_size: The maximum size of a batch to send to Kinesis. Messages will be queued until the batch size is reached, after which they will then be uploaded. If unspecified the default will be 1.

Setting batch_size to 1, solves the issue, i.e. data is uploaded as it comes to the stream. Did I misread the documentation or is it a bug?

[MikeDombo]

Thank you for bringing this to our attention you are correct that the default is not 1, but 500 (the maximum batch size). We will take this feedback and update the documentation to reflect this as soon as possible.

[bartek-aws]

Thanks for confirming the bug. Could you please keep this issue open until the fix to the documentation is merged with master?