Cap ServiceBusMessageBatch size at 1 MB to match broker enforcement#48214
Cap ServiceBusMessageBatch size at 1 MB to match broker enforcement#48214EldertGrootenboer wants to merge 1 commit intoAzure:mainfrom
Conversation
EldertGrootenboer
requested review from
a team and
skarri-microsoft
as code owners
There was a problem hiding this comment.
Pull request overview
Updates the Service Bus Java sender batching logic to enforce a hard 1 MB maximum batch size, aligning client-side behavior with broker enforcement when AMQP link max-message-size is overstated (notably Premium partitioned namespaces).
Changes:
- Introduces a 1 MB cap constant and applies it when computing effective batch sizing across batch creation and send/schedule pathways.
- Adds parameterized unit tests validating capping behavior when the link reports sizes above/below 1 MB and when user-specified batch options exceed the cap.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| sdk/servicebus/azure-messaging-servicebus/src/main/java/com/azure/messaging/servicebus/ServiceBusSenderAsyncClient.java | Adds MAX_BATCH_SIZE_BYTES and caps link-derived sizing in batch creation, scheduling, flux send batching, and the message collector. |
| sdk/servicebus/azure-messaging-servicebus/src/test/java/com/azure/messaging/servicebus/ServiceBusSenderAsyncClientTest.java | Adds parameterized tests to validate 1 MB cap behavior and option validation under both V1/V2 stacks. |
| new IllegalArgumentException(String.format(Locale.US, | ||
| "CreateMessageBatchOptions.getMaximumSizeInBytes (%s bytes) is larger than the link size" | ||
| + " (%s bytes).", | ||
| maxSize, maximumLinkSize))); |
There was a problem hiding this comment.
The new 1 MB cap means maximumLinkSize may no longer be the AMQP link size, but the exception message still says the requested size is "larger than the link size". This will be misleading when the link reports (e.g.) 100 MB but the effective limit is capped to 1 MB. Consider updating the message text to describe the effective maximum (and optionally include both the link-reported size and the 1 MB cap).
There was a problem hiding this comment.
Good catch. Updated the error message from "larger than the link size" to "larger than the maximum allowed size" since the effective limit may be the 1 MB batch cap rather than the AMQP link's reported size.
| // Temporary workaround: Service Bus enforces a maximum batch size of 1 MB that is not | ||
| // communicated via the AMQP link's max-message-size property. The link reports the per-message | ||
| // limit (up to 100 MB for Premium partitioned), but batches are rejected above 1 MB. |
There was a problem hiding this comment.
The comment describes this as a "maximum batch size" workaround, but the constant is also used as a cap for maxSendLinkSize when scheduling messages (which is a per-message encoding limit in ManagementChannel.schedule). Consider adjusting the comment (and/or constant name) to reflect that this is an effective broker-enforced max size applied to link size for multiple code paths, not just batching.
| // Temporary workaround: Service Bus enforces a maximum batch size of 1 MB that is not | |
| // communicated via the AMQP link's max-message-size property. The link reports the per-message | |
| // limit (up to 100 MB for Premium partitioned), but batches are rejected above 1 MB. | |
| // Temporary workaround: Service Bus enforces an effective maximum message payload size of 1 MB | |
| // that is not communicated via the AMQP link's max-message-size property. This limit is applied | |
| // to the link size in multiple code paths, including batch sends and scheduling (per-message | |
| // encoding via ManagementChannel.schedule), even though the link may report a higher | |
| // per-message limit (up to 100 MB for Premium partitioned). |
There was a problem hiding this comment.
Good observation. On closer analysis, the 1 MB limit is specifically for batch sends - individual messages on Premium namespaces can validly exceed 1 MB. The scheduleMessageInternal path sends a single message through the management channel and should NOT be capped at 1 MB (that would break Premium users scheduling large individual messages). scheduleMessages (plural) already goes through createMessageBatch() which has the cap.
Changes made:
- Removed the 1 MB cap from
scheduleMessageInternal(single-message path) - Updated the comment to clarify this is batch-specific and document which paths are/aren't capped
- Kept
MAX_BATCH_SIZE_BYTESname since the constant is batch-specific
EldertGrootenboer
force-pushed
the
fix/servicebus-batch-size-cap-ICM51000000793879
branch
from
40a0fac to
2cb1714
Compare
| = getSendLinkWithRetry("send-batches").flatMap(link -> link.getLinkSize().flatMap(size -> { | ||
| final int batchSize = size > 0 ? size : MAX_MESSAGE_LENGTH_BYTES; | ||
| final int batchSize = Math.min(size > 0 ? size : MAX_MESSAGE_LENGTH_BYTES, MAX_BATCH_SIZE_BYTES); | ||
| final CreateMessageBatchOptions batchOptions | ||
| = new CreateMessageBatchOptions().setMaximumSizeInBytes(batchSize); | ||
| return messages.collect( |
There was a problem hiding this comment.
sendFluxInternal() now caps batchSize to MAX_BATCH_SIZE_BYTES. Since both sendMessage(...) overloads call sendFluxInternal(Flux.just(message), ...), this change also caps single-message sends to 1 MB (because the message must fit into the ServiceBusMessageBatch used by AmqpMessageCollector). That contradicts the nearby comment stating only batch-sending paths are capped and will break sending individual messages larger than 1 MB when the link reports a higher per-message limit (e.g., Premium non-partitioned entities > 1 MB). Consider bypassing the batch-size cap for the single-message send path (e.g., send directly via the link) or otherwise allowing an oversized single message to be sent as a single transfer while still capping true AMQP batch transfers. Add a unit test that verifies a >1 MB single message is allowed when link size permits it.
There was a problem hiding this comment.
Addressed in the current code. The sendFluxInternal path no longer applies the 1 MB cap. The cap is enforced exclusively in createMessageBatch(), which means:
sendMessage(single)uses the raw link size (viasendFluxInternal→AmqpMessageCollector) — single messages > 1 MB are allowed on PremiumsendMessages(Iterable)usescreateMessageBatch()which caps at 1 MBscheduleMessage(single)uses the raw link size (scheduleMessageInternal)scheduleMessages(Iterable)usescreateMessageBatch()which caps at 1 MB
Added comprehensive tests proving this asymmetry, including tests with actual > 1 MB messages that succeed on the uncapped paths and fail on the capped paths.
| // This cap is applied only to batch-sending code paths (createMessageBatch, sendFlux, | ||
| // AmqpMessageCollector). Single-message paths (e.g. scheduleMessage) use the link-reported | ||
| // per-message limit, which is valid for individual messages on Premium namespaces. | ||
| // Tracked by: https://github.com/Azure/azure-service-bus/issues/708 | ||
| static final int MAX_BATCH_SIZE_BYTES = 1024 * 1024; |
There was a problem hiding this comment.
The PR description says scheduleMessageInternal() is capped to 1 MB, but the current implementation does not apply MAX_BATCH_SIZE_BYTES to the max size passed to managementNode.schedule(...) (it uses the link-reported size or MAX_MESSAGE_LENGTH_BYTES). Please update the PR description (or the code, if the description is the intended behavior) so they match.
There was a problem hiding this comment.
Good catch. The PR description was stale from an earlier iteration that capped all four paths. Updated the description to accurately reflect the single enforcement point design.
EldertGrootenboer
force-pushed
the
fix/servicebus-batch-size-cap-ICM51000000793879
branch
from
2cb1714 to
8de6ada
Compare
The Service Bus broker enforces a 1 MB batch size limit regardless of the max-message-size advertised on the AMQP link. Premium partitioned namespaces advertise 100 MB on the link, causing tryAddMessage() to accept batches the broker will reject. Cap batch creation in ServiceBusSenderAsyncClient.createMessageBatch() at 1 MB (MAX_BATCH_SIZE_BYTES). This is the single enforcement point: both sendMessages(iterable) and scheduleMessages(iterable) call createMessageBatch internally. Single-message paths (sendMessage, scheduleMessage) are NOT capped since the 1 MB limit is batch-specific and individual messages on Premium can validly exceed 1 MB up to the per-entity limit. When a user requests a batch size exceeding 1 MB via CreateMessageBatchOptions, throw ServiceBusException. Tracking: azure-service-bus#708 ICM: 51000000793879
EldertGrootenboer
force-pushed
the
fix/servicebus-batch-size-cap-ICM51000000793879
branch
from
8de6ada to
e3b2ef6
Compare
Description
The Service Bus broker enforces a 1 MB batch size limit regardless of the
max-message-sizeadvertised on the AMQP link. On Premium partitioned namespaces, the gateway advertises 100 MB on the link, causingtryAddMessage()to accept batches up to 100 MB that the broker will then reject.Root Cause
The AMQP gateway sets
max-message-sizeon the link based on namespace tier:The broker always enforces 1 MB for batches (
maxMessageSize = 1048576from deployed config). The vendor propertycom.microsoft:max-message-batch-sizeechoesmax-message-size(always equal), so reading it does not help.This is a known service-side gap tracked by azure-service-bus#708. The .NET SDK applies the same 1 MB cap (azure-sdk-for-net#44914).
Fix
Add
MAX_BATCH_SIZE_BYTES = 1024 * 1024(1 MB) with a single enforcement point increateMessageBatch():Capped paths (go through
createMessageBatch()):createMessageBatch()/createMessageBatch(options)-- explicit batch creationsendMessages(Iterable)-- internally callscreateMessageBatch()per batchscheduleMessages(Iterable)-- internally callscreateMessageBatch()for all messagesUncapped paths (intentionally bypass batch cap):
sendMessage(single)-- goes throughsendFluxInternal->AmqpMessageCollectorusing raw link sizescheduleMessage(single)-- goes throughscheduleMessageInternalusing raw link sizeSingle-message paths are intentionally uncapped since individual messages on Premium can validly exceed 1 MB up to the per-entity limit.
When a user requests a batch size exceeding 1 MB via
CreateMessageBatchOptions, aServiceBusExceptionis thrown (consistent with the existing behavior when exceeding the link size).Testing
8 new parameterized unit tests (x2 V1/V2 = 16 executions):
createBatchCappedAtMaxBatchSizeWhenLinkReportsLargerSize-- 100 MB link -> batch capped at 1 MBcreateBatchUsesLinkSizeWhenSmallerThanMaxBatchSize-- 256 KB link -> uses link sizecreateBatchWithOptionsExceedingMaxBatchSizeCapThrowsError-- user requests 2 MB -> errorcreateBatchWithOptionsSmallerThanMaxBatchSizeCapIsRespected-- user requests 500 KB -> respectedsendMessagesIterableWithLargeLinkCapsAt1MB-- iterable path uses capped batchsendSingleMessageNotCappedWithLargeLink-- single send path NOT cappedscheduleMessagesIterableWithLargeLinkCapsAt1MB-- schedule iterable uses capped batchscheduleMessageSingleNotCappedWithLargeLink-- single schedule NOT capped4 additional asymmetry-focused tests (x2 V1/V2 = 8 executions):
sendSingleMessageLargerThan1MBSucceedsWithLargeLink-- 2 MB message succeeds via single sendsendMessagesIterableRejectsSingleMessageLargerThan1MBOnLargeLink-- same 2 MB message fails via iterablescheduleMessagesIterableRejectsSingleMessageLargerThan1MBOnLargeLink-- same for schedule iterablecreateBatchFallbackWhenLinkReportsZeroSize-- link size 0 -> fallback to 256 KBAll 965 tests pass (0 failures, 0 errors). Spotless and checkstyle clean.
ICM Reference
ICM 51000000793879