HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated #9580

jojochuang · 2026-01-03T14:21:25Z

What changes were proposed in this pull request?

HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated

Please describe your PR in detail:

Similar to HDDS-14199 but for Ratis pipelines.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13579

How was this patch tested?

Doc only.

szetszwo · 2026-01-03T21:08:51Z

@jojochuang , thanks for working on this!

Could you limit the line length to 120 characters like the code? It is easier to comment on it.

Discovered and corrected the documentation for how the number of EC pipelines is calculated. The previous analysis was incorrect. - `ErasureCoding.md` is updated to describe the two new properties `ozone.scm.ec.pipeline.minimum` and `ozone.scm.ec.pipeline.per.volume.factor` and the `max()` logic used to determine the target number of pipelines. - `ProductionDeployment.md` is updated to reference the correct and existing configuration property for tuning EC pipelines. Change-Id: I393dc60d8745da2b2bb7899530665a108956446d

Change-Id: I0ec667c21155436eb6a0654782b43b48636f75d5

jojochuang · 2026-01-04T00:49:41Z

thanks for review. updated.

ashishkumar50

@jojochuang Thanks for writing this up.

ashishkumar50 · 2026-01-05T08:13:01Z

hadoop-hdds/docs/content/feature/multi-raft-support.md

+recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the
+global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for
+`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the `NumOpenPipelines` metric in SCM to see if the
+actual number of pipelines aligns with your configured targets.


I think NumOpenPipelines metrics doesn't exist. We can either use admin command to see number of open pipelines or may use recon as well to see the open pipelines.

To clarify, the SCM web UI has a section Pipeline Statistics, and it pulls the metrics from:

{ "name": "Hadoop:service=SCMPipelineManager,name=SCMPipelineManagerInfo", "modelerType": "org.apache.hadoop.hdds.scm.pipeline.PipelineManagerImpl", "PipelineInfo": [ { "key": "CLOSED", "value": 0 }, { "key": "ALLOCATED", "value": 0 }, { "key": "OPEN", "value": 1 }, { "key": "DORMANT", "value": 0 } ] },

Change-Id: I2537905761cf45d23cdb3701b2f0c94e7ff2485a

szetszwo

@jojochuang , thanks for working on this!

The code is quite distributed so it is not easy to understand the exact calculation. Please see the comments inlined (not yet reviewed the example).

szetszwo · 2026-01-07T16:26:03Z

hadoop-hdds/docs/content/feature/multi-raft-support.md


+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum


... FACTOR_THREE ...

Let's use ReplicationFactor.THREE, which is a client API.

... three properties ... that define the maximum

three configuration properties ... that limit the

szetszwo · 2026-01-07T16:31:53Z

hadoop-hdds/docs/content/feature/multi-raft-support.md

+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.


... a cluster-wide level, datanode level, and metadata disk level, respectively.

a cluster-wide level and a datanode level.

szetszwo · 2026-01-07T16:37:38Z

hadoop-hdds/docs/content/feature/multi-raft-support.md

+1.  **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+    *   **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+        across the entire cluster. This acts as a final cap on the total number of pipelines.
+    *   **Default Value**: `0` (which means no global limit is enforced by default).


... no global limit is enforced by default).

no global limit by default).

szetszwo · 2026-01-07T17:05:45Z

hadoop-hdds/docs/content/feature/multi-raft-support.md

+2.  **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+    *   **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+        every datanode. This is one of two ways to calculate a cluster-wide target.
+    *   **Default Value**: `2`


This is tricky since we have a bug; see HDDS-14369.

szetszwo · 2026-01-07T17:07:02Z

hadoop-hdds/docs/content/feature/multi-raft-support.md

+    *   **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
+
+3.  **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
+    *   **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.


... This property is used ... is explicitly set to 0.

This property takes effect ... is not set to a positive number.

szetszwo · 2026-01-07T17:13:41Z

hadoop-hdds/docs/content/feature/multi-raft-support.md

+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.


Replace this sentence with

"The number of pipelines created by SCM is restricted by these limits."

szetszwo · 2026-01-07T17:26:56Z

hadoop-hdds/docs/content/feature/multi-raft-support.md

+SCM will create pipelines until the most restrictive limit is met.
+
+1.  **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+    *   **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines


Remove FACTOR_THREE since the code actually includes ReplicationFactor.ONE pipeline; see

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java

Lines 123 to 128 in 4fd8360

if (pipelineNumberLimit > 0) {

int factorOnePipelineCount = pipelineStateManager

.getPipelines(RatisReplicationConfig.getInstance(ReplicationFactor.ONE)).size();

int allowedOpenPipelines = pipelineNumberLimit - factorOnePipelineCount;

return openPipelines >= allowedOpenPipelines;

}

szetszwo · 2026-01-07T17:29:16Z

hadoop-hdds/docs/content/feature/multi-raft-support.md

+    *   **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+        every datanode. This is one of two ways to calculate a cluster-wide target.
+    *   **Default Value**: `2`
+    *   **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.


* **Cluster-wide Limit Calculation**: If this property is set, the number of pipelines in the cluster is in addition limited by `(<this value> * <number of healthy datanodes>) / 3`.

(revised)

szetszwo · 2026-01-07T17:38:49Z

hadoop-hdds/docs/content/feature/multi-raft-support.md

+    *   **Calculation**: The limit for each datanode is
+        `(<this value> * <number of metadata disks on that datanode>)`.
+        The total cluster-wide target is the sum of all individual datanode limits, divided by 3.


Remove this section since it seems not true (or I cannot find the code enforcing it). I do see that the value is used for filtering datanodes.

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java

Lines 104 to 131 in 4fd8360

private boolean exceedPipelineNumberLimit(RatisReplicationConfig replicationConfig) {

// Apply limits only for replication factor THREE

if (replicationConfig.getReplicationFactor() != ReplicationFactor.THREE) {

return false;

}

PipelineStateManager pipelineStateManager = getPipelineStateManager();

int totalActivePipelines = pipelineStateManager.getPipelines(replicationConfig).size();

int closedPipelines = pipelineStateManager.getPipelines(replicationConfig, PipelineState.CLOSED).size();

int openPipelines = totalActivePipelines - closedPipelines;

// Check per-datanode pipeline limit

if (maxPipelinePerDatanode > 0) {

int healthyNodeCount = getNodeManager()

.getNodeCount(NodeStatus.inServiceHealthy());

int allowedOpenPipelines = (maxPipelinePerDatanode * healthyNodeCount)

/ replicationConfig.getRequiredNodes();

return openPipelines >= allowedOpenPipelines;

}

// Check global pipeline limit

if (pipelineNumberLimit > 0) {

int factorOnePipelineCount = pipelineStateManager

.getPipelines(RatisReplicationConfig.getInstance(ReplicationFactor.ONE)).size();

int allowedOpenPipelines = pipelineNumberLimit - factorOnePipelineCount;

return openPipelines >= allowedOpenPipelines;

}

// No limits are set

return false;

}

szetszwo · 2026-01-07T17:43:01Z

hadoop-hdds/docs/content/feature/multi-raft-support.md

+
+2.  **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+    *   **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+        every datanode. This is one of two ways to calculate a cluster-wide target.


Remove the sentence "This is one of two ways to calculate a cluster-wide target." since the Dynamic Limit
is not used in RatisPipelineProvider.exceedPipelineNumberLimit(..).

jojochuang requested review from ashishkumar50, siddhantsangwan, sodonnel, sumitagrawl and szetszwo January 3, 2026 14:21

jojochuang added 2 commits January 3, 2026 16:49

Limit to 120 characters

c4deea7

Change-Id: I0ec667c21155436eb6a0654782b43b48636f75d5

jojochuang force-pushed the HDDS-13579 branch from 199bd3c to c4deea7 Compare January 4, 2026 00:49

adoroszlai added the documentation Improvements or additions to documentation label Jan 5, 2026

ashishkumar50 reviewed Jan 5, 2026

View reviewed changes

Update text for monitoring number of pipelines.

f8060d9

Change-Id: I2537905761cf45d23cdb3701b2f0c94e7ff2485a

jojochuang requested a review from ashishkumar50 January 7, 2026 17:36

szetszwo reviewed Jan 7, 2026

View reviewed changes


		### Calculating Ratis Pipeline Limits

		The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum

	if (pipelineNumberLimit > 0) {
	int factorOnePipelineCount = pipelineStateManager
	.getPipelines(RatisReplicationConfig.getInstance(ReplicationFactor.ONE)).size();
	int allowedOpenPipelines = pipelineNumberLimit - factorOnePipelineCount;
	return openPipelines >= allowedOpenPipelines;
	}

	private boolean exceedPipelineNumberLimit(RatisReplicationConfig replicationConfig) {
	// Apply limits only for replication factor THREE
	if (replicationConfig.getReplicationFactor() != ReplicationFactor.THREE) {
	return false;
	}

	PipelineStateManager pipelineStateManager = getPipelineStateManager();
	int totalActivePipelines = pipelineStateManager.getPipelines(replicationConfig).size();
	int closedPipelines = pipelineStateManager.getPipelines(replicationConfig, PipelineState.CLOSED).size();
	int openPipelines = totalActivePipelines - closedPipelines;
	// Check per-datanode pipeline limit
	if (maxPipelinePerDatanode > 0) {
	int healthyNodeCount = getNodeManager()
	.getNodeCount(NodeStatus.inServiceHealthy());
	int allowedOpenPipelines = (maxPipelinePerDatanode * healthyNodeCount)
	/ replicationConfig.getRequiredNodes();
	return openPipelines >= allowedOpenPipelines;
	}
	// Check global pipeline limit
	if (pipelineNumberLimit > 0) {
	int factorOnePipelineCount = pipelineStateManager
	.getPipelines(RatisReplicationConfig.getInstance(ReplicationFactor.ONE)).size();
	int allowedOpenPipelines = pipelineNumberLimit - factorOnePipelineCount;
	return openPipelines >= allowedOpenPipelines;
	}
	// No limits are set
	return false;
	}

HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated #9580

Are you sure you want to change the base?

HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated #9580

Conversation

jojochuang commented Jan 3, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

szetszwo commented Jan 3, 2026

Uh oh!

jojochuang commented Jan 4, 2026

Uh oh!

ashishkumar50 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

szetszwo Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

szetszwo Jan 7, 2026 •

edited

Loading

szetszwo Jan 7, 2026 •

edited

Loading

szetszwo Jan 7, 2026 •

edited

Loading