DAOS-18387 test: recovery/ddb.py test_recovery_ddb_ls MD-on-SSD Support

[shimizukko]

To support MD-on-SSD for ddb, we need to support two commands. ddb prov_mem and ddb ls with --db_path.

Update ddb_utils.py to support the new commands.

Add check_ram_used in recovery_utils.py to detect whether the system is MD-on-SSD.

Update test_recovery_ddb_ls to support MD-on-SSD with the new ddb commands.

We need to update the test yaml to run on MD-on-SSD/HW Medium, but that will break other tests in ddb.py because they don't support MD-on-SSD yet. Keep the original tests as ddb_pmem.py and ddb_pmem.yaml and keep running them on VM (except test_recovery_ddb_ls because that's updated in this PR).

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_recovery_ddb_ls DdbPMEMTest

Steps for the author:

• Commit message follows the guidelines.
• Appropriate Features or Test-tag pragmas were used.
• Appropriate Functional Test Stages were run.
• At least two positive code reviews including at least one code owner from each category referenced in the PR.
• Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

• Gatekeeper requested (daos-gatekeeper added as a reviewer).

[github-actions]

Ticket title is 'CR Test Update - recovery/ddb.py test_recovery_ddb_ls MD-on-SSD Support'
Status is 'In Progress'
Labels: 'catastrophic_recovery'
https://daosio.atlassian.net/browse/DAOS-18387

[daosbuild3]

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17332/1/display/redirect

[daosbuild3]

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17332/5/execution/node/857/log

[daosbuild3]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17332/5/execution/node/898/log

[shimizukko]

@phender @dinghwah
In Git diff, it looks like ddb_pmem.py and ddb_pmem.yaml are new, but they appear so because I removed test_recovery_ddb_ls from there and renamed the files by adding _pmem. Please focus on the rest of the files.

I have two questions:

1. In recovery_utils.py, I added check_ram_used to determine whether the system is MD-on-SSD. The logic is that if the test runs on HW Medium and the server config has ram field, it must be running on MD-on-SSD. Is this okay or is there any other better way?

2. On MD-on-SSD, we need to load/mount the pool dir to a new location. That location can be anywhere, but I chose /mnt/daos_load to make it consistent with the existing pattern. At the end of the test, I call umount and rm -rf on /mnt/daos_load. Is this okay?

Thanks.