Pilot 9050: fixup the preupload timeout causing resume issue

[colorzzr]

Summary

After investigation of this issue, it is caused by network timeout during the pre-upload in python version. The resumable json will treat the timeout batch AS unregistered, while backend already registered them(This issue only happened when timeout). Then mismatching happens and block resumable upload. Another PR will increase the timeout in bff-cli service. This PR is meant to fix the mismatching issue:

1. using newly v2 api to get items by status. If any registered item matches with manifest info, it will proceed as resumable upload.
   a. old cli version will still use v1 endpoint.
2. fix up the manifest missing when resumable upload.

api updates:

1. https://github.com/PilotDataPlatform/bff-cli/pull/173
2. https://github.com/PilotDataPlatform/metadata/pull/365
3. https://github.com/PilotDataPlatform/bff-web/pull/796

JIRA Issues

Pilot 9050

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Refactor or reformatting

Testing

Are there any new or updated tests to validate the changes?

• Yes
• No

Test Directions

unittest:

• adding new test cases for duplication check and resumable manifest.
  manual testing:
• for mismatching, we can manually alter the manifest.json to create mismatching, moving registered items into unregistered fields.
• for resumable manifest issue, ctrl+c the resume process. the manifest should contains correct number of informations

Versions

• Pilot Release Version: 2.18.0
• Compatible Version: 2.18.0

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	0	0	100%
pilotcli.py	0	59	0%	5, 7–10, 12–16, 19–20, 22–23, 25–26, 28–32, 34–37, 39, 41–42, 44–45, 48, 51–57, 59, 61–64, 66–71, 74–80, 83–85
commands
__init__.py	0	0	100%
container_registry.py	0	19	59%	16, 22–24, 31–33, 41–44, 50–53, 62–65
dataset.py	0	9	89%	20, 38–39, 47, 82–83, 92, 126, 130
entry_point.py	0	11	81%	36, 42, 70, 93–95, 97–101
file.py	0	38	86%	45, 156–159, 163, 168, 171–173, 304–311, 313, 315, 329–335, 337, 366, 368, 372, 418, 421, 436–438, 444–445
folder.py	0	1	96%	18
project.py	0	2	95%	18, 61
user.py	0	9	81%	28, 50–51, 59–60, 68–69, 90–91
configs
__init__.py	0	0	100%
app_config.py	0	0	100%
config.py	0	0	100%
user_config.py	0	23	81%	33, 73, 82, 93, 126, 135–136, 139, 149, 155, 159, 163, 167, 171–172, 176, 180–181, 185, 189–190, 198, 202
utils.py	0	35	36%	13–14, 16–19, 22–23, 25, 27, 31–32, 34, 36–37, 40, 43, 45–47, 49–51, 54–57, 59, 67, 69–72, 74–75
models
__init__.py	0	0	100%
enums.py	0	0	100%
item.py	0	0	100%
service_meta_class.py	0	1	80%	9
singleton.py	0	0	100%
upload_form.py	0	4	33%	28, 40–42
resources
custom_error.py	0	0	100%
custom_help.py	0	0	100%
services
__init__.py	0	0	100%
services/clients
__init__.py	0	0	100%
base_auth_client.py	0	1	96%	53
base_client.py	0	2	95%	49, 100
services/container_registry_manager
container_registry_manager.py	0	108	16%	18–19, 22–25, 28–32, 35–43, 47–59, 61–63, 67–81, 83–85, 89–98, 100–102, 106–109, 124–135, 139–142, 146–161, 163, 165–168
services/crypto
__init__.py	0	0	100%
crypto.py	0	19	40%	35, 43–47, 49, 59–61, 69–75, 77–78
services/dataset_manager
dataset_detail.py	0	22	70%	37, 58, 65–70, 72–74, 76–85, 87
dataset_download.py	0	22	81%	51, 61–63, 72–74, 90, 92–93, 95–96, 98, 100–102, 110–113, 147, 162
dataset_list.py	0	8	79%	32–37, 39, 48
model.py	0	0	100%
services/file_manager
__init__.py	0	0	100%
file_list.py	0	12	84%	58–64, 84–86, 98, 100
file_manifests.py	0	25	78%	69, 180–194, 196, 201–205, 207–209
file_tag.py	0	40	28%	22–23, 27–31, 33, 36–48, 50–52, 54–55, 59–61, 64–68, 70–73, 75–76
services/file_manager/file_download
__init__.py	0	0	100%
download_client.py	0	125	57%	52–61, 79–80, 125–126, 128–130, 142–143, 148–149, 153–156, 158–161, 163–168, 171, 173–174, 176–178, 181, 183, 186, 190, 193–194, 196, 200, 214–216, 222–223, 232–233, 235–237, 277, 279, 295, 315–316, 321–331, 333–334, 337–338, 343–344, 348–353, 357–358, 361–363, 365–376, 383, 388, 403, 407–409, 411–424, 426
model.py	0	0	100%
services/file_manager/file_metadata
__init__.py	0	0	100%
file_metadata_client.py	0	3	95%	90–92
folder_client.py	0	4	91%	49, 78–80
services/file_manager/file_move
__init__.py	0	0	100%
file_move_client.py	0	37	57%	57–61, 63, 71, 76–80, 83–91, 93, 95–100, 111, 113–117, 119, 169–170
services/file_manager/file_trash
__init__.py	0	0	100%
file_trash_client.py	0	2	96%	65, 86
utils.py	0	3	91%	35, 59, 66
services/file_manager/file_upload
__init__.py	0	0	100%
exception.py	0	0	100%
file_upload.py	0	18	91%	94–96, 108, 136, 170–172, 420–421, 424–426, 428, 435, 456, 459, 463
models.py	0	2	96%	20, 40
upload_client.py	0	49	77%	99–102, 142, 226, 239–242, 244–257, 259, 261–265, 267–271, 273–274, 377–378, 380, 450, 456–457, 462–465, 467–468
upload_validator.py	0	13	71%	51–55, 57, 61–63, 66–67, 72, 74
services/logger_services
__init__.py	0	0	100%
debugging_log.py	0	0	100%
log_functions.py	0	0	100%
services/output_manager
__init__.py	0	0	100%
error_handler.py	0	0	100%
help_page.py	0	7	91%	13–17, 19, 21
message_handler.py	0	54	74%	24, 46, 51, 66–68, 73, 101, 106, 113, 118, 123, 127, 143, 178, 188, 197, 215, 237, 281–292, 303–308, 310–316, 327, 331–332, 336–337, 339–340, 344, 348, 352
services/project_manager
__init__.py	0	0	100%
project.py	0	6	83%	38, 44–45, 47–49
services/user_authentication
__init__.py	0	0	100%
decorator.py	0	2	93%	25, 28
token_manager.py	0	8	88%	41–45, 72–73, 90
user_login_logout.py	0	16	83%	25–26, 37, 122, 126–132, 134–136, 140–141
utils
__init__.py	0	0	100%
aggregated.py	0	21	88%	51, 61–63, 77–79, 111, 125–126, 139, 169–170, 172, 201, 209, 226–227, 271–272, 299
TOTAL	3668	840	77%

[Copilot]

Pull request overview

This PR fixes a resumable upload issue caused by network timeouts during pre-upload operations. When timeouts occur, the local manifest can become mismatched with the backend state, treating already-registered items as unregistered.

Changes:

• Updated duplication check API from v1 to v2 endpoint to separately identify ACTIVE vs REGISTERED items
• Fixed manifest key inconsistency (changed from local_path to object_path for unregistered items)
• Added logic to reconcile registered items found on backend with local manifest during resume
• Updated Python version support from 3.10-3.11 to 3.10-3.13
• Consolidated imports and updated copyright years to 2026

Reviewed changes

Copilot reviewed 100 out of 101 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pyproject.toml	Version bump to 3.21.0 and Python version range update
poetry.lock	Dependency updates for new Poetry version and Python support
app/services/file_manager/file_upload/upload_client.py	API v2 migration, three-way return for duplication check
app/services/file_manager/file_upload/file_upload.py	Resume logic to handle registered items mismatch
app/services/file_manager/file_upload/exception.py	Added proper exception message to super().init
tests/app/services/file_manager/file_upload/test_upload_client.py	Updated tests for new three-way return signature
tests/app/services/file_manager/file_upload/test_file_upload.py	Added comprehensive tests for manifest batch processing
Multiple files	Copyright year updates and import consolidation

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[colorzzr]

also tested with batch size 50 with folder size of 10 to 1000.