Skip to content

[FIX] LocalFileClient with aiofiles for CI pipeline without minio#153

Merged
JonnyTran merged 8 commits intodevelopfrom
fix/localfilestorage-aiofiles
Sep 9, 2025
Merged

[FIX] LocalFileClient with aiofiles for CI pipeline without minio#153
JonnyTran merged 8 commits intodevelopfrom
fix/localfilestorage-aiofiles

Conversation

@JonnyTran
Copy link
Member

@JonnyTran JonnyTran commented Sep 9, 2025

This pull request refactors the S3 file handling logic in extralit_server/contexts/files.py to improve type safety, support S3 versioning, and streamline code. It introduces explicit typing using types-aiobotocore-s3, enhances object listing to include version history, and consolidates S3 operations for clarity and maintainability. Additionally, test setup and cleanup logic is updated for consistency.

S3 Client Typing and Dependency Improvements

  • Added the types-aiobotocore-s3 package to dependencies and updated S3 client usage throughout the codebase to use explicit "S3Client" type annotations for better type safety and IDE support. ([[1]](https://github.com/Extralit/extralit/pull/153/files#diff-53090ab35012181a50db3fa18151687645076c863af0553dbdac56bcb9e81b20R72), [[2]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L18-R18), [[3]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24R27-R36), [[4]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L48-R53))

S3 Object Versioning Support

  • Refactored the list_objects function to support listing all object versions and delete markers using list_object_versions, enabling more robust version history and recovery features. ([extralit-server/src/extralit_server/contexts/files.pyR179-R243](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24R179-R243))
  • Updated get_object and related functions to handle S3 object versioning by accepting an optional version_id and passing it to S3 API calls, ensuring correct retrieval of specific object versions. ([[1]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L230-R255), [[2]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L239-R273), [[3]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24R282))

S3 Operation Refactoring

  • Consolidated S3 operations by removing redundant helper functions (e.g., put_object_to_s3, delete_object_from_s3, create_bucket_in_s3) and replacing them with direct calls to the S3 client, simplifying the codebase and reducing indirection. ([[1]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L61-R72), [[2]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L88-L111), [[3]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L316-R351), [[4]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L326-R375), [[5]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L349-R389), [[6]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L390-R430))

Test Setup Consistency

  • Updated integration test setup and cleanup logic to use test_ (underscore) as the workspace, dataset, and user name prefix instead of test- (hyphen), ensuring consistency and avoiding potential naming conflicts. ([[1]](https://github.com/Extralit/extralit/pull/153/files#diff-b4dd08ddabe8f6274e63aa52c317d5739de4e022bf71f07996cd30d85d040638L27-R27), [[2]](https://github.com/Extralit/extralit/pull/153/files#diff-b4dd08ddabe8f6274e63aa52c317d5739de4e022bf71f07996cd30d85d040638L36-R46))

Bucket Creation and Versioning

  • Improved bucket creation logic to enable versioning by default using put_bucket_versioning, ensuring that new buckets support object versioning out-of-the-box. ([extralit-server/src/extralit_server/contexts/files.pyL326-R375](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L326-R375))

Let me know if you want to dive deeper into any of these changes!

@JonnyTran JonnyTran requested a review from a team as a code owner September 9, 2025 00:09
@JonnyTran JonnyTran self-assigned this Sep 9, 2025
…ce, including methods for bucket and object management, versioning, and metadata handling. Enhance create_s3_client to fallback to LocalFileClient if S3 is not configured.
@JonnyTran JonnyTran changed the title [FIX] localfilestorage with aiofiles [FIX] LocalFileClient with aiofiles for CI pipeline without minio Sep 9, 2025
…storage directory creation in extralit.yml; modify Dockerfile to install only postgresql dependencies.
…ncio-runner package and upgrade pytest-asyncio to version 1.1.0. Remove duplicate document tests from integration tests and refactor file search logic in LocalFileClient.
@JonnyTran JonnyTran requested a review from a team as a code owner September 9, 2025 00:54
… filtering logic for valid objects in API responses. Adjust integration tests to reflect new workspace naming convention.
@JonnyTran JonnyTran merged commit e2afbe3 into develop Sep 9, 2025
6 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments