[FIX] LocalFileClient with aiofiles for CI pipeline without minio#153
Merged
[FIX] LocalFileClient with aiofiles for CI pipeline without minio#153
aiofiles for CI pipeline without minio#153Conversation
…ce, including methods for bucket and object management, versioning, and metadata handling. Enhance create_s3_client to fallback to LocalFileClient if S3 is not configured.
aiofilesaiofiles for CI pipeline without minio
…age and remove MinIO service configuration
…storage directory creation in extralit.yml; modify Dockerfile to install only postgresql dependencies.
…ncio-runner package and upgrade pytest-asyncio to version 1.1.0. Remove duplicate document tests from integration tests and refactor file search logic in LocalFileClient.
… filtering logic for valid objects in API responses. Adjust integration tests to reflect new workspace naming convention.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request refactors the S3 file handling logic in
extralit_server/contexts/files.pyto improve type safety, support S3 versioning, and streamline code. It introduces explicit typing usingtypes-aiobotocore-s3, enhances object listing to include version history, and consolidates S3 operations for clarity and maintainability. Additionally, test setup and cleanup logic is updated for consistency.S3 Client Typing and Dependency Improvements
types-aiobotocore-s3package to dependencies and updated S3 client usage throughout the codebase to use explicit"S3Client"type annotations for better type safety and IDE support. ([[1]](https://github.com/Extralit/extralit/pull/153/files#diff-53090ab35012181a50db3fa18151687645076c863af0553dbdac56bcb9e81b20R72),[[2]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L18-R18),[[3]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24R27-R36),[[4]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L48-R53))S3 Object Versioning Support
list_objectsfunction to support listing all object versions and delete markers usinglist_object_versions, enabling more robust version history and recovery features. ([extralit-server/src/extralit_server/contexts/files.pyR179-R243](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24R179-R243))get_objectand related functions to handle S3 object versioning by accepting an optionalversion_idand passing it to S3 API calls, ensuring correct retrieval of specific object versions. ([[1]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L230-R255),[[2]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L239-R273),[[3]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24R282))S3 Operation Refactoring
put_object_to_s3,delete_object_from_s3,create_bucket_in_s3) and replacing them with direct calls to the S3 client, simplifying the codebase and reducing indirection. ([[1]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L61-R72),[[2]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L88-L111),[[3]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L316-R351),[[4]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L326-R375),[[5]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L349-R389),[[6]](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L390-R430))Test Setup Consistency
test_(underscore) as the workspace, dataset, and user name prefix instead oftest-(hyphen), ensuring consistency and avoiding potential naming conflicts. ([[1]](https://github.com/Extralit/extralit/pull/153/files#diff-b4dd08ddabe8f6274e63aa52c317d5739de4e022bf71f07996cd30d85d040638L27-R27),[[2]](https://github.com/Extralit/extralit/pull/153/files#diff-b4dd08ddabe8f6274e63aa52c317d5739de4e022bf71f07996cd30d85d040638L36-R46))Bucket Creation and Versioning
put_bucket_versioning, ensuring that new buckets support object versioning out-of-the-box. ([extralit-server/src/extralit_server/contexts/files.pyL326-R375](https://github.com/Extralit/extralit/pull/153/files#diff-15da807f093c7a943d28757b744f9c15a6c3e6a3f01bde8035bb6e057e5a7c24L326-R375))Let me know if you want to dive deeper into any of these changes!