Implement async upload methods and support async mode in ExtendedGcsfs #734
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Fixing Block size and consistency options in Extended GCSFS Open (#34) * Separate versioned and non-versioned tests to use different bucket * Update cleanup logic in tests to empty the bucket instead of deleting the bucket * Add consistency and blocksize for ext gcsfs * Merge conflicts resolution * bucket type paramet for fs test for block size and consistency * removed unused mocks variables from some tests * fixing lint errors * fixed small issue with core tests so it can run with exp flag true --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com> * pytest microbenchmarks for seq and random reads both single multi threaded * multiprocess benchmarks * script to run tests * undo settings for bucket names and logs * benchmark script updates * file size and bucket type decorators * file size configuration * removed zonal config * Added README * Readme update * Moving settings and fixture to tests root * Readme update * Readme update * Ignore benchmark pytests in CI * benchmark hook fix * adding skip tests flag * benchmark plugin conditional enablement * Fixing PR Comments, simplifying the configuration by doing auto gen * Fixing PR Comments * default settings --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com>
* Fixing Block size and consistency options in Extended GCSFS Open (#34) * Separate versioned and non-versioned tests to use different bucket * Update cleanup logic in tests to empty the bucket instead of deleting the bucket * Add consistency and blocksize for ext gcsfs * Merge conflicts resolution * bucket type paramet for fs test for block size and consistency * removed unused mocks variables from some tests * fixing lint errors * fixed small issue with core tests so it can run with exp flag true --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com> * pytest microbenchmarks for seq and random reads both single multi threaded * multiprocess benchmarks * script to run tests * undo settings for bucket names and logs * benchmark script updates * file size and bucket type decorators * file size configuration * removed zonal config * Added README * Readme update * Moving settings and fixture to tests root * Readme update * Readme update * Ignore benchmark pytests in CI * benchmark hook fix * adding skip tests flag * benchmark plugin conditional enablement * Fixing PR Comments, simplifying the configuration by doing auto gen * Fixing PR Comments * default settings * added resource monitoring for benchmarks * minor refactoring * moved config to yaml * lint fixes * config yaml update for full read suite * undo zonal file logging changes * simplify single threaded read * bringing back requirements * update readme * csv generation fix when some tests fail * psutil install in cloudbuild * psutil install in cloudbuild * Removing zonal conditional code and updating config --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com>
* Fixing Block size and consistency options in Extended GCSFS Open (#34) * Separate versioned and non-versioned tests to use different bucket * Update cleanup logic in tests to empty the bucket instead of deleting the bucket * Add consistency and blocksize for ext gcsfs * Merge conflicts resolution * bucket type paramet for fs test for block size and consistency * removed unused mocks variables from some tests * fixing lint errors * fixed small issue with core tests so it can run with exp flag true --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com> * pytest microbenchmarks for seq and random reads both single multi threaded * multiprocess benchmarks * script to run tests * undo settings for bucket names and logs * benchmark script updates * file size and bucket type decorators * file size configuration * removed zonal config * Added README * Readme update * Moving settings and fixture to tests root * Readme update * Readme update * Ignore benchmark pytests in CI * benchmark hook fix * adding skip tests flag * benchmark plugin conditional enablement * Fixing PR Comments, simplifying the configuration by doing auto gen * Fixing PR Comments * default settings * added resource monitoring for benchmarks * minor refactoring * moved config to yaml * lint fixes * config yaml update for full read suite * undo zonal file logging changes * simplify single threaded read * bringing back requirements * update readme * csv generation fix when some tests fail * psutil install in cloudbuild * psutil install in cloudbuild * Removing zonal conditional code and updating config * parallel file creation in setup * merge conflicts * merge conflicts - lint fixes * lint issues --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com>
create new zonal bucket for test
… to avoid hitting object mutation limit fix test_upload_chunk_zonal_exception_cleanup getting skipped on mock gcs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Zonal Bucket Write Support in ExtendedGCSFileSystem:
_put_file,_pipe_file,upload_chunk,initiate_upload, andsimple_upload._put_file,_pipe_file, andsimple_uploadto accept afinalize_on_closeargument.upload_chunkto automatically finalize the file once the specified size is reached.ExtendedGCSFileSystemto use lazy initialization for the gRPC client._cat_fileto ensure all logic (including MRD creation) is wrapped within thetry...finallyblock for safer cleanup.ZonalFile Improvements:
ZonalFileto prevent double-closing theAsyncAppendableObjectWriterif the object is already finalized.Testing Infrastructure:
conftest.pyfor better accessibility across test files.zonal_write_mocksto returnNonewhen running against real GCS, ensuring tests adapt automatically.file_pathfixture to generate unique random filenames for every test to avoid exceeding operations rate limit on a single file and ensure test isolation.New & Updated Tests:
put,pipe,ls,cat, andrmto verifyExtendedGCSFileSystemin async mode.upload_chunk,initiate_upload, etc.).test_zonal_fileand updated existing tests to work with both Mock and Real GCS environments using the new isolation fixtures.