Skip to content

Adding a use case example 'shuffle'#15

Open
JaeseungYeom wants to merge 3 commits intoflux-framework:mainfrom
JaeseungYeom:shuffle
Open

Adding a use case example 'shuffle'#15
JaeseungYeom wants to merge 3 commits intoflux-framework:mainfrom
JaeseungYeom:shuffle

Conversation

@JaeseungYeom
Copy link
Contributor

Use case demonstration for input file shuffling with deep learning training.
DYAD needs to be updated to leverage fcntl() to dynamically determine the mode of each individual open file, and handle it accordingly. This code is independent of DYAD and Flux. However, it reuses some of utility functions in DYAD implementation. The source files of those are symbolically linked to avoid duplication.

@ilumsden
Copy link
Collaborator

ilumsden commented Jun 13, 2024

What's left to do:

  • Add script(s) to run with DYAD
  • Integrate into testing/CI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants