Skip to content

Scaling up dataset management #24

New issue
New issue
Open
Open
Scaling up dataset management#24
Labels
hpc_preptask needed to be completed before HPClow_priotraining
@tomiock

Description

@tomiock
tomiock
opened on Sep 13, 2025
  • Pre-tokenize the entire text dataset ahead of time.
  • Pre-process the images ahead of time.
  • Distribute the dataset across nodes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    hpc_preptask needed to be completed before HPClow_priotraining

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions