Skip to content

Fast5 zip#135

Open
robegan21 wants to merge 18 commits intoarq5x:masterfrom
JGI-Bioinformatics:Fast5Zip
Open

Fast5 zip#135
robegan21 wants to merge 18 commits intoarq5x:masterfrom
JGI-Bioinformatics:Fast5Zip

Conversation

@robegan21
Copy link

This patch allows a .zip file (of fast5 files) to be processed efficiently and in the same way that the tarball of fast5 files are processed.

I have found the .zip to have several key advantages over a tarball of fast5 files:

  1. There is about 30% space saved in my data sets
  2. the .zip file is indexed, and efficient to extract a single file, so there is no need to extract the entire run or parse the whole archive
  3. the .zip file can be read in parallel (using multithread.Pool)
  4. the .zip has a checksums to protect against data corruption
  5. extracting zip files on a network drive (GPFS) is about 5-10x faster than working with each file individually, so long as the extraction is saved locally (/tmp) or to memory (python's ZipFile.read() or /dev/shm)

Please consider incorporating this into poretools, as I believe that zips are a workable way forward to managing the millions of fast5 files that are presently being generated per run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant