Skip to content
This repository was archived by the owner on Jul 19, 2024. It is now read-only.
This repository was archived by the owner on Jul 19, 2024. It is now read-only.

Pickle overloaded handler behavior #6

@JostTim

Description

@JostTim

Pickle object should only use dicts to save multiple data objects (removes the overhead of thinking if wee need to load or dump multiple times)
On load :

  • actually save the data IN the pickle instance.
  • With a specific class dict that's only job is to implement a _has_chaged boolean that gets True at the first call to setitem if value is different to previous stored value at the given index
  • Make a backup copy file on first .load() of the instance (a particular day of working) in __picklebackup directory.)
  • If file exists from previous day, we can overwrite it.
    This will prevent processed dataloss if program is stopped during dumpung later, as this can hapen sometimes.

On dump :

  • check data _has_chaged is True :
  • do not dump if it is not the case
    This will remove the overhead of checking if file exists, if data exists in file etc and still using the same file to store processed data during analysis in a transparent way.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions