POC hydra-intergration [do not merge] #30
Conversation
| engine=get_engine(cfg.judge), | ||
| initial_prompt=JUDGE_PROMPT, | ||
| ) | ||
| storage = context.FileStorage(Path.cwd()) # Directory for storing contexts (structured logs) |
There was a problem hiding this comment.
When using hydra, we should just write directly to cwd since hydra auto-changes into a unique output directory for every experimental run
| return False | ||
|
|
||
|
|
||
| @hydra.main(config_path="conf", config_name="adversarial_prompting") |
There was a problem hiding this comment.
this tells hydra to look for a (yaml) config file at conf/adversarial_prompting.yaml
all the attributes specified in that config file become automatically available as CLI arguments
| log.info(f"Git diff:\n{diff}") | ||
|
|
||
|
|
||
| class SymbolicLinkCallback(Callback): |
There was a problem hiding this comment.
this is a slightly ugly (but working) hack to allow the current server implementation to work with the file structure generated by hydra
There was a problem hiding this comment.
Hydra generates the following structure
results/<timestamp1>/<timestamped_context_file>
results/<timestamp2>/<another_timestamped_context_file>
...
what this hack does is to just create symlinks for each context file and folder in the parent directory as follows
results/<timestamped_context_file> -> results/<timestamp1>/<timestamped_context_file>
results/<another_timestamped_context_file> -> results/<timestamp2>/<another_timestamped_context_file>
...
If the server/storage was able to just see the files one level below we could avoid that
|
|
||
| defaults: | ||
| - _self_ | ||
| - override hydra/launcher: joblib # joblib launches runs in parallel; comment out if undesired |
There was a problem hiding this comment.
parallel execution using joblib has some trouble with pydantic and pickling (see cloudpipe/cloudpickle#408) . the issue appears only if pydantic classes are defined in the main script. ==> to avoid the problem define all pydantic classes in a separate module and import them in your main script
| _log_git_sha() | ||
|
|
||
|
|
||
| def _log_git_sha(): |
This is just, for now, just for discussion / knowledge sharing @gavento / @spirali
It shows a simple way of using hydra together with interlab.