Hg model training with torch DataLoader

[annasun28]

What does this PR do? Please describe:

• Update fairseq2 HgTokenizer with some methods required from the base transformers Tokenizer. TBD: whether we should actually subclass and override methods so that we automatically fall back to the base transformers Tokenizer methods?
• Enable FSDP for Hg model (matching the way FSDP wrapping is handled in transformers / accelerate). Note: to override transformer_cls_names_to_wrap, one needs to register a new model family (see the corresponding seamless_next PR for example)
• Fix this issue:
  • fs2 overrides model.state_dict in _FSDP1Model to return fsdp1_local_state_dict (e.g. when checkpointing). If the tensor is sharded unevenly (e.g. if you have a nn.Embedding(2, dim) and sdp.size=4), the key is skipped (https://github.com/facebookresearch/fairseq2/blob/main/src/fairseq2/nn/fsdp/fsdp1.py#L192-L193) which causes a hang when reloading (IIUC due to an allgather in the FSDP pre state dict hook)

Tested with https://github.com/fairinternal/seamless_next/pull/750

Does your PR introduce any breaking changes? If yes, please list them:
N/A

Check list:

• Was the content of this PR discussed and approved via a GitHub issue? (no need for typos or documentation improvements)
• Did you read the contributor guideline?
• Did you make sure that your PR does only one thing instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (no need for typos, documentation, or minor internal changes)

[rsyue]

This looks good. Just wondering if the process granularity is handled by the RecipeModel class during loading. I put barrier() in there because occasionally model info would be on another process and it would not recognize from interleaving.

[annasun28]

Hm, in the seamless_next recipe, I called model = model_hub.load_model(card, config=hg_model_config) before setting up gangs, which I guess is why this was causing an error. Let me try initializing the gangs first and see. But just curious, why are barrier() needed for this HF model, but not needed for others?

[rsyue]

From what I understand, the _load_special_model in factory sets a specific config as opposed to using AutoConfig. Qwen2.5-Omni does not accept AutoConfig because it is sort of special purpose. I noticed with torchrun that sometimes the main process would correctly set the config and others would not, leading to an error: "Invalid model config. Select from BertConfig, LlamaConfig, ..." on the other handles. I assumed this was because they were not properly synced, so thought that letting them all reach that point would resolve it. But it seems like you grab the config directly first in your recipe. I loaded from the card and let it find the config itself.

[rsyue]

Currently working on cleaning this up so it can be merged. ETA is end of next week (around 12/5).