Open
Conversation
…to ensure outputs align with HF Transformer
- Rename olmo2.yaml to olmo.yaml and update model_family to 'olmo' - Fix imports in composition/models.py and composition/tokenizers.py - Fix hub.py import (fairseq2.hub.model -> fairseq2.models.hub) - Add tokenizer exports and OLMO3 aliases to __init__.py - Fix OLMORMSNorm with proper __init__ method in normalization.py - Implement per-layer RoPE support in factory.py for OLMO3 sliding window - Fix yarn_rope.py init order (params before super().__init__) - Fix yarn_rope.py import (unsqueeze from fairseq2.ops) - Update OLMO3 configs: vocab_size=100278, num_layers=64 for 32B - Update test_olmo2.py to use local model paths - Add test_olmo2_all.py and test_olmo3_all.py for comprehensive testing
Fix all E501 line length violations (>88 chars) in the OLMO module: - Reformat long docstrings and comments across all files - Split long dictionary mappings in interop.py for better readability - Wrap long error messages and function arguments - All tests still passing after formatting changes
4975b9a to
1d050dd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do? Please describe:
Add OLMo2 model architecture (1B, 7B, 13B) support in fairseq2
The key architecture changes include:
OLMO2 is similar to the LLaMA architecture with the following differences:
An integration test is added to ensure the output is consistent with HF Transformers. The integration test has passed for the 1B model.
Note:
OLMO2MultiheadAttention inherits from StandardMultiheadAttention (marked
@final)because the only difference is the order of normalization in
_project_q()and_project_kv().Reimplementing the entire class would duplicate ~150 lines of boilerplate code. Right now, the type checker warning is suppressed.
Fixes #1402
Does your PR introduce any breaking changes? If yes, please list them:
List of all backwards-incompatible changes.
Check list: