Bitnet#205

Draft

thejhh wants to merge 21 commits intomainfrom

Commits on May 19, 2025

feat(bitnet): Initial project setup for BitNet implementation (#193)

thejhh
and
Jaakko Heusala
authored
chore: update gitignore for generated files and profiles (#195)

thejhh
and
Jaakko Heusala
authored

Commits on May 20, 2025

feat(bitnet/model): implement model weights and tokenizer integration (#197)

authored
feat(bitnet): implement core model structures and weight loading (#198)

thejhh
and
Jaakko Heusala
authored

Commits on May 21, 2025

174 implement tokenizer llama3 bpe (#199)

thejhh
and
Jaakko Heusala
authored
200 improve pr messages (#201)

thejhh
and
Jaakko Heusala
authored
feat(bitnet): implement token embedding layer (#202)

thejhh
and
Jaakko Heusala
authored
176 set model constants architecture hyperparameters (#203)

thejhh
and
Jaakko Heusala
authored
177 implement rotary positional encoding (#204)

thejhh
and
Jaakko Heusala
authored
feat(tensor): add BitLinear layer and tests (#178) (#206)

thejhh
and
Jaakko Heusala
authored
feat(bitnet): implement Sub-Layer Normalization (SubLN) for #179 (#207)

thejhh
and
Jaakko Heusala
authored
feat(bitnet): implement squared ReLU activation (ReLU²) (#208)

thejhh
and
Jaakko Heusala
authored

Commits on May 22, 2025

feat(bitnet): implement QKV projection for attention (#209)

thejhh
and
Jaakko Heusala
authored
feat(bitnet): implement scaled dot-product attention (#210)

thejhh
and
Jaakko Heusala
authored
Optimize attention weights multiplication with values (#211)

thejhh
and
Jaakko Heusala
authored
feat(math): add attention output projection layer (#184) (#212)

thejhh
and
Jaakko Heusala
authored
feat(math): add FFN sublayer for transformer block (#213)

thejhh
and
Jaakko Heusala
authored
186 integrate attention sublayer pre norm residual (#214)

thejhh
and
Jaakko Heusala
authored
feat(bitnet): implement feed-forward sublayer with pre-norm and residual connections (#215)

thejhh
and
Jaakko Heusala
authored

Commits on May 23, 2025

188 stack transformer blocks (#216)

thejhh
and
Jaakko Heusala
authored
feat(math): implement LM head with error handling (#217)

thejhh
and
Jaakko Heusala
authored