About total parameters

[xyjigsaw]

Great work, can you provide the total number of parameters for the Engram SFT model in the readme table?

[Alichua]

Thanks for your careful reading and for bringing this up.

At the moment, we haven’t added detailed parameter counts to the main table yet, since this project is still in an early exploratory stage and we are actively iterating on the Engram design. The exact parameter size may change as the architecture evolves.

However, the parameter count can be estimated as:

$$ \begin{aligned} P_{total} &\approx L \cdot \bigg[ \\ &\quad \underbrace{(V_2 + V_3) \cdot D_{emb}}_{\text{embedding tables}} \\ &+ \underbrace{2 \cdot (D_{engram} \cdot D_{model} + D_{model})}_{\text{value + key projections}} \\ &+ \underbrace{2 \cdot D_{model}}_{\text{gating norms}} \\ &+ \underbrace{D_{model} \cdot (K + 1)}_{\text{ShortConv (weights + norm)}} \\ \bigg] \end{aligned} $$

For our typical setup:

• 2-gram hash table size ($V_2$): 10,000
• 3-gram hash table size ($V_3$): 1,000
• Engram layers ($L$): 4 ([5, 7, 13, 17])
• $D_{emb}$: 512
• $D_{model}$: 1024
• Kernel size ($K$): 4
• $D_{engram} = (3-1) \times 512 = 1024$

$$ \begin{aligned} P_{total} &\approx 4 \cdot \bigg[ \\ &\quad (10,000 + 1,000) \cdot 512 \\ &+ 2 \cdot (1024 \cdot 1024 + 1024) \\ &+ 2 \cdot 1024 \\ &+ 1024 \cdot (4 + 1) \\ \bigg] \\ &\approx 4 \cdot [ 5.63\text{M} + 2.10\text{M} + 0.002\text{M} + 0.005\text{M} ] \\ &\approx 4 \cdot 7.74\text{M} \\ &\approx \mathbf{30.95\text{M parameters}} \end{aligned} $$

Note: In our current implementation (engram_qwen.py), the hyper-connection multiplier H_mult is hardcoded to 1, i.e., we are using a dense (non-MoE) variant. This differs from DeepSeek’s MoE-style design, where H_mult > 1 enables multiple expert-like branches.

We’ll consider adding summarized parameter statistics to the README once the architecture becomes more stable. Thanks again for the suggestion!