Provide canonical information for llama tokenizer

[kheiss-uwzoo]

Summary

Consolidates all Llama tokenizer documentation into one canonical section and replaces duplicate explanations with links, so future changes (model ID, Hugging Face token, pre‑download behavior) only need to be updated in a single place.

Changes

• Added a dedicated “Llama tokenizer (default)” section in chunking.md that documents:

• The default tokenizer ID (meta-llama/Llama-3.2-1B) and why it’s recommended.

• The requirement for a Hugging Face access token.

• Pre‑download behavior and build‑time configuration (DOWNLOAD_LLAMA_TOKENIZER, HF_ACCESS_TOKEN), with a link to Environment Variables for details.

• Updated the token‑based splitting intro to point to this section instead of repeating the same content.

• Merged the former “Pre-download the Tokenizer” subsection into the new canonical section.

• Converted other mentions into link‑only references (no duplicated text):

• environment-config.md now links to “Llama tokenizer (default)” for DOWNLOAD_LLAMA_TOKENIZER and HF_ACCESS_TOKEN.

• releasenotes-nv-ingest.md release note about the tokenizer being pre‑downloaded now links to the canonical section.

• nv-ingest-python-api.md “Extract Audio” example notes that it uses the default tokenizer and links to the canonical section; code and API usage are unchanged.

Files changed

• extraction/chunking.md

• extraction/environment-config.md

• extraction/releasenotes-nv-ingest.md

• extraction/nv-ingest-python-api.md

Rationale

Reduces the risk of the docs getting out of sync by having a single authoritative description of the Llama tokenizer, with other pages linking to it instead of duplicating the same information.

[sosahi]

was this "{#llama-tokenizer-default}" added intentionally?

[kheiss-uwzoo]

yes

[nkmcalli]

If you intend that to be a section anchor, I don't think it will work across both GitHub and docs.nvidia.com

[kheiss-uwzoo]

Yes - should it be removed?

________________________________ From: sosahi ***@***.***> Sent: Thursday, February 5, 2026 1:12 PM To: NVIDIA/nv-ingest ***@***.***> Cc: Kurt Heiss ***@***.***>; Author ***@***.***> Subject: Re: [NVIDIA/nv-ingest] Provide canonical information for llama tokenizer (PR #1380) @sosahi commented on this pull request.

________________________________ In docs/docs/extraction/chunking.md<#1380 (comment)>:

@@ -76,6 +72,23 @@ ingestor = ingestor.split(

) ``` +### Llama tokenizer (default) {#llama-tokenizer-default} was this "{#llama-tokenizer-default}" added intentionally? — Reply to this email directly, view it on GitHub<#1380 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/B3723WCXQSRWSLJ7ZG4VZRT4KOW3JAVCNFSM6AAAAACUEUJ4CSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTONJZGQ4TCOJVGM>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[nkmcalli]

If you intend that to be a section anchor, I don't think it will work across both GitHub and docs.nvidia.com