Update text-splitter requirement from 0.24 to 0.25

[dependabot]

Updates the requirements on text-splitter to permit the latest version.

Release notes

Sourced from text-splitter's releases.

v0.25.0

Breaking Changes

Rust

• Remove support for rust-tokenizers crate. This crate hasn't been updated in several years and brings in depednencies that have security warnings.

What's New

• Use faster encoding method for tokenizers library, which improves performance with usage of huggingface tokenizers.

Full Changelog: benbrandt/text-splitter@v0.24.2...v0.25.0

Changelog

Sourced from text-splitter's changelog.

v0.25.0

Breaking Changes

Rust

• Remove support for rust-tokenizers crate. This crate hasn't been updated in several years and brings in depednencies that have security warnings.

What's New

• Use faster encoding method for tokenizers library, which improves performance with usage of huggingface tokenizers.

v0.24.2

Fixes

• Python packages target a newer version of libc to hopefully fix header file issues with tree-sitter.

What's New

• MSRV updated to 1.81.0

v0.24.1

What's New

Added a new chunk_char_indices method to the Rust splitters.

use text_splitter::{Characters, ChunkCharIndex, TextSplitter};
let text = "\r\na̐éö̲\r\n";
let splitter = TextSplitter::new(3);
let chunks = splitter.chunk_char_indices(text).collect::<Vec<_>>();
assert_eq!(
vec![
ChunkCharIndex {
chunk: "a̐é",
byte_offset: 2,
char_offset: 2
},
ChunkCharIndex {
chunk: "ö̲",
byte_offset: 7,
char_offset: 5
}
],
chunks
);
</tr></table>

... (truncated)

Commits

• 2fe70b5 ci: remove caches
• 0505991 feat: use encode_fast
• afe6a52 Update benchmark output
• 7e42509 remove rust-tokenizers support
• 7d8533d chore(deps): bump the minor group with 3 updates
• 239fe9c chore(deps): bump zod-to-json-schema in /docs in the minor group
• 9ea65db chore(deps): bump tempfile from 3.19.0 to 3.19.1 in the minor group
• 11f12e4 prep v0.24.2
• 9e03993 ci: remove x86 target
• 4b05233 ci: try building with newer manylinux
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Superseded by #15.