[INFO] cloning repository https://github.com/YdrMaster/ggml-tokenizer
[INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/YdrMaster/ggml-tokenizer" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FYdrMaster%2Fggml-tokenizer", kill_on_drop: false }`
[INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FYdrMaster%2Fggml-tokenizer'...
[INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }`
[INFO] [stdout] ce8462b5ff70ffc6babc97311215655a47c6aa03
[INFO] checking YdrMaster/ggml-tokenizer against 615014f15745532730b9ce7a1d9309f8b4756f93 for pr-160491
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FYdrMaster%2Fggml-tokenizer" "/workspace/builds/worker-7-tc2/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-7-tc2/source'...
[INFO] [stderr] done.
[INFO] started tweaking git repo https://github.com/YdrMaster/ggml-tokenizer
[INFO] finished tweaking git repo https://github.com/YdrMaster/ggml-tokenizer
[INFO] tweaked toml for git repo https://github.com/YdrMaster/ggml-tokenizer written to /workspace/builds/worker-7-tc2/source/Cargo.toml
[INFO] validating manifest of git repo https://github.com/YdrMaster/ggml-tokenizer on toolchain 615014f15745532730b9ce7a1d9309f8b4756f93
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+615014f15745532730b9ce7a1d9309f8b4756f93" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate git repo https://github.com/YdrMaster/ggml-tokenizer already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+615014f15745532730b9ce7a1d9309f8b4756f93" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded ggus v0.4.0
[INFO] [stderr]   Downloaded ggml-quants v0.0.0
[INFO] [stderr]   Downloaded digit-layout v0.2.0
[INFO] [stderr]   Downloaded fancy-regex v0.14.0
[INFO] [stderr]   Downloaded winnow v0.7.6
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-7-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-7-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-m" "1610612736" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:00c5645b54fe3ce5dae1417175e3c7fb6a6646c021d554ecd63a097b8b9f3602" "sleep" "infinity", kill_on_drop: false }`
[INFO] [stdout] 6ad0129e784d041386268666802d28e7d587900399cdc1ddccb26ec545d086af
[INFO] running `Command { std: "docker" "start" "6ad0129e784d041386268666802d28e7d587900399cdc1ddccb26ec545d086af", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "6ad0129e784d041386268666802d28e7d587900399cdc1ddccb26ec545d086af", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-w" "/opt/rustwide/workdir" "--user" "0:0" "6ad0129e784d041386268666802d28e7d587900399cdc1ddccb26ec545d086af" "/opt/rustwide/cargo-home/bin/cargo" "+615014f15745532730b9ce7a1d9309f8b4756f93" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "6ad0129e784d041386268666802d28e7d587900399cdc1ddccb26ec545d086af", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-w" "/opt/rustwide/workdir" "--user" "0:0" "6ad0129e784d041386268666802d28e7d587900399cdc1ddccb26ec545d086af" "/opt/rustwide/cargo-home/bin/cargo" "+615014f15745532730b9ce7a1d9309f8b4756f93" "check" "--frozen" "--all" "--all-targets" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr]    Compiling hashbrown v0.15.2
[INFO] [stderr]    Compiling toml_datetime v0.6.8
[INFO] [stderr]    Compiling winnow v0.7.6
[INFO] [stderr]     Checking bit-vec v0.8.0
[INFO] [stderr]     Checking aho-corasick v1.1.3
[INFO] [stderr]     Checking half v2.6.0
[INFO] [stderr]     Checking log v0.4.27
[INFO] [stderr]     Checking indexmap v2.9.0
[INFO] [stderr]     Checking rayon-core v1.12.1
[INFO] [stderr]    Compiling syn v2.0.100
[INFO] [stderr]     Checking memmap2 v0.9.5
[INFO] [stderr]     Checking bit-set v0.8.0
[INFO] [stderr]     Checking digit-layout v0.2.0
[INFO] [stderr]     Checking rayon v1.10.0
[INFO] [stderr]     Checking regex-automata v0.4.9
[INFO] [stderr]    Compiling toml_edit v0.22.24
[INFO] [stderr]     Checking ggml-quants v0.0.0
[INFO] [stderr]    Compiling proc-macro-crate v3.3.0
[INFO] [stderr]     Checking fancy-regex v0.14.0
[INFO] [stderr]     Checking regex v1.11.1
[INFO] [stderr]    Compiling num_enum_derive v0.7.3
[INFO] [stderr]     Checking num_enum v0.7.3
[INFO] [stderr]     Checking ggus v0.4.0
[INFO] [stderr]     Checking try-tokenize v0.1.0 (/opt/rustwide/workdir)
[INFO] [stdout] warning: unused import: `TokenAttribute`
[INFO] [stdout]  --> src/main.rs:2:20
[INFO] [stdout]   |
[INFO] [stdout] 2 | use common::{NULL, TokenAttribute};
[INFO] [stdout]   |                    ^^^^^^^^^^^^^^
[INFO] [stdout]   |
[INFO] [stdout]   = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `VocabType`
[INFO] [stdout]  --> src/main.rs:3:14
[INFO] [stdout]   |
[INFO] [stdout] 3 | use config::{VocabType, load};
[INFO] [stdout]   |              ^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `untils::llama_escape_whitespace`
[INFO] [stdout]  --> src/main.rs:6:5
[INFO] [stdout]   |
[INFO] [stdout] 6 | use untils::llama_escape_whitespace;
[INFO] [stdout]   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `std::collections::LinkedList`
[INFO] [stdout]  --> src/main.rs:8:5
[INFO] [stdout]   |
[INFO] [stdout] 8 | use std::collections::LinkedList;
[INFO] [stdout]   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused imports: `LlmTokenizerBpeSession` and `LlmTokenizerSpmSession`
[INFO] [stdout]  --> src/common.rs:1:22
[INFO] [stdout]   |
[INFO] [stdout] 1 | use crate::session::{LlmTokenizerBpeSession, LlmTokenizerSpmSession};
[INFO] [stdout]   |                      ^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `VecDeque`
[INFO] [stdout]  --> src/session.rs:3:40
[INFO] [stdout]   |
[INFO] [stdout] 3 |     collections::{BinaryHeap, HashMap, VecDeque},
[INFO] [stdout]   |                                        ^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: type `unicode_cpt_flags` should have an upper camel case name
[INFO] [stdout]    --> src/unicode.rs:469:12
[INFO] [stdout]     |
[INFO] [stdout] 469 | pub struct unicode_cpt_flags {
[INFO] [stdout]     |            ^^^^^^^^^^^^^^^^^ help: convert the identifier to upper camel case: `UnicodeCptFlags`
[INFO] [stdout]     |
[INFO] [stdout]     = note: `#[warn(non_camel_case_types)]` (part of `#[warn(nonstandard_style)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unnecessary parentheses around `if` condition
[INFO] [stdout]    --> src/unicode.rs:593:8
[INFO] [stdout]     |
[INFO] [stdout] 593 |     if (cpt >= '0' as u32 && cpt <= '9' as u32) {
[INFO] [stdout]     |        ^                                      ^
[INFO] [stdout]     |
[INFO] [stdout]     = note: `#[warn(unused_parens)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] help: remove these parentheses
[INFO] [stdout]     |
[INFO] [stdout] 593 -     if (cpt >= '0' as u32 && cpt <= '9' as u32) {
[INFO] [stdout] 593 +     if cpt >= '0' as u32 && cpt <= '9' as u32 {
[INFO] [stdout]     |
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `ggus::GGufMetaError`
[INFO] [stdout]  --> src/untils.rs:1:5
[INFO] [stdout]   |
[INFO] [stdout] 1 | use ggus::GGufMetaError;
[INFO] [stdout]   |     ^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `TokenAttribute`
[INFO] [stdout]  --> src/main.rs:2:20
[INFO] [stdout]   |
[INFO] [stdout] 2 | use common::{NULL, TokenAttribute};
[INFO] [stdout]   |                    ^^^^^^^^^^^^^^
[INFO] [stdout]   |
[INFO] [stdout]   = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `VocabType`
[INFO] [stdout]  --> src/main.rs:3:14
[INFO] [stdout]   |
[INFO] [stdout] 3 | use config::{VocabType, load};
[INFO] [stdout]   |              ^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `untils::llama_escape_whitespace`
[INFO] [stdout]  --> src/main.rs:6:5
[INFO] [stdout]   |
[INFO] [stdout] 6 | use untils::llama_escape_whitespace;
[INFO] [stdout]   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `std::collections::LinkedList`
[INFO] [stdout]  --> src/main.rs:8:5
[INFO] [stdout]   |
[INFO] [stdout] 8 | use std::collections::LinkedList;
[INFO] [stdout]   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused imports: `LlmTokenizerBpeSession` and `LlmTokenizerSpmSession`
[INFO] [stdout]  --> src/common.rs:1:22
[INFO] [stdout]   |
[INFO] [stdout] 1 | use crate::session::{LlmTokenizerBpeSession, LlmTokenizerSpmSession};
[INFO] [stdout]   |                      ^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `VecDeque`
[INFO] [stdout]  --> src/session.rs:3:40
[INFO] [stdout]   |
[INFO] [stdout] 3 |     collections::{BinaryHeap, HashMap, VecDeque},
[INFO] [stdout]   |                                        ^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: type `unicode_cpt_flags` should have an upper camel case name
[INFO] [stdout]    --> src/unicode.rs:469:12
[INFO] [stdout]     |
[INFO] [stdout] 469 | pub struct unicode_cpt_flags {
[INFO] [stdout]     |            ^^^^^^^^^^^^^^^^^ help: convert the identifier to upper camel case: `UnicodeCptFlags`
[INFO] [stdout]     |
[INFO] [stdout]     = note: `#[warn(non_camel_case_types)]` (part of `#[warn(nonstandard_style)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unnecessary parentheses around `if` condition
[INFO] [stdout]    --> src/unicode.rs:593:8
[INFO] [stdout]     |
[INFO] [stdout] 593 |     if (cpt >= '0' as u32 && cpt <= '9' as u32) {
[INFO] [stdout]     |        ^                                      ^
[INFO] [stdout]     |
[INFO] [stdout]     = note: `#[warn(unused_parens)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] help: remove these parentheses
[INFO] [stdout]     |
[INFO] [stdout] 593 -     if (cpt >= '0' as u32 && cpt <= '9' as u32) {
[INFO] [stdout] 593 +     if cpt >= '0' as u32 && cpt <= '9' as u32 {
[INFO] [stdout]     |
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused import: `ggus::GGufMetaError`
[INFO] [stdout]  --> src/untils.rs:1:5
[INFO] [stdout]   |
[INFO] [stdout] 1 | use ggus::GGufMetaError;
[INFO] [stdout]   |     ^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unreachable statement
[INFO] [stdout]    --> src/config.rs:568:25
[INFO] [stdout]     |
[INFO] [stdout] 565 |                         todo!();
[INFO] [stdout]     |                         ------- any code following this expression is unreachable
[INFO] [stdout] ...
[INFO] [stdout] 568 |                         is_prev_special = false;
[INFO] [stdout]     |                         ^^^^^^^^^^^^^^^^^^^^^^^^ unreachable statement
[INFO] [stdout]     |
[INFO] [stdout]     = note: `#[warn(unreachable_code)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unreachable statement
[INFO] [stdout]    --> src/config.rs:568:25
[INFO] [stdout]     |
[INFO] [stdout] 565 |                         todo!();
[INFO] [stdout]     |                         ------- any code following this expression is unreachable
[INFO] [stdout] ...
[INFO] [stdout] 568 |                         is_prev_special = false;
[INFO] [stdout]     |                         ^^^^^^^^^^^^^^^^^^^^^^^^ unreachable statement
[INFO] [stdout]     |
[INFO] [stdout]     = note: `#[warn(unreachable_code)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unreachable statement
[INFO] [stdout]   --> src/session.rs:66:17
[INFO] [stdout]    |
[INFO] [stdout] 65 |                   todo!();
[INFO] [stdout]    |                   ------- any code following this expression is unreachable
[INFO] [stdout] 66 | /                 self.symbols.push(LlmSymbol {
[INFO] [stdout] 67 | |                     prev: -1,
[INFO] [stdout] 68 | |                     next: -1,
[INFO] [stdout] 69 | |                     text: word.to_string(),
[INFO] [stdout] 70 | |                     n: word.len(),
[INFO] [stdout] 71 | |                 });
[INFO] [stdout]    | |___________________^ unreachable statement
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unreachable statement
[INFO] [stdout]   --> src/session.rs:66:17
[INFO] [stdout]    |
[INFO] [stdout] 65 |                   todo!();
[INFO] [stdout]    |                   ------- any code following this expression is unreachable
[INFO] [stdout] 66 | /                 self.symbols.push(LlmSymbol {
[INFO] [stdout] 67 | |                     prev: -1,
[INFO] [stdout] 68 | |                     next: -1,
[INFO] [stdout] 69 | |                     text: word.to_string(),
[INFO] [stdout] 70 | |                     n: word.len(),
[INFO] [stdout] 71 | |                 });
[INFO] [stdout]    | |___________________^ unreachable statement
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: field `score` is never read
[INFO] [stdout]   --> src/common.rs:11:9
[INFO] [stdout]    |
[INFO] [stdout]  9 | pub struct TokenData {
[INFO] [stdout]    |            --------- field in this struct
[INFO] [stdout] 10 |     pub text: String,
[INFO] [stdout] 11 |     pub score: f32,
[INFO] [stdout]    |         ^^^^^
[INFO] [stdout]    |
[INFO] [stdout]    = note: `TokenData` has derived impls for the traits `Clone` and `Debug`, but these are intentionally ignored during dead code analysis
[INFO] [stdout]    = note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: field `score` is never read
[INFO] [stdout]   --> src/common.rs:11:9
[INFO] [stdout]    |
[INFO] [stdout]  9 | pub struct TokenData {
[INFO] [stdout]    |            --------- field in this struct
[INFO] [stdout] 10 |     pub text: String,
[INFO] [stdout] 11 |     pub score: f32,
[INFO] [stdout]    |         ^^^^^
[INFO] [stdout]    |
[INFO] [stdout]    = note: `TokenData` has derived impls for the traits `Clone` and `Debug`, but these are intentionally ignored during dead code analysis
[INFO] [stdout]    = note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: variants `Normalized` and `SingleWord` are never constructed
[INFO] [stdout]   --> src/common.rs:25:5
[INFO] [stdout]    |
[INFO] [stdout] 17 | pub enum TokenAttribute {
[INFO] [stdout]    |          -------------- variants in this enum
[INFO] [stdout] ...
[INFO] [stdout] 25 |     Normalized = 1 << 6,
[INFO] [stdout]    |     ^^^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 28 |     SingleWord = 1 << 9,
[INFO] [stdout]    |     ^^^^^^^^^^
[INFO] [stdout]    |
[INFO] [stdout]    = note: `TokenAttribute` has derived impls for the traits `Debug` and `Clone`, but these are intentionally ignored during dead code analysis
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: variants `Normalized` and `SingleWord` are never constructed
[INFO] [stdout]   --> src/common.rs:25:5
[INFO] [stdout]    |
[INFO] [stdout] 17 | pub enum TokenAttribute {
[INFO] [stdout]    |          -------------- variants in this enum
[INFO] [stdout] ...
[INFO] [stdout] 25 |     Normalized = 1 << 6,
[INFO] [stdout]    |     ^^^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 28 |     SingleWord = 1 << 9,
[INFO] [stdout]    |     ^^^^^^^^^^
[INFO] [stdout]    |
[INFO] [stdout]    = note: `TokenAttribute` has derived impls for the traits `Debug` and `Clone`, but these are intentionally ignored during dead code analysis
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: variants `Spm`, `Wpm`, `Ugm`, and `Rwkv` are never constructed
[INFO] [stdout]    --> src/config.rs:369:5
[INFO] [stdout]     |
[INFO] [stdout] 367 | pub enum VocabType {
[INFO] [stdout]     |          --------- variants in this enum
[INFO] [stdout] 368 |     None = 0, // For models without vocab
[INFO] [stdout] 369 |     Spm = 1,  // LLaMA tokenizer based on byte-level BPE with byte fallback
[INFO] [stdout]     |     ^^^
[INFO] [stdout] 370 |     Bpe = 2,  // GPT-2 tokenizer based on byte-level BPE
[INFO] [stdout] 371 |     Wpm = 3,  // BERT tokenizer based on WordPiece
[INFO] [stdout]     |     ^^^
[INFO] [stdout] 372 |     Ugm = 4,  // T5 tokenizer based on Unigram
[INFO] [stdout]     |     ^^^
[INFO] [stdout] 373 |     Rwkv = 5, // RWKV tokenizer based on greedy tokenization
[INFO] [stdout]     |     ^^^^
[INFO] [stdout]     |
[INFO] [stdout]     = note: `VocabType` has derived impls for the traits `Debug` and `Clone`, but these are intentionally ignored during dead code analysis
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: variants `Spm`, `Wpm`, `Ugm`, and `Rwkv` are never constructed
[INFO] [stdout]    --> src/config.rs:369:5
[INFO] [stdout]     |
[INFO] [stdout] 367 | pub enum VocabType {
[INFO] [stdout]     |          --------- variants in this enum
[INFO] [stdout] 368 |     None = 0, // For models without vocab
[INFO] [stdout] 369 |     Spm = 1,  // LLaMA tokenizer based on byte-level BPE with byte fallback
[INFO] [stdout]     |     ^^^
[INFO] [stdout] 370 |     Bpe = 2,  // GPT-2 tokenizer based on byte-level BPE
[INFO] [stdout] 371 |     Wpm = 3,  // BERT tokenizer based on WordPiece
[INFO] [stdout]     |     ^^^
[INFO] [stdout] 372 |     Ugm = 4,  // T5 tokenizer based on Unigram
[INFO] [stdout]     |     ^^^
[INFO] [stdout] 373 |     Rwkv = 5, // RWKV tokenizer based on greedy tokenization
[INFO] [stdout]     |     ^^^^
[INFO] [stdout]     |
[INFO] [stdout]     = note: `VocabType` has derived impls for the traits `Debug` and `Clone`, but these are intentionally ignored during dead code analysis
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: fields `escape_whitespaces` and `treat_whitespace_as_suffix` are never read
[INFO] [stdout]    --> src/config.rs:399:9
[INFO] [stdout]     |
[INFO] [stdout] 376 | pub struct TokenizerConfig {
[INFO] [stdout]     |            --------------- fields in this struct
[INFO] [stdout] ...
[INFO] [stdout] 399 |     pub escape_whitespaces: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 400 |     pub treat_whitespace_as_suffix: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: methods `n_tokens`, `get_token_data`, and `byte_to_token` are never used
[INFO] [stdout]    --> src/config.rs:457:12
[INFO] [stdout]     |
[INFO] [stdout] 407 | impl TokenizerConfig {
[INFO] [stdout]     | -------------------- methods in this implementation
[INFO] [stdout] ...
[INFO] [stdout] 457 |     pub fn n_tokens(&self) -> u32 {
[INFO] [stdout]     |            ^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 476 |     pub fn get_token_data(&self, id: TokenId) -> TokenData {
[INFO] [stdout]     |            ^^^^^^^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 480 |     pub fn byte_to_token(&self, ch: u8) -> TokenId {
[INFO] [stdout]     |            ^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: associated function `append` is never used
[INFO] [stdout]   --> src/session.rs:50:12
[INFO] [stdout]    |
[INFO] [stdout] 38 | impl LlmTokenizerBpeSession {
[INFO] [stdout]    | --------------------------- associated function in this implementation
[INFO] [stdout] ...
[INFO] [stdout] 50 |     pub fn append(token_id: TokenId, output: &mut Vec<TokenId>) {
[INFO] [stdout]    |            ^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: field `size` is never read
[INFO] [stdout]    --> src/session.rs:221:9
[INFO] [stdout]     |
[INFO] [stdout] 211 | pub struct LlmBigramBpeItem {
[INFO] [stdout]     |            ---------------- field in this struct
[INFO] [stdout] ...
[INFO] [stdout] 221 |     pub size: usize,
[INFO] [stdout]     |         ^^^^
[INFO] [stdout]     |
[INFO] [stdout]     = note: `LlmBigramBpeItem` has derived impls for the traits `Debug` and `Clone`, but these are intentionally ignored during dead code analysis
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: method `is_empty` is never used
[INFO] [stdout]    --> src/session.rs:274:12
[INFO] [stdout]     |
[INFO] [stdout] 255 | impl LlmBigramBpe {
[INFO] [stdout]     | ----------------- method in this implementation
[INFO] [stdout] ...
[INFO] [stdout] 274 |     pub fn is_empty(&self) -> bool {
[INFO] [stdout]     |            ^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: struct `LlmBigramSpm` is never constructed
[INFO] [stdout]    --> src/session.rs:286:12
[INFO] [stdout]     |
[INFO] [stdout] 286 | pub struct LlmBigramSpm {
[INFO] [stdout]     |            ^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: fields `escape_whitespaces` and `treat_whitespace_as_suffix` are never read
[INFO] [stdout]    --> src/config.rs:399:9
[INFO] [stdout]     |
[INFO] [stdout] 376 | pub struct TokenizerConfig {
[INFO] [stdout]     |            --------------- fields in this struct
[INFO] [stdout] ...
[INFO] [stdout] 399 |     pub escape_whitespaces: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 400 |     pub treat_whitespace_as_suffix: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: methods `n_tokens`, `get_token_data`, and `byte_to_token` are never used
[INFO] [stdout]    --> src/config.rs:457:12
[INFO] [stdout]     |
[INFO] [stdout] 407 | impl TokenizerConfig {
[INFO] [stdout]     | -------------------- methods in this implementation
[INFO] [stdout] ...
[INFO] [stdout] 457 |     pub fn n_tokens(&self) -> u32 {
[INFO] [stdout]     |            ^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 476 |     pub fn get_token_data(&self, id: TokenId) -> TokenData {
[INFO] [stdout]     |            ^^^^^^^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 480 |     pub fn byte_to_token(&self, ch: u8) -> TokenId {
[INFO] [stdout]     |            ^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: associated function `append` is never used
[INFO] [stdout]   --> src/session.rs:50:12
[INFO] [stdout]    |
[INFO] [stdout] 38 | impl LlmTokenizerBpeSession {
[INFO] [stdout]    | --------------------------- associated function in this implementation
[INFO] [stdout] ...
[INFO] [stdout] 50 |     pub fn append(token_id: TokenId, output: &mut Vec<TokenId>) {
[INFO] [stdout]    |            ^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: field `size` is never read
[INFO] [stdout]    --> src/session.rs:221:9
[INFO] [stdout]     |
[INFO] [stdout] 211 | pub struct LlmBigramBpeItem {
[INFO] [stdout]     |            ---------------- field in this struct
[INFO] [stdout] ...
[INFO] [stdout] 221 |     pub size: usize,
[INFO] [stdout]     |         ^^^^
[INFO] [stdout]     |
[INFO] [stdout]     = note: `LlmBigramBpeItem` has derived impls for the traits `Debug` and `Clone`, but these are intentionally ignored during dead code analysis
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: method `is_empty` is never used
[INFO] [stdout]    --> src/session.rs:274:12
[INFO] [stdout]     |
[INFO] [stdout] 255 | impl LlmBigramBpe {
[INFO] [stdout]     | ----------------- method in this implementation
[INFO] [stdout] ...
[INFO] [stdout] 274 |     pub fn is_empty(&self) -> bool {
[INFO] [stdout]     |            ^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: struct `LlmBigramSpm` is never constructed
[INFO] [stdout]    --> src/session.rs:286:12
[INFO] [stdout]     |
[INFO] [stdout] 286 | pub struct LlmBigramSpm {
[INFO] [stdout]     |            ^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: struct `LlmTokenizerSpmSession` is never constructed
[INFO] [stdout]    --> src/session.rs:329:12
[INFO] [stdout]     |
[INFO] [stdout] 329 | pub struct LlmTokenizerSpmSession {
[INFO] [stdout]     |            ^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: struct `LlmTokenizerSpmSession` is never constructed
[INFO] [stdout]    --> src/session.rs:329:12
[INFO] [stdout]     |
[INFO] [stdout] 329 | pub struct LlmTokenizerSpmSession {
[INFO] [stdout]     |            ^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: associated items `new`, `tokenize`, `try_add_bigram`, and `resegment` are never used
[INFO] [stdout]    --> src/session.rs:340:12
[INFO] [stdout]     |
[INFO] [stdout] 338 | impl<'a> LlmTokenizerSpmSession {
[INFO] [stdout]     | ------------------------------- associated items in this implementation
[INFO] [stdout] 339 |     /// 创建一个新的 SPM 标记器会话
[INFO] [stdout] 340 |     pub fn new() -> Self {
[INFO] [stdout]     |            ^^^
[INFO] [stdout] ...
[INFO] [stdout] 349 |     pub fn tokenize(&mut self, text: &'a str, output: &mut Vec<u32>, config: &TokenizerConfig) {
[INFO] [stdout]     |            ^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 424 |     fn try_add_bigram(&mut self, left: i32, right: i32, config: &TokenizerConfig) {
[INFO] [stdout]     |        ^^^^^^^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 468 |     fn resegment(&self, symbol: &LlmSymbol, output: &mut Vec<u32>, config: &TokenizerConfig) {
[INFO] [stdout]     |        ^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: associated items `new`, `tokenize`, `try_add_bigram`, and `resegment` are never used
[INFO] [stdout]    --> src/session.rs:340:12
[INFO] [stdout]     |
[INFO] [stdout] 338 | impl<'a> LlmTokenizerSpmSession {
[INFO] [stdout]     | ------------------------------- associated items in this implementation
[INFO] [stdout] 339 |     /// 创建一个新的 SPM 标记器会话
[INFO] [stdout] 340 |     pub fn new() -> Self {
[INFO] [stdout]     |            ^^^
[INFO] [stdout] ...
[INFO] [stdout] 349 |     pub fn tokenize(&mut self, text: &'a str, output: &mut Vec<u32>, config: &TokenizerConfig) {
[INFO] [stdout]     |            ^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 424 |     fn try_add_bigram(&mut self, left: i32, right: i32, config: &TokenizerConfig) {
[INFO] [stdout]     |        ^^^^^^^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 468 |     fn resegment(&self, symbol: &LlmSymbol, output: &mut Vec<u32>, config: &TokenizerConfig) {
[INFO] [stdout]     |        ^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_regex_split_custom` is never used
[INFO] [stdout]    --> src/unicode.rs:149:4
[INFO] [stdout]     |
[INFO] [stdout] 149 | fn unicode_regex_split_custom(text: &str, regex_expr: &str, offsets: &[usize]) -> Vec<usize> {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_regex_split_custom_gpt2` is never used
[INFO] [stdout]    --> src/unicode.rs:165:4
[INFO] [stdout]     |
[INFO] [stdout] 165 | fn unicode_regex_split_custom_gpt2(text: &str, offsets: &[usize]) -> Vec<usize> {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_regex_split_custom` is never used
[INFO] [stdout]    --> src/unicode.rs:149:4
[INFO] [stdout]     |
[INFO] [stdout] 149 | fn unicode_regex_split_custom(text: &str, regex_expr: &str, offsets: &[usize]) -> Vec<usize> {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_regex_split_custom_llama3` is never used
[INFO] [stdout]    --> src/unicode.rs:305:4
[INFO] [stdout]     |
[INFO] [stdout] 305 | fn unicode_regex_split_custom_llama3(text: &str, offsets: &[usize]) -> Vec<usize> {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_regex_split_custom_gpt2` is never used
[INFO] [stdout]    --> src/unicode.rs:165:4
[INFO] [stdout]     |
[INFO] [stdout] 165 | fn unicode_regex_split_custom_gpt2(text: &str, offsets: &[usize]) -> Vec<usize> {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_regex_split_custom_llama3` is never used
[INFO] [stdout]    --> src/unicode.rs:305:4
[INFO] [stdout]     |
[INFO] [stdout] 305 | fn unicode_regex_split_custom_llama3(text: &str, offsets: &[usize]) -> Vec<usize> {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: multiple fields are never read
[INFO] [stdout]    --> src/unicode.rs:470:9
[INFO] [stdout]     |
[INFO] [stdout] 469 | pub struct unicode_cpt_flags {
[INFO] [stdout]     |            ----------------- fields in this struct
[INFO] [stdout] 470 |     pub is_whitespace: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^^
[INFO] [stdout] 471 |     pub is_letter: bool,
[INFO] [stdout]     |         ^^^^^^^^^
[INFO] [stdout] 472 |     pub is_number: bool,
[INFO] [stdout]     |         ^^^^^^^^^
[INFO] [stdout] 473 |     pub is_punctuation: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^^^
[INFO] [stdout] 474 |     pub is_symbol: bool,
[INFO] [stdout]     |         ^^^^^^^^^
[INFO] [stdout] 475 |     pub is_accent_mark: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^^^
[INFO] [stdout] 476 |     pub is_lowercase: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^
[INFO] [stdout] 477 |     pub is_uppercase: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^
[INFO] [stdout] 478 |     pub is_nfd: bool,
[INFO] [stdout]     |         ^^^^^^
[INFO] [stdout]     |
[INFO] [stdout]     = note: `unicode_cpt_flags` has a derived impl for the trait `Clone`, but this is intentionally ignored during dead code analysis
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: multiple fields are never read
[INFO] [stdout]    --> src/unicode.rs:470:9
[INFO] [stdout]     |
[INFO] [stdout] 469 | pub struct unicode_cpt_flags {
[INFO] [stdout]     |            ----------------- fields in this struct
[INFO] [stdout] 470 |     pub is_whitespace: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^^
[INFO] [stdout] 471 |     pub is_letter: bool,
[INFO] [stdout]     |         ^^^^^^^^^
[INFO] [stdout] 472 |     pub is_number: bool,
[INFO] [stdout]     |         ^^^^^^^^^
[INFO] [stdout] 473 |     pub is_punctuation: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^^^
[INFO] [stdout] 474 |     pub is_symbol: bool,
[INFO] [stdout]     |         ^^^^^^^^^
[INFO] [stdout] 475 |     pub is_accent_mark: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^^^
[INFO] [stdout] 476 |     pub is_lowercase: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^
[INFO] [stdout] 477 |     pub is_uppercase: bool,
[INFO] [stdout]     |         ^^^^^^^^^^^^
[INFO] [stdout] 478 |     pub is_nfd: bool,
[INFO] [stdout]     |         ^^^^^^
[INFO] [stdout]     |
[INFO] [stdout]     = note: `unicode_cpt_flags` has a derived impl for the trait `Clone`, but this is intentionally ignored during dead code analysis
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: associated items `UNDEFINED`, `WHITESPACE`, `as_uint`, and `category_flag` are never used
[INFO] [stdout]    --> src/unicode.rs:482:15
[INFO] [stdout]     |
[INFO] [stdout] 481 | impl unicode_cpt_flags {
[INFO] [stdout]     | ---------------------- associated items in this implementation
[INFO] [stdout] 482 |     pub const UNDEFINED: Self = Self {
[INFO] [stdout]     |               ^^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 494 |     pub const WHITESPACE: u32 = 1 << 0;
[INFO] [stdout]     |               ^^^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 501 |     pub fn as_uint(&self) -> u32 {
[INFO] [stdout]     |            ^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 524 |     pub fn category_flag(&self) -> u32 {
[INFO] [stdout]     |            ^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_cpts_from_utf8` is never used
[INFO] [stdout]    --> src/unicode.rs:547:4
[INFO] [stdout]     |
[INFO] [stdout] 547 | fn unicode_cpts_from_utf8(text: &str) -> Vec<u32> {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_cpt_to_utf8` is never used
[INFO] [stdout]    --> src/unicode.rs:551:4
[INFO] [stdout]     |
[INFO] [stdout] 551 | fn unicode_cpt_to_utf8(cpt: u32) -> Result<String, &'static str> {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: associated items `UNDEFINED`, `WHITESPACE`, `as_uint`, and `category_flag` are never used
[INFO] [stdout]    --> src/unicode.rs:482:15
[INFO] [stdout]     |
[INFO] [stdout] 481 | impl unicode_cpt_flags {
[INFO] [stdout]     | ---------------------- associated items in this implementation
[INFO] [stdout] 482 |     pub const UNDEFINED: Self = Self {
[INFO] [stdout]     |               ^^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 494 |     pub const WHITESPACE: u32 = 1 << 0;
[INFO] [stdout]     |               ^^^^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 501 |     pub fn as_uint(&self) -> u32 {
[INFO] [stdout]     |            ^^^^^^^
[INFO] [stdout] ...
[INFO] [stdout] 524 |     pub fn category_flag(&self) -> u32 {
[INFO] [stdout]     |            ^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_cpt_flags_from_cpt` is never used
[INFO] [stdout]    --> src/unicode.rs:588:4
[INFO] [stdout]     |
[INFO] [stdout] 588 | fn unicode_cpt_flags_from_cpt(cpt: u32) -> unicode_cpt_flags {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_tolower` is never used
[INFO] [stdout]    --> src/unicode.rs:615:4
[INFO] [stdout]     |
[INFO] [stdout] 615 | fn unicode_tolower(cpt: u32) -> u32 {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_len_utf8` is never used
[INFO] [stdout]    --> src/unicode.rs:669:8
[INFO] [stdout]     |
[INFO] [stdout] 669 | pub fn unicode_len_utf8(byte: u8) -> usize {
[INFO] [stdout]     |        ^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_cpts_from_utf8` is never used
[INFO] [stdout]    --> src/unicode.rs:547:4
[INFO] [stdout]     |
[INFO] [stdout] 547 | fn unicode_cpts_from_utf8(text: &str) -> Vec<u32> {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_cpt_to_utf8` is never used
[INFO] [stdout]    --> src/unicode.rs:551:4
[INFO] [stdout]     |
[INFO] [stdout] 551 | fn unicode_cpt_to_utf8(cpt: u32) -> Result<String, &'static str> {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_cpt_flags_from_cpt` is never used
[INFO] [stdout]    --> src/unicode.rs:588:4
[INFO] [stdout]     |
[INFO] [stdout] 588 | fn unicode_cpt_flags_from_cpt(cpt: u32) -> unicode_cpt_flags {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_tolower` is never used
[INFO] [stdout]    --> src/unicode.rs:615:4
[INFO] [stdout]     |
[INFO] [stdout] 615 | fn unicode_tolower(cpt: u32) -> u32 {
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `unicode_len_utf8` is never used
[INFO] [stdout]    --> src/unicode.rs:669:8
[INFO] [stdout]     |
[INFO] [stdout] 669 | pub fn unicode_len_utf8(byte: u8) -> usize {
[INFO] [stdout]     |        ^^^^^^^^^^^^^^^^
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stderr]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.54s
[INFO] running `Command { std: "docker" "inspect" "6ad0129e784d041386268666802d28e7d587900399cdc1ddccb26ec545d086af", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "6ad0129e784d041386268666802d28e7d587900399cdc1ddccb26ec545d086af", kill_on_drop: false }`
[INFO] [stdout] 6ad0129e784d041386268666802d28e7d587900399cdc1ddccb26ec545d086af
