[INFO] cloning repository https://github.com/imdigitalashish/LLM-Tokenizer-In-Rust-From-Scratch [INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/imdigitalashish/LLM-Tokenizer-In-Rust-From-Scratch" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fimdigitalashish%2FLLM-Tokenizer-In-Rust-From-Scratch", kill_on_drop: false }` [INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fimdigitalashish%2FLLM-Tokenizer-In-Rust-From-Scratch'... [INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }` [INFO] [stdout] f2bc726c374aacd629536467d27dd7792fc002bd [INFO] testing imdigitalashish/LLM-Tokenizer-In-Rust-From-Scratch against master#414482f6a0d4e7290f614300581a0b55442552a3 for pr-140985 [INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fimdigitalashish%2FLLM-Tokenizer-In-Rust-From-Scratch" "/workspace/builds/worker-1-tc1/source", kill_on_drop: false }` [INFO] [stderr] Cloning into '/workspace/builds/worker-1-tc1/source'... [INFO] [stderr] done. [INFO] validating manifest of git repo https://github.com/imdigitalashish/LLM-Tokenizer-In-Rust-From-Scratch on toolchain 414482f6a0d4e7290f614300581a0b55442552a3 [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+414482f6a0d4e7290f614300581a0b55442552a3" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }` [INFO] started tweaking git repo https://github.com/imdigitalashish/LLM-Tokenizer-In-Rust-From-Scratch [INFO] finished tweaking git repo https://github.com/imdigitalashish/LLM-Tokenizer-In-Rust-From-Scratch [INFO] tweaked toml for git repo https://github.com/imdigitalashish/LLM-Tokenizer-In-Rust-From-Scratch written to /workspace/builds/worker-1-tc1/source/Cargo.toml [INFO] crate git repo https://github.com/imdigitalashish/LLM-Tokenizer-In-Rust-From-Scratch already has a lockfile, it will not be regenerated [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+414482f6a0d4e7290f614300581a0b55442552a3" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }` [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:eea15d5475069c3ef791f10c2c6b5af2ee421ef2da1e680ecce1cba46243983b" "/opt/rustwide/cargo-home/bin/cargo" "+414482f6a0d4e7290f614300581a0b55442552a3" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }` [INFO] [stdout] c1fcc09011cc9c727bb00097276185b0d3b91322e8d5804f31e5d172d3f69525 [INFO] running `Command { std: "docker" "start" "-a" "c1fcc09011cc9c727bb00097276185b0d3b91322e8d5804f31e5d172d3f69525", kill_on_drop: false }` [INFO] running `Command { std: "docker" "inspect" "c1fcc09011cc9c727bb00097276185b0d3b91322e8d5804f31e5d172d3f69525", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "c1fcc09011cc9c727bb00097276185b0d3b91322e8d5804f31e5d172d3f69525", kill_on_drop: false }` [INFO] [stdout] c1fcc09011cc9c727bb00097276185b0d3b91322e8d5804f31e5d172d3f69525 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:eea15d5475069c3ef791f10c2c6b5af2ee421ef2da1e680ecce1cba46243983b" "/opt/rustwide/cargo-home/bin/cargo" "+414482f6a0d4e7290f614300581a0b55442552a3" "build" "--frozen" "--message-format=json", kill_on_drop: false }` [INFO] [stdout] 15a4177c7e0ae0a7af006e6f68a4e481d736a448c39859a04f1a155b65ec9292 [INFO] running `Command { std: "docker" "start" "-a" "15a4177c7e0ae0a7af006e6f68a4e481d736a448c39859a04f1a155b65ec9292", kill_on_drop: false }` [INFO] [stderr] Compiling proc-macro2 v1.0.94 [INFO] [stderr] Compiling unicode-ident v1.0.18 [INFO] [stderr] Compiling serde v1.0.219 [INFO] [stderr] Compiling serde_json v1.0.140 [INFO] [stderr] Compiling memchr v2.7.4 [INFO] [stderr] Compiling itoa v1.0.15 [INFO] [stderr] Compiling ryu v1.0.20 [INFO] [stderr] Compiling quote v1.0.39 [INFO] [stderr] Compiling syn v2.0.99 [INFO] [stderr] Compiling serde_derive v1.0.219 [INFO] [stderr] Compiling tokenizer_in_rust v0.1.0 (/opt/rustwide/workdir) [INFO] [stdout] warning: unused imports: `max` and `min` [INFO] [stdout] --> src/bpe_tokenizer.rs:3:16 [INFO] [stdout] | [INFO] [stdout] 3 | use std::cmp::{max, min}; [INFO] [stdout] | ^^^ ^^^ [INFO] [stdout] | [INFO] [stdout] = note: `#[warn(unused_imports)]` on by default [INFO] [stdout] [INFO] [stdout] [INFO] [stdout] warning: unused import: `Write` [INFO] [stdout] --> src/bpe_tokenizer.rs:6:35 [INFO] [stdout] | [INFO] [stdout] 6 | use std::io::{BufRead, BufReader, Write}; [INFO] [stdout] | ^^^^^ [INFO] [stdout] [INFO] [stdout] [INFO] [stdout] warning: methods `load_vocab_and_merges_from_openai`, `load_vocab_and_merges`, and `get_special_token_id` are never used [INFO] [stdout] --> src/bpe_tokenizer.rs:123:12 [INFO] [stdout] | [INFO] [stdout] 22 | impl RustyBPETokenizer { [INFO] [stdout] | ---------------------- methods in this implementation [INFO] [stdout] ... [INFO] [stdout] 123 | pub fn load_vocab_and_merges_from_openai>( [INFO] [stdout] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [INFO] [stdout] ... [INFO] [stdout] 307 | pub fn load_vocab_and_merges>( [INFO] [stdout] | ^^^^^^^^^^^^^^^^^^^^^ [INFO] [stdout] ... [INFO] [stdout] 340 | pub fn get_special_token_id(&self, token: &str) -> Option { [INFO] [stdout] | ^^^^^^^^^^^^^^^^^^^^ [INFO] [stdout] | [INFO] [stdout] = note: `#[warn(dead_code)]` on by default [INFO] [stdout] [INFO] [stdout] [INFO] [stderr] Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.22s [INFO] running `Command { std: "docker" "inspect" "15a4177c7e0ae0a7af006e6f68a4e481d736a448c39859a04f1a155b65ec9292", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "15a4177c7e0ae0a7af006e6f68a4e481d736a448c39859a04f1a155b65ec9292", kill_on_drop: false }` [INFO] [stdout] 15a4177c7e0ae0a7af006e6f68a4e481d736a448c39859a04f1a155b65ec9292 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:eea15d5475069c3ef791f10c2c6b5af2ee421ef2da1e680ecce1cba46243983b" "/opt/rustwide/cargo-home/bin/cargo" "+414482f6a0d4e7290f614300581a0b55442552a3" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }` [INFO] [stdout] 9f83ccd31deaef64a26f5b7f4c4827c35b2130e383f584ce6bc1a7f5a6aded8a [INFO] running `Command { std: "docker" "start" "-a" "9f83ccd31deaef64a26f5b7f4c4827c35b2130e383f584ce6bc1a7f5a6aded8a", kill_on_drop: false }` [INFO] [stderr] Compiling tokenizer_in_rust v0.1.0 (/opt/rustwide/workdir) [INFO] [stdout] warning: unused imports: `max` and `min` [INFO] [stdout] --> src/bpe_tokenizer.rs:3:16 [INFO] [stdout] | [INFO] [stdout] 3 | use std::cmp::{max, min}; [INFO] [stdout] | ^^^ ^^^ [INFO] [stdout] | [INFO] [stdout] = note: `#[warn(unused_imports)]` on by default [INFO] [stdout] [INFO] [stdout] [INFO] [stdout] warning: unused import: `Write` [INFO] [stdout] --> src/bpe_tokenizer.rs:6:35 [INFO] [stdout] | [INFO] [stdout] 6 | use std::io::{BufRead, BufReader, Write}; [INFO] [stdout] | ^^^^^ [INFO] [stdout] [INFO] [stdout] [INFO] [stdout] warning: methods `load_vocab_and_merges_from_openai`, `load_vocab_and_merges`, and `get_special_token_id` are never used [INFO] [stdout] --> src/bpe_tokenizer.rs:123:12 [INFO] [stdout] | [INFO] [stdout] 22 | impl RustyBPETokenizer { [INFO] [stdout] | ---------------------- methods in this implementation [INFO] [stdout] ... [INFO] [stdout] 123 | pub fn load_vocab_and_merges_from_openai>( [INFO] [stdout] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [INFO] [stdout] ... [INFO] [stdout] 307 | pub fn load_vocab_and_merges>( [INFO] [stdout] | ^^^^^^^^^^^^^^^^^^^^^ [INFO] [stdout] ... [INFO] [stdout] 340 | pub fn get_special_token_id(&self, token: &str) -> Option { [INFO] [stdout] | ^^^^^^^^^^^^^^^^^^^^ [INFO] [stdout] | [INFO] [stdout] = note: `#[warn(dead_code)]` on by default [INFO] [stdout] [INFO] [stdout] [INFO] [stderr] Finished `test` profile [unoptimized + debuginfo] target(s) in 0.30s [INFO] running `Command { std: "docker" "inspect" "9f83ccd31deaef64a26f5b7f4c4827c35b2130e383f584ce6bc1a7f5a6aded8a", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "9f83ccd31deaef64a26f5b7f4c4827c35b2130e383f584ce6bc1a7f5a6aded8a", kill_on_drop: false }` [INFO] [stdout] 9f83ccd31deaef64a26f5b7f4c4827c35b2130e383f584ce6bc1a7f5a6aded8a [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:eea15d5475069c3ef791f10c2c6b5af2ee421ef2da1e680ecce1cba46243983b" "/opt/rustwide/cargo-home/bin/cargo" "+414482f6a0d4e7290f614300581a0b55442552a3" "test" "--frozen", kill_on_drop: false }` [INFO] [stdout] 14d4b58db3b07d4157408a095b0c1b093e561ffc2c04989742be48e5a4ba8c8b [INFO] running `Command { std: "docker" "start" "-a" "14d4b58db3b07d4157408a095b0c1b093e561ffc2c04989742be48e5a4ba8c8b", kill_on_drop: false }` [INFO] [stderr] warning: unused imports: `max` and `min` [INFO] [stderr] --> src/bpe_tokenizer.rs:3:16 [INFO] [stderr] | [INFO] [stderr] 3 | use std::cmp::{max, min}; [INFO] [stderr] | ^^^ ^^^ [INFO] [stderr] | [INFO] [stderr] = note: `#[warn(unused_imports)]` on by default [INFO] [stderr] [INFO] [stderr] warning: unused import: `Write` [INFO] [stderr] --> src/bpe_tokenizer.rs:6:35 [INFO] [stderr] | [INFO] [stderr] 6 | use std::io::{BufRead, BufReader, Write}; [INFO] [stderr] | ^^^^^ [INFO] [stderr] [INFO] [stderr] warning: methods `load_vocab_and_merges_from_openai`, `load_vocab_and_merges`, and `get_special_token_id` are never used [INFO] [stderr] --> src/bpe_tokenizer.rs:123:12 [INFO] [stderr] | [INFO] [stderr] 22 | impl RustyBPETokenizer { [INFO] [stderr] | ---------------------- methods in this implementation [INFO] [stderr] ... [INFO] [stderr] 123 | pub fn load_vocab_and_merges_from_openai>( [INFO] [stderr] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [INFO] [stderr] ... [INFO] [stderr] 307 | pub fn load_vocab_and_merges>( [INFO] [stderr] | ^^^^^^^^^^^^^^^^^^^^^ [INFO] [stderr] ... [INFO] [stderr] 340 | pub fn get_special_token_id(&self, token: &str) -> Option { [INFO] [stderr] | ^^^^^^^^^^^^^^^^^^^^ [INFO] [stderr] | [INFO] [stderr] = note: `#[warn(dead_code)]` on by default [INFO] [stderr] [INFO] [stderr] warning: `tokenizer_in_rust` (bin "tokenizer_in_rust" test) generated 3 warnings (run `cargo fix --bin "tokenizer_in_rust" --tests` to apply 2 suggestions) [INFO] [stderr] Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s [INFO] [stderr] Running unittests src/main.rs (/opt/rustwide/target/debug/deps/tokenizer_in_rust-853abbd12f4252d4) [INFO] [stdout] [INFO] [stdout] running 0 tests [INFO] [stdout] [INFO] [stdout] test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s [INFO] [stdout] [INFO] running `Command { std: "docker" "inspect" "14d4b58db3b07d4157408a095b0c1b093e561ffc2c04989742be48e5a4ba8c8b", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "14d4b58db3b07d4157408a095b0c1b093e561ffc2c04989742be48e5a4ba8c8b", kill_on_drop: false }` [INFO] [stdout] 14d4b58db3b07d4157408a095b0c1b093e561ffc2c04989742be48e5a4ba8c8b