[INFO] fetching crate oxibonsai-eval 0.2.3...
[INFO] testing oxibonsai-eval-0.2.3 against 1.97.0-beta.6+cargoflags=--release for beta-1.98-release-3
[INFO] extracting crate oxibonsai-eval 0.2.3 into /workspace/builds/worker-4-tc1/source
[INFO] started tweaking crates.io crate oxibonsai-eval 0.2.3
[INFO] removed 0 missing tests
[INFO] finished tweaking crates.io crate oxibonsai-eval 0.2.3
[INFO] tweaked toml for crates.io crate oxibonsai-eval 0.2.3 written to /workspace/builds/worker-4-tc1/source/Cargo.toml
[INFO] validating manifest of crates.io crate oxibonsai-eval 0.2.3 on toolchain 1.97.0-beta.6
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.97.0-beta.6" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate crates.io crate oxibonsai-eval 0.2.3 already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.97.0-beta.6" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]     Updating crates.io index
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded tracing-serde v0.2.0
[INFO] [stderr]   Downloaded toml v1.1.3+spec-1.1.0
[INFO] [stderr]   Downloaded thread_local v1.1.10
[INFO] [stderr]   Downloaded monostate-impl v0.1.18
[INFO] [stderr]   Downloaded rawpointer v0.2.1
[INFO] [stderr]   Downloaded monostate v0.1.18
[INFO] [stderr]   Downloaded derive_builder_macro v0.20.2
[INFO] [stderr]   Downloaded page_size v0.6.0
[INFO] [stderr]   Downloaded rayon-cond v0.4.0
[INFO] [stderr]   Downloaded plotters-backend v0.3.7
[INFO] [stderr]   Downloaded plotters-svg v0.3.7
[INFO] [stderr]   Downloaded cast v0.3.0
[INFO] [stderr]   Downloaded rand_chacha v0.10.0
[INFO] [stderr]   Downloaded ciborium-io v0.2.2
[INFO] [stderr]   Downloaded alloca v0.4.0
[INFO] [stderr]   Downloaded dary_heap v0.3.9
[INFO] [stderr]   Downloaded toml_writer v1.1.2+spec-1.1.0
[INFO] [stderr]   Downloaded derive_builder_core v0.20.2
[INFO] [stderr]   Downloaded oxionnx-core v0.1.4
[INFO] [stderr]   Downloaded oorandom v11.1.5
[INFO] [stderr]   Downloaded criterion-plot v0.8.2
[INFO] [stderr]   Downloaded ciborium-ll v0.2.2
[INFO] [stderr]   Downloaded derive_builder v0.20.2
[INFO] [stderr]   Downloaded mach2 v0.6.0
[INFO] [stderr]   Downloaded anes v0.1.6
[INFO] [stderr]   Downloaded safetensors v0.8.0
[INFO] [stderr]   Downloaded oxionnx-proto v0.1.4
[INFO] [stderr]   Downloaded daachorse v1.0.1
[INFO] [stderr]   Downloaded ciborium v0.2.2
[INFO] [stderr]   Downloaded memmap2 v0.9.11
[INFO] [stderr]   Downloaded bytemuck v1.25.2
[INFO] [stderr]   Downloaded matrixmultiply v0.3.11
[INFO] [stderr]   Downloaded oxibonsai-rag v0.2.3
[INFO] [stderr]   Downloaded rand_distr v0.6.0
[INFO] [stderr]   Downloaded cc v1.3.0
[INFO] [stderr]   Downloaded winnow v1.0.4
[INFO] [stderr]   Downloaded uuid v1.24.0
[INFO] [stderr]   Downloaded safe_arch v1.1.0
[INFO] [stderr]   Downloaded unicode-normalization-alignments v0.1.12
[INFO] [stderr]   Downloaded compact_str v0.9.1
[INFO] [stderr]   Downloaded oxibonsai-tokenizer v0.2.3
[INFO] [stderr]   Downloaded oxibonsai-core v0.2.3
[INFO] [stderr]   Downloaded fancy-regex v0.17.0
[INFO] [stderr]   Downloaded criterion v0.8.2
[INFO] [stderr]   Downloaded plotters v0.3.7
[INFO] [stderr]   Downloaded esaxx-rs v0.1.10
[INFO] [stderr]   Downloaded wide v1.5.0
[INFO] [stderr]   Downloaded tokenizers v0.23.1
[INFO] [stderr]   Downloaded ndarray v0.17.2
[INFO] [stderr]   Downloaded oxibonsai-runtime v0.2.3
[INFO] [stderr]   Downloaded oxibonsai-model v0.2.3
[INFO] [stderr]   Downloaded oxibonsai-kernels v0.2.3
[INFO] [stderr]   Downloaded spm_precompiled v0.1.4
[INFO] [stderr]   Downloaded scirs2-core v0.6.2
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-m" "1610612736" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:00c5645b54fe3ce5dae1417175e3c7fb6a6646c021d554ecd63a097b8b9f3602" "sleep" "infinity", kill_on_drop: false }`
[INFO] [stdout] 764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593
[INFO] running `Command { std: "docker" "start" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-w" "/opt/rustwide/workdir" "--user" "0:0" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593" "/opt/rustwide/cargo-home/bin/cargo" "+1.97.0-beta.6" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593" "/opt/rustwide/cargo-home/bin/cargo" "+1.97.0-beta.6" "build" "--frozen" "--message-format=json" "--release", kill_on_drop: false }`
[INFO] [stderr]    Compiling proc-macro2 v1.0.107
[INFO] [stderr]    Compiling quote v1.0.47
[INFO] [stderr]    Compiling serde_core v1.0.229
[INFO] [stderr]    Compiling libc v0.2.189
[INFO] [stderr]    Compiling serde v1.0.229
[INFO] [stderr]    Compiling zerocopy v0.8.55
[INFO] [stderr]    Compiling memchr v2.8.3
[INFO] [stderr]    Compiling libm v0.2.16
[INFO] [stderr]    Compiling crossbeam-utils v0.8.22
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]    Compiling crossbeam-epoch v0.9.20
[INFO] [stderr]    Compiling rand_core v0.10.1
[INFO] [stderr]    Compiling tracing-core v0.1.36
[INFO] [stderr]    Compiling crossbeam-deque v0.8.7
[INFO] [stderr]    Compiling smallvec v1.15.2
[INFO] [stderr]    Compiling thiserror v2.0.19
[INFO] [stderr]    Compiling syn v2.0.119
[INFO] [stderr]    Compiling syn v3.0.3
[INFO] [stderr]    Compiling getrandom v0.4.3
[INFO] [stderr]    Compiling rayon-core v1.13.0
[INFO] [stderr]    Compiling zmij v1.0.23
[INFO] [stderr]    Compiling either v1.16.0
[INFO] [stderr]    Compiling serde_json v1.0.151
[INFO] [stderr]    Compiling ident_case v1.0.1
[INFO] [stderr]    Compiling fnv v1.0.7
[INFO] [stderr]    Compiling strsim v0.11.1
[INFO] [stderr]    Compiling parking_lot_core v0.9.12
[INFO] [stderr]    Compiling aho-corasick v1.1.4
[INFO] [stderr]    Compiling rayon v1.12.0
[INFO] [stderr]    Compiling matrixmultiply v0.3.11
[INFO] [stderr]    Compiling regex-syntax v0.8.11
[INFO] [stderr]    Compiling cpufeatures v0.3.0
[INFO] [stderr]    Compiling foldhash v0.2.0
[INFO] [stderr]    Compiling chacha20 v0.10.1
[INFO] [stderr]    Compiling parking_lot v0.12.5
[INFO] [stderr]    Compiling serde_derive v1.0.229
[INFO] [stderr]    Compiling thiserror-impl v2.0.19
[INFO] [stderr]    Compiling getrandom v0.3.4
[INFO] [stderr]    Compiling rawpointer v0.2.1
[INFO] [stderr]    Compiling rustversion v1.0.23
[INFO] [stderr]    Compiling bytemuck v1.25.2
[INFO] [stderr]    Compiling safe_arch v1.1.0
[INFO] [stderr]    Compiling darling_core v0.20.11
[INFO] [stderr]    Compiling regex-automata v0.4.16
[INFO] [stderr]    Compiling zerocopy-derive v0.8.55
[INFO] [stderr]    Compiling tracing-attributes v0.1.31
[INFO] [stderr]    Compiling rand v0.10.2
[INFO] [stderr]    Compiling darling_macro v0.20.11
[INFO] [stderr]    Compiling darling v0.20.11
[INFO] [stderr]    Compiling approx v0.5.1
[INFO] [stderr]    Compiling num-complex v0.4.6
[INFO] [stderr]    Compiling num-integer v0.1.46
[INFO] [stderr]    Compiling tracing v0.1.44
[INFO] [stderr]    Compiling memmap2 v0.9.11
[INFO] [stderr]    Compiling bitflags v2.13.1
[INFO] [stderr]    Compiling scirs2-core v0.6.2
[INFO] [stderr]    Compiling byteorder v1.5.0
[INFO] [stderr]    Compiling paste v1.0.15
[INFO] [stderr]    Compiling rustix v1.1.4
[INFO] [stderr]    Compiling ahash v0.8.12
[INFO] [stderr]    Compiling chrono v0.4.45
[INFO] [stderr]    Compiling ndarray v0.17.2
[INFO] [stderr]    Compiling rand_distr v0.6.0
[INFO] [stderr]    Compiling derive_builder_core v0.20.2
[INFO] [stderr]    Compiling wide v1.5.0
[INFO] [stderr]    Compiling rand_core v0.9.5
[INFO] [stderr]    Compiling uuid v1.24.0
[INFO] [stderr]    Compiling hashbrown v0.17.1
[INFO] [stderr]    Compiling errno v0.3.14
[INFO] [stderr]    Compiling libloading v0.9.0
[INFO] [stderr]    Compiling anyhow v1.0.104
[INFO] [stderr]    Compiling log v0.4.33
[INFO] [stderr]    Compiling fastrand v2.5.0
[INFO] [stderr]    Compiling esaxx-rs v0.1.10
[INFO] [stderr]    Compiling oxibonsai-kernels v0.2.3
[INFO] [stderr]    Compiling bit-vec v0.8.0
[INFO] [stderr]    Compiling minimal-lexical v0.2.1
[INFO] [stderr]    Compiling tempfile v3.27.0
[INFO] [stderr]    Compiling bit-set v0.8.0
[INFO] [stderr]    Compiling nom v7.1.3
[INFO] [stderr]    Compiling signal-hook-registry v1.4.8
[INFO] [stderr]    Compiling ppv-lite86 v0.2.21
[INFO] [stderr]    Compiling half v2.7.1
[INFO] [stderr]    Compiling derive_builder_macro v0.20.2
[INFO] [stderr]    Compiling castaway v0.2.4
[INFO] [stderr]    Compiling rand_chacha v0.10.0
[INFO] [stderr]    Compiling oxibonsai-core v0.2.3
[INFO] [stderr]    Compiling rand_chacha v0.9.0
[INFO] [stderr]    Compiling oxionnx-core v0.1.4
[INFO] [stderr]    Compiling monostate-impl v0.1.18
[INFO] [stderr]    Compiling tokio-macros v2.7.1
[INFO] [stderr]    Compiling hashbrown v0.16.1
[INFO] [stderr]    Compiling itertools v0.14.0
[INFO] [stderr]    Compiling socket2 v0.6.5
[INFO] [stderr]    Compiling mio v1.2.2
[INFO] [stderr]    Compiling lazy_static v1.5.0
[INFO] [stderr]    Compiling macro_rules_attribute-proc_macro v0.2.2
[INFO] [stderr]    Compiling winnow v1.0.4
[INFO] [stderr]    Compiling static_assertions v1.1.0
[INFO] [stderr]    Compiling bytes v1.12.1
[INFO] [stderr]    Compiling base64 v0.13.1
[INFO] [stderr]    Compiling toml_parser v1.1.2+spec-1.1.0
[INFO] [stderr]    Compiling spm_precompiled v0.1.4
[INFO] [stderr]    Compiling rayon-cond v0.4.0
[INFO] [stderr]    Compiling tokio v1.53.1
[INFO] [stderr]    Compiling compact_str v0.9.1
[INFO] [stderr]    Compiling macro_rules_attribute v0.2.2
[INFO] [stderr]    Compiling sharded-slab v0.1.7
[INFO] [stderr]    Compiling safetensors v0.8.0
[INFO] [stderr]    Compiling monostate v0.1.18
[INFO] [stderr]    Compiling oxionnx-proto v0.1.4
[INFO] [stderr]    Compiling rand v0.9.5
[INFO] [stderr]    Compiling derive_builder v0.20.2
[INFO] [stderr]    Compiling fancy-regex v0.17.0
[INFO] [stderr]    Compiling tracing-log v0.2.0
[INFO] [stderr]    Compiling matchers v0.2.0
[INFO] [stderr]    Compiling regex v1.13.1
[INFO] [stderr]    Compiling tracing-serde v0.2.0
[INFO] [stderr]    Compiling dary_heap v0.3.9
[INFO] [stderr]    Compiling serde_spanned v1.1.1
[INFO] [stderr]    Compiling toml_datetime v1.1.1+spec-1.1.0
[INFO] [stderr]    Compiling unicode-normalization-alignments v0.1.12
[INFO] [stderr]    Compiling thread_local v1.1.10
[INFO] [stderr]    Compiling daachorse v1.0.1
[INFO] [stderr]    Compiling futures-core v0.3.33
[INFO] [stderr]    Compiling unicode_categories v0.1.1
[INFO] [stderr]    Compiling nu-ansi-term v0.50.3
[INFO] [stderr]    Compiling toml_writer v1.1.2+spec-1.1.0
[INFO] [stderr]    Compiling tracing-subscriber v0.3.23
[INFO] [stderr]    Compiling toml v1.1.3+spec-1.1.0
[INFO] [stderr]    Compiling tokenizers v0.23.1
[INFO] [stderr]    Compiling oxibonsai-rag v0.2.3
[INFO] [stderr]    Compiling tokio-stream v0.1.19
[INFO] [stderr]    Compiling oxibonsai-tokenizer v0.2.3
[INFO] [stderr]    Compiling oxibonsai-model v0.2.3
[INFO] [stderr]    Compiling oxibonsai-runtime v0.2.3
[INFO] [stderr]    Compiling oxibonsai-eval v0.2.3 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `release` profile [optimized] target(s) in 7m 53s
[INFO] running `Command { std: "docker" "inspect" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593" "/opt/rustwide/cargo-home/bin/cargo" "+1.97.0-beta.6" "test" "--frozen" "--no-run" "--message-format=json" "--release", kill_on_drop: false }`
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]    Compiling shlex v2.0.1
[INFO] [stderr]    Compiling fancy-regex v0.17.0
[INFO] [stderr]    Compiling ciborium-io v0.2.2
[INFO] [stderr]    Compiling plotters-backend v0.3.7
[INFO] [stderr]    Compiling clap_builder v4.6.2
[INFO] [stderr]    Compiling itertools v0.13.0
[INFO] [stderr]    Compiling wait-timeout v0.2.1
[INFO] [stderr]    Compiling ciborium-ll v0.2.2
[INFO] [stderr]    Compiling cc v1.3.0
[INFO] [stderr]    Compiling plotters-svg v0.3.7
[INFO] [stderr]    Compiling same-file v1.0.6
[INFO] [stderr]    Compiling cast v0.3.0
[INFO] [stderr]    Compiling walkdir v2.5.0
[INFO] [stderr]    Compiling rusty-fork v0.3.1
[INFO] [stderr]    Compiling ciborium v0.2.2
[INFO] [stderr]    Compiling tokenizers v0.23.1
[INFO] [stderr]    Compiling num-integer v0.1.46
[INFO] [stderr]    Compiling num-complex v0.4.6
[INFO] [stderr]    Compiling approx v0.5.1
[INFO] [stderr]    Compiling chrono v0.4.45
[INFO] [stderr]    Compiling rand_distr v0.6.0
[INFO] [stderr]    Compiling ndarray v0.17.2
[INFO] [stderr]    Compiling alloca v0.4.0
[INFO] [stderr]    Compiling criterion-plot v0.8.2
[INFO] [stderr]    Compiling plotters v0.3.7
[INFO] [stderr]    Compiling rand_xorshift v0.4.0
[INFO] [stderr]    Compiling tinytemplate v1.2.1
[INFO] [stderr]    Compiling clap v4.6.4
[INFO] [stderr]    Compiling page_size v0.6.0
[INFO] [stderr]    Compiling anes v0.1.6
[INFO] [stderr]    Compiling oorandom v11.1.5
[INFO] [stderr]    Compiling proptest v1.11.0
[INFO] [stderr]    Compiling criterion v0.8.2
[INFO] [stderr]    Compiling scirs2-core v0.6.2
[INFO] [stderr]    Compiling oxibonsai-kernels v0.2.3
[INFO] [stderr]    Compiling oxibonsai-model v0.2.3
[INFO] [stderr]    Compiling oxibonsai-runtime v0.2.3
[INFO] [stderr]    Compiling oxibonsai-eval v0.2.3 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `release` profile [optimized] target(s) in 3m 12s
[INFO] running `Command { std: "docker" "inspect" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593" "/opt/rustwide/cargo-home/bin/cargo" "+1.97.0-beta.6" "test" "--frozen" "--release", kill_on_drop: false }`
[INFO] [stderr]     Finished `release` profile [optimized] target(s) in 0.42s
[INFO] [stderr]      Running unittests src/lib.rs (/opt/rustwide/target/release/deps/oxibonsai_eval-7c68e082172d60da)
[INFO] [stdout] 
[INFO] [stdout] running 32 tests
[INFO] [stdout] test boolq::tests::extract_answer_leading_whitespace ... ok
[INFO] [stdout] test boolq::tests::extract_answer_none_for_garbage ... ok
[INFO] [stdout] test boolq::tests::extract_answer_short_string_no_panic ... ok
[INFO] [stdout] test boolq::tests::extract_answer_case_insensitive ... ok
[INFO] [stdout] test boolq::tests::extract_answer_no ... ok
[INFO] [stdout] test boolq::tests::extract_answer_yes ... ok
[INFO] [stdout] test tests::test_accuracy_result_pct ... ok
[INFO] [stdout] test tests::test_bits_per_byte ... ok
[INFO] [stdout] test tests::test_eval_dataset_from_jsonl ... ok
[INFO] [stdout] test tests::test_eval_dataset_sample_deterministic ... ok
[INFO] [stdout] test tests::test_eval_dataset_split ... ok
[INFO] [stdout] test tests::test_eval_report_summary ... ok
[INFO] [stdout] test tests::test_eval_report_to_json ... ok
[INFO] [stdout] test tests::test_eval_report_to_markdown ... ok
[INFO] [stdout] test tests::test_exact_match_evaluator_normalized ... ok
[INFO] [stdout] test tests::test_mc_dataset_filter_by_subject ... ok
[INFO] [stdout] test tests::test_mc_dataset_subjects ... ok
[INFO] [stdout] test tests::test_mc_evaluator_extract_answer ... ok
[INFO] [stdout] test tests::test_mc_evaluator_format_question ... ok
[INFO] [stdout] test tests::test_exact_match_evaluator_basic ... ok
[INFO] [stdout] test tests::test_mc_evaluator_score_incorrect ... ok
[INFO] [stdout] test tests::test_percentile_median ... ok
[INFO] [stdout] test tests::test_percentile_p95 ... ok
[INFO] [stdout] test tests::test_perplexity_batch ... ok
[INFO] [stdout] test winogrande::tests::winogrande_as_mc_dataset_correct_answer_index_one_for_answer2 ... ok
[INFO] [stdout] test winogrande::tests::winogrande_as_mc_dataset_correct_answer_index_zero_for_answer1 ... ok
[INFO] [stdout] test winogrande::tests::winogrande_dataset_len ... ok
[INFO] [stdout] test tests::test_perplexity_uniform_predictions ... ok
[INFO] [stdout] test winogrande::tests::winogrande_empty_dataset ... ok
[INFO] [stdout] test tests::test_mc_evaluator_score_correct ... ok
[INFO] [stdout] test tests::test_perplexity_perfect_predictions ... ok
[INFO] [stdout] test tests::test_throughput_result_meets_target ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/accuracy_logit_tests.rs (/opt/rustwide/target/release/deps/accuracy_logit_tests-39a6b78e193c56e2)
[INFO] [stdout] 
[INFO] [stdout] running 4 tests
[INFO] [stdout] test score_empty_slate_not_credited_as_correct ... ok
[INFO] [stdout] test score_normal_slate_picks_argmax ... ok
[INFO] [stderr]      Running tests/arc_tests.rs (/opt/rustwide/target/release/deps/arc_tests-efae0ef43cfbc0a4)
[INFO] [stdout] test score_partial_nan_slate_picks_best_real_value ... ok
[INFO] [stdout] test score_all_nan_slate_not_credited_as_correct ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] running 10 tests
[INFO] [stdout] test arc_empty_dataset_returns_zero_accuracy ... ok
[INFO] [stdout] test arc_evaluate_completions_all_correct ... ok
[INFO] [stdout] test arc_evaluate_completions_all_wrong ... ok
[INFO] [stdout] test arc_evaluate_logits_picks_max ... ok
[INFO] [stdout] test arc_evaluator_easy_creation ... ok
[INFO] [stdout] test arc_evaluator_challenge_creation ... ok
[INFO] [stdout] test arc_evaluate_completions_partial ... ok
[INFO] [stdout] test arc_five_choice_question ... ok
[INFO] [stdout] test arc_result_from_accuracy_result ... ok
[INFO] [stdout] test arc_split_name ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/bleu_tests.rs (/opt/rustwide/target/release/deps/bleu_tests-a96c4a3d77d40c14)
[INFO] [stdout] 
[INFO] [stdout] running 19 tests
[INFO] [stdout] test bleu_empty_candidate_zero ... ok
[INFO] [stdout] test bleu_disjoint_is_zero ... ok
[INFO] [stdout] test bleu_in_unit_interval ... ok
[INFO] [stdout] test bleu_identical_is_one ... ok
[INFO] [stdout] test bleu_no_references_zero ... ok
[INFO] [stdout] test bleu_tokens_must_overlap ... ok
[INFO] [stdout] test brevity_penalty_equal_length ... ok
[INFO] [stdout] test brevity_penalty_applied_when_short ... ok
[INFO] [stdout] test brevity_penalty_one_when_longer ... ok
[INFO] [stdout] test corpus_bleu_empty_input ... ok
[INFO] [stdout] test corpus_bleu_accumulates ... ok
[INFO] [stdout] test corpus_bleu_identical_is_one ... ok
[INFO] [stdout] test multi_reference_picks_best_match ... ok
[INFO] [stdout] test smoothing_add_one_recovers_positive_sparse ... ok
[INFO] [stdout] test corpus_greater_or_equal_than_some_sentence ... ok
[INFO] [stdout] test smoothing_none_collapses_when_missing_higher_order ... ok
[INFO] [stdout] test multi_reference_length_closest_used ... ok
[INFO] [stdout] test smoothing_none_vs_add_one_ordering ... ok
[INFO] [stdout] test smoothing_exp_decay_strictly_positive ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/boolq_tests.rs (/opt/rustwide/target/release/deps/boolq_tests-4ccd3e839812a251)
[INFO] [stdout] 
[INFO] [stdout] running 16 tests
[INFO] [stdout] test boolq_accuracy_pct_matches_accuracy ... ok
[INFO] [stdout] test boolq_build_prompt_format ... ok
[INFO] [stdout] test boolq_empty_dataset ... ok
[INFO] [stdout] test boolq_default_impl ... ok
[INFO] [stdout] test boolq_evaluate_completions_all_wrong ... ok
[INFO] [stdout] test boolq_evaluate_completions_counts_yes_no ... ok
[INFO] [stdout] test boolq_evaluate_completions_perfect ... ok
[INFO] [stdout] test boolq_evaluate_logits_no ... ok
[INFO] [stdout] test boolq_evaluate_logits_yes ... ok
[INFO] [stdout] test boolq_extract_none_for_ambiguous ... ok
[INFO] [stdout] test boolq_extract_no_variants ... ok
[INFO] [stdout] test boolq_extract_yes_variants ... ok
[INFO] [stdout] test boolq_leading_whitespace ... ok
[INFO] [stdout] test boolq_score_correct_yes ... ok
[INFO] [stdout] test boolq_short_string_no_panic ... ok
[INFO] [stdout] test boolq_score_correct_no ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/bootstrap_tests.rs (/opt/rustwide/target/release/deps/bootstrap_tests-d52b95dabe682e71)
[INFO] [stdout] 
[INFO] [stdout] running 10 tests
[INFO] [stdout] test bootstrap_bounds_within_sample_range ... ok
[INFO] [stdout] test bootstrap_ci_constant_sample_has_zero_width ... ok
[INFO] [stdout] test bootstrap_ci_contains_mean ... ok
[INFO] [stdout] test bootstrap_confidence_out_of_range_errors ... ok
[INFO] [stdout] test bootstrap_empty_samples_errors ... ok
[INFO] [stdout] test bootstrap_different_seeds_may_differ ... ok
[INFO] [stdout] test bootstrap_preserves_confidence_level ... ok
[INFO] [stdout] test bootstrap_seed_is_deterministic ... ok
[INFO] [stdout] test bootstrap_zero_resamples_zero_width ... ok
[INFO] [stdout] test bootstrap_width_tightens_with_more_resamples ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/calibration_tests.rs (/opt/rustwide/target/release/deps/calibration_tests-86ab49963104066d)
[INFO] [stdout] 
[INFO] [stdout] running 15 tests
[INFO] [stdout] test brier_label_out_of_range_errors ... ok
[INFO] [stdout] test brier_length_mismatch_errors ... ok
[INFO] [stdout] test brier_is_in_unit_interval_two_class ... ok
[INFO] [stdout] test calibration_all_perfect ... ok
[INFO] [stdout] test brier_perfect_is_zero ... ok
[INFO] [stdout] test ece_bin_boundary_last_bin_inclusive ... ok
[INFO] [stdout] test ece_empty_is_zero ... ok
[INFO] [stdout] test ece_in_unit_interval ... ok
[INFO] [stdout] test ece_length_mismatch_errors ... ok
[INFO] [stdout] test ece_n_bins_one_behaves_as_global_gap ... ok
[INFO] [stdout] test ece_overconfident_model ... ok
[INFO] [stdout] test ece_perfect_calibration_high_confidence ... ok
[INFO] [stdout] test ece_uniform_known_value ... ok
[INFO] [stdout] test nll_length_mismatch_errors ... ok
[INFO] [stdout] test nll_perfectly_confident_is_near_zero ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/chrf_tests.rs (/opt/rustwide/target/release/deps/chrf_tests-0c7098a6f5362029)
[INFO] [stdout] 
[INFO] [stdout] running 15 tests
[INFO] [stdout] test chrf_both_empty_is_one ... ok
[INFO] [stdout] test chrf_default_order_is_six ... ok
[INFO] [stdout] test chrf_disjoint_is_zero_or_tiny ... ok
[INFO] [stdout] test chrf_default_beta_is_two ... ok
[INFO] [stdout] test chrf_handles_multibyte_utf8 ... ok
[INFO] [stdout] test chrf_one_empty_is_zero ... ok
[INFO] [stdout] test chrf_identical_is_one ... ok
[INFO] [stdout] test chrf_order_sensitivity_char ... ok
[INFO] [stdout] test chrf_partial_match_between_disjoint_and_identical ... ok
[INFO] [stdout] test chrf_plus_plus_has_word_order_two ... ok
[INFO] [stdout] test chrf_score_in_unit_interval ... ok
[INFO] [stdout] test chrf_utf8_vs_ascii_are_independent ... ok
[INFO] [stdout] test chrf_whitespace_matters_for_char_ngrams ... ok
[INFO] [stdout] test chrf_with_custom_order ... ok
[INFO] [stdout] test chrf_with_lower_beta_weights_precision ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/gsm8k_tests.rs (/opt/rustwide/target/release/deps/gsm8k_tests-15c52de0488916be)
[INFO] [stdout] 
[INFO] [stdout] running 15 tests
[INFO] [stdout] test evaluate_dataset_empty ... ok
[INFO] [stdout] test evaluate_dataset_all_correct ... ok
[INFO] [stdout] test evaluate_dataset_partial ... ok
[INFO] [stdout] test evaluate_dataset_skips_none_expected_output ... ok
[INFO] [stdout] test extract_final_answer_decimal ... ok
[INFO] [stdout] test extract_final_answer_negative ... ok
[INFO] [stdout] test extract_final_answer_none_when_missing ... ok
[INFO] [stdout] test extract_final_answer_picks_last_marker ... ok
[INFO] [stdout] test extract_final_answer_simple_integer ... ok
[INFO] [stdout] test gsm8k_result_helper_methods ... ok
[INFO] [stdout] test extract_final_answer_with_commas ... ok
[INFO] [stdout] test score_no_marker_in_completion ... ok
[INFO] [stdout] test score_integer_vs_integer ... ok
[INFO] [stdout] test score_float_vs_integer ... ok
[INFO] [stdout] test score_wrong_answer ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/hellaswag_tests.rs (/opt/rustwide/target/release/deps/hellaswag_tests-a9889eeaefe41ac3)
[INFO] [stdout] 
[INFO] [stdout] running 13 tests
[INFO] [stdout] test test_hellaswag_4_choices_enforced ... ok
[INFO] [stdout] test test_hellaswag_all_wrong_completions ... ok
[INFO] [stdout] test test_hellaswag_accuracy_pct ... ok
[INFO] [stdout] test test_hellaswag_dataset_from_items ... ok
[INFO] [stdout] test test_hellaswag_as_mc_dataset_conversion ... ok
[INFO] [stdout] test test_hellaswag_dataset_len ... ok
[INFO] [stdout] test test_hellaswag_default_impl ... ok
[INFO] [stdout] test test_hellaswag_empty_dataset ... ok
[INFO] [stdout] test test_hellaswag_logit_argmax_correct ... ok
[INFO] [stdout] test test_hellaswag_logit_argmax_wrong ... ok
[INFO] [stdout] test test_hellaswag_partial_score ... ok
[INFO] [stdout] test test_hellaswag_perfect_completions ... ok
[INFO] [stdout] test test_hellaswag_from_jsonl ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/meteor_tests.rs (/opt/rustwide/target/release/deps/meteor_tests-4b49b19e1deeac9b)
[INFO] [stdout] 
[INFO] [stdout] running 15 tests
[INFO] [stdout] test align_tokens_full_match ... ok
[INFO] [stdout] test meteor_both_empty_is_one ... ok
[INFO] [stdout] test align_tokens_each_ref_used_once ... ok
[INFO] [stdout] test align_tokens_no_overlap ... ok
[INFO] [stdout] test meteor_default_gamma_beta ... ok
[INFO] [stdout] test meteor_fragmentation_penalises_shuffled ... ok
[INFO] [stdout] test meteor_disjoint_is_zero ... ok
[INFO] [stdout] test meteor_identical_is_one ... ok
[INFO] [stdout] test meteor_default_alpha_is_09 ... ok
[INFO] [stdout] test meteor_multi_empty_refs_is_zero ... ok
[INFO] [stdout] test meteor_multi_picks_max ... ok
[INFO] [stderr]      Running tests/mmlu_tests.rs (/opt/rustwide/target/release/deps/mmlu_tests-dd802be0dce586b0)
[INFO] [stdout] test meteor_one_empty_is_zero ... ok
[INFO] [stdout] test meteor_partial_overlap_positive ... ok
[INFO] [stdout] test meteor_range_is_unit_interval ... ok
[INFO] [stdout] test meteor_recall_precision_bounds ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] running 20 tests
[INFO] [stdout] test by_subject_completions_correct_subjects ... ok
[INFO] [stdout] test completions_partial_accuracy ... ok
[INFO] [stdout] test accuracy_pct_matches_accuracy_times_100 ... ok
[INFO] [stdout] test empty_dataset_completions_zero_accuracy ... ok
[INFO] [stdout] test empty_dataset_logits_zero_accuracy ... ok
[INFO] [stdout] test by_subject_logits_correct_subjects ... ok
[INFO] [stdout] test completions_all_wrong_zero_accuracy ... ok
[INFO] [stdout] test evaluate_by_subject_completions_returns_subject_map ... ok
[INFO] [stdout] test completions_all_correct_100pct ... ok
[INFO] [stdout] test evaluate_by_subject_logits_returns_subject_map ... ok
[INFO] [stdout] test evaluator_new_default_are_equivalent ... ok
[INFO] [stdout] test fewer_completions_than_questions_only_annotated_scored ... ok
[INFO] [stdout] test fewer_logits_than_questions_only_annotated_scored ... ok
[INFO] [stdout] test logits_all_correct_100pct ... ok
[INFO] [stdout] test logits_all_wrong_zero_accuracy ... ok
[INFO] [stdout] test logits_partial_accuracy ... ok
[INFO] [stdout] test multiple_subjects_independent_breakdown ... ok
[INFO] [stdout] test subject_extracted_from_id_no_slash ... ok
[INFO] [stdout] test subject_extracted_from_id_when_field_absent ... ok
[INFO] [stdout] test subject_field_takes_priority_over_id ... ok
[INFO] [stderr]      Running tests/property_tests.rs (/opt/rustwide/target/release/deps/property_tests-eb452e9bb1283273)
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] running 11 tests
[INFO] [stdout] test prop_ece_any_bin_count_in_range ... ok
[INFO] [stdout] test prop_ece_bounded ... ok
[INFO] [stdout] test prop_ece_clamped_inputs_ok ... ok
[INFO] [stdout] test prop_f1_identical_is_one ... ok
[INFO] [stdout] test prop_bleu_in_unit_interval ... ok
[INFO] [stdout] test prop_bleu_identical_is_one ... ok
[INFO] [stdout] test prop_f1_in_unit_interval ... ok
[INFO] [stdout] test prop_chrf_identical_is_one ... ok
[INFO] [stdout] test prop_chrf_in_unit_interval ... ok
[INFO] [stdout] test prop_chrf_symmetric_in_range ... ok
[INFO] [stdout] test prop_bleu_precisions_in_unit_interval ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/qa_tests.rs (/opt/rustwide/target/release/deps/qa_tests-9e5ffaf9ac74a589)
[INFO] [stdout] 
[INFO] [stdout] running 18 tests
[INFO] [stdout] test corpus_em_f1_empty ... ok
[INFO] [stdout] test corpus_em_f1_averages ... ok
[INFO] [stdout] test exact_match_distinct_strings ... ok
[INFO] [stdout] test em_vs_f1_divergence ... ok
[INFO] [stdout] test exact_match_identical_after_normalisation ... ok
[INFO] [stdout] test f1_both_empty_one ... ok
[INFO] [stdout] test f1_identical_is_one ... ok
[INFO] [stdout] test f1_empty_prediction_and_nonempty_ref_zero ... ok
[INFO] [stdout] test f1_partial_overlap ... ok
[INFO] [stdout] test normalize_collapses_whitespace ... ok
[INFO] [stdout] test normalize_empty_is_empty ... ok
[INFO] [stdout] test normalize_keeps_article_inside_token ... ok
[INFO] [stdout] test normalize_lowercases ... ok
[INFO] [stdout] test normalize_removes_articles ... ok
[INFO] [stdout] test normalize_strips_punctuation ... ok
[INFO] [stdout] test normalize_tokens_split ... ok
[INFO] [stdout] test score_multi_empty_refs_zero ... ok
[INFO] [stderr]      Running tests/rouge_tests.rs (/opt/rustwide/target/release/deps/rouge_tests-10ae7a55c7425a19)
[INFO] [stdout] test score_multi_picks_max ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] running 16 tests
[INFO] [stdout] test corpus_rouge_single_pair ... ok
[INFO] [stdout] test corpus_rouge_summary_nonempty ... ok
[INFO] [stdout] test ngram_counts_unigrams ... ok
[INFO] [stdout] test rouge_1_disjoint ... ok
[INFO] [stdout] test ngram_counts_bigrams ... ok
[INFO] [stdout] test rouge_1_identical ... ok
[INFO] [stdout] test rouge_1_partial ... ok
[INFO] [stdout] test rouge_1_recall_focus ... ok
[INFO] [stdout] test rouge_2_disjoint ... ok
[INFO] [stdout] test rouge_2_identical ... ok
[INFO] [stdout] test rouge_l_identical ... ok
[INFO] [stdout] test rouge_l_lcs_length_correct ... ok
[INFO] [stdout] test rouge_multi_ref_takes_max ... ok
[INFO] [stdout] test rouge_s_identical ... ok
[INFO] [stdout] test tokenize_basic ... ok
[INFO] [stdout] test tokenize_punctuation ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/streaming_tests.rs (/opt/rustwide/target/release/deps/streaming_tests-8f6eb3c26b9bf4d4)
[INFO] [stdout] 
[INFO] [stdout] running 10 tests
[INFO] [stdout] test online_accuracy_all_correct ... ok
[INFO] [stdout] test online_accuracy_empty_is_zero ... ok
[INFO] [stdout] test online_accuracy_mixed_outcomes ... ok
[INFO] [stdout] test online_accuracy_in_unit_interval ... ok
[INFO] [stdout] test online_perplexity_batch_equivalent_to_streaming ... ok
[INFO] [stdout] test online_perplexity_empty_is_infinity ... ok
[INFO] [stdout] test online_perplexity_partial_early_stop_stable ... ok
[INFO] [stdout] test online_perplexity_perfect_prediction_one ... ok
[INFO] [stdout] test online_perplexity_reset ... ok
[INFO] [stdout] test online_accuracy_reset ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/truthfulqa_tests.rs (/opt/rustwide/target/release/deps/truthfulqa_tests-b9c0be529040f6b7)
[INFO] [stdout] 
[INFO] [stdout] running 23 tests
[INFO] [stdout] test test_mc1_all_nan_logits_not_credited_as_correct ... ok
[INFO] [stdout] test test_mc1_all_wrong ... ok
[INFO] [stdout] test test_mc1_empty_logits_skipped_regardless_of_gold_index ... ok
[INFO] [stdout] test test_mc1_empty_logits_not_credited_as_correct ... ok
[INFO] [stdout] test test_mc1_mode_is_mc1 ... ok
[INFO] [stdout] test test_mc1_out_of_range_correct_idx_counts_as_skipped ... ok
[INFO] [stdout] test test_mc1_partial ... ok
[INFO] [stdout] test test_mc1_partial_nan_logits_picks_best_real_value ... ok
[INFO] [stdout] test test_mc1_perfect_score ... ok
[INFO] [stdout] test test_mc1_single_item ... ok
[INFO] [stdout] test test_mc2_all_in_range_correct_indices_scored_normally ... ok
[INFO] [stdout] test test_mc2_all_correct_mass_on_correct ... ok
[INFO] [stdout] test test_mc2_all_mass_on_wrong ... ok
[INFO] [stdout] test test_mc2_all_nan_logits_scores_zero_not_nan ... ok
[INFO] [stdout] test test_mc2_continuous_score_not_just_01 ... ok
[INFO] [stdout] test test_mc2_mixed_correct_incorrect ... ok
[INFO] [stdout] test test_mc2_mode_is_mc2 ... ok
[INFO] [stdout] test test_mc2_partial_out_of_range_correct_indices_counts_as_skipped ... ok
[INFO] [stdout] test test_mc2_single_correct_only ... ok
[INFO] [stdout] test test_truthfulqa_dataset_from_items ... ok
[INFO] [stdout] test test_mc2_softmax_normalization ... ok
[INFO] [stdout] test test_truthfulqa_empty_mc1 ... ok
[INFO] [stdout] test test_truthfulqa_empty_mc2 ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 23 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/winogrande_tests.rs (/opt/rustwide/target/release/deps/winogrande_tests-323dbdbf3396d8f8)
[INFO] [stdout] 
[INFO] [stdout] running 12 tests
[INFO] [stdout] test winogrande_as_mc_dataset_answer2_maps_to_index1 ... ok
[INFO] [stdout] test winogrande_default_impl ... ok
[INFO] [stdout] test winogrande_as_mc_dataset_has_two_choices ... ok
[INFO] [stdout] test winogrande_batch_mixed_answers ... ok
[INFO] [stdout] test winogrande_evaluate_completions_all_wrong ... ok
[INFO] [stdout] test winogrande_construction_and_len ... ok
[INFO] [stdout] test winogrande_evaluate_completions_partial ... ok
[INFO] [stdout] test winogrande_empty_dataset_returns_zero_accuracy ... ok
[INFO] [stdout] test winogrande_evaluate_logits_argmax ... ok
[INFO] [stdout] test winogrande_evaluate_logits_argmax_wrong ... ok
[INFO] [stdout] test winogrande_evaluate_completions_perfect ... ok
[INFO] [stdout] test winogrande_accuracy_pct_matches_accuracy ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stderr]    Doc-tests oxibonsai_eval
[INFO] [stdout] 
[INFO] [stdout] running 8 tests
[INFO] [stdout] test src/hellaswag.rs - hellaswag (line 10) ... ok
[INFO] [stdout] test src/gsm8k.rs - gsm8k (line 15) ... ok
[INFO] [stdout] test src/truthfulqa.rs - truthfulqa (line 17) ... ok
[INFO] [stdout] test src/boolq.rs - boolq (line 8) ... ok
[INFO] [stdout] test src/mmlu.rs - mmlu (line 19) ... ok
[INFO] [stdout] test src/winogrande.rs - winogrande (line 9) ... ok
[INFO] [stdout] test src/lib.rs - (line 25) ... ok
[INFO] [stdout] test src/arc.rs - arc (line 10) ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.34s
[INFO] [stdout] 
[INFO] running `Command { std: "docker" "inspect" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593", kill_on_drop: false }`
[INFO] [stdout] 764050aa962aec8fbd238be5b7fe8890fa8aad9c47cc7b26b45b0aacdc4bd593