[INFO] fetching crate rullama-eval 0.12.0...
[INFO] testing rullama-eval-0.12.0 against 1.99.0-beta.8 for beta-1.100-2
[INFO] extracting crate rullama-eval 0.12.0 into /workspace/builds/worker-2-tc1/source
[INFO] started tweaking crates.io crate rullama-eval 0.12.0
[INFO] removed 0 missing tests
[INFO] finished tweaking crates.io crate rullama-eval 0.12.0
[INFO] tweaked toml for crates.io crate rullama-eval 0.12.0 written to /workspace/builds/worker-2-tc1/source/Cargo.toml
[INFO] validating manifest of crates.io crate rullama-eval 0.12.0 on toolchain 1.99.0-beta.8
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.99.0-beta.8" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate crates.io crate rullama-eval 0.12.0 already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.99.0-beta.8" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Updating crates.io index
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded serde_yml v0.0.12
[INFO] [stderr]   Downloaded libyml v0.0.5
[INFO] [stderr]   Downloaded tokio v1.52.2
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-2-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-2-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-m" "1610612736" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:3111399a4047eeb3a02b7a90e478d715f38a8c6669b5c4b49d30a17385265909" "sleep" "infinity", kill_on_drop: false }`
[INFO] [stdout] c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618
[INFO] running `Command { std: "docker" "start" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-w" "/opt/rustwide/workdir" "--user" "0:0" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618" "/opt/rustwide/cargo-home/bin/cargo" "+1.99.0-beta.8" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618" "/opt/rustwide/cargo-home/bin/cargo" "+1.99.0-beta.8" "build" "--frozen" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr]    Compiling memchr v2.8.0
[INFO] [stderr]    Compiling hashbrown v0.17.0
[INFO] [stderr]    Compiling regex-syntax v0.8.10
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]    Compiling serde_yml v0.0.12
[INFO] [stderr]    Compiling serde_json v1.0.149
[INFO] [stderr]    Compiling quote v1.0.45
[INFO] [stderr]    Compiling libyml v0.0.5
[INFO] [stderr]    Compiling syn v2.0.117
[INFO] [stderr]    Compiling aho-corasick v1.1.4
[INFO] [stderr]    Compiling indexmap v2.14.0
[INFO] [stderr]    Compiling regex-automata v0.4.14
[INFO] [stderr]    Compiling serde_derive v1.0.228
[INFO] [stderr]    Compiling tokio-macros v2.7.0
[INFO] [stderr]    Compiling tracing-attributes v0.1.31
[INFO] [stderr]    Compiling async-trait v0.1.89
[INFO] [stderr]    Compiling tokio v1.52.2
[INFO] [stderr]    Compiling regex v1.12.3
[INFO] [stderr]    Compiling tracing v0.1.44
[INFO] [stderr]    Compiling serde v1.0.228
[INFO] [stderr]    Compiling chrono v0.4.44
[INFO] [stderr]    Compiling rullama-eval v0.12.0 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 21.16s
[INFO] running `Command { std: "docker" "inspect" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618" "/opt/rustwide/cargo-home/bin/cargo" "+1.99.0-beta.8" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr]    Compiling libc v0.2.186
[INFO] [stderr]    Compiling smallvec v1.15.1
[INFO] [stderr]    Compiling getrandom v0.4.2
[INFO] [stderr]    Compiling rustix v1.1.4
[INFO] [stderr]    Compiling linux-raw-sys v0.12.1
[INFO] [stderr]    Compiling bitflags v2.11.1
[INFO] [stderr]    Compiling bytes v1.11.1
[INFO] [stderr]    Compiling fastrand v2.4.1
[INFO] [stderr]    Compiling parking_lot_core v0.9.12
[INFO] [stderr]    Compiling errno v0.3.14
[INFO] [stderr]    Compiling mio v1.2.0
[INFO] [stderr]    Compiling socket2 v0.6.3
[INFO] [stderr]    Compiling signal-hook-registry v1.4.8
[INFO] [stderr]    Compiling parking_lot v0.12.5
[INFO] [stderr]    Compiling tokio v1.52.2
[INFO] [stderr]    Compiling tempfile v3.27.0
[INFO] [stderr]    Compiling rullama-eval v0.12.0 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 16.21s
[INFO] running `Command { std: "docker" "inspect" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618" "/opt/rustwide/cargo-home/bin/cargo" "+1.99.0-beta.8" "test" "--frozen", kill_on_drop: false }`
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
[INFO] [stderr]      Running unittests src/lib.rs (/opt/rustwide/target/debug/deps/rullama_eval-df385e968a5a6861)
[INFO] [stdout] 
[INFO] [stdout] running 96 tests
[INFO] [stdout] test adversarial::tests::test_prompt_injection_builder ... ok
[INFO] [stdout] test adversarial::tests::test_standard_suite_all_injection_cases_expect_rejection ... ok
[INFO] [stdout] test adversarial::tests::test_ambiguous_instruction_builder ... ok
[INFO] [stdout] test adversarial::tests::test_json_round_trip ... ok
[INFO] [stdout] test adversarial::tests::test_standard_suite_non_empty ... ok
[INFO] [stdout] test adversarial::tests::test_with_expect_rejection_override ... ok
[INFO] [stdout] test case::tests::test_always_pass_case ... ok
[INFO] [stdout] test case::tests::test_always_fail_case ... ok
[INFO] [stdout] test case::tests::test_stochastic_case_rate ... ok
[INFO] [stdout] test case::tests::test_stochastic_case_reproducible ... ok
[INFO] [stdout] test adversarial::tests::test_budget_exhaustion_builder ... ok
[INFO] [stdout] test fault_report::tests::test_new_capability_constructor ... ok
[INFO] [stdout] test fault_report::tests::test_priority_new_capability ... ok
[INFO] [stdout] test fault_report::tests::test_new_capability_when_regression_suite_provided_but_no_matching_baseline ... ok
[INFO] [stdout] test adversarial::tests::test_missing_context_builder ... ok
[INFO] [stdout] test fault_report::tests::test_priority_consistent_failure ... ok
[INFO] [stdout] test fault_report::tests::test_no_fault_for_passing_case_without_regression_suite ... ok
[INFO] [stdout] test fault_report::tests::test_priority_regression_scaled_by_drop ... ok
[INFO] [stdout] test fault_report::tests::test_no_fault_when_within_tolerance ... ok
[INFO] [stdout] test fault_report::tests::test_consistent_failure_detected ... ok
[INFO] [stdout] test fault_report::tests::test_priority_flaky ... ok
[INFO] [stdout] test fault_report::tests::test_priority_regression_capped_at_10 ... ok
[INFO] [stdout] test fault_report::tests::test_regression_constructor_sets_fields ... ok
[INFO] [stdout] test fixtures::tests::evaluate_fails_on_tool_sequence_mismatch ... ok
[INFO] [stdout] test fault_report::tests::test_results_sorted_by_priority_descending ... ok
[INFO] [stdout] test fault_report::tests::test_sample_errors_collected ... ok
[INFO] [stdout] test fixtures::tests::evaluate_fails_on_missing_substring ... ok
[INFO] [stdout] test ranking_metrics::tests::test_mrr_second_is_relevant ... ok
[INFO] [stdout] test fixtures::tests::evaluate_passes_when_all_assertions_hold ... ok
[INFO] [stdout] test ranking_metrics::tests::test_ndcg_all_zero_relevance ... ok
[INFO] [stdout] test ranking_metrics::tests::test_mrr_no_relevant ... ok
[INFO] [stdout] test fault_report::tests::test_regression_detected_when_drop_exceeds_tolerance ... ok
[INFO] [stdout] test fixtures::tests::evaluate_regex_assertion ... ok
[INFO] [stdout] test fixtures::tests::fixture_case_bridges_to_trial_result ... ok
[INFO] [stdout] test ranking_metrics::tests::test_ndcg_empty ... ok
[INFO] [stdout] test ranking_metrics::tests::test_mrr_empty ... ok
[INFO] [stdout] test ranking_metrics::tests::test_mrr_first_is_relevant ... ok
[INFO] [stdout] test ranking_metrics::tests::test_ndcg_k_truncates ... ok
[INFO] [stdout] test ranking_metrics::tests::test_ndcg_k_zero_means_all ... ok
[INFO] [stdout] test ranking_metrics::tests::test_ndcg_worst_ranking ... ok
[INFO] [stdout] test ranking_metrics::tests::test_precision_at_k_all_relevant ... ok
[INFO] [stdout] test recorder::tests::test_args_fingerprint_same_for_same_args ... ok
[INFO] [stdout] test recorder::tests::test_diff_both_empty ... ok
[INFO] [stdout] test fixtures::tests::load_fixtures_from_tmpdir_in_sorted_order ... ok
[INFO] [stdout] test recorder::tests::test_diff_partial_match ... ok
[INFO] [stdout] test recorder::tests::test_diff_empty_vs_expected ... ok
[INFO] [stdout] test recorder::tests::test_levenshtein_identical ... ok
[INFO] [stdout] test recorder::tests::test_record_and_retrieve ... ok
[INFO] [stdout] test recorder::tests::test_levenshtein_insert_delete ... ok
[INFO] [stdout] test recorder::tests::test_levenshtein_single_substitution ... ok
[INFO] [stdout] test recorder::tests::test_reset_clears_calls ... ok
[INFO] [stdout] test ranking_metrics::tests::test_precision_at_k_truncates ... ok
[INFO] [stdout] test ranking_metrics::tests::test_precision_at_k_zero_k_means_all ... ok
[INFO] [stdout] test regression::tests::test_check_passes_when_no_regression ... ok
[INFO] [stdout] test regression::tests::test_check_skips_low_trial_count ... ok
[INFO] [stdout] test regression::tests::test_improved_categories ... ok
[INFO] [stdout] test stability_tests::tests::test_goal_injection_points_20iter_interval5 ... ok
[INFO] [stdout] test recorder::tests::test_call_names ... ok
[INFO] [stdout] test stability_tests::tests::test_goal_injection_simulation_matches_expected ... ok
[INFO] [stdout] test regression::tests::test_baseline_creation ... ok
[INFO] [stdout] test recorder::tests::test_diff_exact_match ... ok
[INFO] [stdout] test stability_tests::tests::test_goal_preservation_case_succeeds ... ok
[INFO] [stdout] test regression::tests::test_check_fails_on_regression_above_threshold ... ok
[INFO] [stdout] test stability_tests::tests::test_loop_detection_case_succeeds_when_loop_fires ... ok
[INFO] [stdout] test stability_tests::tests::test_loop_sim_does_not_fire_diverse ... ok
[INFO] [stdout] test stability_tests::tests::test_loop_sim_fires_at_correct_step ... ok
[INFO] [stdout] test ranking_metrics::tests::test_precision_at_k_half_relevant ... ok
[INFO] [stdout] test stability_tests::tests::test_loop_detection_case_fails_when_no_loop_fires ... ok
[INFO] [stdout] test ranking_metrics::tests::test_ndcg_perfect_ranking ... ok
[INFO] [stdout] test regression::tests::test_record_baselines_from_suite_result ... ok
[INFO] [stdout] test regression::tests::test_check_passes_regression_within_threshold ... ok
[INFO] [stdout] test recorder::tests::test_args_fingerprint_differs_for_different_args ... ok
[INFO] [stdout] test stability_tests::tests::test_loop_sim_short_run_no_loop ... ok
[INFO] [stdout] test regression::tests::test_json_roundtrip ... ok
[INFO] [stdout] test stability_tests::tests::test_goal_injection_points_15iter_interval10 ... ok
[INFO] [stdout] test suite::tests::test_suite_all_fail ... ok
[INFO] [stdout] test trial::tests::test_percentile_single_element ... ok
[INFO] [stdout] test trial::tests::test_evaluation_stats_empty ... ok
[INFO] [stdout] test trial::tests::test_evaluation_stats_mixed ... ok
[INFO] [stdout] test trial::tests::test_percentile_interpolation ... ok
[INFO] [stdout] test trial::tests::test_wilson_ci_contains_true_rate ... ok
[INFO] [stdout] test trial::tests::test_wilson_ci_all_successes ... ok
[INFO] [stdout] test trial::tests::test_wilson_ci_no_successes ... ok
[INFO] [stdout] test trial::tests::test_wilson_ci_zero_trials ... ok
[INFO] [stdout] test trial::tests::test_trial_failure_builder ... ok
[INFO] [stdout] test trial::tests::test_trial_success_builder ... ok
[INFO] [stdout] test stability_tests::tests::test_full_stability_suite_runs ... ok
[INFO] [stdout] test suite::tests::test_suite_n_trials_minimum_one ... ok
[INFO] [stdout] test trial::tests::test_evaluation_stats_all_success ... ok
[INFO] [stdout] test stability_tests::tests::test_loop_sim_fires_immediately ... ok
[INFO] [stdout] test suite::tests::test_confidence_interval_in_suite_result ... ok
[INFO] [stdout] test suite::tests::test_failing_cases_filter ... ok
[INFO] [stdout] test suite::tests::test_run_case_returns_correct_count ... ok
[INFO] [stdout] test suite::tests::test_suite_all_pass ... ok
[INFO] [stdout] test suite::tests::test_suite_multiple_cases ... ok
[INFO] [stdout] test trial::tests::test_trial_with_meta ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 96 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stderr]      Running tests/fixtures_suite.rs (/opt/rustwide/target/debug/deps/fixtures_suite-34339b86a9b9d8c4)
[INFO] [stdout] running 1 test
[INFO] [stdout] test shipped_yaml_fixtures_parse ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stderr]    Doc-tests rullama_eval
[INFO] [stdout] 
[INFO] [stdout] running 5 tests
[INFO] [stdout] test src/case.rs - case::EvaluationCase (line 15) ... ignored
[INFO] [stdout] test src/recorder.rs - recorder::ToolSequenceRecorder (line 131) ... ignored
[INFO] [stdout] test src/regression.rs - regression (line 10) ... ignored
[INFO] [stdout] test src/suite.rs - suite::EvaluationSuite (line 83) ... ignored
[INFO] [stdout] test src/ranking_metrics.rs - ranking_metrics (line 16) ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 1 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stdout] all doctests ran in 0.60s; merged doctests compilation took 0.58s
[INFO] running `Command { std: "docker" "inspect" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618", kill_on_drop: false }`
[INFO] [stdout] c3fb66cc682b28f100f43483dfa88973eed5855912addce861fbe80e4ee40618
