[INFO] fetching crate rustfs-io-metrics 1.0.0...
[INFO] testing rustfs-io-metrics-1.0.0 against 1.99.0-beta.8 for beta-1.100-2
[INFO] extracting crate rustfs-io-metrics 1.0.0 into /workspace/builds/worker-4-tc1/source
[INFO] started tweaking crates.io crate rustfs-io-metrics 1.0.0
[INFO] removed 0 missing examples
[INFO] removed 0 missing tests
[INFO] finished tweaking crates.io crate rustfs-io-metrics 1.0.0
[INFO] tweaked toml for crates.io crate rustfs-io-metrics 1.0.0 written to /workspace/builds/worker-4-tc1/source/Cargo.toml
[INFO] validating manifest of crates.io crate rustfs-io-metrics 1.0.0 on toolchain 1.99.0-beta.8
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.99.0-beta.8" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate crates.io crate rustfs-io-metrics 1.0.0 already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.99.0-beta.8" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Updating crates.io index
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded rustfs-s3-types v1.0.0
[INFO] [stderr]   Downloaded rustfs-s3-ops v1.0.0
[INFO] [stderr]   Downloaded chunked_transfer v1.5.0
[INFO] [stderr]   Downloaded sketches-ddsketch v0.3.1
[INFO] [stderr]   Downloaded hotpath-macros v0.25.1
[INFO] [stderr]   Downloaded find-msvc-tools v0.1.12
[INFO] [stderr]   Downloaded ascii v1.1.0
[INFO] [stderr]   Downloaded metrics-util v0.20.4
[INFO] [stderr]   Downloaded ordered-float v5.5.0
[INFO] [stderr]   Downloaded objc2-open-directory v0.3.2
[INFO] [stderr]   Downloaded tiny_http v0.12.0
[INFO] [stderr]   Downloaded cc v1.4.6
[INFO] [stderr]   Downloaded objc2-foundation v0.3.2
[INFO] [stderr]   Downloaded zlib-rs v0.6.7
[INFO] [stderr]   Downloaded sysinfo v0.39.6
[INFO] [stderr]   Downloaded hdrhistogram v7.6.0
[INFO] [stderr]   Downloaded hotpath v0.25.1
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-m" "1610612736" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:3111399a4047eeb3a02b7a90e478d715f38a8c6669b5c4b49d30a17385265909" "sleep" "infinity", kill_on_drop: false }`
[INFO] [stdout] 84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2
[INFO] running `Command { std: "docker" "start" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-w" "/opt/rustwide/workdir" "--user" "0:0" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2" "/opt/rustwide/cargo-home/bin/cargo" "+1.99.0-beta.8" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2" "/opt/rustwide/cargo-home/bin/cargo" "+1.99.0-beta.8" "build" "--frozen" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr]    Compiling serde_core v1.0.229
[INFO] [stderr]    Compiling hotpath v0.25.1
[INFO] [stderr]    Compiling memchr v2.8.3
[INFO] [stderr]    Compiling libc v0.2.189
[INFO] [stderr]    Compiling rustversion v1.0.23
[INFO] [stderr]    Compiling serde_json v1.0.151
[INFO] [stderr]    Compiling serde v1.0.229
[INFO] [stderr]    Compiling hotpath-macros v0.25.1
[INFO] [stderr]    Compiling tracing-core v0.1.36
[INFO] [stderr]    Compiling tokio v1.53.1
[INFO] [stderr]    Compiling syn v3.0.5
[INFO] [stderr]    Compiling syn v2.0.119
[INFO] [stderr]    Compiling rapidhash v4.5.1
[INFO] [stderr]    Compiling metrics v0.24.6
[INFO] [stderr]    Compiling num_cpus v1.17.0
[INFO] [stderr]    Compiling sysinfo v0.39.6
[INFO] [stderr]    Compiling serde_derive v1.0.229
[INFO] [stderr]    Compiling thiserror-impl v2.0.20
[INFO] [stderr]    Compiling thiserror v2.0.20
[INFO] [stderr]    Compiling rustfs-s3-types v1.0.0
[INFO] [stderr]    Compiling rustfs-s3-ops v1.0.0
[INFO] [stderr]    Compiling tracing-attributes v0.1.31
[INFO] [stderr]    Compiling tracing v0.1.44
[INFO] [stderr]    Compiling rustfs-io-metrics v1.0.0 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 18.23s
[INFO] running `Command { std: "docker" "inspect" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2" "/opt/rustwide/cargo-home/bin/cargo" "+1.99.0-beta.8" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr]    Compiling zerocopy v0.8.57
[INFO] [stderr]    Compiling crossbeam-utils v0.8.23
[INFO] [stderr]    Compiling serde v1.0.229
[INFO] [stderr]    Compiling crossbeam-epoch v0.9.21
[INFO] [stderr]    Compiling getrandom v0.3.4
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]    Compiling crossbeam-deque v0.8.8
[INFO] [stderr]    Compiling find-msvc-tools v0.1.12
[INFO] [stderr]    Compiling syn v2.0.119
[INFO] [stderr]    Compiling syn v3.0.5
[INFO] [stderr]    Compiling serde_json v1.0.151
[INFO] [stderr]    Compiling regex-syntax v0.8.11
[INFO] [stderr]    Compiling bitflags v2.13.2
[INFO] [stderr]    Compiling clap_lex v1.1.1
[INFO] [stderr]    Compiling smallvec v1.16.1
[INFO] [stderr]    Compiling itertools v0.13.0
[INFO] [stderr]    Compiling indexmap v2.14.2
[INFO] [stderr]    Compiling nibble_vec v0.1.0
[INFO] [stderr]    Compiling clap_builder v4.6.7
[INFO] [stderr]    Compiling hashbrown v0.16.1
[INFO] [stderr]    Compiling page_size v0.6.0
[INFO] [stderr]    Compiling radix_trie v0.2.1
[INFO] [stderr]    Compiling cc v1.4.6
[INFO] [stderr]    Compiling rand_core v0.9.5
[INFO] [stderr]    Compiling raw-cpuid v11.6.0
[INFO] [stderr]    Compiling sketches-ddsketch v0.3.1
[INFO] [stderr]    Compiling aho-corasick v1.1.5
[INFO] [stderr]    Compiling rand_xoshiro v0.7.0
[INFO] [stderr]    Compiling plotters v0.3.7
[INFO] [stderr]    Compiling ordered-float v5.5.0
[INFO] [stderr]    Compiling rayon-core v1.13.0
[INFO] [stderr]    Compiling regex-automata v0.4.18
[INFO] [stderr]    Compiling rayon v1.12.0
[INFO] [stderr]    Compiling alloca v0.4.0
[INFO] [stderr]    Compiling criterion-plot v0.8.2
[INFO] [stderr]    Compiling clap v4.6.7
[INFO] [stderr]    Compiling quanta v0.12.6
[INFO] [stderr]    Compiling regex v1.13.1
[INFO] [stderr]    Compiling zerocopy-derive v0.8.57
[INFO] [stderr]    Compiling tracing-attributes v0.1.31
[INFO] [stderr]    Compiling serde_derive v1.0.229
[INFO] [stderr]    Compiling tokio-macros v2.7.2
[INFO] [stderr]    Compiling thiserror-impl v2.0.20
[INFO] [stderr]    Compiling tokio v1.53.1
[INFO] [stderr]    Compiling thiserror v2.0.20
[INFO] [stderr]    Compiling tracing v0.1.44
[INFO] [stderr]    Compiling rustfs-s3-types v1.0.0
[INFO] [stderr]    Compiling tinytemplate v1.2.1
[INFO] [stderr]    Compiling rustfs-s3-ops v1.0.0
[INFO] [stderr]    Compiling rustfs-io-metrics v1.0.0 (/opt/rustwide/workdir)
[INFO] [stderr]    Compiling half v2.7.1
[INFO] [stderr]    Compiling ppv-lite86 v0.2.21
[INFO] [stderr]    Compiling ciborium-ll v0.2.2
[INFO] [stderr]    Compiling ciborium v0.2.2
[INFO] [stderr]    Compiling rand_chacha v0.9.0
[INFO] [stderr]    Compiling rand v0.9.5
[INFO] [stderr]    Compiling criterion v0.8.2
[INFO] [stderr]    Compiling metrics-util v0.20.4
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 42.65s
[INFO] running `Command { std: "docker" "inspect" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2" "/opt/rustwide/cargo-home/bin/cargo" "+1.99.0-beta.8" "test" "--frozen", kill_on_drop: false }`
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 0.16s
[INFO] [stderr]      Running unittests src/lib.rs (/opt/rustwide/target/debug/deps/rustfs_io_metrics-f6852c13532d5cbc)
[INFO] [stdout] 
[INFO] [stdout] running 120 tests
[INFO] [stdout] test cache_config::tests::test_cache_config_validation ... ok
[INFO] [stdout] test bandwidth::tests::test_bandwidth_monitor_records_samples ... ok
[INFO] [stdout] test adaptive_ttl::tests::test_access_tracker_clear ... ok
[INFO] [stdout] test cache_config::tests::test_adaptive_ttl ... ok
[INFO] [stdout] test cache_config::tests::test_cache_config_default ... ok
[INFO] [stdout] test autotuner::tests::test_metrics_history ... ok
[INFO] [stdout] test cache_config::tests::test_cache_health_status ... ok
[INFO] [stdout] test cache_config::tests::test_cache_stats ... ok
[INFO] [stdout] test collector::tests::test_collector_creation ... ok
[INFO] [stdout] test collector::tests::test_record_io_basic ... ok
[INFO] [stdout] test autotuner::tests::test_autotuner_creation ... ok
[INFO] [stdout] test collector::tests::test_percentiles_throttled_but_mean_updates_per_op ... ok
[INFO] [stdout] test autotuner::tests::test_autotuner_with_config ... ok
[INFO] [stdout] test backpressure_metrics::tests::record_helpers_emit_their_metrics ... ok
[INFO] [stdout] test adaptive_ttl::tests::test_access_tracker_hot_cold ... ok
[INFO] [stdout] test adaptive_ttl::tests::test_adaptive_ttl_stats ... ok
[INFO] [stdout] test deadlock_metrics::tests::record_helpers_emit_their_metrics ... ok
[INFO] [stdout] test collector::tests::test_read_write_distinction ... ok
[INFO] [stdout] test collector::tests::test_sample_limit ... ok
[INFO] [stdout] test adaptive_ttl::tests::test_access_tracker ... ok
[INFO] [stdout] test global_metrics::tests::test_global_metrics_recording ... ok
[INFO] [stdout] test cache_config::tests::test_calculate_priority ... ok
[INFO] [stdout] test cache_config::tests::test_should_evict_early ... ok
[INFO] [stdout] test collector::tests::low_frequency_percentiles_recompute_on_time ... ok
[INFO] [stdout] test collector::tests::test_latency_percentiles ... ok
[INFO] [stdout] test adaptive_ttl::tests::test_access_tracker_top_keys ... ok
[INFO] [stdout] test adaptive_ttl::tests::record_helpers_emit_their_metrics ... ok
[INFO] [stdout] test internode_metrics::tests::cluster_peer_should_bypass_ignores_unknown_and_online_peers ... ok
[INFO] [stdout] test internode_metrics::tests::cluster_peer_should_bypass_is_self_healing ... ok
[INFO] [stdout] test internode_metrics::tests::cluster_servers_offline_total_name_is_stable ... ok
[INFO] [stdout] test internode_metrics::tests::msgpack_json_decode_counter_tracks_codec_direction_and_message ... ok
[INFO] [stdout] test internode_metrics::tests::operation_stage_duration_records_low_cardinality_stage_labels ... ok
[INFO] [stdout] test internode_metrics::tests::peer_health_key_is_normalized_across_address_forms ... ok
[INFO] [stdout] test internode_metrics::tests::replay_cache_metrics_record_state_eviction_and_overflow_scope ... ok
[INFO] [stdout] test internode_metrics::tests::classified_and_retry_metrics_update_counters ... ok
[INFO] [stdout] test internode_metrics::tests::cluster_peer_observed_online_status_reports_known_and_unknown_peers ... ok
[INFO] [stdout] test internode_metrics::tests::replay_scope_fallback_counter_updates_snapshot_and_resets ... ok
[INFO] [stdout] test internode_metrics::tests::rpc_auth_failure_counter_records_low_cardinality_labels ... ok
[INFO] [stdout] test io_metrics::tests::record_helpers_emit_their_metrics ... ok
[INFO] [stdout] test internode_metrics::tests::signature_v1_fallback_counter_updates_snapshot_and_resets ... ok
[INFO] [stdout] test internode_metrics::tests::direct_internode_metrics_emit_stable_server_label ... ok
[INFO] [stdout] test internode_metrics::tests::cluster_peer_flips_offline_after_threshold_and_back_online ... ok
[INFO] [stdout] test internode_metrics::tests::cluster_peer_threshold_is_clamped_to_at_least_one ... ok
[INFO] [stdout] test internode_metrics::tests::msgpack_json_decode_error_counter_tracks_codec_direction_and_message ... ok
[INFO] [stdout] test internode_metrics::tests::msgpack_json_fallback_counter_separates_the_two_directions ... ok
[INFO] [stdout] test adaptive_ttl::tests::test_access_record ... ok
[INFO] [stdout] test io_metrics::tests::test_io_scheduler_stats ... ok
[INFO] [stdout] test list_objects_metrics::tests::gather_scan_amplification_falls_back_to_the_scan_count_on_an_empty_page ... ok
[INFO] [stdout] test list_objects_metrics::tests::index_fallback_counts_once_per_reason ... ok
[INFO] [stdout] test lock_metrics::tests::test_record_object_lock_diag_enabled ... ok
[INFO] [stdout] test global_metrics::tests::test_global_metrics_instance ... ok
[INFO] [stdout] test list_objects_metrics::tests::index_serving_reports_verification_amplification_against_returned_objects ... ok
[INFO] [stdout] test internode_metrics::tests::operation_metric_names_and_low_cardinality_values_are_stable ... ok
[INFO] [stdout] test list_objects_metrics::tests::local_read_dir_passes_the_whole_directory_sentinel_through_as_the_limit ... ok
[INFO] [stdout] test performance::tests::test_io_recording ... ok
[INFO] [stdout] test performance::tests::test_l1_hit_rate ... ok
[INFO] [stdout] test performance::tests::test_metrics_creation ... ok
[INFO] [stdout] test performance::tests::test_error_recording ... ok
[INFO] [stdout] test internode_metrics::tests::snapshot_reports_recorded_values ... ok
[INFO] [stdout] test global_metrics::tests::test_global_metrics_singleton ... ok
[INFO] [stdout] test lock_metrics::tests::test_record_object_lock_diag_slow_events ... ok
[INFO] [stdout] test performance::tests::test_cache_hit_rate ... ok
[INFO] [stdout] test performance::tests::test_concurrent_tracking ... ok
[INFO] [stdout] test list_objects_metrics::tests::merge_records_a_zero_read_quorum_rather_than_skipping_it ... ok
[INFO] [stdout] test s3_api_metrics::tests::s3_op_metrics_snapshot_reports_recorded_totals ... ok
[INFO] [stdout] test s3_api_metrics::tests::record_s3_op_labels_by_op_only_no_bucket ... ok
[INFO] [stdout] test process_lock_metrics::tests::held_lock_counters_round_trip ... ok
[INFO] [stdout] test process_lock_metrics::tests::parse_kv_u64_round_trip ... ok
[INFO] [stdout] test lock_metrics::tests::test_record_object_lock_diag_hold_duration ... ok
[INFO] [stdout] test lock_metrics::tests::record_helpers_emit_their_metrics ... ok
[INFO] [stdout] test lock_metrics::tests::test_record_object_lock_diag_acquire_duration ... ok
[INFO] [stdout] test s3_api_metrics::tests::record_s3_op_cardinality_bounded_by_distinct_ops ... ok
[INFO] [stdout] test internode_metrics::tests::operation_metric_descriptors_include_backend_and_operation_labels ... ok
[INFO] [stdout] test s3_http_metrics::tests::request_operation_is_scoped_and_first_dispatch_wins ... ok
[INFO] [stdout] test sampler::process::tests::process_status_snapshot_maps_sysinfo_values ... ok
[INFO] [stdout] test s3_http_metrics::tests::request_guard_preserves_operation_across_polls_and_finishes_once ... ok
[INFO] [stdout] test internode_metrics::tests::operation_metrics_also_update_aggregate_snapshot ... ok
[INFO] [stdout] test tests::metadata_fanout_lifecycle_records_named_histograms ... ok
[INFO] [stdout] test s3_http_metrics::tests::outcome_counters_distinguish_partial_and_complete_write_failure ... ok
[INFO] [stdout] test tests::cgroup_memory_split_skips_absent_fields ... ok
[INFO] [stdout] test tests::put_commit_lock_admission_labels_are_static_and_gated ... ok
[INFO] [stdout] test tests::failure_helpers_emit_their_metrics ... ok
[INFO] [stdout] test tests::io_scheduler_and_bandwidth_helpers_emit_their_metrics ... ok
[INFO] [stdout] test tests::s3_operation_helpers_emit_their_metrics ... ok
[INFO] [stdout] test tests::test_ec_encode_inflight_bytes_tracking ... ok
[INFO] [stdout] test tests::put_stage_sync_tail_labels_are_static_and_gated ... ok
[INFO] [stdout] test tests::put_rename_code_level_metrics_are_static_and_gated ... ok
[INFO] [stdout] test tests::system_resource_helpers_emit_their_metrics ... ok
[INFO] [stdout] test tests::test_ec_encode_producer_peak_exports_the_high_water_mark ... ok
[INFO] [stdout] test tests::test_get_object_buffered_bytes_guard ... ok
[INFO] [stdout] test tests::test_get_object_buffered_bytes_guard_saturates_on_underflow ... ok
[INFO] [stdout] test tests::put_stage_gate_does_not_disable_basic_put_metrics ... ok
[INFO] [stdout] test tests::test_get_stage_metrics_disabled_by_default ... ok
[INFO] [stdout] test tests::test_ec_encode_payload_stage_guards_are_noop_when_disabled ... ok
[INFO] [stdout] test tests::test_record_batch_processor_observation ... ok
[INFO] [stdout] test tests::test_ec_encode_producer_and_writer_stage_guards_aggregate_and_settle ... ok
[INFO] [stdout] test tests::test_put_stage_metrics_disabled_by_default ... ok
[INFO] [stdout] test tests::test_get_object_size_buckets_match_issue714_matrix ... ok
[INFO] [stdout] test tests::test_put_stage_timer_follows_metrics_switch ... ok
[INFO] [stdout] test tests::test_ec_encode_peak_publish_does_not_regress_after_out_of_order_cas ... ok
[INFO] [stdout] test tests::test_record_stage_duration_generic ... ok
[INFO] [stdout] test tests::test_record_get_object_stage_metrics ... ok
[INFO] [stdout] test tests::test_record_get_object_path_and_stage ... ok
[INFO] [stdout] test sampler::process::tests::independent_samplers_observe_the_same_process ... ok
[INFO] [stdout] test tests::test_get_object_shard_locality_observe_only_metrics_smoke ... ok
[INFO] [stdout] test tests::test_put_object_diagnostic_buckets ... ok
[INFO] [stdout] test timeout_metrics::tests::record_helpers_emit_their_metrics ... ok
[INFO] [stdout] test lock_metrics::tests::test_lock_metrics_summary ... ok
[INFO] [stdout] test process_lock_metrics::platform::tests::parse_proc_self_io_extracts_expected_fields ... ok
[INFO] [stdout] test process_lock_metrics::platform::tests::parse_vm_peak ... ok
[INFO] [stdout] test s3_http_metrics::tests::completion_observer_sees_each_dispatched_request_once ... ok
[INFO] [stdout] test tests::test_record_get_object_fill_metrics ... ok
[INFO] [stdout] test timeout_metrics::tests::test_timeout_metrics_summary ... ok
[INFO] [stdout] test tests::test_ec_encode_payload_stage_guards_track_concurrent_ownership ... ok
[INFO] [stdout] test tests::test_record_put_object_path_and_stage ... ok
[INFO] [stdout] test tests::zero_copy_and_bytes_pool_helpers_emit_their_metrics ... ok
[INFO] [stdout] test sampler::process::tests::combined_snapshot_agrees_with_the_individual_ones_on_per_process_facts ... FAILED
[INFO] [stdout] test tests::test_metrics_enabled_toggle_gates_common_recorders ... ok
[INFO] [stdout] test zero_copy_tests::test_metric_names ... ok
[INFO] [stdout] test zero_copy_tests::zero_copy_helpers_emit_their_metrics ... ok
[INFO] [stdout] 
[INFO] [stdout] failures:
[INFO] [stdout] 
[INFO] [stdout] ---- sampler::process::tests::combined_snapshot_agrees_with_the_individual_ones_on_per_process_facts stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'sampler::process::tests::combined_snapshot_agrees_with_the_individual_ones_on_per_process_facts' (1835) panicked at src/sampler/process.rs:207:9:
[INFO] [stdout] assertion `left == right` failed: the status enum and its numeric projection must stay in step
[INFO] [stdout]   left: 1
[INFO] [stdout]  right: 0
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5bf267608ad1 - std[70759c8f55707aa4]::backtrace_rs::backtrace::libunwind::trace
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5bf267608ad1 - std[70759c8f55707aa4]::backtrace_rs::backtrace::trace_unsynchronized::<std[70759c8f55707aa4]::sys::backtrace::_print_fmt::{closure#1}>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5bf267608ad1 - std[70759c8f55707aa4]::sys::backtrace::_print_fmt
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/sys/backtrace.rs:74:9
[INFO] [stdout]    3:     0x5bf267608ad1 - <<std[70759c8f55707aa4]::sys::backtrace::BacktraceLock>::print::DisplayBacktrace as core[df12db4294e9bfd3]::fmt::Display>::fmt
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/sys/backtrace.rs:44:26
[INFO] [stdout]    4:     0x5bf26761e22a - <core[df12db4294e9bfd3]::fmt::rt::Argument>::fmt
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x5bf26761e22a - core[df12db4294e9bfd3]::fmt::write
[INFO] [stdout]    6:     0x5bf26760d3dc - core[df12db4294e9bfd3]::io::write::default_write_fmt::<alloc[2182bb758b4b3781]::vec::Vec<u8>>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/io/write.rs:402:11
[INFO] [stdout]    7:     0x5bf26760d3dc - <alloc[2182bb758b4b3781]::vec::Vec<u8> as core[df12db4294e9bfd3]::io::write::Write>::write_fmt
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/io/write.rs:335:13
[INFO] [stdout]    8:     0x5bf2675e3dc6 - <std[70759c8f55707aa4]::sys::backtrace::BacktraceLock>::print
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/sys/backtrace.rs:47:9
[INFO] [stdout]    9:     0x5bf2675e3dc6 - std[70759c8f55707aa4]::panicking::default_hook::{closure#0}
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x5bf267600a79 - std[70759c8f55707aa4]::panicking::default_hook
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x5bf2674a7170 - <alloc[2182bb758b4b3781]::boxed::Box<dyn for<'a, 'b> core[df12db4294e9bfd3]::ops::function::Fn<(&'a std[70759c8f55707aa4]::panic::PanicHookInfo<'b>,), Output = ()> + core[df12db4294e9bfd3]::marker::Send + core[df12db4294e9bfd3]::marker::Sync> as core[df12db4294e9bfd3]::ops::function::Fn<(&std[70759c8f55707aa4]::panic::PanicHookInfo,)>>::call
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/alloc/src/boxed.rs:2334:9
[INFO] [stdout]   12:     0x5bf2674a7170 - test[9d35eded1c95d3be]::test_main_inner::<test[9d35eded1c95d3be]::test_main_static::{closure#0}>::{closure#0}
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/test/src/lib.rs:155:21
[INFO] [stdout]   13:     0x5bf267600c32 - <alloc[2182bb758b4b3781]::boxed::Box<dyn for<'a, 'b> core[df12db4294e9bfd3]::ops::function::Fn<(&'a std[70759c8f55707aa4]::panic::PanicHookInfo<'b>,), Output = ()> + core[df12db4294e9bfd3]::marker::Send + core[df12db4294e9bfd3]::marker::Sync> as core[df12db4294e9bfd3]::ops::function::Fn<(&std[70759c8f55707aa4]::panic::PanicHookInfo,)>>::call
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/alloc/src/boxed.rs:2334:9
[INFO] [stdout]   14:     0x5bf267600c32 - std[70759c8f55707aa4]::panicking::panic_with_hook
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panicking.rs:823:13
[INFO] [stdout]   15:     0x5bf2675e3e72 - std[70759c8f55707aa4]::panicking::panic_handler::{closure#0}
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panicking.rs:688:13
[INFO] [stdout]   16:     0x5bf2675dc999 - std[70759c8f55707aa4]::sys::backtrace::__rust_end_short_backtrace::<std[70759c8f55707aa4]::panicking::panic_handler::{closure#0}, !>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/sys/backtrace.rs:182:18
[INFO] [stdout]   17:     0x5bf2675e4a6d - __rustc[8fa7c3cbc660c2b3]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panicking.rs:679:5
[INFO] [stdout]   18:     0x5bf26761ea4c - core[df12db4294e9bfd3]::panicking::panic_fmt
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x5bf26761e943 - core[df12db4294e9bfd3]::panicking::assert_failed_inner
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/panicking.rs:447:23
[INFO] [stdout]   20:     0x5bf2673c6859 - core[df12db4294e9bfd3]::panicking::assert_failed::<i64, i64>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/panicking.rs:407:5
[INFO] [stdout]   21:     0x5bf267413feb - rustfs_io_metrics[21af34ccc206022]::sampler::process::tests::combined_snapshot_agrees_with_the_individual_ones_on_per_process_facts
[INFO] [stdout]                                at /opt/rustwide/workdir/src/sampler/process.rs:207:9
[INFO] [stdout]   22:     0x5bf2674102b7 - rustfs_io_metrics[21af34ccc206022]::sampler::process::tests::combined_snapshot_agrees_with_the_individual_ones_on_per_process_facts::{closure#0}
[INFO] [stdout]                                at /opt/rustwide/workdir/src/sampler/process.rs:191:80
[INFO] [stdout]   23:     0x5bf267402866 - <rustfs_io_metrics[21af34ccc206022]::sampler::process::tests::combined_snapshot_agrees_with_the_individual_ones_on_per_process_facts::{closure#0} as core[df12db4294e9bfd3]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5bf26749a45b - <fn() -> core[df12db4294e9bfd3]::result::Result<(), alloc[2182bb758b4b3781]::string::String> as core[df12db4294e9bfd3]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   25:     0x5bf26749a45b - test[9d35eded1c95d3be]::__rust_begin_short_backtrace::<core[df12db4294e9bfd3]::result::Result<(), alloc[2182bb758b4b3781]::string::String>, fn() -> core[df12db4294e9bfd3]::result::Result<(), alloc[2182bb758b4b3781]::string::String>>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/test/src/lib.rs:733:18
[INFO] [stdout]   26:     0x5bf2674a7ac5 - test[9d35eded1c95d3be]::run_test_in_process::{closure#0}
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/test/src/lib.rs:756:74
[INFO] [stdout]   27:     0x5bf2674a7ac5 - <core[df12db4294e9bfd3]::panic::unwind_safe::AssertUnwindSafe<test[9d35eded1c95d3be]::run_test_in_process::{closure#0}> as core[df12db4294e9bfd3]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/panic/unwind_safe.rs:275:9
[INFO] [stdout]   28:     0x5bf2674a7ac5 - std[70759c8f55707aa4]::panicking::catch_unwind::do_call::<core[df12db4294e9bfd3]::panic::unwind_safe::AssertUnwindSafe<test[9d35eded1c95d3be]::run_test_in_process::{closure#0}>, core[df12db4294e9bfd3]::result::Result<(), alloc[2182bb758b4b3781]::string::String>>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panicking.rs:576:43
[INFO] [stdout]   29:     0x5bf2674a7ac5 - std[70759c8f55707aa4]::panicking::catch_unwind::<core[df12db4294e9bfd3]::result::Result<(), alloc[2182bb758b4b3781]::string::String>, core[df12db4294e9bfd3]::panic::unwind_safe::AssertUnwindSafe<test[9d35eded1c95d3be]::run_test_in_process::{closure#0}>>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panicking.rs:544:19
[INFO] [stdout]   30:     0x5bf2674a7ac5 - std[70759c8f55707aa4]::panic::catch_unwind::<core[df12db4294e9bfd3]::panic::unwind_safe::AssertUnwindSafe<test[9d35eded1c95d3be]::run_test_in_process::{closure#0}>, core[df12db4294e9bfd3]::result::Result<(), alloc[2182bb758b4b3781]::string::String>>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panic.rs:359:14
[INFO] [stdout]   31:     0x5bf2674a7ac5 - test[9d35eded1c95d3be]::run_test_in_process
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/test/src/lib.rs:756:27
[INFO] [stdout]   32:     0x5bf2674a7ac5 - test[9d35eded1c95d3be]::run_test::{closure#0}
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/test/src/lib.rs:677:43
[INFO] [stdout]   33:     0x5bf2674a1384 - test[9d35eded1c95d3be]::run_test::{closure#1}
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/test/src/lib.rs:707:41
[INFO] [stdout]   34:     0x5bf2674a1384 - std[70759c8f55707aa4]::sys::backtrace::__rust_begin_short_backtrace::<test[9d35eded1c95d3be]::run_test::{closure#1}, ()>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/sys/backtrace.rs:166:18
[INFO] [stdout]   35:     0x5bf2674aac22 - std[70759c8f55707aa4]::thread::lifecycle::spawn_unchecked::<test[9d35eded1c95d3be]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/thread/lifecycle.rs:70:13
[INFO] [stdout]   36:     0x5bf2674aac22 - <core[df12db4294e9bfd3]::panic::unwind_safe::AssertUnwindSafe<std[70759c8f55707aa4]::thread::lifecycle::spawn_unchecked<test[9d35eded1c95d3be]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}> as core[df12db4294e9bfd3]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/panic/unwind_safe.rs:275:9
[INFO] [stdout]   37:     0x5bf2674aac22 - std[70759c8f55707aa4]::panicking::catch_unwind::do_call::<core[df12db4294e9bfd3]::panic::unwind_safe::AssertUnwindSafe<std[70759c8f55707aa4]::thread::lifecycle::spawn_unchecked<test[9d35eded1c95d3be]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>, ()>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panicking.rs:576:43
[INFO] [stdout]   38:     0x5bf2674aac22 - std[70759c8f55707aa4]::panicking::catch_unwind::<(), core[df12db4294e9bfd3]::panic::unwind_safe::AssertUnwindSafe<std[70759c8f55707aa4]::thread::lifecycle::spawn_unchecked<test[9d35eded1c95d3be]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panicking.rs:544:19
[INFO] [stdout]   39:     0x5bf2674aac22 - std[70759c8f55707aa4]::panic::catch_unwind::<core[df12db4294e9bfd3]::panic::unwind_safe::AssertUnwindSafe<std[70759c8f55707aa4]::thread::lifecycle::spawn_unchecked<test[9d35eded1c95d3be]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>, ()>
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/panic.rs:359:14
[INFO] [stdout]   40:     0x5bf2674aac22 - std[70759c8f55707aa4]::thread::lifecycle::spawn_unchecked::<test[9d35eded1c95d3be]::run_test::{closure#1}, ()>::{closure#1}
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/thread/lifecycle.rs:68:26
[INFO] [stdout]   41:     0x5bf2674aac22 - <std[70759c8f55707aa4]::thread::lifecycle::spawn_unchecked<test[9d35eded1c95d3be]::run_test::{closure#1}, ()>::{closure#1} as core[df12db4294e9bfd3]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   42:     0x5bf267608009 - <alloc[2182bb758b4b3781]::boxed::Box<dyn core[df12db4294e9bfd3]::ops::function::FnOnce<(), Output = ()> + core[df12db4294e9bfd3]::marker::Send> as core[df12db4294e9bfd3]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/alloc/src/boxed.rs:2320:9
[INFO] [stdout]   43:     0x5bf267608009 - <std[70759c8f55707aa4]::sys::thread::unix::Thread>::new::thread_start
[INFO] [stdout]                                at /rustc/dfb0accb7f508ae8be965d8629557221c889dd98/library/std/src/sys/thread/unix.rs:123:17
[INFO] [stdout]   44:     0x74e7ecb06dfa - <unknown>
[INFO] [stdout]   45:     0x74e7ecb9a3d4 - clone
[INFO] [stdout]   46:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] failures:
[INFO] [stdout]     sampler::process::tests::combined_snapshot_agrees_with_the_individual_ones_on_per_process_facts
[INFO] [stdout] 
[INFO] [stdout] test result: FAILED. 119 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
[INFO] [stdout] 
[INFO] [stderr] error: test failed, to rerun pass `--lib`
[INFO] running `Command { std: "docker" "inspect" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2", kill_on_drop: false }`
[INFO] [stdout] 84f7f0b58ee57bdd19a97471c87f91e5ff07afe8e6e0350c6c9443d053b155a2
