[INFO] cloning repository https://github.com/Infer-Lab/inferlab
[INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/Infer-Lab/inferlab" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FInfer-Lab%2Finferlab", kill_on_drop: false }`
[INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FInfer-Lab%2Finferlab'...
[INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }`
[INFO] [stdout] e413643da56c886cbf87858c29491182f2beaf64
[INFO] testing Infer-Lab/inferlab against 1.98.0-beta.1 for beta-1.98-1
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FInfer-Lab%2Finferlab" "/workspace/builds/worker-1-tc2/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-1-tc2/source'...
[INFO] [stderr] done.
[INFO] removed /workspace/builds/worker-1-tc2/source/rust-toolchain.toml
[INFO] started tweaking git repo https://github.com/Infer-Lab/inferlab
[INFO] finished tweaking git repo https://github.com/Infer-Lab/inferlab
[INFO] tweaked toml for git repo https://github.com/Infer-Lab/inferlab written to /workspace/builds/worker-1-tc2/source/Cargo.toml
[INFO] validating manifest of git repo https://github.com/Infer-Lab/inferlab on toolchain 1.98.0-beta.1
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.98.0-beta.1" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate git repo https://github.com/Infer-Lab/inferlab already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.98.0-beta.1" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-m" "1610612736" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:3d5ced03c013a94a2f102a4510f48a6e9184255caf5fd8244f58017bde7f5210" "sleep" "infinity", kill_on_drop: false }`
[INFO] [stdout] 9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27
[INFO] running `Command { std: "docker" "start" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-w" "/opt/rustwide/workdir" "--user" "0:0" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27" "/opt/rustwide/cargo-home/bin/cargo" "+1.98.0-beta.1" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27" "/opt/rustwide/cargo-home/bin/cargo" "+1.98.0-beta.1" "build" "--frozen" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr]    Compiling quote v1.0.46
[INFO] [stderr]    Compiling log v0.4.33
[INFO] [stderr]    Compiling serde_json v1.0.150
[INFO] [stderr]    Compiling ref-cast v1.0.25
[INFO] [stderr]    Compiling dyn-clone v1.0.20
[INFO] [stderr]    Compiling nix v0.31.3
[INFO] [stderr]    Compiling getrandom v0.4.3
[INFO] [stderr]    Compiling serde_spanned v1.1.1
[INFO] [stderr]    Compiling toml_datetime v1.1.1+spec-1.1.0
[INFO] [stderr]    Compiling time-core v0.1.9
[INFO] [stderr]    Compiling deranged v0.5.8
[INFO] [stderr]    Compiling libyaml-rs v0.3.0
[INFO] [stderr]    Compiling sha2 v0.10.9
[INFO] [stderr]    Compiling wait-timeout v0.2.1
[INFO] [stderr]    Compiling fs2 v0.4.3
[INFO] [stderr]    Compiling tempfile v3.27.0
[INFO] [stderr]    Compiling tracing v0.1.44
[INFO] [stderr]    Compiling toml v1.1.2+spec-1.1.0
[INFO] [stderr]    Compiling syn v2.0.118
[INFO] [stderr]    Compiling axum-core v0.5.6
[INFO] [stderr]    Compiling ctrlc v3.5.2
[INFO] [stderr]    Compiling time v0.3.52
[INFO] [stderr]    Compiling synstructure v0.13.2
[INFO] [stderr]    Compiling serde_derive_internals v0.29.1
[INFO] [stderr]    Compiling zerofrom-derive v0.1.7
[INFO] [stderr]    Compiling yoke-derive v0.8.2
[INFO] [stderr]    Compiling zerovec-derive v0.11.3
[INFO] [stderr]    Compiling displaydoc v0.2.6
[INFO] [stderr]    Compiling tokio-macros v2.7.0
[INFO] [stderr]    Compiling serde_derive v1.0.228
[INFO] [stderr]    Compiling futures-macro v0.3.32
[INFO] [stderr]    Compiling ref-cast-impl v1.0.25
[INFO] [stderr]    Compiling thiserror-impl v2.0.18
[INFO] [stderr]    Compiling async-stream-impl v0.3.6
[INFO] [stderr]    Compiling clap_derive v4.6.1
[INFO] [stderr]    Compiling schemars_derive v1.2.1
[INFO] [stderr]    Compiling async-stream v0.3.6
[INFO] [stderr]    Compiling tokio v1.52.3
[INFO] [stderr]    Compiling zerofrom v0.1.8
[INFO] [stderr]    Compiling futures-util v0.3.32
[INFO] [stderr]    Compiling yoke v0.8.3
[INFO] [stderr]    Compiling thiserror v2.0.18
[INFO] [stderr]    Compiling zerovec v0.11.6
[INFO] [stderr]    Compiling zerotrie v0.2.4
[INFO] [stderr]    Compiling agent-plugin-installer v0.5.0
[INFO] [stderr]    Compiling clap v4.6.1
[INFO] [stderr]    Compiling tinystr v0.8.3
[INFO] [stderr]    Compiling potential_utf v0.1.5
[INFO] [stderr]    Compiling icu_collections v2.2.0
[INFO] [stderr]    Compiling icu_locale_core v2.2.0
[INFO] [stderr]    Compiling serde v1.0.228
[INFO] [stderr]    Compiling icu_provider v2.2.0
[INFO] [stderr]    Compiling icu_properties v2.2.0
[INFO] [stderr]    Compiling icu_normalizer v2.2.0
[INFO] [stderr]    Compiling serde_urlencoded v0.7.1
[INFO] [stderr]    Compiling schemars v1.2.1
[INFO] [stderr]    Compiling yaml_serde v0.10.4
[INFO] [stderr]    Compiling hyper v1.10.1
[INFO] [stderr]    Compiling tower v0.5.3
[INFO] [stderr]    Compiling tokio-util v0.7.18
[INFO] [stderr]    Compiling inferlab-protocol v0.1.0 (/opt/rustwide/workdir/crates/inferlab-protocol)
[INFO] [stderr]    Compiling idna_adapter v1.2.2
[INFO] [stderr]    Compiling idna v1.1.0
[INFO] [stderr]    Compiling url v2.5.8
[INFO] [stderr]    Compiling hyper-util v0.1.20
[INFO] [stderr]    Compiling tower-http v0.6.11
[INFO] [stderr]    Compiling reqwest v0.13.4
[INFO] [stderr]    Compiling axum v0.8.9
[INFO] [stderr]    Compiling inferlab-proxy v0.1.0 (/opt/rustwide/workdir/crates/inferlab-proxy)
[INFO] [stderr]    Compiling inferlab v0.1.0 (/opt/rustwide/workdir/crates/inferlab)
[INFO] [stderr]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 08s
[INFO] running `Command { std: "docker" "inspect" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27" "/opt/rustwide/cargo-home/bin/cargo" "+1.98.0-beta.1" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr]    Compiling console v0.16.4
[INFO] [stderr]    Compiling similar v2.7.0
[INFO] [stderr]    Compiling anyhow v1.0.103
[INFO] [stderr]    Compiling inferlab-protocol v0.1.0 (/opt/rustwide/workdir/crates/inferlab-protocol)
[INFO] [stderr]    Compiling inferlab-proxy v0.1.0 (/opt/rustwide/workdir/crates/inferlab-proxy)
[INFO] [stderr]    Compiling insta v1.48.0
[INFO] [stderr]    Compiling inferlab v0.1.0 (/opt/rustwide/workdir/crates/inferlab)
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 27.40s
[INFO] running `Command { std: "docker" "inspect" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27" "/opt/rustwide/cargo-home/bin/cargo" "+1.98.0-beta.1" "test" "--frozen", kill_on_drop: false }`
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 0.33s
[INFO] [stderr]      Running unittests src/lib.rs (/opt/rustwide/target/debug/deps/inferlab-87c2f620598f2283)
[INFO] [stdout] 
[INFO] [stdout] running 43 tests
[INFO] [stdout] test image::context::tests::dockerfile_runs_postprocess_then_checks_through_the_entrypoint ... ok
[INFO] [stdout] test image::context::tests::entrypoint_contract_digest_tracks_rendered_text ... ok
[INFO] [stdout] test image::context::tests::entrypoint_excludes_credential_named_activation ... ok
[INFO] [stdout] test image::context::tests::entrypoint_projection_applies_the_v0_v1_rules ... ok
[INFO] [stdout] test image::context::tests::entrypoint_rejects_shell_active_values ... ok
[INFO] [stdout] test image::context::tests::guard_accepts_portable_text ... ok
[INFO] [stdout] test image::context::tests::manual_activation_merge_matches_real_pixi ... ignored
[INFO] [stdout] test image::context::tests::guard_rejects_machine_private_facts_in_portable_text ... ok
[INFO] [stdout] test image::context::tests::pixi_platform_maps_supported_targets ... ok
[INFO] [stdout] test image::runtime::tests::multi_reference_values_are_rejected ... ok
[INFO] [stdout] test profiler::tests::escapes_splice_ahead_of_the_managed_launch_tail ... ok
[INFO] [stdout] test image::context::tests::editable_identities_cover_external_paths_only ... ok
[INFO] [stdout] test image::runtime::tests::check_evidence_reconstructs_from_framed_builder_log ... ok
[INFO] [stdout] test image::context::tests::activation_env_composes_features_per_pixi_precedence ... ok
[INFO] [stdout] test image::context::tests::locked_closure_digest_tracks_pinned_packages_only ... ok
[INFO] [stdout] test image::tool::tests::lossy_tail_survives_multibyte_boundaries ... ok
[INFO] [stdout] test image::runtime::tests::cache_adoption_lost_race_adopts_the_published_artifact ... ok
[INFO] [stdout] test image::runtime::tests::cache_adoption_publishes_atomically_and_drops_the_payload ... ok
[INFO] [stdout] test profiler::tests::prepares_profiled_process_without_changing_the_serving_command ... ok
[INFO] [stdout] test image::runtime::tests::redirects_source_set_references_and_leaves_the_rest ... ok
[INFO] [stdout] test server::network::tests::link_local_only_probes_have_no_candidate ... ok
[INFO] [stdout] test profiler::tests::static_range_plan_maps_windows_to_one_based_reports ... ok
[INFO] [stdout] test server::runtime::tests::hardware_coverage_mismatch_and_empty_output_are_loud ... ok
[INFO] [stdout] test profiler::tests::escapes_splice_ahead_of_the_managed_start_tail ... ok
[INFO] [stdout] test server::runtime::tests::zero_assigned_devices_record_the_driver_without_claiming_inventory ... ok
[INFO] [stdout] test image::context::tests::unmodeled_activation_branches_are_rejected ... ok
[INFO] [stdout] test workload::adaptive::tests::probes_initial_rates_then_bisects_the_pass_fail_bracket ... ok
[INFO] [stdout] test profiler::tests::ssh_control_script_quotes_escape_values_with_metacharacters ... ok
[INFO] [stdout] test server::runtime::tests::hardware_rows_parse_through_banner_noise_in_index_order ... ok
[INFO] [stdout] test server::network::tests::common_rdma_interface_wins_over_a_link_local_default_route ... ok
[INFO] [stdout] test server::network::tests::selection_falls_back_to_a_common_routable_interface ... ok
[INFO] [stdout] test server::runtime::tests::rejects_a_reused_process_identity ... ok
[INFO] [stdout] test shell::tests::shell_quote_preserves_single_quotes ... ok
[INFO] [stdout] test workload::runtime::tests::openai_smoke_requires_a_nonempty_choices_array_with_text ... ok
[INFO] [stdout] test resolve::tests::effective_parallelism_preserves_explicit_role_components ... ok
[INFO] [stdout] test workspace::tests::escape_env_keys_must_be_posix_identifiers ... ok
[INFO] [stdout] test workspace::tests::managed_and_dedicated_escape_options_are_rejected_in_both_lists ... ok
[INFO] [stdout] test workspace::tests::role_escapes_merge_into_profile_escapes ... ok
[INFO] [stdout] test workspace::tests::standalone_terminators_are_rejected_in_both_lists ... ok
[INFO] [stdout] test profiler::tests::missing_range_report_is_capture_failure_evidence ... ok
[INFO] [stdout] test server::tests::multi_role_launch_failure_rolls_back_the_first_role ... ok
[INFO] [stdout] test workspace::tests::source_digest_script_text_is_pinned ... ok
[INFO] [stdout] test server::runtime::tests::termination_waits_for_the_group_after_the_launcher_exits ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 42 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.17s
[INFO] [stdout] 
[INFO] [stderr]      Running unittests src/main.rs (/opt/rustwide/target/debug/deps/inferlab-6967e8f653f67b10)
[INFO] [stdout] 
[INFO] [stdout] running 0 tests
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/agent.rs (/opt/rustwide/target/debug/deps/agent-c78cabc977f018d5)
[INFO] [stdout] 
[INFO] [stdout] running 9 tests
[INFO] [stdout] test license_output_failure_is_an_error_not_a_panic ... ok
[INFO] [stdout] test license_prints_the_notice ... ok
[INFO] [stdout] test preflight_failure_still_emits_the_report ... ok
[INFO] [stdout] test doctor_reports_missing_native_clis ... ok
[INFO] [stdout] test install_fails_loudly_before_any_cli_on_a_broken_package ... ok
[INFO] [stdout] test a_gate_failure_on_one_runtime_skips_the_other ... ok
[INFO] [stdout] test a_spawn_failure_after_a_mutation_keeps_the_mutation_in_the_report ... ok
[INFO] [stdout] test install_validates_the_shipped_package_and_drives_both_clis ... ok
[INFO] [stdout] test native_failure_still_emits_the_report_then_fails_loudly ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.16s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/cli.rs (/opt/rustwide/target/debug/deps/cli-d5d40d35fc5838c7)
[INFO] [stdout] 
[INFO] [stdout] running 1 test
[INFO] [stdout] test help_is_a_runnable_minimal_surface ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/dry_run.rs (/opt/rustwide/target/debug/deps/dry_run-2c74c858d6908e17)
[INFO] [stdout] 
[INFO] [stdout] running 45 tests
[INFO] [stdout] test fragment_type_errors_name_their_position ... ok
[INFO] [stdout] test identifier_declared_by_two_files_is_rejected_naming_both ... ok
[INFO] [stdout] test digest_visible_links_may_not_ride_machine_local_links ... ok
[INFO] [stdout] test explicit_local_bindings_file_replaces_the_default ... ok
[INFO] [stdout] test escaping_source_links_are_rejected ... ok
[INFO] [stdout] test integration_must_be_selected_by_the_pixi_manifest ... ok
[INFO] [stdout] test identity_uncovered_targets_are_rejected ... ok
[INFO] [stdout] test missing_local_bindings_error_guides_the_operator ... ok
[INFO] [stdout] test explicit_case_and_server_override_preserve_provenance ... ok
[INFO] [stdout] test gpu_groups_can_place_multiple_ranks_on_one_machine ... ok
[INFO] [stdout] test insufficient_devices_are_reported_after_lowering ... ok
[INFO] [stdout] test chains_through_covered_link_directories_are_accepted ... ok
[INFO] [stdout] test case_and_invocation_roles_must_belong_to_the_selected_topology ... ok
[INFO] [stdout] test ignored_links_to_in_root_content_are_machine_local ... ok
[INFO] [stdout] test empty_fragment_directory_leaves_the_single_file_workspace_unchanged ... ok
[INFO] [stdout] test dirty_workspace_reports_a_digest_and_effective_values ... ok
[INFO] [stdout] test dirty_submodule_state_changes_workspace_evidence ... ok
[INFO] [stdout] test missing_weight_binding_is_reported_before_lowering ... ok
[INFO] [stdout] test built_in_proxy_prefers_the_local_machine_in_a_remote_first_placement ... ok
[INFO] [stdout] test override_outside_server_settings_is_rejected ... ok
[INFO] [stdout] test missing_eval_toolchain_reports_the_explicit_install_action ... ok
[INFO] [stdout] test machine_binding_selects_runtime_cache_storage_root ... ok
[INFO] [stdout] test schema_version_in_a_fragment_is_rejected ... ok
[INFO] [stdout] test placement_role_must_belong_to_the_resolved_topology ... ok
[INFO] [stdout] test ordered_two_node_placement_is_allocated_before_process_rendering ... ok
[INFO] [stdout] test symlinked_inferlab_directory_is_rejected ... ok
[INFO] [stdout] test out_of_source_set_bridge_links_are_contained ... ok
[INFO] [stdout] test symlink_cycles_are_rejected ... ok
[INFO] [stdout] test submodule_ignore_rules_govern_submodule_targets ... ok
[INFO] [stdout] test definitions_split_across_fragments_resolve_identically ... ok
[INFO] [stdout] test dangling_submodule_links_do_not_kill_the_digest ... ok
[INFO] [stdout] test unresolved_typed_reference_is_rejected ... ok
[INFO] [stdout] test protocol_version_mismatch_names_both_versions_and_the_remedy ... ok
[INFO] [stdout] test unknown_pixi_environment_is_rejected_before_lowering ... ok
[INFO] [stdout] test internal_source_links_are_permitted ... ok
[INFO] [stdout] test symlinked_source_set_components_are_rejected ... ok
[INFO] [stdout] test recipe_capture_selects_one_workload_and_prepares_the_server ... ok
[INFO] [stdout] test symlinked_workspace_files_are_rejected ... ok
[INFO] [stdout] test unavailable_pixi_environment_reports_the_locked_install_action ... ok
[INFO] [stdout] test two_node_resolution_rejects_placements_without_a_common_routable_interface ... ok
[INFO] [stdout] test static_npmd_on_one_machine_allocates_disjoint_replicas_and_a_public_proxy ... ok
[INFO] [stdout] test scratchpad_state_stays_outside_source_identity ... ok
[INFO] [stdout] test uncovered_links_are_rejected_regardless_of_tracking_state ... ok
[INFO] [stdout] test serve_and_recipe_dry_run_share_the_default_case ... ok
[INFO] [stderr]      Running tests/environment.rs (/opt/rustwide/target/debug/deps/environment-46dd97e57534d2f8)
[INFO] [stdout] test submodule_untracked_links_enter_the_source_digest ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 45 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.43s
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] running 3 tests
[INFO] [stdout] test real_pixi_clean_prefix_lock_and_locked_install ... ignored, requires real Pixi and conda-forge access
[INFO] [stdout] test env_lock_restores_manifest_and_previous_lock_when_full_lock_fails ... ok
[INFO] [stdout] test env_lock_stages_build_dependencies_and_leaves_the_full_lock ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 2 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.05s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/error_registry.rs (/opt/rustwide/target/debug/deps/error_registry-f6c2014b231ec650)
[INFO] [stdout] 
[INFO] [stdout] running 1 test
[INFO] [stderr]      Running tests/harness.rs (/opt/rustwide/target/debug/deps/harness-104bb77e96e13366)
[INFO] [stdout] test the_shipped_registry_and_the_emitted_codes_agree ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] running 16 tests
[INFO] [stdout] test claimed_ports_are_never_rehanded_in_process ... ok
[INFO] [stdout] test reclaim_creates_fresh_when_the_lease_vanished ... ok
[INFO] [stdout] test port_lease_is_exclusive_while_owner_lives ... ok
[INFO] [stdout] test lease_sweep_unlinks_exactly_the_stale_shapes ... ok
[INFO] [stdout] test sweep_defers_to_the_directory_lock ... ok
[INFO] [stdout] test reclaim_rejudges_under_the_directory_lock ... ok
[INFO] [stdout] test port_lease_reclaims_dead_owners_only ... ok
[INFO] [stdout] test the_guard_file_is_never_judged_or_swept ... ok
[INFO] [stdout] test sweep_spares_recycled_group_identities ... ok
[INFO] [stdout] test reserved_ports_are_distinct_and_leased ... ok
[INFO] [stdout] test sweep_skips_groups_with_live_owners ... ok
[INFO] [stdout] test guard_reaps_registered_groups_on_drop ... ok
[INFO] [stdout] test guard_spares_groups_of_other_workspaces ... ok
[INFO] [stdout] test sweep_reclaims_groups_with_dead_owners ... ok
[INFO] [stdout] test guard_reaps_registered_groups_on_panic ... ok
[INFO] [stderr]      Running tests/image.rs (/opt/rustwide/target/debug/deps/image-c62ea7405954ae9d)
[INFO] [stdout] test fixture_server_shim_registers_and_is_reaped ... ok
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.25s
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] running 35 tests
[INFO] [stdout] test dirty_workspace_is_rejected ... ok
[INFO] [stdout] test all_unproducible_platforms_fail_resolution ... ok
[INFO] [stdout] test hung_remote_removal_expires_with_deadline_evidence ... FAILED
[INFO] [stdout] test dry_run_reports_dedup_and_eligibility ... ok
[INFO] [stdout] test confirmed_removal_with_failed_process_cleanup_is_not_verified ... FAILED
[INFO] [stdout] test failing_entry_check_aborts_before_package_builds ... ok
[INFO] [stdout] test invalid_container_hardware_declarations_are_rejected_at_load ... ok
[INFO] [stdout] test invalid_check_declarations_fail_at_load ... ok
[INFO] [stdout] test external_image_missing_on_a_machine_rejects_naming_the_pull ... FAILED
[INFO] [stdout] test container_hardware_facts_are_lowered_as_declared ... ok
[INFO] [stdout] test external_two_machine_serving_resolves_per_machine_facts ... FAILED
[INFO] [stdout] test pass_env_value_declarations_are_rejected ... ok
[INFO] [stdout] test image_backed_multi_machine_is_rejected_at_the_distribution_boundary ... ok
[INFO] [stdout] test missing_builder_binding_is_rejected ... ok
[INFO] [stdout] test docker_exit_removal_reason_is_distinct_from_the_deadline ... FAILED
[INFO] [stdout] test incompatible_external_selections_are_rejected ... ok
[INFO] [stdout] test adapter_container_device_is_declared_not_guessed ... ok
[INFO] [stdout] test mutating_package_build_fails_the_assembly ... ok
[INFO] [stdout] test selection_rejections_precede_integration_invocation ... ok
[INFO] [stdout] test zero_adapter_timeout_is_rejected_at_load ... ok
[INFO] [stdout] test unknown_external_integration_claim_is_rejected_at_load ... ok
[INFO] [stdout] test image_backed_capture_is_rejected ... ok
[INFO] [stdout] test mutating_then_failing_package_build_still_reports_the_mutation ... ok
[INFO] [stdout] test swallowed_ssh_handle_removes_the_created_container ... FAILED
[INFO] [stdout] test image_backed_launch_needs_no_local_environment ... ok
[INFO] [stdout] test external_image_recipe_runs_with_unqualified_evidence ... ok
[INFO] [stdout] test unconfirmed_launch_removal_never_claims_verified_cleanup ... FAILED
[INFO] [stdout] test oversized_adapter_diagnostics_do_not_deadlock ... ok
[INFO] [stdout] test structured_rejection_attempts_no_container_removal ... ok
[INFO] [stdout] test incompatible_image_selections_are_rejected ... ok
[INFO] [stdout] test native_only_image_succeeds ... ok
[INFO] [stdout] test timed_out_adapter_container_is_removed ... ok
[INFO] [stdout] test serve_start_from_image_admits_manual_bench_and_stops ... ok
[INFO] [stdout] test image_backed_recipe_runs_from_the_selected_record ... ok
[INFO] [stdout] test closed_loop_builds_validates_and_scopes_platforms ... ok
[INFO] [stdout] 
[INFO] [stdout] failures:
[INFO] [stdout] 
[INFO] [stdout] ---- hung_remote_removal_expires_with_deadline_evidence stdout ----
[INFO] [stdout] Error: "stdout is not JSON (EOF while parsing a value at line 1 column 0); stdout: ; stderr: error[E1004]: invalid configuration: network resolution failed: no common routable communication interface across machines [local, remote]; verify that every machine has a common global IPv4 interface and that RDMA links are up\n"
[INFO] [stdout] 
[INFO] [stdout] ---- confirmed_removal_with_failed_process_cleanup_is_not_verified stdout ----
[INFO] [stdout] Error: "stdout is not JSON (EOF while parsing a value at line 1 column 0); stdout: ; stderr: error[E1004]: invalid configuration: network resolution failed: no common routable communication interface across machines [local, remote]; verify that every machine has a common global IPv4 interface and that RDMA links are up\n"
[INFO] [stdout] 
[INFO] [stdout] ---- external_image_missing_on_a_machine_rejects_naming_the_pull stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'external_image_missing_on_a_machine_rejects_naming_the_pull' (6629) panicked at crates/inferlab/tests/image.rs:2379:5:
[INFO] [stdout] the rejection names the machine and the exact pull command: error[E1004]: invalid configuration: network resolution failed: no common routable communication interface across machines [local, remote]; verify that every machine has a common global IPv4 interface and that RDMA links are up
[INFO] [stdout] 
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x62f760a901a1 - std[73adb7dc35730857]::backtrace_rs::backtrace::libunwind::trace
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x62f760a901a1 - std[73adb7dc35730857]::backtrace_rs::backtrace::trace_unsynchronized::<std[73adb7dc35730857]::sys::backtrace::_print_fmt::{closure#1}>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x62f760a901a1 - std[73adb7dc35730857]::sys::backtrace::_print_fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:74:9
[INFO] [stdout]    3:     0x62f760a901a1 - <<std[73adb7dc35730857]::sys::backtrace::BacktraceLock>::print::DisplayBacktrace as core[6883ba1bc0fe4ed1]::fmt::Display>::fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:44:26
[INFO] [stdout]    4:     0x62f760aa57ca - <core[6883ba1bc0fe4ed1]::fmt::rt::Argument>::fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x62f760aa57ca - core[6883ba1bc0fe4ed1]::fmt::write
[INFO] [stdout]    6:     0x62f760a94b7c - std[73adb7dc35730857]::io::default_write_fmt::<alloc[55a36b64bcbf2c0d]::vec::Vec<u8>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/io/mod.rs:626:11
[INFO] [stdout]    7:     0x62f760a94b7c - <alloc[55a36b64bcbf2c0d]::vec::Vec<u8> as std[73adb7dc35730857]::io::Write>::write_fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/io/mod.rs:1730:13
[INFO] [stdout]    8:     0x62f760a6b3a6 - <std[73adb7dc35730857]::sys::backtrace::BacktraceLock>::print
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:47:9
[INFO] [stdout]    9:     0x62f760a6b3a6 - std[73adb7dc35730857]::panicking::default_hook::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x62f760a868d9 - std[73adb7dc35730857]::panicking::default_hook
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x62f7609d5710 - <alloc[55a36b64bcbf2c0d]::boxed::Box<dyn for<'a, 'b> core[6883ba1bc0fe4ed1]::ops::function::Fn<(&'a std[73adb7dc35730857]::panic::PanicHookInfo<'b>,), Output = ()> + core[6883ba1bc0fe4ed1]::marker::Send + core[6883ba1bc0fe4ed1]::marker::Sync> as core[6883ba1bc0fe4ed1]::ops::function::Fn<(&std[73adb7dc35730857]::panic::PanicHookInfo,)>>::call
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/alloc/src/boxed.rs:2333:9
[INFO] [stdout]   12:     0x62f7609d5710 - test[980ffaebd391d06d]::test_main_inner::<test[980ffaebd391d06d]::test_main_static::{closure#0}>::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:155:21
[INFO] [stdout]   13:     0x62f760a86a92 - <alloc[55a36b64bcbf2c0d]::boxed::Box<dyn for<'a, 'b> core[6883ba1bc0fe4ed1]::ops::function::Fn<(&'a std[73adb7dc35730857]::panic::PanicHookInfo<'b>,), Output = ()> + core[6883ba1bc0fe4ed1]::marker::Send + core[6883ba1bc0fe4ed1]::marker::Sync> as core[6883ba1bc0fe4ed1]::ops::function::Fn<(&std[73adb7dc35730857]::panic::PanicHookInfo,)>>::call
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/alloc/src/boxed.rs:2333:9
[INFO] [stdout]   14:     0x62f760a86a92 - std[73adb7dc35730857]::panicking::panic_with_hook
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:823:13
[INFO] [stdout]   15:     0x62f760a6b452 - std[73adb7dc35730857]::panicking::panic_handler::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:688:13
[INFO] [stdout]   16:     0x62f760a629b9 - std[73adb7dc35730857]::sys::backtrace::__rust_end_short_backtrace::<std[73adb7dc35730857]::panicking::panic_handler::{closure#0}, !>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:182:18
[INFO] [stdout]   17:     0x62f760a6c2fd - __rustc[a7b7b02e776dd976]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:679:5
[INFO] [stdout]   18:     0x62f760aa5f9c - core[6883ba1bc0fe4ed1]::panicking::panic_fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x62f76098bd15 - image[5abd30ebf079914b]::external_image_missing_on_a_machine_rejects_naming_the_pull
[INFO] [stdout]                                at /opt/rustwide/workdir/crates/inferlab/tests/image.rs:2379:5
[INFO] [stdout]   20:     0x62f76095d207 - image[5abd30ebf079914b]::external_image_missing_on_a_machine_rejects_naming_the_pull::{closure#0}
[INFO] [stdout]                                at /opt/rustwide/workdir/crates/inferlab/tests/image.rs:2357:69
[INFO] [stdout]   21:     0x62f7609b4ec6 - <image[5abd30ebf079914b]::external_image_missing_on_a_machine_rejects_naming_the_pull::{closure#0} as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   22:     0x62f7609c8a4b - <fn() -> core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   23:     0x62f7609c8a4b - test[980ffaebd391d06d]::__rust_begin_short_backtrace::<core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>, fn() -> core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:724:18
[INFO] [stdout]   24:     0x62f7609d6095 - test[980ffaebd391d06d]::run_test_in_process::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:747:74
[INFO] [stdout]   25:     0x62f7609d6095 - <core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/panic/unwind_safe.rs:275:9
[INFO] [stdout]   26:     0x62f7609d6095 - std[73adb7dc35730857]::panicking::catch_unwind::do_call::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}>, core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:576:43
[INFO] [stdout]   27:     0x62f7609d6095 - std[73adb7dc35730857]::panicking::catch_unwind::<core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>, core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:544:19
[INFO] [stdout]   28:     0x62f7609d6095 - std[73adb7dc35730857]::panic::catch_unwind::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}>, core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panic.rs:359:14
[INFO] [stdout]   29:     0x62f7609d6095 - test[980ffaebd391d06d]::run_test_in_process
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:747:27
[INFO] [stdout]   30:     0x62f7609d6095 - test[980ffaebd391d06d]::run_test::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:668:43
[INFO] [stdout]   31:     0x62f7609d0b44 - test[980ffaebd391d06d]::run_test::{closure#1}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:698:41
[INFO] [stdout]   32:     0x62f7609d0b44 - std[73adb7dc35730857]::sys::backtrace::__rust_begin_short_backtrace::<test[980ffaebd391d06d]::run_test::{closure#1}, ()>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:166:18
[INFO] [stdout]   33:     0x62f7609d91e2 - std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked::<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/thread/lifecycle.rs:70:13
[INFO] [stdout]   34:     0x62f7609d91e2 - <core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/panic/unwind_safe.rs:275:9
[INFO] [stdout]   35:     0x62f7609d91e2 - std[73adb7dc35730857]::panicking::catch_unwind::do_call::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>, ()>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:576:43
[INFO] [stdout]   36:     0x62f7609d91e2 - std[73adb7dc35730857]::panicking::catch_unwind::<(), core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:544:19
[INFO] [stdout]   37:     0x62f7609d91e2 - std[73adb7dc35730857]::panic::catch_unwind::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>, ()>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panic.rs:359:14
[INFO] [stdout]   38:     0x62f7609d91e2 - std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked::<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/thread/lifecycle.rs:68:26
[INFO] [stdout]   39:     0x62f7609d91e2 - <std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1} as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   40:     0x62f760a8ebcf - <alloc[55a36b64bcbf2c0d]::boxed::Box<dyn core[6883ba1bc0fe4ed1]::ops::function::FnOnce<(), Output = ()> + core[6883ba1bc0fe4ed1]::marker::Send> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/alloc/src/boxed.rs:2319:9
[INFO] [stdout]   41:     0x62f760a8ebcf - <std[73adb7dc35730857]::sys::thread::unix::Thread>::new::thread_start
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/thread/unix.rs:123:17
[INFO] [stdout]   42:     0x795e445eeaa4 - <unknown>
[INFO] [stdout]   43:     0x795e4467ba64 - clone
[INFO] [stdout]   44:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- external_two_machine_serving_resolves_per_machine_facts stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'external_two_machine_serving_resolves_per_machine_facts' (6631) panicked at crates/inferlab/tests/image.rs:1935:5:
[INFO] [stdout] two-machine external dry-run failed: error[E1004]: invalid configuration: network resolution failed: no common routable communication interface across machines [local, remote]; verify that every machine has a common global IPv4 interface and that RDMA links are up
[INFO] [stdout] 
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x62f760a901a1 - std[73adb7dc35730857]::backtrace_rs::backtrace::libunwind::trace
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x62f760a901a1 - std[73adb7dc35730857]::backtrace_rs::backtrace::trace_unsynchronized::<std[73adb7dc35730857]::sys::backtrace::_print_fmt::{closure#1}>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x62f760a901a1 - std[73adb7dc35730857]::sys::backtrace::_print_fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:74:9
[INFO] [stdout]    3:     0x62f760a901a1 - <<std[73adb7dc35730857]::sys::backtrace::BacktraceLock>::print::DisplayBacktrace as core[6883ba1bc0fe4ed1]::fmt::Display>::fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:44:26
[INFO] [stdout]    4:     0x62f760aa57ca - <core[6883ba1bc0fe4ed1]::fmt::rt::Argument>::fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x62f760aa57ca - core[6883ba1bc0fe4ed1]::fmt::write
[INFO] [stdout]    6:     0x62f760a94b7c - std[73adb7dc35730857]::io::default_write_fmt::<alloc[55a36b64bcbf2c0d]::vec::Vec<u8>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/io/mod.rs:626:11
[INFO] [stdout]    7:     0x62f760a94b7c - <alloc[55a36b64bcbf2c0d]::vec::Vec<u8> as std[73adb7dc35730857]::io::Write>::write_fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/io/mod.rs:1730:13
[INFO] [stdout]    8:     0x62f760a6b3a6 - <std[73adb7dc35730857]::sys::backtrace::BacktraceLock>::print
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:47:9
[INFO] [stdout]    9:     0x62f760a6b3a6 - std[73adb7dc35730857]::panicking::default_hook::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x62f760a868d9 - std[73adb7dc35730857]::panicking::default_hook
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x62f7609d5710 - <alloc[55a36b64bcbf2c0d]::boxed::Box<dyn for<'a, 'b> core[6883ba1bc0fe4ed1]::ops::function::Fn<(&'a std[73adb7dc35730857]::panic::PanicHookInfo<'b>,), Output = ()> + core[6883ba1bc0fe4ed1]::marker::Send + core[6883ba1bc0fe4ed1]::marker::Sync> as core[6883ba1bc0fe4ed1]::ops::function::Fn<(&std[73adb7dc35730857]::panic::PanicHookInfo,)>>::call
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/alloc/src/boxed.rs:2333:9
[INFO] [stdout]   12:     0x62f7609d5710 - test[980ffaebd391d06d]::test_main_inner::<test[980ffaebd391d06d]::test_main_static::{closure#0}>::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:155:21
[INFO] [stdout]   13:     0x62f760a86a92 - <alloc[55a36b64bcbf2c0d]::boxed::Box<dyn for<'a, 'b> core[6883ba1bc0fe4ed1]::ops::function::Fn<(&'a std[73adb7dc35730857]::panic::PanicHookInfo<'b>,), Output = ()> + core[6883ba1bc0fe4ed1]::marker::Send + core[6883ba1bc0fe4ed1]::marker::Sync> as core[6883ba1bc0fe4ed1]::ops::function::Fn<(&std[73adb7dc35730857]::panic::PanicHookInfo,)>>::call
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/alloc/src/boxed.rs:2333:9
[INFO] [stdout]   14:     0x62f760a86a92 - std[73adb7dc35730857]::panicking::panic_with_hook
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:823:13
[INFO] [stdout]   15:     0x62f760a6b452 - std[73adb7dc35730857]::panicking::panic_handler::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:688:13
[INFO] [stdout]   16:     0x62f760a629b9 - std[73adb7dc35730857]::sys::backtrace::__rust_end_short_backtrace::<std[73adb7dc35730857]::panicking::panic_handler::{closure#0}, !>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:182:18
[INFO] [stdout]   17:     0x62f760a6c2fd - __rustc[a7b7b02e776dd976]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:679:5
[INFO] [stdout]   18:     0x62f760aa5f9c - core[6883ba1bc0fe4ed1]::panicking::panic_fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x62f760985b59 - image[5abd30ebf079914b]::external_two_machine_serving_resolves_per_machine_facts
[INFO] [stdout]                                at /opt/rustwide/workdir/crates/inferlab/tests/image.rs:1935:5
[INFO] [stdout]   20:     0x62f76095d177 - image[5abd30ebf079914b]::external_two_machine_serving_resolves_per_machine_facts::{closure#0}
[INFO] [stdout]                                at /opt/rustwide/workdir/crates/inferlab/tests/image.rs:1919:65
[INFO] [stdout]   21:     0x62f7609b4e06 - <image[5abd30ebf079914b]::external_two_machine_serving_resolves_per_machine_facts::{closure#0} as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   22:     0x62f7609c8a4b - <fn() -> core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   23:     0x62f7609c8a4b - test[980ffaebd391d06d]::__rust_begin_short_backtrace::<core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>, fn() -> core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:724:18
[INFO] [stdout]   24:     0x62f7609d6095 - test[980ffaebd391d06d]::run_test_in_process::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:747:74
[INFO] [stdout]   25:     0x62f7609d6095 - <core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/panic/unwind_safe.rs:275:9
[INFO] [stdout]   26:     0x62f7609d6095 - std[73adb7dc35730857]::panicking::catch_unwind::do_call::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}>, core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:576:43
[INFO] [stdout]   27:     0x62f7609d6095 - std[73adb7dc35730857]::panicking::catch_unwind::<core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>, core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:544:19
[INFO] [stdout]   28:     0x62f7609d6095 - std[73adb7dc35730857]::panic::catch_unwind::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}>, core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panic.rs:359:14
[INFO] [stdout]   29:     0x62f7609d6095 - test[980ffaebd391d06d]::run_test_in_process
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:747:27
[INFO] [stdout]   30:     0x62f7609d6095 - test[980ffaebd391d06d]::run_test::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:668:43
[INFO] [stdout]   31:     0x62f7609d0b44 - test[980ffaebd391d06d]::run_test::{closure#1}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:698:41
[INFO] [stdout]   32:     0x62f7609d0b44 - std[73adb7dc35730857]::sys::backtrace::__rust_begin_short_backtrace::<test[980ffaebd391d06d]::run_test::{closure#1}, ()>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:166:18
[INFO] [stdout]   33:     0x62f7609d91e2 - std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked::<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/thread/lifecycle.rs:70:13
[INFO] [stdout]   34:     0x62f7609d91e2 - <core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/panic/unwind_safe.rs:275:9
[INFO] [stdout]   35:     0x62f7609d91e2 - std[73adb7dc35730857]::panicking::catch_unwind::do_call::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>, ()>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:576:43
[INFO] [stdout]   36:     0x62f7609d91e2 - std[73adb7dc35730857]::panicking::catch_unwind::<(), core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:544:19
[INFO] [stdout]   37:     0x62f7609d91e2 - std[73adb7dc35730857]::panic::catch_unwind::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>, ()>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panic.rs:359:14
[INFO] [stdout]   38:     0x62f7609d91e2 - std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked::<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/thread/lifecycle.rs:68:26
[INFO] [stdout]   39:     0x62f7609d91e2 - <std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1} as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   40:     0x62f760a8ebcf - <alloc[55a36b64bcbf2c0d]::boxed::Box<dyn core[6883ba1bc0fe4ed1]::ops::function::FnOnce<(), Output = ()> + core[6883ba1bc0fe4ed1]::marker::Send> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/alloc/src/boxed.rs:2319:9
[INFO] [stdout]   41:     0x62f760a8ebcf - <std[73adb7dc35730857]::sys::thread::unix::Thread>::new::thread_start
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/thread/unix.rs:123:17
[INFO] [stdout]   42:     0x795e445eeaa4 - <unknown>
[INFO] [stdout]   43:     0x795e4467ba64 - clone
[INFO] [stdout]   44:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- docker_exit_removal_reason_is_distinct_from_the_deadline stdout ----
[INFO] [stdout] Error: "stdout is not JSON (EOF while parsing a value at line 1 column 0); stdout: ; stderr: error[E1004]: invalid configuration: network resolution failed: no common routable communication interface across machines [local, remote]; verify that every machine has a common global IPv4 interface and that RDMA links are up\n"
[INFO] [stdout] 
[INFO] [stdout] ---- swallowed_ssh_handle_removes_the_created_container stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'swallowed_ssh_handle_removes_the_created_container' (9625) panicked at crates/inferlab/tests/image.rs:2111:5:
[INFO] [stdout] the launch failure removed the remote container: image inspect --format {{.Id}} example.com/fixture-vllm@sha256:abababababababababababababababababababababababababababababababab
[INFO] [stdout] run --rm --interactive --cidfile /tmp/.tmpRZLtFs/cid --mount type=bind,source=/tmp/.tmpfj461X/.pixi/envs/adapter/site/inferlab_adapter_sdk,target=/inferlab-adapter/inferlab_adapter_sdk,readonly --mount type=bind,source=/tmp/.tmpfj461X/.pixi/envs/adapter/site/inferlab_integration_vllm,target=/inferlab-adapter/inferlab_integration_vllm,readonly --env PYTHONPATH=/inferlab-adapter --entrypoint python3 example.com/fixture-vllm@sha256:abababababababababababababababababababababababababababababababab -m inferlab_integration_vllm
[INFO] [stdout] run --rm --interactive --cidfile /tmp/.tmpf3ET2t/cid --mount type=bind,source=/tmp/.tmpfj461X/.pixi/envs/adapter/site/inferlab_adapter_sdk,target=/inferlab-adapter/inferlab_adapter_sdk,readonly --mount type=bind,source=/tmp/.tmpfj461X/.pixi/envs/adapter/site/inferlab_integration_vllm,target=/inferlab-adapter/inferlab_integration_vllm,readonly --env PYTHONPATH=/inferlab-adapter --entrypoint python3 example.com/fixture-vllm@sha256:abababababababababababababababababababababababababababababababab -m inferlab_integration_vllm
[INFO] [stdout] 
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x62f760a901a1 - std[73adb7dc35730857]::backtrace_rs::backtrace::libunwind::trace
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x62f760a901a1 - std[73adb7dc35730857]::backtrace_rs::backtrace::trace_unsynchronized::<std[73adb7dc35730857]::sys::backtrace::_print_fmt::{closure#1}>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x62f760a901a1 - std[73adb7dc35730857]::sys::backtrace::_print_fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:74:9
[INFO] [stdout]    3:     0x62f760a901a1 - <<std[73adb7dc35730857]::sys::backtrace::BacktraceLock>::print::DisplayBacktrace as core[6883ba1bc0fe4ed1]::fmt::Display>::fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:44:26
[INFO] [stdout]    4:     0x62f760aa57ca - <core[6883ba1bc0fe4ed1]::fmt::rt::Argument>::fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x62f760aa57ca - core[6883ba1bc0fe4ed1]::fmt::write
[INFO] [stdout]    6:     0x62f760a94b7c - std[73adb7dc35730857]::io::default_write_fmt::<alloc[55a36b64bcbf2c0d]::vec::Vec<u8>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/io/mod.rs:626:11
[INFO] [stdout]    7:     0x62f760a94b7c - <alloc[55a36b64bcbf2c0d]::vec::Vec<u8> as std[73adb7dc35730857]::io::Write>::write_fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/io/mod.rs:1730:13
[INFO] [stdout]    8:     0x62f760a6b3a6 - <std[73adb7dc35730857]::sys::backtrace::BacktraceLock>::print
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:47:9
[INFO] [stdout]    9:     0x62f760a6b3a6 - std[73adb7dc35730857]::panicking::default_hook::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x62f760a868d9 - std[73adb7dc35730857]::panicking::default_hook
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x62f7609d5710 - <alloc[55a36b64bcbf2c0d]::boxed::Box<dyn for<'a, 'b> core[6883ba1bc0fe4ed1]::ops::function::Fn<(&'a std[73adb7dc35730857]::panic::PanicHookInfo<'b>,), Output = ()> + core[6883ba1bc0fe4ed1]::marker::Send + core[6883ba1bc0fe4ed1]::marker::Sync> as core[6883ba1bc0fe4ed1]::ops::function::Fn<(&std[73adb7dc35730857]::panic::PanicHookInfo,)>>::call
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/alloc/src/boxed.rs:2333:9
[INFO] [stdout]   12:     0x62f7609d5710 - test[980ffaebd391d06d]::test_main_inner::<test[980ffaebd391d06d]::test_main_static::{closure#0}>::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:155:21
[INFO] [stdout]   13:     0x62f760a86a92 - <alloc[55a36b64bcbf2c0d]::boxed::Box<dyn for<'a, 'b> core[6883ba1bc0fe4ed1]::ops::function::Fn<(&'a std[73adb7dc35730857]::panic::PanicHookInfo<'b>,), Output = ()> + core[6883ba1bc0fe4ed1]::marker::Send + core[6883ba1bc0fe4ed1]::marker::Sync> as core[6883ba1bc0fe4ed1]::ops::function::Fn<(&std[73adb7dc35730857]::panic::PanicHookInfo,)>>::call
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/alloc/src/boxed.rs:2333:9
[INFO] [stdout]   14:     0x62f760a86a92 - std[73adb7dc35730857]::panicking::panic_with_hook
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:823:13
[INFO] [stdout]   15:     0x62f760a6b452 - std[73adb7dc35730857]::panicking::panic_handler::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:688:13
[INFO] [stdout]   16:     0x62f760a629b9 - std[73adb7dc35730857]::sys::backtrace::__rust_end_short_backtrace::<std[73adb7dc35730857]::panicking::panic_handler::{closure#0}, !>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:182:18
[INFO] [stdout]   17:     0x62f760a6c2fd - __rustc[a7b7b02e776dd976]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:679:5
[INFO] [stdout]   18:     0x62f760aa5f9c - core[6883ba1bc0fe4ed1]::panicking::panic_fmt
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x62f76097fb78 - image[5abd30ebf079914b]::swallowed_ssh_handle_removes_the_created_container
[INFO] [stdout]                                at /opt/rustwide/workdir/crates/inferlab/tests/image.rs:2111:5
[INFO] [stdout]   20:     0x62f76095d087 - image[5abd30ebf079914b]::swallowed_ssh_handle_removes_the_created_container::{closure#0}
[INFO] [stdout]                                at /opt/rustwide/workdir/crates/inferlab/tests/image.rs:2085:60
[INFO] [stdout]   21:     0x62f7609b4cc6 - <image[5abd30ebf079914b]::swallowed_ssh_handle_removes_the_created_container::{closure#0} as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   22:     0x62f7609c8a4b - <fn() -> core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   23:     0x62f7609c8a4b - test[980ffaebd391d06d]::__rust_begin_short_backtrace::<core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>, fn() -> core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:724:18
[INFO] [stdout]   24:     0x62f7609d6095 - test[980ffaebd391d06d]::run_test_in_process::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:747:74
[INFO] [stdout]   25:     0x62f7609d6095 - <core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/panic/unwind_safe.rs:275:9
[INFO] [stdout]   26:     0x62f7609d6095 - std[73adb7dc35730857]::panicking::catch_unwind::do_call::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}>, core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:576:43
[INFO] [stdout]   27:     0x62f7609d6095 - std[73adb7dc35730857]::panicking::catch_unwind::<core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>, core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:544:19
[INFO] [stdout]   28:     0x62f7609d6095 - std[73adb7dc35730857]::panic::catch_unwind::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<test[980ffaebd391d06d]::run_test_in_process::{closure#0}>, core[6883ba1bc0fe4ed1]::result::Result<(), alloc[55a36b64bcbf2c0d]::string::String>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panic.rs:359:14
[INFO] [stdout]   29:     0x62f7609d6095 - test[980ffaebd391d06d]::run_test_in_process
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:747:27
[INFO] [stdout]   30:     0x62f7609d6095 - test[980ffaebd391d06d]::run_test::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:668:43
[INFO] [stdout]   31:     0x62f7609d0b44 - test[980ffaebd391d06d]::run_test::{closure#1}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/test/src/lib.rs:698:41
[INFO] [stdout]   32:     0x62f7609d0b44 - std[73adb7dc35730857]::sys::backtrace::__rust_begin_short_backtrace::<test[980ffaebd391d06d]::run_test::{closure#1}, ()>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/backtrace.rs:166:18
[INFO] [stdout]   33:     0x62f7609d91e2 - std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked::<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/thread/lifecycle.rs:70:13
[INFO] [stdout]   34:     0x62f7609d91e2 - <core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/panic/unwind_safe.rs:275:9
[INFO] [stdout]   35:     0x62f7609d91e2 - std[73adb7dc35730857]::panicking::catch_unwind::do_call::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>, ()>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:576:43
[INFO] [stdout]   36:     0x62f7609d91e2 - std[73adb7dc35730857]::panicking::catch_unwind::<(), core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panicking.rs:544:19
[INFO] [stdout]   37:     0x62f7609d91e2 - std[73adb7dc35730857]::panic::catch_unwind::<core[6883ba1bc0fe4ed1]::panic::unwind_safe::AssertUnwindSafe<std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>, ()>
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/panic.rs:359:14
[INFO] [stdout]   38:     0x62f7609d91e2 - std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked::<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/thread/lifecycle.rs:68:26
[INFO] [stdout]   39:     0x62f7609d91e2 - <std[73adb7dc35730857]::thread::lifecycle::spawn_unchecked<test[980ffaebd391d06d]::run_test::{closure#1}, ()>::{closure#1} as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   40:     0x62f760a8ebcf - <alloc[55a36b64bcbf2c0d]::boxed::Box<dyn core[6883ba1bc0fe4ed1]::ops::function::FnOnce<(), Output = ()> + core[6883ba1bc0fe4ed1]::marker::Send> as core[6883ba1bc0fe4ed1]::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/alloc/src/boxed.rs:2319:9
[INFO] [stdout]   41:     0x62f760a8ebcf - <std[73adb7dc35730857]::sys::thread::unix::Thread>::new::thread_start
[INFO] [stdout]                                at /rustc/6b3fa26749ab40e159b2dd5cf577acaaf5902772/library/std/src/sys/thread/unix.rs:123:17
[INFO] [stdout]   42:     0x795e445eeaa4 - <unknown>
[INFO] [stdout]   43:     0x795e4467ba64 - clone
[INFO] [stdout]   44:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- unconfirmed_launch_removal_never_claims_verified_cleanup stdout ----
[INFO] [stdout] Error: "stdout is not JSON (EOF while parsing a value at line 1 column 0); stdout: ; stderr: error[E1004]: invalid configuration: network resolution failed: no common routable communication interface across machines [local, remote]; verify that every machine has a common global IPv4 interface and that RDMA links are up\n"
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] failures:
[INFO] [stdout]     confirmed_removal_with_failed_process_cleanup_is_not_verified
[INFO] [stdout]     docker_exit_removal_reason_is_distinct_from_the_deadline
[INFO] [stdout]     external_image_missing_on_a_machine_rejects_naming_the_pull
[INFO] [stdout]     external_two_machine_serving_resolves_per_machine_facts
[INFO] [stdout]     hung_remote_removal_expires_with_deadline_evidence
[INFO] [stdout]     swallowed_ssh_handle_removes_the_created_container
[INFO] [stdout]     unconfirmed_launch_removal_never_claims_verified_cleanup
[INFO] [stdout] 
[INFO] [stdout] test result: FAILED. 28 passed; 7 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.18s
[INFO] [stdout] 
[INFO] [stderr] error: test failed, to rerun pass `-p inferlab --test image`
[INFO] running `Command { std: "docker" "inspect" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27", kill_on_drop: false }`
[INFO] [stdout] 9689d965e6dd4677ed71f84cd39de5de71196a6a88719fef2d983322c7994b27
