[INFO] cloning repository https://github.com/llm-d-incubation/hermes [INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/llm-d-incubation/hermes" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fllm-d-incubation%2Fhermes", kill_on_drop: false }` [INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fllm-d-incubation%2Fhermes'... [INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }` [INFO] [stdout] 9937cdee66cc904174409844eed37558be1f34ff [INFO] testing llm-d-incubation/hermes against master#d98a5da813da67eb189387b8ccfb73cf481275d8+rustflags=-Copt-level=3 for pr-138759-retry [INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fllm-d-incubation%2Fhermes" "/workspace/builds/worker-4-tc1/source", kill_on_drop: false }` [INFO] [stderr] Cloning into '/workspace/builds/worker-4-tc1/source'... [INFO] [stderr] done. [INFO] started tweaking git repo https://github.com/llm-d-incubation/hermes [INFO] finished tweaking git repo https://github.com/llm-d-incubation/hermes [INFO] tweaked toml for git repo https://github.com/llm-d-incubation/hermes written to /workspace/builds/worker-4-tc1/source/Cargo.toml [INFO] validating manifest of git repo https://github.com/llm-d-incubation/hermes on toolchain d98a5da813da67eb189387b8ccfb73cf481275d8 [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }` [INFO] crate git repo https://github.com/llm-d-incubation/hermes already has a lockfile, it will not be regenerated [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }` [INFO] [stderr] Updating crates.io index [INFO] [stderr] Downloading crates ... [INFO] [stderr] Downloaded wasi v0.14.7+wasi-0.2.4 [INFO] [stderr] Downloaded wasm-bindgen-shared v0.2.104 [INFO] [stderr] Downloaded wasm-bindgen v0.2.104 [INFO] [stderr] Downloaded wasm-bindgen-macro v0.2.104 [INFO] [stderr] Downloaded csv-core v0.1.12 [INFO] [stderr] Downloaded wit-bindgen v0.46.0 [INFO] [stderr] Downloaded windows-targets v0.53.4 [INFO] [stderr] Downloaded libredox v0.1.10 [INFO] [stderr] Downloaded pest_derive v2.8.2 [INFO] [stderr] Downloaded schannel v0.1.28 [INFO] [stderr] Downloaded pest_generator v2.8.2 [INFO] [stderr] Downloaded windows-core v0.62.1 [INFO] [stderr] Downloaded anstyle v1.0.12 [INFO] [stderr] Downloaded anyhow v1.0.100 [INFO] [stderr] Downloaded kube-core v2.0.1 [INFO] [stderr] Downloaded insta v1.43.2 [INFO] [stderr] Downloaded security-framework v3.5.1 [INFO] [stderr] Downloaded pest v2.8.2 [INFO] [stderr] Downloaded wasip2 v1.0.1+wasi-0.2.4 [INFO] [stderr] Downloaded minijinja v2.12.0 [INFO] [stderr] Downloaded clap_builder v4.5.48 [INFO] [stderr] Downloaded regex v1.11.3 [INFO] [stderr] Downloaded hyper-util v0.1.17 [INFO] [stderr] Downloaded chrono v0.4.42 [INFO] [stderr] Downloaded kube-client v2.0.1 [INFO] [stderr] Downloaded cc v1.2.39 [INFO] [stderr] Downloaded gimli v0.32.3 [INFO] [stderr] Downloaded object v0.37.3 [INFO] [stderr] Downloaded clap v4.5.48 [INFO] [stderr] Downloaded ucd-trie v0.1.7 [INFO] [stderr] Downloaded rustls v0.23.32 [INFO] [stderr] Downloaded jsonpath-rust v0.7.5 [INFO] [stderr] Downloaded rustls-webpki v0.103.6 [INFO] [stderr] Downloaded thiserror v2.0.17 [INFO] [stderr] Downloaded addr2line v0.25.1 [INFO] [stderr] Downloaded backtrace v0.3.76 [INFO] [stderr] Downloaded nu-ansi-term v0.50.1 [INFO] [stderr] Downloaded clap_derive v4.5.47 [INFO] [stderr] Downloaded console v0.15.11 [INFO] [stderr] Downloaded rustls-native-certs v0.8.1 [INFO] [stderr] Downloaded tokio-rustls v0.26.4 [INFO] [stderr] Downloaded pest_meta v2.8.2 [INFO] [stderr] Downloaded find-msvc-tools v0.1.2 [INFO] [stderr] Downloaded regex-automata v0.4.11 [INFO] [stderr] Downloaded iana-time-zone v0.1.64 [INFO] [stderr] Downloaded thiserror-impl v2.0.17 [INFO] [stderr] Downloaded kube-derive v2.0.1 [INFO] [stderr] Downloaded kube v2.0.1 [INFO] [stderr] Downloaded owo-colors v4.2.3 [INFO] [stderr] Downloaded ref-cast-impl v1.0.25 [INFO] [stderr] Downloaded pem v3.0.5 [INFO] [stderr] Downloaded libc v0.2.176 [INFO] [stderr] Downloaded secrecy v0.10.3 [INFO] [stderr] Downloaded windows-interface v0.59.2 [INFO] [stderr] Downloaded is-terminal v0.4.16 [INFO] [stderr] Downloaded ref-cast v1.0.25 [INFO] [stderr] Downloaded security-framework-sys v2.15.0 [INFO] [stderr] Downloaded windows-implement v0.60.1 [INFO] [stderr] Downloaded windows-strings v0.5.0 [INFO] [stderr] Downloaded js-sys v0.3.81 [INFO] [stderr] Downloaded csv v1.3.1 [INFO] [stderr] Downloaded windows-result v0.4.0 [INFO] [stderr] Downloaded serde-value v0.7.0 [INFO] [stderr] Downloaded wasm-bindgen-backend v0.2.104 [INFO] [stderr] Downloaded wasm-bindgen-macro-support v0.2.104 [INFO] [stderr] Downloaded windows-sys v0.61.1 [INFO] [stderr] Downloaded k8s-openapi v0.26.0 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }` [INFO] [stdout] 366053546c278014db8b0e4391c75548d0b2eab964b737a75b8a80b8adcadb6e [INFO] running `Command { std: "docker" "start" "-a" "366053546c278014db8b0e4391c75548d0b2eab964b737a75b8a80b8adcadb6e", kill_on_drop: false }` [INFO] running `Command { std: "docker" "inspect" "366053546c278014db8b0e4391c75548d0b2eab964b737a75b8a80b8adcadb6e", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "366053546c278014db8b0e4391c75548d0b2eab964b737a75b8a80b8adcadb6e", kill_on_drop: false }` [INFO] [stdout] 366053546c278014db8b0e4391c75548d0b2eab964b737a75b8a80b8adcadb6e [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid -Copt-level=3" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "build" "--frozen" "--message-format=json", kill_on_drop: false }` [INFO] [stdout] d3f452bc6120e25a11fe36e19975bcab0aab05d0cfacb3b1e99641c8c62046f1 [INFO] running `Command { std: "docker" "start" "-a" "d3f452bc6120e25a11fe36e19975bcab0aab05d0cfacb3b1e99641c8c62046f1", kill_on_drop: false }` [INFO] [stderr] Compiling unicode-ident v1.0.19 [INFO] [stderr] Compiling quote v1.0.41 [INFO] [stderr] Compiling libc v0.2.176 [INFO] [stderr] Compiling memchr v2.7.6 [INFO] [stderr] Compiling serde_core v1.0.228 [INFO] [stderr] Compiling serde v1.0.228 [INFO] [stderr] Compiling log v0.4.28 [INFO] [stderr] Compiling find-msvc-tools v0.1.2 [INFO] [stderr] Compiling proc-macro2 v1.0.101 [INFO] [stderr] Compiling thiserror v2.0.17 [INFO] [stderr] Compiling cc v1.2.39 [INFO] [stderr] Compiling zeroize v1.8.1 [INFO] [stderr] Compiling num-traits v0.2.19 [INFO] [stderr] Compiling serde_json v1.0.145 [INFO] [stderr] Compiling rustls-pki-types v1.12.0 [INFO] [stderr] Compiling syn v2.0.106 [INFO] [stderr] Compiling ucd-trie v0.1.7 [INFO] [stderr] Compiling rustls v0.23.32 [INFO] [stderr] Compiling ref-cast v1.0.25 [INFO] [stderr] Compiling parking_lot_core v0.9.11 [INFO] [stderr] Compiling socket2 v0.6.0 [INFO] [stderr] Compiling parking_lot v0.12.4 [INFO] [stderr] Compiling mio v1.0.4 [INFO] [stderr] Compiling signal-hook-registry v1.4.6 [INFO] [stderr] Compiling getrandom v0.2.16 [INFO] [stderr] Compiling k8s-openapi v0.26.0 [INFO] [stderr] Compiling aho-corasick v1.1.3 [INFO] [stderr] Compiling hashbrown v0.16.0 [INFO] [stderr] Compiling regex-syntax v0.8.6 [INFO] [stderr] Compiling iana-time-zone v0.1.64 [INFO] [stderr] Compiling indexmap v2.11.4 [INFO] [stderr] Compiling ordered-float v2.10.1 [INFO] [stderr] Compiling rustls-native-certs v0.8.1 [INFO] [stderr] Compiling serde_derive_internals v0.29.1 [INFO] [stderr] Compiling regex-automata v0.4.11 [INFO] [stderr] Compiling ring v0.17.14 [INFO] [stderr] Compiling darling_core v0.21.3 [INFO] [stderr] Compiling dirs-sys-next v0.1.2 [INFO] [stderr] Compiling unsafe-libyaml v0.2.11 [INFO] [stderr] Compiling regex v1.11.3 [INFO] [stderr] Compiling rustls-webpki v0.103.6 [INFO] [stderr] Compiling bitflags v2.9.4 [INFO] [stderr] Compiling anstyle v1.0.12 [INFO] [stderr] Compiling anstream v0.6.20 [INFO] [stderr] Compiling dirs-next v2.0.0 [INFO] [stderr] Compiling crypto-common v0.1.6 [INFO] [stderr] Compiling secrecy v0.10.3 [INFO] [stderr] Compiling csv-core v0.1.12 [INFO] [stderr] Compiling pem v3.0.5 [INFO] [stderr] Compiling owo-colors v4.2.3 [INFO] [stderr] Compiling anyhow v1.0.100 [INFO] [stderr] Compiling either v1.15.0 [INFO] [stderr] Compiling dirs-sys v0.4.1 [INFO] [stderr] Compiling clap_builder v4.5.48 [INFO] [stderr] Compiling digest v0.10.7 [INFO] [stderr] Compiling term v0.7.0 [INFO] [stderr] Compiling getrandom v0.3.3 [INFO] [stderr] Compiling is-terminal v0.4.16 [INFO] [stderr] Compiling tracing-log v0.2.0 [INFO] [stderr] Compiling hermes v0.2.5 (/opt/rustwide/workdir) [INFO] [stderr] Compiling encode_unicode v1.0.0 [INFO] [stderr] Compiling nu-ansi-term v0.50.1 [INFO] [stderr] Compiling unicode-width v0.1.14 [INFO] [stderr] Compiling tracing-subscriber v0.3.20 [INFO] [stderr] Compiling sha2 v0.10.9 [INFO] [stderr] Compiling uuid v1.18.1 [INFO] [stderr] Compiling dirs v5.0.1 [INFO] [stderr] Compiling serde_derive v1.0.228 [INFO] [stderr] Compiling tokio-macros v2.5.0 [INFO] [stderr] Compiling thiserror-impl v2.0.17 [INFO] [stderr] Compiling futures-macro v0.3.31 [INFO] [stderr] Compiling tracing-attributes v0.1.30 [INFO] [stderr] Compiling ref-cast-impl v1.0.25 [INFO] [stderr] Compiling tokio v1.47.1 [INFO] [stderr] Compiling derive_more-impl v2.0.1 [INFO] [stderr] Compiling futures-util v0.3.31 [INFO] [stderr] Compiling pest v2.8.2 [INFO] [stderr] Compiling schemars_derive v1.0.4 [INFO] [stderr] Compiling pest_meta v2.8.2 [INFO] [stderr] Compiling hyper v1.7.0 [INFO] [stderr] Compiling chrono v0.4.42 [INFO] [stderr] Compiling pest_generator v2.8.2 [INFO] [stderr] Compiling tracing v0.1.41 [INFO] [stderr] Compiling hyper-util v0.1.17 [INFO] [stderr] Compiling serde-value v0.7.0 [INFO] [stderr] Compiling derive_more v2.0.1 [INFO] [stderr] Compiling tokio-util v0.7.16 [INFO] [stderr] Compiling tokio-rustls v0.26.4 [INFO] [stderr] Compiling futures-executor v0.3.31 [INFO] [stderr] Compiling darling_macro v0.21.3 [INFO] [stderr] Compiling hyper-rustls v0.27.7 [INFO] [stderr] Compiling hyper-timeout v0.5.2 [INFO] [stderr] Compiling tower v0.5.2 [INFO] [stderr] Compiling futures v0.3.31 [INFO] [stderr] Compiling tower-http v0.6.6 [INFO] [stderr] Compiling serde_yaml v0.9.34+deprecated [INFO] [stderr] Compiling pest_derive v2.8.2 [INFO] [stderr] Compiling jsonpath-rust v0.7.5 [INFO] [stderr] Compiling schemars v1.0.4 [INFO] [stderr] Compiling darling v0.21.3 [INFO] [stderr] Compiling csv v1.3.1 [INFO] [stderr] Compiling clap_derive v4.5.47 [INFO] [stderr] Compiling kube-derive v2.0.1 [INFO] [stderr] Compiling prettytable-rs v0.10.0 [INFO] [stderr] Compiling minijinja v2.12.0 [INFO] [stderr] Compiling tokio-stream v0.1.17 [INFO] [stderr] Compiling kube-core v2.0.1 [INFO] [stderr] Compiling kube-client v2.0.1 [INFO] [stderr] Compiling clap v4.5.48 [INFO] [stderr] Compiling kube v2.0.1 [INFO] [stderr] Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 46s [INFO] running `Command { std: "docker" "inspect" "d3f452bc6120e25a11fe36e19975bcab0aab05d0cfacb3b1e99641c8c62046f1", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "d3f452bc6120e25a11fe36e19975bcab0aab05d0cfacb3b1e99641c8c62046f1", kill_on_drop: false }` [INFO] [stdout] d3f452bc6120e25a11fe36e19975bcab0aab05d0cfacb3b1e99641c8c62046f1 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid -Copt-level=3" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }` [INFO] [stdout] 07e8abf570968afea07c7a6e69789b88ba8248ceed25f4338d00d4ff8971e106 [INFO] running `Command { std: "docker" "start" "-a" "07e8abf570968afea07c7a6e69789b88ba8248ceed25f4338d00d4ff8971e106", kill_on_drop: false }` [INFO] [stderr] Compiling console v0.15.11 [INFO] [stderr] Compiling similar v2.7.0 [INFO] [stderr] Compiling insta v1.43.2 [INFO] [stderr] Compiling hermes v0.2.5 (/opt/rustwide/workdir) [INFO] [stderr] Finished `test` profile [unoptimized + debuginfo] target(s) in 20.28s [INFO] running `Command { std: "docker" "inspect" "07e8abf570968afea07c7a6e69789b88ba8248ceed25f4338d00d4ff8971e106", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "07e8abf570968afea07c7a6e69789b88ba8248ceed25f4338d00d4ff8971e106", kill_on_drop: false }` [INFO] [stdout] 07e8abf570968afea07c7a6e69789b88ba8248ceed25f4338d00d4ff8971e106 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid -Copt-level=3" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "test" "--frozen", kill_on_drop: false }` [INFO] [stdout] 669cc7210aa060e91f121443f08ffd8200dd9fe578a7195af0f9f52e076a50d6 [INFO] running `Command { std: "docker" "start" "-a" "669cc7210aa060e91f121443f08ffd8200dd9fe578a7195af0f9f52e076a50d6", kill_on_drop: false }` [INFO] [stderr] Finished `test` profile [unoptimized + debuginfo] target(s) in 0.20s [INFO] [stderr] Running unittests src/lib.rs (/opt/rustwide/target/debug/deps/hermes-0cef5cab3b34b0b9) [INFO] [stdout] [INFO] [stdout] running 5 tests [INFO] [stdout] test cache::tests::test_humanize_duration ... ok [INFO] [stdout] test topology_selector::tests::test_gke_selector_different_topologies ... FAILED [INFO] [stdout] test analyzer::tests::test_analyze_node_gke_with_rdma ... FAILED [INFO] [stdout] test topology_selector::tests::test_coreweave_selector_same_leafgroup ... FAILED [INFO] [stdout] test analyzer::tests::test_determine_topology_strategy_gke_with_fabric ... FAILED [INFO] [stdout] [INFO] [stdout] failures: [INFO] [stdout] [INFO] [stdout] ---- topology_selector::tests::test_gke_selector_different_topologies stdout ---- [INFO] [stdout] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [INFO] [stdout] Snapshot file: src/snapshots/hermes__topology_selector__tests__gke_selector_different_topologies.snap [INFO] [stdout] Snapshot: gke_selector_different_topologies [INFO] [stdout] Source: src/topology_selector.rs:301 [INFO] [stdout] ──────────────────────────────────────────────────────────────────────────────── [INFO] [stdout] Expression: reason [INFO] [stdout] ──────────────────────────────────────────────────────────────────────────────── [INFO] [stdout] +new results [INFO] [stdout] ────────────┬─────────────────────────────────────────────────────────────────── [INFO] [stdout] 1 │+GKE fallback: GKE RDMA RDMA nodes (topology may differ) [INFO] [stdout] ────────────┴─────────────────────────────────────────────────────────────────── [INFO] [stdout] [INFO] [stdout] thread 'topology_selector::tests::test_gke_selector_different_topologies' panicked at src/topology_selector.rs:301:9: [INFO] [stdout] called `Result::unwrap()` on an `Err` value: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" } [INFO] [stdout] stack backtrace: [INFO] [stdout] 0: 0x5869d8c43fd2 - std::backtrace_rs::backtrace::libunwind::trace::hd39b1f53d3cf9745 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9 [INFO] [stdout] 1: 0x5869d8c43fd2 - std::backtrace_rs::backtrace::trace_unsynchronized::he91d9a75d4e3972b [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14 [INFO] [stdout] 2: 0x5869d8c43fd2 - std::sys::backtrace::_print_fmt::hca46938f8c6e22cf [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:66:9 [INFO] [stdout] 3: 0x5869d8c43fd2 - ::fmt::ha499add612cccf8e [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:39:26 [INFO] [stdout] 4: 0x5869d8c6d233 - core::fmt::rt::Argument::fmt::hd21145b75a833b7a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/rt.rs:173:76 [INFO] [stdout] 5: 0x5869d8c6d233 - core::fmt::write::hb10c956f5235c8a4 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/mod.rs:1465:25 [INFO] [stdout] 6: 0x5869d8c40003 - std::io::default_write_fmt::hdb7615052be2ba4d [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:639:11 [INFO] [stdout] 7: 0x5869d8c40003 - std::io::Write::write_fmt::he1bcd251ec6e4153 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:1954:13 [INFO] [stdout] 8: 0x5869d8c43e22 - std::sys::backtrace::BacktraceLock::print::hb47c770ef659fd10 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:42:9 [INFO] [stdout] 9: 0x5869d8c4587c - std::panicking::default_hook::{{closure}}::hdda8afb9d457a22c [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:300:27 [INFO] [stdout] 10: 0x5869d8c456d2 - std::panicking::default_hook::h7c46b44874fe5c9a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:324:9 [INFO] [stdout] 11: 0x5869d8b6ec44 - as core::ops::function::Fn>::call::h178a5fcedee41e2f [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9 [INFO] [stdout] 12: 0x5869d8b6ec44 - test::test_main_with_exit_callback::{{closure}}::h951a41e0149d6d5d [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:145:21 [INFO] [stdout] 13: 0x5869d8c4625b - as core::ops::function::Fn>::call::h13602080f5b63276 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9 [INFO] [stdout] 14: 0x5869d8c4625b - std::panicking::rust_panic_with_hook::ha6cb99ed099eb1c5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:841:13 [INFO] [stdout] 15: 0x5869d8c4602a - std::panicking::begin_panic_handler::{{closure}}::he11808bc797ee921 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:706:13 [INFO] [stdout] 16: 0x5869d8c444c9 - std::sys::backtrace::__rust_end_short_backtrace::h9418807cb7346258 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:168:18 [INFO] [stdout] 17: 0x5869d8c45cbd - __rustc[18fb429eef004894]::rust_begin_unwind [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:697:5 [INFO] [stdout] 18: 0x5869d8c6b0e0 - core::panicking::panic_fmt::hd890aeb12c3a3fc3 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panicking.rs:75:14 [INFO] [stdout] 19: 0x5869d8c6b486 - core::result::unwrap_failed::h31c9775ec1a50121 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1761:5 [INFO] [stdout] 20: 0x5869d8b2c12b - core::result::Result::unwrap::h074a3ed8f442fb80 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1167:23 [INFO] [stdout] 21: 0x5869d8b2c12b - hermes::topology_selector::tests::test_gke_selector_different_topologies::h89a8115c74219aff [INFO] [stdout] at /opt/rustwide/workdir/src/topology_selector.rs:301:9 [INFO] [stdout] 22: 0x5869d8b2cd49 - hermes::topology_selector::tests::test_gke_selector_different_topologies::{{closure}}::hc357d4ae787d0dba [INFO] [stdout] at /opt/rustwide/workdir/src/topology_selector.rs:287:48 [INFO] [stdout] 23: 0x5869d8b2cd49 - core::ops::function::FnOnce::call_once::h561ca1e689e1760c [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 24: 0x5869d8b743cb - core::ops::function::FnOnce::call_once::h2869fb5b0a2b0bdc [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 25: 0x5869d8b743cb - test::__rust_begin_short_backtrace::h7dd7142bd62fa711 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:648:18 [INFO] [stdout] 26: 0x5869d8b735be - test::run_test_in_process::{{closure}}::h43a753f038d36b3f [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:74 [INFO] [stdout] 27: 0x5869d8b735be - as core::ops::function::FnOnce<()>>::call_once::h0ca95dd3e12d1e16 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9 [INFO] [stdout] 28: 0x5869d8b735be - std::panicking::catch_unwind::do_call::he308587d70ac34ba [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40 [INFO] [stdout] 29: 0x5869d8b735be - std::panicking::catch_unwind::h30dcba31973e8fb0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19 [INFO] [stdout] 30: 0x5869d8b735be - std::panic::catch_unwind::hfb68364e5621fbee [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14 [INFO] [stdout] 31: 0x5869d8b735be - test::run_test_in_process::hcdcc2977903b998a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:27 [INFO] [stdout] 32: 0x5869d8b735be - test::run_test::{{closure}}::h544a6550958c5d14 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:592:43 [INFO] [stdout] 33: 0x5869d8b37134 - test::run_test::{{closure}}::hb172e48ebe2b92c7 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:622:41 [INFO] [stdout] 34: 0x5869d8b37134 - std::sys::backtrace::__rust_begin_short_backtrace::h7e79d8706638bea0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:152:18 [INFO] [stdout] 35: 0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc4a275f1e71b8ab0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:559:17 [INFO] [stdout] 36: 0x5869d8b3ab0a - as core::ops::function::FnOnce<()>>::call_once::h01ed0d242df78cfd [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9 [INFO] [stdout] 37: 0x5869d8b3ab0a - std::panicking::catch_unwind::do_call::h052f373fb905fee0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40 [INFO] [stdout] 38: 0x5869d8b3ab0a - std::panicking::catch_unwind::hb425d20c8ffb09c8 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19 [INFO] [stdout] 39: 0x5869d8b3ab0a - std::panic::catch_unwind::ha0d5dfbf18fdeda5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14 [INFO] [stdout] 40: 0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::hbe79182bd37949c7 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:557:30 [INFO] [stdout] 41: 0x5869d8b3ab0a - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc096c4a06972fde5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 42: 0x5869d8c49497 - as core::ops::function::FnOnce>::call_once::h47377e27fb938a26 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9 [INFO] [stdout] 43: 0x5869d8c49497 - as core::ops::function::FnOnce>::call_once::h72f1fe5d095abf57 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9 [INFO] [stdout] 44: 0x5869d8c49497 - std::sys::pal::unix::thread::Thread::new::thread_start::h0a4d8e1b9c0d38cf [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/pal/unix/thread.rs:97:17 [INFO] [stdout] 45: 0x7b805ffc1aa4 - [INFO] [stdout] 46: 0x7b806004ea34 - clone [INFO] [stdout] 47: 0x0 - [INFO] [stdout] [INFO] [stdout] ---- analyzer::tests::test_analyze_node_gke_with_rdma stdout ---- [INFO] [stdout] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [INFO] [stdout] Snapshot file: src/snapshots/hermes__analyzer__tests__analyze_node_gke_with_rdma.snap [INFO] [stdout] Snapshot: analyze_node_gke_with_rdma [INFO] [stdout] Source: src/analyzer.rs:564 [INFO] [stdout] ──────────────────────────────────────────────────────────────────────────────── [INFO] [stdout] Expression: result [INFO] [stdout] ──────────────────────────────────────────────────────────────────────────────── [INFO] [stdout] +new results [INFO] [stdout] ────────────┬─────────────────────────────────────────────────────────────────── [INFO] [stdout] 1 │+name: test-node-1 [INFO] [stdout] 2 │+rdma_capability: Capable [INFO] [stdout] 3 │+rdma_type: GKE RDMA [INFO] [stdout] 4 │+rdma_resource: 1 RDMA interfaces [INFO] [stdout] 5 │+platform_type: GKE [INFO] [stdout] 6 │+topology_block: us-central1-a-gpu-pool [INFO] [stdout] 7 │+topology_detection: [INFO] [stdout] 8 │+ topology_type: Zone [INFO] [stdout] 9 │+ detection_method: GKE zone+nodepool topology [INFO] [stdout] 10 │+ confidence: Medium [INFO] [stdout] 11 │+ib_speed: ~ [INFO] [stdout] 12 │+ib_fabric: ~ [INFO] [stdout] 13 │+ib_ports: ~ [INFO] [stdout] 14 │+leafgroup: ~ [INFO] [stdout] 15 │+superpod: ~ [INFO] [stdout] 16 │+neighbors: [] [INFO] [stdout] 17 │+mellanox_nics: [] [INFO] [stdout] 18 │+node_labels: [INFO] [stdout] 19 │+ cloud.google.com/gke-nodepool: gpu-pool [INFO] [stdout] 20 │+ topology.gke.io/zone: us-central1-a [INFO] [stdout] 21 │+gpu_count: 8 [INFO] [stdout] 22 │+gpu_type: NVIDIA GPU [INFO] [stdout] 23 │+gpu_allocatable: 8 [INFO] [stdout] 24 │+gpu_allocated: ~ [INFO] [stdout] 25 │+cpu_allocatable: ~ [INFO] [stdout] 26 │+cpu_allocated: ~ [INFO] [stdout] 27 │+memory_allocatable: ~ [INFO] [stdout] 28 │+memory_allocated: ~ [INFO] [stdout] 29 │+platform_data: [INFO] [stdout] 30 │+ Gke: [INFO] [stdout] 31 │+ nodepool: gpu-pool [INFO] [stdout] 32 │+ machine_family: ~ [INFO] [stdout] 33 │+ zone: us-central1-a [INFO] [stdout] 34 │+ rdma_interfaces: [] [INFO] [stdout] 35 │+ pci_topology: ~ [INFO] [stdout] 36 │+ fabric_domain: ~ [INFO] [stdout] 37 │+ topology_block: ~ [INFO] [stdout] 38 │+ topology_subblock: ~ [INFO] [stdout] 39 │+ topology_host: ~ [INFO] [stdout] 40 │+image_cache_status: Unknown [INFO] [stdout] 41 │+image_cache_checked_at: ~ [INFO] [stdout] ────────────┴─────────────────────────────────────────────────────────────────── [INFO] [stdout] [INFO] [stdout] thread 'analyzer::tests::test_analyze_node_gke_with_rdma' panicked at src/analyzer.rs:564:9: [INFO] [stdout] called `Result::unwrap()` on an `Err` value: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" } [INFO] [stdout] stack backtrace: [INFO] [stdout] 0: 0x5869d8c43fd2 - std::backtrace_rs::backtrace::libunwind::trace::hd39b1f53d3cf9745 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9 [INFO] [stdout] 1: 0x5869d8c43fd2 - std::backtrace_rs::backtrace::trace_unsynchronized::he91d9a75d4e3972b [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14 [INFO] [stdout] 2: 0x5869d8c43fd2 - std::sys::backtrace::_print_fmt::hca46938f8c6e22cf [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:66:9 [INFO] [stdout] 3: 0x5869d8c43fd2 - ::fmt::ha499add612cccf8e [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:39:26 [INFO] [stdout] 4: 0x5869d8c6d233 - core::fmt::rt::Argument::fmt::hd21145b75a833b7a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/rt.rs:173:76 [INFO] [stdout] 5: 0x5869d8c6d233 - core::fmt::write::hb10c956f5235c8a4 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/mod.rs:1465:25 [INFO] [stdout] 6: 0x5869d8c40003 - std::io::default_write_fmt::hdb7615052be2ba4d [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:639:11 [INFO] [stdout] 7: 0x5869d8c40003 - std::io::Write::write_fmt::he1bcd251ec6e4153 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:1954:13 [INFO] [stdout] 8: 0x5869d8c43e22 - std::sys::backtrace::BacktraceLock::print::hb47c770ef659fd10 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:42:9 [INFO] [stdout] 9: 0x5869d8c4587c - std::panicking::default_hook::{{closure}}::hdda8afb9d457a22c [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:300:27 [INFO] [stdout] 10: 0x5869d8c456d2 - std::panicking::default_hook::h7c46b44874fe5c9a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:324:9 [INFO] [stdout] 11: 0x5869d8b6ec44 - as core::ops::function::Fn>::call::h178a5fcedee41e2f [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9 [INFO] [stdout] 12: 0x5869d8b6ec44 - test::test_main_with_exit_callback::{{closure}}::h951a41e0149d6d5d [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:145:21 [INFO] [stdout] 13: 0x5869d8c4625b - as core::ops::function::Fn>::call::h13602080f5b63276 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9 [INFO] [stdout] 14: 0x5869d8c4625b - std::panicking::rust_panic_with_hook::ha6cb99ed099eb1c5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:841:13 [INFO] [stdout] 15: 0x5869d8c4602a - std::panicking::begin_panic_handler::{{closure}}::he11808bc797ee921 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:706:13 [INFO] [stdout] 16: 0x5869d8c444c9 - std::sys::backtrace::__rust_end_short_backtrace::h9418807cb7346258 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:168:18 [INFO] [stdout] 17: 0x5869d8c45cbd - __rustc[18fb429eef004894]::rust_begin_unwind [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:697:5 [INFO] [stdout] 18: 0x5869d8c6b0e0 - core::panicking::panic_fmt::hd890aeb12c3a3fc3 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panicking.rs:75:14 [INFO] [stdout] 19: 0x5869d8c6b486 - core::result::unwrap_failed::h31c9775ec1a50121 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1761:5 [INFO] [stdout] 20: 0x5869d8b32e22 - core::result::Result::unwrap::h074a3ed8f442fb80 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1167:23 [INFO] [stdout] 21: 0x5869d8b32e22 - hermes::analyzer::tests::test_analyze_node_gke_with_rdma::h05d8ddf345ab797b [INFO] [stdout] at /opt/rustwide/workdir/src/analyzer.rs:564:9 [INFO] [stdout] 22: 0x5869d8b2cd69 - hermes::analyzer::tests::test_analyze_node_gke_with_rdma::{{closure}}::hab94dffc314ca28c [INFO] [stdout] at /opt/rustwide/workdir/src/analyzer.rs:559:41 [INFO] [stdout] 23: 0x5869d8b2cd69 - core::ops::function::FnOnce::call_once::h99281a2a8ab61265 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 24: 0x5869d8b743cb - core::ops::function::FnOnce::call_once::h2869fb5b0a2b0bdc [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 25: 0x5869d8b743cb - test::__rust_begin_short_backtrace::h7dd7142bd62fa711 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:648:18 [INFO] [stdout] 26: 0x5869d8b735be - test::run_test_in_process::{{closure}}::h43a753f038d36b3f [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:74 [INFO] [stdout] 27: 0x5869d8b735be - as core::ops::function::FnOnce<()>>::call_once::h0ca95dd3e12d1e16 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9 [INFO] [stdout] 28: 0x5869d8b735be - std::panicking::catch_unwind::do_call::he308587d70ac34ba [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40 [INFO] [stdout] 29: 0x5869d8b735be - std::panicking::catch_unwind::h30dcba31973e8fb0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19 [INFO] [stdout] 30: 0x5869d8b735be - std::panic::catch_unwind::hfb68364e5621fbee [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14 [INFO] [stdout] 31: 0x5869d8b735be - test::run_test_in_process::hcdcc2977903b998a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:27 [INFO] [stdout] 32: 0x5869d8b735be - test::run_test::{{closure}}::h544a6550958c5d14 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:592:43 [INFO] [stdout] 33: 0x5869d8b37134 - test::run_test::{{closure}}::hb172e48ebe2b92c7 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:622:41 [INFO] [stdout] 34: 0x5869d8b37134 - std::sys::backtrace::__rust_begin_short_backtrace::h7e79d8706638bea0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:152:18 [INFO] [stdout] 35: 0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc4a275f1e71b8ab0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:559:17 [INFO] [stdout] 36: 0x5869d8b3ab0a - as core::ops::function::FnOnce<()>>::call_once::h01ed0d242df78cfd [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9 [INFO] [stdout] 37: 0x5869d8b3ab0a - std::panicking::catch_unwind::do_call::h052f373fb905fee0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40 [INFO] [stdout] 38: 0x5869d8b3ab0a - std::panicking::catch_unwind::hb425d20c8ffb09c8 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19 [INFO] [stdout] 39: 0x5869d8b3ab0a - std::panic::catch_unwind::ha0d5dfbf18fdeda5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14 [INFO] [stdout] 40: 0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::hbe79182bd37949c7 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:557:30 [INFO] [stdout] 41: 0x5869d8b3ab0a - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc096c4a06972fde5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 42: 0x5869d8c49497 - as core::ops::function::FnOnce>::call_once::h47377e27fb938a26 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9 [INFO] [stdout] 43: 0x5869d8c49497 - as core::ops::function::FnOnce>::call_once::h72f1fe5d095abf57 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9 [INFO] [stdout] 44: 0x5869d8c49497 - std::sys::pal::unix::thread::Thread::new::thread_start::h0a4d8e1b9c0d38cf [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/pal/unix/thread.rs:97:17 [INFO] [stdout] 45: 0x7b805ffc1aa4 - [INFO] [stdout] 46: 0x7b806004ea34 - clone [INFO] [stdout] 47: 0x0 - [INFO] [stdout] [INFO] [stdout] ---- topology_selector::tests::test_coreweave_selector_same_leafgroup stdout ---- [INFO] [stdout] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [INFO] [stdout] Snapshot file: src/snapshots/hermes__topology_selector__tests__coreweave_selector_same_leafgroup.snap [INFO] [stdout] Snapshot: coreweave_selector_same_leafgroup [INFO] [stdout] Source: src/topology_selector.rs:283 [INFO] [stdout] ──────────────────────────────────────────────────────────────────────────────── [INFO] [stdout] Expression: reason [INFO] [stdout] ──────────────────────────────────────────────────────────────────────────────── [INFO] [stdout] +new results [INFO] [stdout] ────────────┬─────────────────────────────────────────────────────────────────── [INFO] [stdout] 1 │+Optimal CoreWeave same-topology: RoCE RDMA within leafgroup 'lg1' [INFO] [stdout] ────────────┴─────────────────────────────────────────────────────────────────── [INFO] [stdout] [INFO] [stdout] thread 'topology_selector::tests::test_coreweave_selector_same_leafgroup' panicked at src/topology_selector.rs:283:9: [INFO] [stdout] called `Result::unwrap()` on an `Err` value: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" } [INFO] [stdout] stack backtrace: [INFO] [stdout] 0: 0x5869d8c43fd2 - std::backtrace_rs::backtrace::libunwind::trace::hd39b1f53d3cf9745 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9 [INFO] [stdout] 1: 0x5869d8c43fd2 - std::backtrace_rs::backtrace::trace_unsynchronized::he91d9a75d4e3972b [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14 [INFO] [stdout] 2: 0x5869d8c43fd2 - std::sys::backtrace::_print_fmt::hca46938f8c6e22cf [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:66:9 [INFO] [stdout] 3: 0x5869d8c43fd2 - ::fmt::ha499add612cccf8e [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:39:26 [INFO] [stdout] 4: 0x5869d8c6d233 - core::fmt::rt::Argument::fmt::hd21145b75a833b7a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/rt.rs:173:76 [INFO] [stdout] 5: 0x5869d8c6d233 - core::fmt::write::hb10c956f5235c8a4 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/mod.rs:1465:25 [INFO] [stdout] 6: 0x5869d8c40003 - std::io::default_write_fmt::hdb7615052be2ba4d [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:639:11 [INFO] [stdout] 7: 0x5869d8c40003 - std::io::Write::write_fmt::he1bcd251ec6e4153 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:1954:13 [INFO] [stdout] 8: 0x5869d8c43e22 - std::sys::backtrace::BacktraceLock::print::hb47c770ef659fd10 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:42:9 [INFO] [stdout] 9: 0x5869d8c4587c - std::panicking::default_hook::{{closure}}::hdda8afb9d457a22c [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:300:27 [INFO] [stdout] 10: 0x5869d8c456d2 - std::panicking::default_hook::h7c46b44874fe5c9a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:324:9 [INFO] [stdout] 11: 0x5869d8b6ec44 - as core::ops::function::Fn>::call::h178a5fcedee41e2f [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9 [INFO] [stdout] 12: 0x5869d8b6ec44 - test::test_main_with_exit_callback::{{closure}}::h951a41e0149d6d5d [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:145:21 [INFO] [stdout] 13: 0x5869d8c4625b - as core::ops::function::Fn>::call::h13602080f5b63276 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9 [INFO] [stdout] 14: 0x5869d8c4625b - std::panicking::rust_panic_with_hook::ha6cb99ed099eb1c5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:841:13 [INFO] [stdout] 15: 0x5869d8c4602a - std::panicking::begin_panic_handler::{{closure}}::he11808bc797ee921 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:706:13 [INFO] [stdout] 16: 0x5869d8c444c9 - std::sys::backtrace::__rust_end_short_backtrace::h9418807cb7346258 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:168:18 [INFO] [stdout] 17: 0x5869d8c45cbd - __rustc[18fb429eef004894]::rust_begin_unwind [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:697:5 [INFO] [stdout] 18: 0x5869d8c6b0e0 - core::panicking::panic_fmt::hd890aeb12c3a3fc3 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panicking.rs:75:14 [INFO] [stdout] 19: 0x5869d8c6b486 - core::result::unwrap_failed::h31c9775ec1a50121 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1761:5 [INFO] [stdout] 20: 0x5869d8b2bc12 - core::result::Result::unwrap::h074a3ed8f442fb80 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1167:23 [INFO] [stdout] 21: 0x5869d8b2bc12 - hermes::topology_selector::tests::test_coreweave_selector_same_leafgroup::hd488f93a33cb186a [INFO] [stdout] at /opt/rustwide/workdir/src/topology_selector.rs:283:9 [INFO] [stdout] 22: 0x5869d8b2cd29 - hermes::topology_selector::tests::test_coreweave_selector_same_leafgroup::{{closure}}::h56d42425170cbd91 [INFO] [stdout] at /opt/rustwide/workdir/src/topology_selector.rs:266:48 [INFO] [stdout] 23: 0x5869d8b2cd29 - core::ops::function::FnOnce::call_once::h0e24a792ddd76543 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 24: 0x5869d8b743cb - core::ops::function::FnOnce::call_once::h2869fb5b0a2b0bdc [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 25: 0x5869d8b743cb - test::__rust_begin_short_backtrace::h7dd7142bd62fa711 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:648:18 [INFO] [stdout] 26: 0x5869d8b735be - test::run_test_in_process::{{closure}}::h43a753f038d36b3f [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:74 [INFO] [stdout] 27: 0x5869d8b735be - as core::ops::function::FnOnce<()>>::call_once::h0ca95dd3e12d1e16 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9 [INFO] [stdout] 28: 0x5869d8b735be - std::panicking::catch_unwind::do_call::he308587d70ac34ba [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40 [INFO] [stdout] 29: 0x5869d8b735be - std::panicking::catch_unwind::h30dcba31973e8fb0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19 [INFO] [stdout] 30: 0x5869d8b735be - std::panic::catch_unwind::hfb68364e5621fbee [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14 [INFO] [stdout] 31: 0x5869d8b735be - test::run_test_in_process::hcdcc2977903b998a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:27 [INFO] [stdout] 32: 0x5869d8b735be - test::run_test::{{closure}}::h544a6550958c5d14 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:592:43 [INFO] [stdout] 33: 0x5869d8b37134 - test::run_test::{{closure}}::hb172e48ebe2b92c7 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:622:41 [INFO] [stdout] 34: 0x5869d8b37134 - std::sys::backtrace::__rust_begin_short_backtrace::h7e79d8706638bea0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:152:18 [INFO] [stdout] 35: 0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc4a275f1e71b8ab0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:559:17 [INFO] [stdout] 36: 0x5869d8b3ab0a - as core::ops::function::FnOnce<()>>::call_once::h01ed0d242df78cfd [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9 [INFO] [stdout] 37: 0x5869d8b3ab0a - std::panicking::catch_unwind::do_call::h052f373fb905fee0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40 [INFO] [stdout] 38: 0x5869d8b3ab0a - std::panicking::catch_unwind::hb425d20c8ffb09c8 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19 [INFO] [stdout] 39: 0x5869d8b3ab0a - std::panic::catch_unwind::ha0d5dfbf18fdeda5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14 [INFO] [stdout] 40: 0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::hbe79182bd37949c7 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:557:30 [INFO] [stdout] 41: 0x5869d8b3ab0a - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc096c4a06972fde5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 42: 0x5869d8c49497 - as core::ops::function::FnOnce>::call_once::h47377e27fb938a26 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9 [INFO] [stdout] 43: 0x5869d8c49497 - as core::ops::function::FnOnce>::call_once::h72f1fe5d095abf57 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9 [INFO] [stdout] 44: 0x5869d8c49497 - std::sys::pal::unix::thread::Thread::new::thread_start::h0a4d8e1b9c0d38cf [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/pal/unix/thread.rs:97:17 [INFO] [stdout] 45: 0x7b805ffc1aa4 - [INFO] [stdout] 46: 0x7b806004ea34 - clone [INFO] [stdout] 47: 0x0 - [INFO] [stdout] [INFO] [stdout] ---- analyzer::tests::test_determine_topology_strategy_gke_with_fabric stdout ---- [INFO] [stdout] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [INFO] [stdout] Snapshot file: src/snapshots/hermes__analyzer__tests__determine_topology_strategy_gke_with_fabric.snap [INFO] [stdout] Snapshot: determine_topology_strategy_gke_with_fabric [INFO] [stdout] Source: src/analyzer.rs:575 [INFO] [stdout] ──────────────────────────────────────────────────────────────────────────────── [INFO] [stdout] Expression: result [INFO] [stdout] ──────────────────────────────────────────────────────────────────────────────── [INFO] [stdout] +new results [INFO] [stdout] ────────────┬─────────────────────────────────────────────────────────────────── [INFO] [stdout] 1 │+topology_type: Hardware [INFO] [stdout] 2 │+detection_method: GKE RDMA fabric domain analysis [INFO] [stdout] 3 │+confidence: High [INFO] [stdout] ────────────┴─────────────────────────────────────────────────────────────────── [INFO] [stdout] [INFO] [stdout] thread 'analyzer::tests::test_determine_topology_strategy_gke_with_fabric' panicked at src/analyzer.rs:575:9: [INFO] [stdout] called `Result::unwrap()` on an `Err` value: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" } [INFO] [stdout] stack backtrace: [INFO] [stdout] 0: 0x5869d8c43fd2 - std::backtrace_rs::backtrace::libunwind::trace::hd39b1f53d3cf9745 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9 [INFO] [stdout] 1: 0x5869d8c43fd2 - std::backtrace_rs::backtrace::trace_unsynchronized::he91d9a75d4e3972b [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14 [INFO] [stdout] 2: 0x5869d8c43fd2 - std::sys::backtrace::_print_fmt::hca46938f8c6e22cf [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:66:9 [INFO] [stdout] 3: 0x5869d8c43fd2 - ::fmt::ha499add612cccf8e [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:39:26 [INFO] [stdout] 4: 0x5869d8c6d233 - core::fmt::rt::Argument::fmt::hd21145b75a833b7a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/rt.rs:173:76 [INFO] [stdout] 5: 0x5869d8c6d233 - core::fmt::write::hb10c956f5235c8a4 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/mod.rs:1465:25 [INFO] [stdout] 6: 0x5869d8c40003 - std::io::default_write_fmt::hdb7615052be2ba4d [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:639:11 [INFO] [stdout] 7: 0x5869d8c40003 - std::io::Write::write_fmt::he1bcd251ec6e4153 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:1954:13 [INFO] [stdout] 8: 0x5869d8c43e22 - std::sys::backtrace::BacktraceLock::print::hb47c770ef659fd10 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:42:9 [INFO] [stdout] 9: 0x5869d8c4587c - std::panicking::default_hook::{{closure}}::hdda8afb9d457a22c [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:300:27 [INFO] [stdout] 10: 0x5869d8c456d2 - std::panicking::default_hook::h7c46b44874fe5c9a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:324:9 [INFO] [stdout] 11: 0x5869d8b6ec44 - as core::ops::function::Fn>::call::h178a5fcedee41e2f [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9 [INFO] [stdout] 12: 0x5869d8b6ec44 - test::test_main_with_exit_callback::{{closure}}::h951a41e0149d6d5d [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:145:21 [INFO] [stdout] 13: 0x5869d8c4625b - as core::ops::function::Fn>::call::h13602080f5b63276 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9 [INFO] [stdout] 14: 0x5869d8c4625b - std::panicking::rust_panic_with_hook::ha6cb99ed099eb1c5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:841:13 [INFO] [stdout] 15: 0x5869d8c4602a - std::panicking::begin_panic_handler::{{closure}}::he11808bc797ee921 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:706:13 [INFO] [stdout] 16: 0x5869d8c444c9 - std::sys::backtrace::__rust_end_short_backtrace::h9418807cb7346258 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:168:18 [INFO] [stdout] 17: 0x5869d8c45cbd - __rustc[18fb429eef004894]::rust_begin_unwind [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:697:5 [INFO] [stdout] 18: 0x5869d8c6b0e0 - core::panicking::panic_fmt::hd890aeb12c3a3fc3 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panicking.rs:75:14 [INFO] [stdout] 19: 0x5869d8c6b486 - core::result::unwrap_failed::h31c9775ec1a50121 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1761:5 [INFO] [stdout] 20: 0x5869d8b333c6 - core::result::Result::unwrap::h074a3ed8f442fb80 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1167:23 [INFO] [stdout] 21: 0x5869d8b333c6 - hermes::analyzer::tests::test_determine_topology_strategy_gke_with_fabric::h68942f48812a90bc [INFO] [stdout] at /opt/rustwide/workdir/src/analyzer.rs:575:9 [INFO] [stdout] 22: 0x5869d8b2cfa9 - hermes::analyzer::tests::test_determine_topology_strategy_gke_with_fabric::{{closure}}::h84cafd842465e6ae [INFO] [stdout] at /opt/rustwide/workdir/src/analyzer.rs:570:58 [INFO] [stdout] 23: 0x5869d8b2cfa9 - core::ops::function::FnOnce::call_once::hf3836bf5eab9cf69 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 24: 0x5869d8b743cb - core::ops::function::FnOnce::call_once::h2869fb5b0a2b0bdc [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 25: 0x5869d8b743cb - test::__rust_begin_short_backtrace::h7dd7142bd62fa711 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:648:18 [INFO] [stdout] 26: 0x5869d8b735be - test::run_test_in_process::{{closure}}::h43a753f038d36b3f [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:74 [INFO] [stdout] 27: 0x5869d8b735be - as core::ops::function::FnOnce<()>>::call_once::h0ca95dd3e12d1e16 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9 [INFO] [stdout] 28: 0x5869d8b735be - std::panicking::catch_unwind::do_call::he308587d70ac34ba [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40 [INFO] [stdout] 29: 0x5869d8b735be - std::panicking::catch_unwind::h30dcba31973e8fb0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19 [INFO] [stdout] 30: 0x5869d8b735be - std::panic::catch_unwind::hfb68364e5621fbee [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14 [INFO] [stdout] 31: 0x5869d8b735be - test::run_test_in_process::hcdcc2977903b998a [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:27 [INFO] [stdout] 32: 0x5869d8b735be - test::run_test::{{closure}}::h544a6550958c5d14 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:592:43 [INFO] [stdout] 33: 0x5869d8b37134 - test::run_test::{{closure}}::hb172e48ebe2b92c7 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:622:41 [INFO] [stdout] 34: 0x5869d8b37134 - std::sys::backtrace::__rust_begin_short_backtrace::h7e79d8706638bea0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:152:18 [INFO] [stdout] 35: 0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc4a275f1e71b8ab0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:559:17 [INFO] [stdout] 36: 0x5869d8b3ab0a - as core::ops::function::FnOnce<()>>::call_once::h01ed0d242df78cfd [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9 [INFO] [stdout] 37: 0x5869d8b3ab0a - std::panicking::catch_unwind::do_call::h052f373fb905fee0 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40 [INFO] [stdout] 38: 0x5869d8b3ab0a - std::panicking::catch_unwind::hb425d20c8ffb09c8 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19 [INFO] [stdout] 39: 0x5869d8b3ab0a - std::panic::catch_unwind::ha0d5dfbf18fdeda5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14 [INFO] [stdout] 40: 0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::hbe79182bd37949c7 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:557:30 [INFO] [stdout] 41: 0x5869d8b3ab0a - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc096c4a06972fde5 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5 [INFO] [stdout] 42: 0x5869d8c49497 - as core::ops::function::FnOnce>::call_once::h47377e27fb938a26 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9 [INFO] [stdout] 43: 0x5869d8c49497 - as core::ops::function::FnOnce>::call_once::h72f1fe5d095abf57 [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9 [INFO] [stdout] 44: 0x5869d8c49497 - std::sys::pal::unix::thread::Thread::new::thread_start::h0a4d8e1b9c0d38cf [INFO] [stdout] at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/pal/unix/thread.rs:97:17 [INFO] [stdout] 45: 0x7b805ffc1aa4 - [INFO] [stdout] 46: 0x7b806004ea34 - clone [INFO] [stdout] 47: 0x0 - [INFO] [stdout] [INFO] [stdout] [INFO] [stdout] failures: [INFO] [stdout] analyzer::tests::test_analyze_node_gke_with_rdma [INFO] [stdout] analyzer::tests::test_determine_topology_strategy_gke_with_fabric [INFO] [stdout] topology_selector::tests::test_coreweave_selector_same_leafgroup [INFO] [stdout] topology_selector::tests::test_gke_selector_different_topologies [INFO] [stdout] [INFO] [stdout] test result: FAILED. 1 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.50s [INFO] [stdout] [INFO] [stderr] error: test failed, to rerun pass `--lib` [INFO] running `Command { std: "docker" "inspect" "669cc7210aa060e91f121443f08ffd8200dd9fe578a7195af0f9f52e076a50d6", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "669cc7210aa060e91f121443f08ffd8200dd9fe578a7195af0f9f52e076a50d6", kill_on_drop: false }` [INFO] [stdout] 669cc7210aa060e91f121443f08ffd8200dd9fe578a7195af0f9f52e076a50d6