[INFO] cloning repository https://github.com/llm-d-incubation/hermes
[INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/llm-d-incubation/hermes" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fllm-d-incubation%2Fhermes", kill_on_drop: false }`
[INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fllm-d-incubation%2Fhermes'...
[INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }`
[INFO] [stdout] 9937cdee66cc904174409844eed37558be1f34ff
[INFO] testing llm-d-incubation/hermes against master#d98a5da813da67eb189387b8ccfb73cf481275d8+rustflags=-Copt-level=3 for pr-138759-retry
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fllm-d-incubation%2Fhermes" "/workspace/builds/worker-4-tc1/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-4-tc1/source'...
[INFO] [stderr] done.
[INFO] started tweaking git repo https://github.com/llm-d-incubation/hermes
[INFO] finished tweaking git repo https://github.com/llm-d-incubation/hermes
[INFO] tweaked toml for git repo https://github.com/llm-d-incubation/hermes written to /workspace/builds/worker-4-tc1/source/Cargo.toml
[INFO] validating manifest of git repo https://github.com/llm-d-incubation/hermes on toolchain d98a5da813da67eb189387b8ccfb73cf481275d8
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate git repo https://github.com/llm-d-incubation/hermes already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Updating crates.io index
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded wasi v0.14.7+wasi-0.2.4
[INFO] [stderr]   Downloaded wasm-bindgen-shared v0.2.104
[INFO] [stderr]   Downloaded wasm-bindgen v0.2.104
[INFO] [stderr]   Downloaded wasm-bindgen-macro v0.2.104
[INFO] [stderr]   Downloaded csv-core v0.1.12
[INFO] [stderr]   Downloaded wit-bindgen v0.46.0
[INFO] [stderr]   Downloaded windows-targets v0.53.4
[INFO] [stderr]   Downloaded libredox v0.1.10
[INFO] [stderr]   Downloaded pest_derive v2.8.2
[INFO] [stderr]   Downloaded schannel v0.1.28
[INFO] [stderr]   Downloaded pest_generator v2.8.2
[INFO] [stderr]   Downloaded windows-core v0.62.1
[INFO] [stderr]   Downloaded anstyle v1.0.12
[INFO] [stderr]   Downloaded anyhow v1.0.100
[INFO] [stderr]   Downloaded kube-core v2.0.1
[INFO] [stderr]   Downloaded insta v1.43.2
[INFO] [stderr]   Downloaded security-framework v3.5.1
[INFO] [stderr]   Downloaded pest v2.8.2
[INFO] [stderr]   Downloaded wasip2 v1.0.1+wasi-0.2.4
[INFO] [stderr]   Downloaded minijinja v2.12.0
[INFO] [stderr]   Downloaded clap_builder v4.5.48
[INFO] [stderr]   Downloaded regex v1.11.3
[INFO] [stderr]   Downloaded hyper-util v0.1.17
[INFO] [stderr]   Downloaded chrono v0.4.42
[INFO] [stderr]   Downloaded kube-client v2.0.1
[INFO] [stderr]   Downloaded cc v1.2.39
[INFO] [stderr]   Downloaded gimli v0.32.3
[INFO] [stderr]   Downloaded object v0.37.3
[INFO] [stderr]   Downloaded clap v4.5.48
[INFO] [stderr]   Downloaded ucd-trie v0.1.7
[INFO] [stderr]   Downloaded rustls v0.23.32
[INFO] [stderr]   Downloaded jsonpath-rust v0.7.5
[INFO] [stderr]   Downloaded rustls-webpki v0.103.6
[INFO] [stderr]   Downloaded thiserror v2.0.17
[INFO] [stderr]   Downloaded addr2line v0.25.1
[INFO] [stderr]   Downloaded backtrace v0.3.76
[INFO] [stderr]   Downloaded nu-ansi-term v0.50.1
[INFO] [stderr]   Downloaded clap_derive v4.5.47
[INFO] [stderr]   Downloaded console v0.15.11
[INFO] [stderr]   Downloaded rustls-native-certs v0.8.1
[INFO] [stderr]   Downloaded tokio-rustls v0.26.4
[INFO] [stderr]   Downloaded pest_meta v2.8.2
[INFO] [stderr]   Downloaded find-msvc-tools v0.1.2
[INFO] [stderr]   Downloaded regex-automata v0.4.11
[INFO] [stderr]   Downloaded iana-time-zone v0.1.64
[INFO] [stderr]   Downloaded thiserror-impl v2.0.17
[INFO] [stderr]   Downloaded kube-derive v2.0.1
[INFO] [stderr]   Downloaded kube v2.0.1
[INFO] [stderr]   Downloaded owo-colors v4.2.3
[INFO] [stderr]   Downloaded ref-cast-impl v1.0.25
[INFO] [stderr]   Downloaded pem v3.0.5
[INFO] [stderr]   Downloaded libc v0.2.176
[INFO] [stderr]   Downloaded secrecy v0.10.3
[INFO] [stderr]   Downloaded windows-interface v0.59.2
[INFO] [stderr]   Downloaded is-terminal v0.4.16
[INFO] [stderr]   Downloaded ref-cast v1.0.25
[INFO] [stderr]   Downloaded security-framework-sys v2.15.0
[INFO] [stderr]   Downloaded windows-implement v0.60.1
[INFO] [stderr]   Downloaded windows-strings v0.5.0
[INFO] [stderr]   Downloaded js-sys v0.3.81
[INFO] [stderr]   Downloaded csv v1.3.1
[INFO] [stderr]   Downloaded windows-result v0.4.0
[INFO] [stderr]   Downloaded serde-value v0.7.0
[INFO] [stderr]   Downloaded wasm-bindgen-backend v0.2.104
[INFO] [stderr]   Downloaded wasm-bindgen-macro-support v0.2.104
[INFO] [stderr]   Downloaded windows-sys v0.61.1
[INFO] [stderr]   Downloaded k8s-openapi v0.26.0
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] [stdout] 366053546c278014db8b0e4391c75548d0b2eab964b737a75b8a80b8adcadb6e
[INFO] running `Command { std: "docker" "start" "-a" "366053546c278014db8b0e4391c75548d0b2eab964b737a75b8a80b8adcadb6e", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "366053546c278014db8b0e4391c75548d0b2eab964b737a75b8a80b8adcadb6e", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "366053546c278014db8b0e4391c75548d0b2eab964b737a75b8a80b8adcadb6e", kill_on_drop: false }`
[INFO] [stdout] 366053546c278014db8b0e4391c75548d0b2eab964b737a75b8a80b8adcadb6e
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid -Copt-level=3" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "build" "--frozen" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] d3f452bc6120e25a11fe36e19975bcab0aab05d0cfacb3b1e99641c8c62046f1
[INFO] running `Command { std: "docker" "start" "-a" "d3f452bc6120e25a11fe36e19975bcab0aab05d0cfacb3b1e99641c8c62046f1", kill_on_drop: false }`
[INFO] [stderr]    Compiling unicode-ident v1.0.19
[INFO] [stderr]    Compiling quote v1.0.41
[INFO] [stderr]    Compiling libc v0.2.176
[INFO] [stderr]    Compiling memchr v2.7.6
[INFO] [stderr]    Compiling serde_core v1.0.228
[INFO] [stderr]    Compiling serde v1.0.228
[INFO] [stderr]    Compiling log v0.4.28
[INFO] [stderr]    Compiling find-msvc-tools v0.1.2
[INFO] [stderr]    Compiling proc-macro2 v1.0.101
[INFO] [stderr]    Compiling thiserror v2.0.17
[INFO] [stderr]    Compiling cc v1.2.39
[INFO] [stderr]    Compiling zeroize v1.8.1
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]    Compiling serde_json v1.0.145
[INFO] [stderr]    Compiling rustls-pki-types v1.12.0
[INFO] [stderr]    Compiling syn v2.0.106
[INFO] [stderr]    Compiling ucd-trie v0.1.7
[INFO] [stderr]    Compiling rustls v0.23.32
[INFO] [stderr]    Compiling ref-cast v1.0.25
[INFO] [stderr]    Compiling parking_lot_core v0.9.11
[INFO] [stderr]    Compiling socket2 v0.6.0
[INFO] [stderr]    Compiling parking_lot v0.12.4
[INFO] [stderr]    Compiling mio v1.0.4
[INFO] [stderr]    Compiling signal-hook-registry v1.4.6
[INFO] [stderr]    Compiling getrandom v0.2.16
[INFO] [stderr]    Compiling k8s-openapi v0.26.0
[INFO] [stderr]    Compiling aho-corasick v1.1.3
[INFO] [stderr]    Compiling hashbrown v0.16.0
[INFO] [stderr]    Compiling regex-syntax v0.8.6
[INFO] [stderr]    Compiling iana-time-zone v0.1.64
[INFO] [stderr]    Compiling indexmap v2.11.4
[INFO] [stderr]    Compiling ordered-float v2.10.1
[INFO] [stderr]    Compiling rustls-native-certs v0.8.1
[INFO] [stderr]    Compiling serde_derive_internals v0.29.1
[INFO] [stderr]    Compiling regex-automata v0.4.11
[INFO] [stderr]    Compiling ring v0.17.14
[INFO] [stderr]    Compiling darling_core v0.21.3
[INFO] [stderr]    Compiling dirs-sys-next v0.1.2
[INFO] [stderr]    Compiling unsafe-libyaml v0.2.11
[INFO] [stderr]    Compiling regex v1.11.3
[INFO] [stderr]    Compiling rustls-webpki v0.103.6
[INFO] [stderr]    Compiling bitflags v2.9.4
[INFO] [stderr]    Compiling anstyle v1.0.12
[INFO] [stderr]    Compiling anstream v0.6.20
[INFO] [stderr]    Compiling dirs-next v2.0.0
[INFO] [stderr]    Compiling crypto-common v0.1.6
[INFO] [stderr]    Compiling secrecy v0.10.3
[INFO] [stderr]    Compiling csv-core v0.1.12
[INFO] [stderr]    Compiling pem v3.0.5
[INFO] [stderr]    Compiling owo-colors v4.2.3
[INFO] [stderr]    Compiling anyhow v1.0.100
[INFO] [stderr]    Compiling either v1.15.0
[INFO] [stderr]    Compiling dirs-sys v0.4.1
[INFO] [stderr]    Compiling clap_builder v4.5.48
[INFO] [stderr]    Compiling digest v0.10.7
[INFO] [stderr]    Compiling term v0.7.0
[INFO] [stderr]    Compiling getrandom v0.3.3
[INFO] [stderr]    Compiling is-terminal v0.4.16
[INFO] [stderr]    Compiling tracing-log v0.2.0
[INFO] [stderr]    Compiling hermes v0.2.5 (/opt/rustwide/workdir)
[INFO] [stderr]    Compiling encode_unicode v1.0.0
[INFO] [stderr]    Compiling nu-ansi-term v0.50.1
[INFO] [stderr]    Compiling unicode-width v0.1.14
[INFO] [stderr]    Compiling tracing-subscriber v0.3.20
[INFO] [stderr]    Compiling sha2 v0.10.9
[INFO] [stderr]    Compiling uuid v1.18.1
[INFO] [stderr]    Compiling dirs v5.0.1
[INFO] [stderr]    Compiling serde_derive v1.0.228
[INFO] [stderr]    Compiling tokio-macros v2.5.0
[INFO] [stderr]    Compiling thiserror-impl v2.0.17
[INFO] [stderr]    Compiling futures-macro v0.3.31
[INFO] [stderr]    Compiling tracing-attributes v0.1.30
[INFO] [stderr]    Compiling ref-cast-impl v1.0.25
[INFO] [stderr]    Compiling tokio v1.47.1
[INFO] [stderr]    Compiling derive_more-impl v2.0.1
[INFO] [stderr]    Compiling futures-util v0.3.31
[INFO] [stderr]    Compiling pest v2.8.2
[INFO] [stderr]    Compiling schemars_derive v1.0.4
[INFO] [stderr]    Compiling pest_meta v2.8.2
[INFO] [stderr]    Compiling hyper v1.7.0
[INFO] [stderr]    Compiling chrono v0.4.42
[INFO] [stderr]    Compiling pest_generator v2.8.2
[INFO] [stderr]    Compiling tracing v0.1.41
[INFO] [stderr]    Compiling hyper-util v0.1.17
[INFO] [stderr]    Compiling serde-value v0.7.0
[INFO] [stderr]    Compiling derive_more v2.0.1
[INFO] [stderr]    Compiling tokio-util v0.7.16
[INFO] [stderr]    Compiling tokio-rustls v0.26.4
[INFO] [stderr]    Compiling futures-executor v0.3.31
[INFO] [stderr]    Compiling darling_macro v0.21.3
[INFO] [stderr]    Compiling hyper-rustls v0.27.7
[INFO] [stderr]    Compiling hyper-timeout v0.5.2
[INFO] [stderr]    Compiling tower v0.5.2
[INFO] [stderr]    Compiling futures v0.3.31
[INFO] [stderr]    Compiling tower-http v0.6.6
[INFO] [stderr]    Compiling serde_yaml v0.9.34+deprecated
[INFO] [stderr]    Compiling pest_derive v2.8.2
[INFO] [stderr]    Compiling jsonpath-rust v0.7.5
[INFO] [stderr]    Compiling schemars v1.0.4
[INFO] [stderr]    Compiling darling v0.21.3
[INFO] [stderr]    Compiling csv v1.3.1
[INFO] [stderr]    Compiling clap_derive v4.5.47
[INFO] [stderr]    Compiling kube-derive v2.0.1
[INFO] [stderr]    Compiling prettytable-rs v0.10.0
[INFO] [stderr]    Compiling minijinja v2.12.0
[INFO] [stderr]    Compiling tokio-stream v0.1.17
[INFO] [stderr]    Compiling kube-core v2.0.1
[INFO] [stderr]    Compiling kube-client v2.0.1
[INFO] [stderr]    Compiling clap v4.5.48
[INFO] [stderr]    Compiling kube v2.0.1
[INFO] [stderr]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 46s
[INFO] running `Command { std: "docker" "inspect" "d3f452bc6120e25a11fe36e19975bcab0aab05d0cfacb3b1e99641c8c62046f1", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "d3f452bc6120e25a11fe36e19975bcab0aab05d0cfacb3b1e99641c8c62046f1", kill_on_drop: false }`
[INFO] [stdout] d3f452bc6120e25a11fe36e19975bcab0aab05d0cfacb3b1e99641c8c62046f1
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid -Copt-level=3" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] 07e8abf570968afea07c7a6e69789b88ba8248ceed25f4338d00d4ff8971e106
[INFO] running `Command { std: "docker" "start" "-a" "07e8abf570968afea07c7a6e69789b88ba8248ceed25f4338d00d4ff8971e106", kill_on_drop: false }`
[INFO] [stderr]    Compiling console v0.15.11
[INFO] [stderr]    Compiling similar v2.7.0
[INFO] [stderr]    Compiling insta v1.43.2
[INFO] [stderr]    Compiling hermes v0.2.5 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 20.28s
[INFO] running `Command { std: "docker" "inspect" "07e8abf570968afea07c7a6e69789b88ba8248ceed25f4338d00d4ff8971e106", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "07e8abf570968afea07c7a6e69789b88ba8248ceed25f4338d00d4ff8971e106", kill_on_drop: false }`
[INFO] [stdout] 07e8abf570968afea07c7a6e69789b88ba8248ceed25f4338d00d4ff8971e106
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid -Copt-level=3" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+d98a5da813da67eb189387b8ccfb73cf481275d8" "test" "--frozen", kill_on_drop: false }`
[INFO] [stdout] 669cc7210aa060e91f121443f08ffd8200dd9fe578a7195af0f9f52e076a50d6
[INFO] running `Command { std: "docker" "start" "-a" "669cc7210aa060e91f121443f08ffd8200dd9fe578a7195af0f9f52e076a50d6", kill_on_drop: false }`
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 0.20s
[INFO] [stderr]      Running unittests src/lib.rs (/opt/rustwide/target/debug/deps/hermes-0cef5cab3b34b0b9)
[INFO] [stdout] 
[INFO] [stdout] running 5 tests
[INFO] [stdout] test cache::tests::test_humanize_duration ... ok
[INFO] [stdout] test topology_selector::tests::test_gke_selector_different_topologies ... FAILED
[INFO] [stdout] test analyzer::tests::test_analyze_node_gke_with_rdma ... FAILED
[INFO] [stdout] test topology_selector::tests::test_coreweave_selector_same_leafgroup ... FAILED
[INFO] [stdout] test analyzer::tests::test_determine_topology_strategy_gke_with_fabric ... FAILED
[INFO] [stdout] 
[INFO] [stdout] failures:
[INFO] [stdout] 
[INFO] [stdout] ---- topology_selector::tests::test_gke_selector_different_topologies stdout ----
[INFO] [stdout] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] [stdout] Snapshot file: src/snapshots/hermes__topology_selector__tests__gke_selector_different_topologies.snap
[INFO] [stdout] Snapshot: gke_selector_different_topologies
[INFO] [stdout] Source: src/topology_selector.rs:301
[INFO] [stdout] ────────────────────────────────────────────────────────────────────────────────
[INFO] [stdout] Expression: reason
[INFO] [stdout] ────────────────────────────────────────────────────────────────────────────────
[INFO] [stdout] +new results
[INFO] [stdout] ────────────┬───────────────────────────────────────────────────────────────────
[INFO] [stdout]           1 │+GKE fallback: GKE RDMA RDMA nodes (topology may differ)
[INFO] [stdout] ────────────┴───────────────────────────────────────────────────────────────────
[INFO] [stdout] 
[INFO] [stdout] thread 'topology_selector::tests::test_gke_selector_different_topologies' panicked at src/topology_selector.rs:301:9:
[INFO] [stdout] called `Result::unwrap()` on an `Err` value: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" }
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5869d8c43fd2 - std::backtrace_rs::backtrace::libunwind::trace::hd39b1f53d3cf9745
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5869d8c43fd2 - std::backtrace_rs::backtrace::trace_unsynchronized::he91d9a75d4e3972b
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5869d8c43fd2 - std::sys::backtrace::_print_fmt::hca46938f8c6e22cf
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:66:9
[INFO] [stdout]    3:     0x5869d8c43fd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::ha499add612cccf8e
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:39:26
[INFO] [stdout]    4:     0x5869d8c6d233 - core::fmt::rt::Argument::fmt::hd21145b75a833b7a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/rt.rs:173:76
[INFO] [stdout]    5:     0x5869d8c6d233 - core::fmt::write::hb10c956f5235c8a4
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/mod.rs:1465:25
[INFO] [stdout]    6:     0x5869d8c40003 - std::io::default_write_fmt::hdb7615052be2ba4d
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5869d8c40003 - std::io::Write::write_fmt::he1bcd251ec6e4153
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:1954:13
[INFO] [stdout]    8:     0x5869d8c43e22 - std::sys::backtrace::BacktraceLock::print::hb47c770ef659fd10
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:42:9
[INFO] [stdout]    9:     0x5869d8c4587c - std::panicking::default_hook::{{closure}}::hdda8afb9d457a22c
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:300:27
[INFO] [stdout]   10:     0x5869d8c456d2 - std::panicking::default_hook::h7c46b44874fe5c9a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:324:9
[INFO] [stdout]   11:     0x5869d8b6ec44 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h178a5fcedee41e2f
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   12:     0x5869d8b6ec44 - test::test_main_with_exit_callback::{{closure}}::h951a41e0149d6d5d
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5869d8c4625b - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h13602080f5b63276
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   14:     0x5869d8c4625b - std::panicking::rust_panic_with_hook::ha6cb99ed099eb1c5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:841:13
[INFO] [stdout]   15:     0x5869d8c4602a - std::panicking::begin_panic_handler::{{closure}}::he11808bc797ee921
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:706:13
[INFO] [stdout]   16:     0x5869d8c444c9 - std::sys::backtrace::__rust_end_short_backtrace::h9418807cb7346258
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:168:18
[INFO] [stdout]   17:     0x5869d8c45cbd - __rustc[18fb429eef004894]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:697:5
[INFO] [stdout]   18:     0x5869d8c6b0e0 - core::panicking::panic_fmt::hd890aeb12c3a3fc3
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panicking.rs:75:14
[INFO] [stdout]   19:     0x5869d8c6b486 - core::result::unwrap_failed::h31c9775ec1a50121
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1761:5
[INFO] [stdout]   20:     0x5869d8b2c12b - core::result::Result<T,E>::unwrap::h074a3ed8f442fb80
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1167:23
[INFO] [stdout]   21:     0x5869d8b2c12b - hermes::topology_selector::tests::test_gke_selector_different_topologies::h89a8115c74219aff
[INFO] [stdout]                                at /opt/rustwide/workdir/src/topology_selector.rs:301:9
[INFO] [stdout]   22:     0x5869d8b2cd49 - hermes::topology_selector::tests::test_gke_selector_different_topologies::{{closure}}::hc357d4ae787d0dba
[INFO] [stdout]                                at /opt/rustwide/workdir/src/topology_selector.rs:287:48
[INFO] [stdout]   23:     0x5869d8b2cd49 - core::ops::function::FnOnce::call_once::h561ca1e689e1760c
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5869d8b743cb - core::ops::function::FnOnce::call_once::h2869fb5b0a2b0bdc
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   25:     0x5869d8b743cb - test::__rust_begin_short_backtrace::h7dd7142bd62fa711
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:648:18
[INFO] [stdout]   26:     0x5869d8b735be - test::run_test_in_process::{{closure}}::h43a753f038d36b3f
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:74
[INFO] [stdout]   27:     0x5869d8b735be - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h0ca95dd3e12d1e16
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   28:     0x5869d8b735be - std::panicking::catch_unwind::do_call::he308587d70ac34ba
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40
[INFO] [stdout]   29:     0x5869d8b735be - std::panicking::catch_unwind::h30dcba31973e8fb0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19
[INFO] [stdout]   30:     0x5869d8b735be - std::panic::catch_unwind::hfb68364e5621fbee
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14
[INFO] [stdout]   31:     0x5869d8b735be - test::run_test_in_process::hcdcc2977903b998a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:27
[INFO] [stdout]   32:     0x5869d8b735be - test::run_test::{{closure}}::h544a6550958c5d14
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:592:43
[INFO] [stdout]   33:     0x5869d8b37134 - test::run_test::{{closure}}::hb172e48ebe2b92c7
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:622:41
[INFO] [stdout]   34:     0x5869d8b37134 - std::sys::backtrace::__rust_begin_short_backtrace::h7e79d8706638bea0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:152:18
[INFO] [stdout]   35:     0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc4a275f1e71b8ab0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:559:17
[INFO] [stdout]   36:     0x5869d8b3ab0a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h01ed0d242df78cfd
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   37:     0x5869d8b3ab0a - std::panicking::catch_unwind::do_call::h052f373fb905fee0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40
[INFO] [stdout]   38:     0x5869d8b3ab0a - std::panicking::catch_unwind::hb425d20c8ffb09c8
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19
[INFO] [stdout]   39:     0x5869d8b3ab0a - std::panic::catch_unwind::ha0d5dfbf18fdeda5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14
[INFO] [stdout]   40:     0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::hbe79182bd37949c7
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:557:30
[INFO] [stdout]   41:     0x5869d8b3ab0a - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc096c4a06972fde5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   42:     0x5869d8c49497 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h47377e27fb938a26
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9
[INFO] [stdout]   43:     0x5869d8c49497 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h72f1fe5d095abf57
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9
[INFO] [stdout]   44:     0x5869d8c49497 - std::sys::pal::unix::thread::Thread::new::thread_start::h0a4d8e1b9c0d38cf
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/pal/unix/thread.rs:97:17
[INFO] [stdout]   45:     0x7b805ffc1aa4 - <unknown>
[INFO] [stdout]   46:     0x7b806004ea34 - clone
[INFO] [stdout]   47:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- analyzer::tests::test_analyze_node_gke_with_rdma stdout ----
[INFO] [stdout] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] [stdout] Snapshot file: src/snapshots/hermes__analyzer__tests__analyze_node_gke_with_rdma.snap
[INFO] [stdout] Snapshot: analyze_node_gke_with_rdma
[INFO] [stdout] Source: src/analyzer.rs:564
[INFO] [stdout] ────────────────────────────────────────────────────────────────────────────────
[INFO] [stdout] Expression: result
[INFO] [stdout] ────────────────────────────────────────────────────────────────────────────────
[INFO] [stdout] +new results
[INFO] [stdout] ────────────┬───────────────────────────────────────────────────────────────────
[INFO] [stdout]           1 │+name: test-node-1
[INFO] [stdout]           2 │+rdma_capability: Capable
[INFO] [stdout]           3 │+rdma_type: GKE RDMA
[INFO] [stdout]           4 │+rdma_resource: 1 RDMA interfaces
[INFO] [stdout]           5 │+platform_type: GKE
[INFO] [stdout]           6 │+topology_block: us-central1-a-gpu-pool
[INFO] [stdout]           7 │+topology_detection:
[INFO] [stdout]           8 │+  topology_type: Zone
[INFO] [stdout]           9 │+  detection_method: GKE zone+nodepool topology
[INFO] [stdout]          10 │+  confidence: Medium
[INFO] [stdout]          11 │+ib_speed: ~
[INFO] [stdout]          12 │+ib_fabric: ~
[INFO] [stdout]          13 │+ib_ports: ~
[INFO] [stdout]          14 │+leafgroup: ~
[INFO] [stdout]          15 │+superpod: ~
[INFO] [stdout]          16 │+neighbors: []
[INFO] [stdout]          17 │+mellanox_nics: []
[INFO] [stdout]          18 │+node_labels:
[INFO] [stdout]          19 │+  cloud.google.com/gke-nodepool: gpu-pool
[INFO] [stdout]          20 │+  topology.gke.io/zone: us-central1-a
[INFO] [stdout]          21 │+gpu_count: 8
[INFO] [stdout]          22 │+gpu_type: NVIDIA GPU
[INFO] [stdout]          23 │+gpu_allocatable: 8
[INFO] [stdout]          24 │+gpu_allocated: ~
[INFO] [stdout]          25 │+cpu_allocatable: ~
[INFO] [stdout]          26 │+cpu_allocated: ~
[INFO] [stdout]          27 │+memory_allocatable: ~
[INFO] [stdout]          28 │+memory_allocated: ~
[INFO] [stdout]          29 │+platform_data:
[INFO] [stdout]          30 │+  Gke:
[INFO] [stdout]          31 │+    nodepool: gpu-pool
[INFO] [stdout]          32 │+    machine_family: ~
[INFO] [stdout]          33 │+    zone: us-central1-a
[INFO] [stdout]          34 │+    rdma_interfaces: []
[INFO] [stdout]          35 │+    pci_topology: ~
[INFO] [stdout]          36 │+    fabric_domain: ~
[INFO] [stdout]          37 │+    topology_block: ~
[INFO] [stdout]          38 │+    topology_subblock: ~
[INFO] [stdout]          39 │+    topology_host: ~
[INFO] [stdout]          40 │+image_cache_status: Unknown
[INFO] [stdout]          41 │+image_cache_checked_at: ~
[INFO] [stdout] ────────────┴───────────────────────────────────────────────────────────────────
[INFO] [stdout] 
[INFO] [stdout] thread 'analyzer::tests::test_analyze_node_gke_with_rdma' panicked at src/analyzer.rs:564:9:
[INFO] [stdout] called `Result::unwrap()` on an `Err` value: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" }
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5869d8c43fd2 - std::backtrace_rs::backtrace::libunwind::trace::hd39b1f53d3cf9745
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5869d8c43fd2 - std::backtrace_rs::backtrace::trace_unsynchronized::he91d9a75d4e3972b
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5869d8c43fd2 - std::sys::backtrace::_print_fmt::hca46938f8c6e22cf
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:66:9
[INFO] [stdout]    3:     0x5869d8c43fd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::ha499add612cccf8e
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:39:26
[INFO] [stdout]    4:     0x5869d8c6d233 - core::fmt::rt::Argument::fmt::hd21145b75a833b7a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/rt.rs:173:76
[INFO] [stdout]    5:     0x5869d8c6d233 - core::fmt::write::hb10c956f5235c8a4
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/mod.rs:1465:25
[INFO] [stdout]    6:     0x5869d8c40003 - std::io::default_write_fmt::hdb7615052be2ba4d
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5869d8c40003 - std::io::Write::write_fmt::he1bcd251ec6e4153
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:1954:13
[INFO] [stdout]    8:     0x5869d8c43e22 - std::sys::backtrace::BacktraceLock::print::hb47c770ef659fd10
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:42:9
[INFO] [stdout]    9:     0x5869d8c4587c - std::panicking::default_hook::{{closure}}::hdda8afb9d457a22c
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:300:27
[INFO] [stdout]   10:     0x5869d8c456d2 - std::panicking::default_hook::h7c46b44874fe5c9a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:324:9
[INFO] [stdout]   11:     0x5869d8b6ec44 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h178a5fcedee41e2f
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   12:     0x5869d8b6ec44 - test::test_main_with_exit_callback::{{closure}}::h951a41e0149d6d5d
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5869d8c4625b - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h13602080f5b63276
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   14:     0x5869d8c4625b - std::panicking::rust_panic_with_hook::ha6cb99ed099eb1c5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:841:13
[INFO] [stdout]   15:     0x5869d8c4602a - std::panicking::begin_panic_handler::{{closure}}::he11808bc797ee921
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:706:13
[INFO] [stdout]   16:     0x5869d8c444c9 - std::sys::backtrace::__rust_end_short_backtrace::h9418807cb7346258
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:168:18
[INFO] [stdout]   17:     0x5869d8c45cbd - __rustc[18fb429eef004894]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:697:5
[INFO] [stdout]   18:     0x5869d8c6b0e0 - core::panicking::panic_fmt::hd890aeb12c3a3fc3
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panicking.rs:75:14
[INFO] [stdout]   19:     0x5869d8c6b486 - core::result::unwrap_failed::h31c9775ec1a50121
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1761:5
[INFO] [stdout]   20:     0x5869d8b32e22 - core::result::Result<T,E>::unwrap::h074a3ed8f442fb80
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1167:23
[INFO] [stdout]   21:     0x5869d8b32e22 - hermes::analyzer::tests::test_analyze_node_gke_with_rdma::h05d8ddf345ab797b
[INFO] [stdout]                                at /opt/rustwide/workdir/src/analyzer.rs:564:9
[INFO] [stdout]   22:     0x5869d8b2cd69 - hermes::analyzer::tests::test_analyze_node_gke_with_rdma::{{closure}}::hab94dffc314ca28c
[INFO] [stdout]                                at /opt/rustwide/workdir/src/analyzer.rs:559:41
[INFO] [stdout]   23:     0x5869d8b2cd69 - core::ops::function::FnOnce::call_once::h99281a2a8ab61265
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5869d8b743cb - core::ops::function::FnOnce::call_once::h2869fb5b0a2b0bdc
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   25:     0x5869d8b743cb - test::__rust_begin_short_backtrace::h7dd7142bd62fa711
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:648:18
[INFO] [stdout]   26:     0x5869d8b735be - test::run_test_in_process::{{closure}}::h43a753f038d36b3f
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:74
[INFO] [stdout]   27:     0x5869d8b735be - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h0ca95dd3e12d1e16
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   28:     0x5869d8b735be - std::panicking::catch_unwind::do_call::he308587d70ac34ba
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40
[INFO] [stdout]   29:     0x5869d8b735be - std::panicking::catch_unwind::h30dcba31973e8fb0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19
[INFO] [stdout]   30:     0x5869d8b735be - std::panic::catch_unwind::hfb68364e5621fbee
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14
[INFO] [stdout]   31:     0x5869d8b735be - test::run_test_in_process::hcdcc2977903b998a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:27
[INFO] [stdout]   32:     0x5869d8b735be - test::run_test::{{closure}}::h544a6550958c5d14
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:592:43
[INFO] [stdout]   33:     0x5869d8b37134 - test::run_test::{{closure}}::hb172e48ebe2b92c7
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:622:41
[INFO] [stdout]   34:     0x5869d8b37134 - std::sys::backtrace::__rust_begin_short_backtrace::h7e79d8706638bea0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:152:18
[INFO] [stdout]   35:     0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc4a275f1e71b8ab0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:559:17
[INFO] [stdout]   36:     0x5869d8b3ab0a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h01ed0d242df78cfd
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   37:     0x5869d8b3ab0a - std::panicking::catch_unwind::do_call::h052f373fb905fee0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40
[INFO] [stdout]   38:     0x5869d8b3ab0a - std::panicking::catch_unwind::hb425d20c8ffb09c8
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19
[INFO] [stdout]   39:     0x5869d8b3ab0a - std::panic::catch_unwind::ha0d5dfbf18fdeda5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14
[INFO] [stdout]   40:     0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::hbe79182bd37949c7
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:557:30
[INFO] [stdout]   41:     0x5869d8b3ab0a - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc096c4a06972fde5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   42:     0x5869d8c49497 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h47377e27fb938a26
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9
[INFO] [stdout]   43:     0x5869d8c49497 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h72f1fe5d095abf57
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9
[INFO] [stdout]   44:     0x5869d8c49497 - std::sys::pal::unix::thread::Thread::new::thread_start::h0a4d8e1b9c0d38cf
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/pal/unix/thread.rs:97:17
[INFO] [stdout]   45:     0x7b805ffc1aa4 - <unknown>
[INFO] [stdout]   46:     0x7b806004ea34 - clone
[INFO] [stdout]   47:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- topology_selector::tests::test_coreweave_selector_same_leafgroup stdout ----
[INFO] [stdout] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] [stdout] Snapshot file: src/snapshots/hermes__topology_selector__tests__coreweave_selector_same_leafgroup.snap
[INFO] [stdout] Snapshot: coreweave_selector_same_leafgroup
[INFO] [stdout] Source: src/topology_selector.rs:283
[INFO] [stdout] ────────────────────────────────────────────────────────────────────────────────
[INFO] [stdout] Expression: reason
[INFO] [stdout] ────────────────────────────────────────────────────────────────────────────────
[INFO] [stdout] +new results
[INFO] [stdout] ────────────┬───────────────────────────────────────────────────────────────────
[INFO] [stdout]           1 │+Optimal CoreWeave same-topology: RoCE RDMA within leafgroup 'lg1'
[INFO] [stdout] ────────────┴───────────────────────────────────────────────────────────────────
[INFO] [stdout] 
[INFO] [stdout] thread 'topology_selector::tests::test_coreweave_selector_same_leafgroup' panicked at src/topology_selector.rs:283:9:
[INFO] [stdout] called `Result::unwrap()` on an `Err` value: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" }
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5869d8c43fd2 - std::backtrace_rs::backtrace::libunwind::trace::hd39b1f53d3cf9745
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5869d8c43fd2 - std::backtrace_rs::backtrace::trace_unsynchronized::he91d9a75d4e3972b
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5869d8c43fd2 - std::sys::backtrace::_print_fmt::hca46938f8c6e22cf
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:66:9
[INFO] [stdout]    3:     0x5869d8c43fd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::ha499add612cccf8e
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:39:26
[INFO] [stdout]    4:     0x5869d8c6d233 - core::fmt::rt::Argument::fmt::hd21145b75a833b7a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/rt.rs:173:76
[INFO] [stdout]    5:     0x5869d8c6d233 - core::fmt::write::hb10c956f5235c8a4
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/mod.rs:1465:25
[INFO] [stdout]    6:     0x5869d8c40003 - std::io::default_write_fmt::hdb7615052be2ba4d
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5869d8c40003 - std::io::Write::write_fmt::he1bcd251ec6e4153
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:1954:13
[INFO] [stdout]    8:     0x5869d8c43e22 - std::sys::backtrace::BacktraceLock::print::hb47c770ef659fd10
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:42:9
[INFO] [stdout]    9:     0x5869d8c4587c - std::panicking::default_hook::{{closure}}::hdda8afb9d457a22c
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:300:27
[INFO] [stdout]   10:     0x5869d8c456d2 - std::panicking::default_hook::h7c46b44874fe5c9a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:324:9
[INFO] [stdout]   11:     0x5869d8b6ec44 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h178a5fcedee41e2f
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   12:     0x5869d8b6ec44 - test::test_main_with_exit_callback::{{closure}}::h951a41e0149d6d5d
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5869d8c4625b - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h13602080f5b63276
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   14:     0x5869d8c4625b - std::panicking::rust_panic_with_hook::ha6cb99ed099eb1c5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:841:13
[INFO] [stdout]   15:     0x5869d8c4602a - std::panicking::begin_panic_handler::{{closure}}::he11808bc797ee921
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:706:13
[INFO] [stdout]   16:     0x5869d8c444c9 - std::sys::backtrace::__rust_end_short_backtrace::h9418807cb7346258
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:168:18
[INFO] [stdout]   17:     0x5869d8c45cbd - __rustc[18fb429eef004894]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:697:5
[INFO] [stdout]   18:     0x5869d8c6b0e0 - core::panicking::panic_fmt::hd890aeb12c3a3fc3
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panicking.rs:75:14
[INFO] [stdout]   19:     0x5869d8c6b486 - core::result::unwrap_failed::h31c9775ec1a50121
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1761:5
[INFO] [stdout]   20:     0x5869d8b2bc12 - core::result::Result<T,E>::unwrap::h074a3ed8f442fb80
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1167:23
[INFO] [stdout]   21:     0x5869d8b2bc12 - hermes::topology_selector::tests::test_coreweave_selector_same_leafgroup::hd488f93a33cb186a
[INFO] [stdout]                                at /opt/rustwide/workdir/src/topology_selector.rs:283:9
[INFO] [stdout]   22:     0x5869d8b2cd29 - hermes::topology_selector::tests::test_coreweave_selector_same_leafgroup::{{closure}}::h56d42425170cbd91
[INFO] [stdout]                                at /opt/rustwide/workdir/src/topology_selector.rs:266:48
[INFO] [stdout]   23:     0x5869d8b2cd29 - core::ops::function::FnOnce::call_once::h0e24a792ddd76543
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5869d8b743cb - core::ops::function::FnOnce::call_once::h2869fb5b0a2b0bdc
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   25:     0x5869d8b743cb - test::__rust_begin_short_backtrace::h7dd7142bd62fa711
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:648:18
[INFO] [stdout]   26:     0x5869d8b735be - test::run_test_in_process::{{closure}}::h43a753f038d36b3f
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:74
[INFO] [stdout]   27:     0x5869d8b735be - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h0ca95dd3e12d1e16
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   28:     0x5869d8b735be - std::panicking::catch_unwind::do_call::he308587d70ac34ba
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40
[INFO] [stdout]   29:     0x5869d8b735be - std::panicking::catch_unwind::h30dcba31973e8fb0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19
[INFO] [stdout]   30:     0x5869d8b735be - std::panic::catch_unwind::hfb68364e5621fbee
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14
[INFO] [stdout]   31:     0x5869d8b735be - test::run_test_in_process::hcdcc2977903b998a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:27
[INFO] [stdout]   32:     0x5869d8b735be - test::run_test::{{closure}}::h544a6550958c5d14
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:592:43
[INFO] [stdout]   33:     0x5869d8b37134 - test::run_test::{{closure}}::hb172e48ebe2b92c7
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:622:41
[INFO] [stdout]   34:     0x5869d8b37134 - std::sys::backtrace::__rust_begin_short_backtrace::h7e79d8706638bea0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:152:18
[INFO] [stdout]   35:     0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc4a275f1e71b8ab0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:559:17
[INFO] [stdout]   36:     0x5869d8b3ab0a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h01ed0d242df78cfd
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   37:     0x5869d8b3ab0a - std::panicking::catch_unwind::do_call::h052f373fb905fee0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40
[INFO] [stdout]   38:     0x5869d8b3ab0a - std::panicking::catch_unwind::hb425d20c8ffb09c8
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19
[INFO] [stdout]   39:     0x5869d8b3ab0a - std::panic::catch_unwind::ha0d5dfbf18fdeda5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14
[INFO] [stdout]   40:     0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::hbe79182bd37949c7
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:557:30
[INFO] [stdout]   41:     0x5869d8b3ab0a - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc096c4a06972fde5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   42:     0x5869d8c49497 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h47377e27fb938a26
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9
[INFO] [stdout]   43:     0x5869d8c49497 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h72f1fe5d095abf57
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9
[INFO] [stdout]   44:     0x5869d8c49497 - std::sys::pal::unix::thread::Thread::new::thread_start::h0a4d8e1b9c0d38cf
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/pal/unix/thread.rs:97:17
[INFO] [stdout]   45:     0x7b805ffc1aa4 - <unknown>
[INFO] [stdout]   46:     0x7b806004ea34 - clone
[INFO] [stdout]   47:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- analyzer::tests::test_determine_topology_strategy_gke_with_fabric stdout ----
[INFO] [stdout] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] [stdout] Snapshot file: src/snapshots/hermes__analyzer__tests__determine_topology_strategy_gke_with_fabric.snap
[INFO] [stdout] Snapshot: determine_topology_strategy_gke_with_fabric
[INFO] [stdout] Source: src/analyzer.rs:575
[INFO] [stdout] ────────────────────────────────────────────────────────────────────────────────
[INFO] [stdout] Expression: result
[INFO] [stdout] ────────────────────────────────────────────────────────────────────────────────
[INFO] [stdout] +new results
[INFO] [stdout] ────────────┬───────────────────────────────────────────────────────────────────
[INFO] [stdout]           1 │+topology_type: Hardware
[INFO] [stdout]           2 │+detection_method: GKE RDMA fabric domain analysis
[INFO] [stdout]           3 │+confidence: High
[INFO] [stdout] ────────────┴───────────────────────────────────────────────────────────────────
[INFO] [stdout] 
[INFO] [stdout] thread 'analyzer::tests::test_determine_topology_strategy_gke_with_fabric' panicked at src/analyzer.rs:575:9:
[INFO] [stdout] called `Result::unwrap()` on an `Err` value: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" }
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5869d8c43fd2 - std::backtrace_rs::backtrace::libunwind::trace::hd39b1f53d3cf9745
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5869d8c43fd2 - std::backtrace_rs::backtrace::trace_unsynchronized::he91d9a75d4e3972b
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5869d8c43fd2 - std::sys::backtrace::_print_fmt::hca46938f8c6e22cf
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:66:9
[INFO] [stdout]    3:     0x5869d8c43fd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::ha499add612cccf8e
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:39:26
[INFO] [stdout]    4:     0x5869d8c6d233 - core::fmt::rt::Argument::fmt::hd21145b75a833b7a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/rt.rs:173:76
[INFO] [stdout]    5:     0x5869d8c6d233 - core::fmt::write::hb10c956f5235c8a4
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/fmt/mod.rs:1465:25
[INFO] [stdout]    6:     0x5869d8c40003 - std::io::default_write_fmt::hdb7615052be2ba4d
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5869d8c40003 - std::io::Write::write_fmt::he1bcd251ec6e4153
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/io/mod.rs:1954:13
[INFO] [stdout]    8:     0x5869d8c43e22 - std::sys::backtrace::BacktraceLock::print::hb47c770ef659fd10
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:42:9
[INFO] [stdout]    9:     0x5869d8c4587c - std::panicking::default_hook::{{closure}}::hdda8afb9d457a22c
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:300:27
[INFO] [stdout]   10:     0x5869d8c456d2 - std::panicking::default_hook::h7c46b44874fe5c9a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:324:9
[INFO] [stdout]   11:     0x5869d8b6ec44 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h178a5fcedee41e2f
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   12:     0x5869d8b6ec44 - test::test_main_with_exit_callback::{{closure}}::h951a41e0149d6d5d
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5869d8c4625b - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h13602080f5b63276
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   14:     0x5869d8c4625b - std::panicking::rust_panic_with_hook::ha6cb99ed099eb1c5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:841:13
[INFO] [stdout]   15:     0x5869d8c4602a - std::panicking::begin_panic_handler::{{closure}}::he11808bc797ee921
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:706:13
[INFO] [stdout]   16:     0x5869d8c444c9 - std::sys::backtrace::__rust_end_short_backtrace::h9418807cb7346258
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:168:18
[INFO] [stdout]   17:     0x5869d8c45cbd - __rustc[18fb429eef004894]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:697:5
[INFO] [stdout]   18:     0x5869d8c6b0e0 - core::panicking::panic_fmt::hd890aeb12c3a3fc3
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panicking.rs:75:14
[INFO] [stdout]   19:     0x5869d8c6b486 - core::result::unwrap_failed::h31c9775ec1a50121
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1761:5
[INFO] [stdout]   20:     0x5869d8b333c6 - core::result::Result<T,E>::unwrap::h074a3ed8f442fb80
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/result.rs:1167:23
[INFO] [stdout]   21:     0x5869d8b333c6 - hermes::analyzer::tests::test_determine_topology_strategy_gke_with_fabric::h68942f48812a90bc
[INFO] [stdout]                                at /opt/rustwide/workdir/src/analyzer.rs:575:9
[INFO] [stdout]   22:     0x5869d8b2cfa9 - hermes::analyzer::tests::test_determine_topology_strategy_gke_with_fabric::{{closure}}::h84cafd842465e6ae
[INFO] [stdout]                                at /opt/rustwide/workdir/src/analyzer.rs:570:58
[INFO] [stdout]   23:     0x5869d8b2cfa9 - core::ops::function::FnOnce::call_once::hf3836bf5eab9cf69
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5869d8b743cb - core::ops::function::FnOnce::call_once::h2869fb5b0a2b0bdc
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   25:     0x5869d8b743cb - test::__rust_begin_short_backtrace::h7dd7142bd62fa711
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:648:18
[INFO] [stdout]   26:     0x5869d8b735be - test::run_test_in_process::{{closure}}::h43a753f038d36b3f
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:74
[INFO] [stdout]   27:     0x5869d8b735be - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h0ca95dd3e12d1e16
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   28:     0x5869d8b735be - std::panicking::catch_unwind::do_call::he308587d70ac34ba
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40
[INFO] [stdout]   29:     0x5869d8b735be - std::panicking::catch_unwind::h30dcba31973e8fb0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19
[INFO] [stdout]   30:     0x5869d8b735be - std::panic::catch_unwind::hfb68364e5621fbee
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14
[INFO] [stdout]   31:     0x5869d8b735be - test::run_test_in_process::hcdcc2977903b998a
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:671:27
[INFO] [stdout]   32:     0x5869d8b735be - test::run_test::{{closure}}::h544a6550958c5d14
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:592:43
[INFO] [stdout]   33:     0x5869d8b37134 - test::run_test::{{closure}}::hb172e48ebe2b92c7
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/test/src/lib.rs:622:41
[INFO] [stdout]   34:     0x5869d8b37134 - std::sys::backtrace::__rust_begin_short_backtrace::h7e79d8706638bea0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/backtrace.rs:152:18
[INFO] [stdout]   35:     0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc4a275f1e71b8ab0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:559:17
[INFO] [stdout]   36:     0x5869d8b3ab0a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h01ed0d242df78cfd
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   37:     0x5869d8b3ab0a - std::panicking::catch_unwind::do_call::h052f373fb905fee0
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:589:40
[INFO] [stdout]   38:     0x5869d8b3ab0a - std::panicking::catch_unwind::hb425d20c8ffb09c8
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panicking.rs:552:19
[INFO] [stdout]   39:     0x5869d8b3ab0a - std::panic::catch_unwind::ha0d5dfbf18fdeda5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/panic.rs:359:14
[INFO] [stdout]   40:     0x5869d8b3ab0a - std::thread::Builder::spawn_unchecked_::{{closure}}::hbe79182bd37949c7
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/thread/mod.rs:557:30
[INFO] [stdout]   41:     0x5869d8b3ab0a - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc096c4a06972fde5
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   42:     0x5869d8c49497 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h47377e27fb938a26
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9
[INFO] [stdout]   43:     0x5869d8c49497 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h72f1fe5d095abf57
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/alloc/src/boxed.rs:1971:9
[INFO] [stdout]   44:     0x5869d8c49497 - std::sys::pal::unix::thread::Thread::new::thread_start::h0a4d8e1b9c0d38cf
[INFO] [stdout]                                at /rustc/d98a5da813da67eb189387b8ccfb73cf481275d8/library/std/src/sys/pal/unix/thread.rs:97:17
[INFO] [stdout]   45:     0x7b805ffc1aa4 - <unknown>
[INFO] [stdout]   46:     0x7b806004ea34 - clone
[INFO] [stdout]   47:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] failures:
[INFO] [stdout]     analyzer::tests::test_analyze_node_gke_with_rdma
[INFO] [stdout]     analyzer::tests::test_determine_topology_strategy_gke_with_fabric
[INFO] [stdout]     topology_selector::tests::test_coreweave_selector_same_leafgroup
[INFO] [stdout]     topology_selector::tests::test_gke_selector_different_topologies
[INFO] [stdout] 
[INFO] [stdout] test result: FAILED. 1 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.50s
[INFO] [stdout] 
[INFO] [stderr] error: test failed, to rerun pass `--lib`
[INFO] running `Command { std: "docker" "inspect" "669cc7210aa060e91f121443f08ffd8200dd9fe578a7195af0f9f52e076a50d6", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "669cc7210aa060e91f121443f08ffd8200dd9fe578a7195af0f9f52e076a50d6", kill_on_drop: false }`
[INFO] [stdout] 669cc7210aa060e91f121443f08ffd8200dd9fe578a7195af0f9f52e076a50d6
