[INFO] cloning repository https://github.com/famedly/hedwig
[INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/famedly/hedwig" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Ffamedly%2Fhedwig", kill_on_drop: false }`
[INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Ffamedly%2Fhedwig'...
[INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }`
[INFO] [stdout] 672df0359f43560ac7c93ed3e228df75d309968f
[INFO] testing famedly/hedwig against 1.90.0 for beta-1.91-2
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Ffamedly%2Fhedwig" "/workspace/builds/worker-5-tc1/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-5-tc1/source'...
[INFO] [stderr] done.
[INFO] started tweaking git repo https://github.com/famedly/hedwig
[INFO] finished tweaking git repo https://github.com/famedly/hedwig
[INFO] tweaked toml for git repo https://github.com/famedly/hedwig written to /workspace/builds/worker-5-tc1/source/Cargo.toml
[INFO] validating manifest of git repo https://github.com/famedly/hedwig on toolchain 1.90.0
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.90.0" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate git repo https://github.com/famedly/hedwig already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.90.0" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]     Updating git repository `https://github.com/famedly/axum-opentelemetry-middleware.git`
[INFO] [stderr]     Updating crates.io index
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded firebae-cm v0.1.0
[INFO] [stderr]   Downloaded opentelemetry v0.21.0
[INFO] [stderr]   Downloaded gcp_auth v0.8.1
[INFO] [stderr]   Downloaded opentelemetry-prometheus v0.14.1
[INFO] [stderr]   Downloaded axum-extra v0.8.0
[INFO] [stderr]   Downloaded prometheus v0.13.4
[INFO] [stderr]   Downloaded cargo_metadata v0.19.1
[INFO] [stderr]   Downloaded opentelemetry_sdk v0.21.2
[INFO] [stderr]   Downloaded vergen v9.0.4
[INFO] [stderr]   Downloaded vergen-gitcl v1.0.5
[INFO] [stderr]   Downloaded color-spantrace v0.2.1
[INFO] [stderr]   Downloaded sysinfo v0.33.1
[INFO] [stderr]   Downloaded color-eyre v0.6.3
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-5-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-5-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+1.90.0" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] [stdout] 911ab1bad3518a8a2be4bc37c20eac8d90c971187c3e71345c464fd84c9a8209
[INFO] running `Command { std: "docker" "start" "-a" "911ab1bad3518a8a2be4bc37c20eac8d90c971187c3e71345c464fd84c9a8209", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "911ab1bad3518a8a2be4bc37c20eac8d90c971187c3e71345c464fd84c9a8209", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "911ab1bad3518a8a2be4bc37c20eac8d90c971187c3e71345c464fd84c9a8209", kill_on_drop: false }`
[INFO] [stdout] 911ab1bad3518a8a2be4bc37c20eac8d90c971187c3e71345c464fd84c9a8209
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-5-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-5-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+1.90.0" "build" "--frozen" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] 7bc493dbc03ae0ce9e433e6952359a727afedff0cf316d2a5e1a07cd6320a712
[INFO] running `Command { std: "docker" "start" "-a" "7bc493dbc03ae0ce9e433e6952359a727afedff0cf316d2a5e1a07cd6320a712", kill_on_drop: false }`
[INFO] [stderr]    Compiling unicode-ident v1.0.16
[INFO] [stderr]    Compiling libc v0.2.169
[INFO] [stderr]    Compiling once_cell v1.20.3
[INFO] [stderr]    Compiling itoa v1.0.14
[INFO] [stderr]    Compiling rustversion v1.0.19
[INFO] [stderr]    Compiling bytes v1.10.0
[INFO] [stderr]    Compiling serde v1.0.217
[INFO] [stderr]    Compiling cc v1.2.13
[INFO] [stderr]    Compiling ryu v1.0.19
[INFO] [stderr]    Compiling hashbrown v0.15.2
[INFO] [stderr]    Compiling thiserror v2.0.11
[INFO] [stderr]    Compiling litemap v0.7.4
[INFO] [stderr]    Compiling crossbeam-utils v0.8.21
[INFO] [stderr]    Compiling httparse v1.10.0
[INFO] [stderr]    Compiling serde_json v1.0.138
[INFO] [stderr]    Compiling proc-macro2 v1.0.93
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]    Compiling semver v1.0.25
[INFO] [stderr]    Compiling rayon-core v1.12.1
[INFO] [stderr]    Compiling tracing-core v0.1.33
[INFO] [stderr]    Compiling crossbeam-epoch v0.9.18
[INFO] [stderr]    Compiling http v0.2.12
[INFO] [stderr]    Compiling byteorder v1.5.0
[INFO] [stderr]    Compiling camino v1.1.9
[INFO] [stderr]    Compiling anyhow v1.0.95
[INFO] [stderr]    Compiling quote v1.0.38
[INFO] [stderr]    Compiling ahash v0.7.8
[INFO] [stderr]    Compiling aho-corasick v1.1.3
[INFO] [stderr]    Compiling either v1.13.0
[INFO] [stderr]    Compiling syn v2.0.98
[INFO] [stderr]    Compiling indexmap v2.7.1
[INFO] [stderr]    Compiling crossbeam-deque v0.8.6
[INFO] [stderr]    Compiling tracing-log v0.2.0
[INFO] [stderr]    Compiling thread_local v1.1.8
[INFO] [stderr]    Compiling deranged v0.3.11
[INFO] [stderr]    Compiling rustls v0.21.12
[INFO] [stderr]    Compiling num_threads v0.1.7
[INFO] [stderr]    Compiling protobuf v2.28.0
[INFO] [stderr]    Compiling vergen-lib v0.1.6
[INFO] [stderr]    Compiling axum-core v0.3.4
[INFO] [stderr]    Compiling vergen v9.0.4
[INFO] [stderr]    Compiling axum v0.6.20
[INFO] [stderr]    Compiling vergen-gitcl v1.0.5
[INFO] [stderr]    Compiling crossbeam-channel v0.5.14
[INFO] [stderr]    Compiling ring v0.17.8
[INFO] [stderr]    Compiling sha1 v0.10.6
[INFO] [stderr]    Compiling prometheus v0.13.4
[INFO] [stderr]    Compiling rustix v0.38.44
[INFO] [stderr]    Compiling parking_lot_core v0.9.10
[INFO] [stderr]    Compiling getrandom v0.2.15
[INFO] [stderr]    Compiling signal-hook-registry v1.4.2
[INFO] [stderr]    Compiling socket2 v0.5.8
[INFO] [stderr]    Compiling parking_lot v0.12.3
[INFO] [stderr]    Compiling mio v1.0.3
[INFO] [stderr]    Compiling tracing-subscriber v0.3.19
[INFO] [stderr]    Compiling rand_core v0.6.4
[INFO] [stderr]    Compiling http-body v0.4.6
[INFO] [stderr]    Compiling headers-core v0.2.0
[INFO] [stderr]    Compiling rayon v1.10.0
[INFO] [stderr]    Compiling hashbrown v0.12.3
[INFO] [stderr]    Compiling time v0.3.37
[INFO] [stderr]    Compiling headers v0.3.9
[INFO] [stderr]    Compiling ordered-float v4.6.0
[INFO] [stderr]    Compiling backtrace v0.3.71
[INFO] [stderr]    Compiling ring v0.16.20
[INFO] [stderr]    Compiling bitflags v2.8.0
[INFO] [stderr]    Compiling linux-raw-sys v0.4.15
[INFO] [stderr]    Compiling eyre v0.6.12
[INFO] [stderr]    Compiling dlv-list v0.3.0
[INFO] [stderr]    Compiling regex-automata v0.4.9
[INFO] [stderr]    Compiling gimli v0.28.1
[INFO] [stderr]    Compiling glob v0.3.2
[INFO] [stderr]    Compiling rustls-native-certs v0.6.3
[INFO] [stderr]    Compiling miniz_oxide v0.7.4
[INFO] [stderr]    Compiling ordered-multimap v0.4.3
[INFO] [stderr]    Compiling dirs-sys-next v0.1.2
[INFO] [stderr]    Compiling object v0.32.2
[INFO] [stderr]    Compiling indenter v0.3.3
[INFO] [stderr]    Compiling owo-colors v3.5.0
[INFO] [stderr]    Compiling rustc-demangle v0.1.24
[INFO] [stderr]    Compiling chrono v0.4.39
[INFO] [stderr]    Compiling dirs-next v2.0.0
[INFO] [stderr]    Compiling rust-ini v0.18.0
[INFO] [stderr]    Compiling sysinfo v0.33.1
[INFO] [stderr]    Compiling pathdiff v0.2.3
[INFO] [stderr]    Compiling which v4.4.2
[INFO] [stderr]    Compiling addr2line v0.21.0
[INFO] [stderr]    Compiling regex v1.11.1
[INFO] [stderr]    Compiling synstructure v0.13.1
[INFO] [stderr]    Compiling darling_core v0.20.10
[INFO] [stderr]    Compiling serde_derive v1.0.217
[INFO] [stderr]    Compiling zerofrom-derive v0.1.5
[INFO] [stderr]    Compiling futures-macro v0.3.31
[INFO] [stderr]    Compiling tokio-macros v2.5.0
[INFO] [stderr]    Compiling tracing-attributes v0.1.28
[INFO] [stderr]    Compiling yoke-derive v0.7.5
[INFO] [stderr]    Compiling zerovec-derive v0.10.3
[INFO] [stderr]    Compiling displaydoc v0.2.5
[INFO] [stderr]    Compiling thiserror-impl v2.0.11
[INFO] [stderr]    Compiling icu_provider_macros v1.5.0
[INFO] [stderr]    Compiling thiserror-impl v1.0.69
[INFO] [stderr]    Compiling async-trait v0.1.86
[INFO] [stderr]    Compiling tokio v1.43.0
[INFO] [stderr]    Compiling futures-util v0.3.31
[INFO] [stderr]    Compiling pin-project-internal v1.1.9
[INFO] [stderr]    Compiling zerocopy-derive v0.7.35
[INFO] [stderr]    Compiling rustls-webpki v0.101.7
[INFO] [stderr]    Compiling pest v2.7.15
[INFO] [stderr]    Compiling tracing v0.1.41
[INFO] [stderr]    Compiling sct v0.7.1
[INFO] [stderr]    Compiling zerofrom v0.1.5
[INFO] [stderr]    Compiling axum-macros v0.3.8
[INFO] [stderr]    Compiling tracing-error v0.2.1
[INFO] [stderr]    Compiling yoke v0.7.5
[INFO] [stderr]    Compiling color-spantrace v0.2.1
[INFO] [stderr]    Compiling zerovec v0.10.4
[INFO] [stderr]    Compiling darling_macro v0.20.10
[INFO] [stderr]    Compiling thiserror v1.0.69
[INFO] [stderr]    Compiling zerocopy v0.7.35
[INFO] [stderr]    Compiling pin-project v1.1.9
[INFO] [stderr]    Compiling pest_meta v2.7.15
[INFO] [stderr]    Compiling opentelemetry v0.21.0
[INFO] [stderr]    Compiling darling v0.20.10
[INFO] [stderr]    Compiling derive_builder_core v0.20.2
[INFO] [stderr]    Compiling tracing-futures v0.2.5
[INFO] [stderr]    Compiling color-eyre v0.6.3
[INFO] [stderr]    Compiling pest_generator v2.7.15
[INFO] [stderr]    Compiling tinystr v0.7.6
[INFO] [stderr]    Compiling icu_collections v1.5.0
[INFO] [stderr]    Compiling ppv-lite86 v0.2.20
[INFO] [stderr]    Compiling icu_locid v1.5.0
[INFO] [stderr]    Compiling derive_builder_macro v0.20.2
[INFO] [stderr]    Compiling rand_chacha v0.3.1
[INFO] [stderr]    Compiling rand v0.8.5
[INFO] [stderr]    Compiling derive_builder v0.20.2
[INFO] [stderr]    Compiling futures-executor v0.3.31
[INFO] [stderr]    Compiling pest_derive v2.7.15
[INFO] [stderr]    Compiling futures v0.3.31
[INFO] [stderr]    Compiling icu_provider v1.5.0
[INFO] [stderr]    Compiling serde_urlencoded v0.7.1
[INFO] [stderr]    Compiling serde_path_to_error v0.1.16
[INFO] [stderr]    Compiling toml v0.5.11
[INFO] [stderr]    Compiling json5 v0.4.1
[INFO] [stderr]    Compiling ron v0.7.1
[INFO] [stderr]    Compiling icu_locid_transform v1.5.0
[INFO] [stderr]    Compiling icu_properties v1.5.1
[INFO] [stderr]    Compiling config v0.13.4
[INFO] [stderr]    Compiling tokio-util v0.7.13
[INFO] [stderr]    Compiling tower v0.4.13
[INFO] [stderr]    Compiling tokio-stream v0.1.17
[INFO] [stderr]    Compiling tokio-rustls v0.24.1
[INFO] [stderr]    Compiling cargo-platform v0.1.9
[INFO] [stderr]    Compiling h2 v0.3.26
[INFO] [stderr]    Compiling opentelemetry_sdk v0.21.2
[INFO] [stderr]    Compiling rustc_version v0.4.1
[INFO] [stderr]    Compiling tracing-appender v0.2.3
[INFO] [stderr]    Compiling cargo_metadata v0.19.1
[INFO] [stderr]    Compiling icu_normalizer v1.5.0
[INFO] [stderr]    Compiling idna_adapter v1.2.0
[INFO] [stderr]    Compiling opentelemetry-prometheus v0.14.1
[INFO] [stderr]    Compiling idna v1.0.3
[INFO] [stderr]    Compiling url v2.5.4
[INFO] [stderr]    Compiling matrix-hedwig v2.0.0 (/opt/rustwide/workdir)
[INFO] [stderr]    Compiling hyper v0.14.32
[INFO] [stderr]    Compiling hyper-rustls v0.24.2
[INFO] [stderr]    Compiling reqwest v0.11.27
[INFO] [stderr]    Compiling gcp_auth v0.8.1
[INFO] [stderr]    Compiling firebae-cm v0.1.0
[INFO] [stderr]    Compiling axum-opentelemetry-middleware v0.2.1 (https://github.com/famedly/axum-opentelemetry-middleware.git?rev=f8da4a9642fcc2a5671e4f2eaa5831b94c62464f#f8da4a96)
[INFO] [stderr]    Compiling axum-extra v0.8.0
[INFO] [stderr]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 46s
[INFO] running `Command { std: "docker" "inspect" "7bc493dbc03ae0ce9e433e6952359a727afedff0cf316d2a5e1a07cd6320a712", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "7bc493dbc03ae0ce9e433e6952359a727afedff0cf316d2a5e1a07cd6320a712", kill_on_drop: false }`
[INFO] [stdout] 7bc493dbc03ae0ce9e433e6952359a727afedff0cf316d2a5e1a07cd6320a712
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-5-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-5-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+1.90.0" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] 7db0f79bc961f112ca0d26af69f17f4de4ef54254cdb0eb7a04b09360deeab1f
[INFO] running `Command { std: "docker" "start" "-a" "7db0f79bc961f112ca0d26af69f17f4de4ef54254cdb0eb7a04b09360deeab1f", kill_on_drop: false }`
[INFO] [stderr]    Compiling vergen v9.0.4
[INFO] [stderr]    Compiling vergen-gitcl v1.0.5
[INFO] [stderr]    Compiling matrix-hedwig v2.0.0 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 15.87s
[INFO] running `Command { std: "docker" "inspect" "7db0f79bc961f112ca0d26af69f17f4de4ef54254cdb0eb7a04b09360deeab1f", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "7db0f79bc961f112ca0d26af69f17f4de4ef54254cdb0eb7a04b09360deeab1f", kill_on_drop: false }`
[INFO] [stdout] 7db0f79bc961f112ca0d26af69f17f4de4ef54254cdb0eb7a04b09360deeab1f
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-5-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-5-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:e90291280db7d1fac5b66fc6dad9f9662629e7365a55743daf9bdf73ebc4ea79" "/opt/rustwide/cargo-home/bin/cargo" "+1.90.0" "test" "--frozen", kill_on_drop: false }`
[INFO] [stdout] 17a0606e883f3ce3cef9d2216ddd73c1ce84991f2b9bd01c32acce31a8134e9e
[INFO] running `Command { std: "docker" "start" "-a" "17a0606e883f3ce3cef9d2216ddd73c1ce84991f2b9bd01c32acce31a8134e9e", kill_on_drop: false }`
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 0.45s
[INFO] [stderr]      Running unittests src/lib.rs (/opt/rustwide/target/debug/deps/matrix_hedwig-bb7eb8f97039d266)
[INFO] [stdout] 
[INFO] [stdout] running 0 tests
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stderr]      Running unittests src/main.rs (/opt/rustwide/target/debug/deps/matrix_hedwig-ebf4d681220d4d77)
[INFO] [stdout] 
[INFO] [stdout] running 0 tests
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stderr]      Running tests/pusher.rs (/opt/rustwide/target/debug/deps/pusher-4afbf3da5abb52f7)
[INFO] [stdout] 
[INFO] [stdout] running 6 tests
[INFO] [stdout] test bad_json ... ok
[INFO] [stdout] test push_body_limit ... ok
[INFO] [stdout] test many_requests ... ok
[INFO] [stdout] test normal_operation ... FAILED
[INFO] [stdout] test fcm_failure ... ok
[INFO] [stdout] test many_devices ... FAILED
[INFO] [stdout] 
[INFO] [stdout] failures:
[INFO] [stdout] 
[INFO] [stdout] ---- normal_operation stdout ----
[INFO] [stdout] OpenTelemetry metrics error occurred. Metrics error: reader is shut down or not registered
[INFO] [stdout] 
[INFO] [stdout] thread 'normal_operation' panicked at tests/pusher.rs:211:5:
[INFO] [stdout] assertion `left == right` failed
[INFO] [stdout]   left: ""
[INFO] [stdout]  right: "# TYPE devices_total counter\ndevices_total{otel_scope_name=\"axum-opentelemetry\"} 8\n# TYPE http_requests_duration_seconds histogram\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"0\"} 0\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"5\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"10\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"25\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"50\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"75\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"100\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"250\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"500\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"750\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"1000\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"2500\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"5000\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"7500\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"10000\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"+Inf\"} 8\nhttp_requests_duration_seconds_sum{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} FLOAT\nhttp_requests_duration_seconds_count{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} 8\n# TYPE http_requests_total counter\nhttp_requests_total{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} 8\n# TYPE notifications_total counter\nnotifications_total{otel_scope_name=\"axum-opentelemetry\"} 8\n# HELP otel_scope_info Instrumentation Scope metadata\n# TYPE otel_scope_info gauge\notel_scope_info{otel_scope_name=\"axum-opentelemetry\"} 1\n# HELP pushes_successful_total Successful pushes\n# TYPE pushes_successful_total counter\npushes_successful_total{device_type=\"Android\",otel_scope_name=\"axum-opentelemetry\"} 2\npushes_successful_total{device_type=\"AndroidLegacy\",otel_scope_name=\"axum-opentelemetry\"} 2\npushes_successful_total{device_type=\"Ios\",otel_scope_name=\"axum-opentelemetry\"} 2\npushes_successful_total{device_type=\"None\",otel_scope_name=\"axum-opentelemetry\"} 2\n# HELP target_info Target metadata\n# TYPE target_info gauge\ntarget_info{service_name=\"Hedwig\"} 1\n"
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x61eb552e6a12 - std::backtrace_rs::backtrace::libunwind::trace::h2d45396358f41939
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x61eb552e6a12 - std::backtrace_rs::backtrace::trace_unsynchronized::hffcefc0b67f1d6e2
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x61eb552e6a12 - std::sys::backtrace::_print_fmt::hd72f71d23b436b92
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/backtrace.rs:66:9
[INFO] [stdout]    3:     0x61eb552e6a12 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hdcfcb6d4c8489523
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/backtrace.rs:39:26
[INFO] [stdout]    4:     0x61eb5530f193 - core::fmt::rt::Argument::fmt::h2c56b3114963061a
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/fmt/rt.rs:173:76
[INFO] [stdout]    5:     0x61eb5530f193 - core::fmt::write::h8a494366950f23bb
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/fmt/mod.rs:1468:25
[INFO] [stdout]    6:     0x61eb552e3683 - std::io::default_write_fmt::h7b8824096454f323
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x61eb552e3683 - std::io::Write::write_fmt::h4e71294925c334d0
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/io/mod.rs:1954:13
[INFO] [stdout]    8:     0x61eb552e6862 - std::sys::backtrace::BacktraceLock::print::hb2a626a81e06b2dc
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/backtrace.rs:42:9
[INFO] [stdout]    9:     0x61eb552e7f99 - std::panicking::default_hook::{{closure}}::h4f78485264f12d10
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:300:27
[INFO] [stdout]   10:     0x61eb552e7dce - std::panicking::default_hook::h2c66fc99e962531d
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:324:9
[INFO] [stdout]   11:     0x61eb54b57c44 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hb6bd65493727d71a
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   12:     0x61eb54b57c44 - test::test_main_with_exit_callback::{{closure}}::h60217ec76a0ece4a
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x61eb552e8a6e - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h318e4efb8c1a5689
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   14:     0x61eb552e8a6e - std::panicking::rust_panic_with_hook::h33ac55f64bbd807d
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:841:13
[INFO] [stdout]   15:     0x61eb552e873a - std::panicking::begin_panic_handler::{{closure}}::h30e7cb89678a57fe
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:706:13
[INFO] [stdout]   16:     0x61eb552e6f19 - std::sys::backtrace::__rust_end_short_backtrace::hed60f27456c16ced
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/backtrace.rs:174:18
[INFO] [stdout]   17:     0x61eb552e83cd - __rustc[de2ca18b4c54d5b8]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:697:5
[INFO] [stdout]   18:     0x61eb5530c3b0 - core::panicking::panic_fmt::h62f63d096dd276af
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panicking.rs:75:14
[INFO] [stdout]   19:     0x61eb5530c7bd - core::panicking::assert_failed_inner::h102b4539a88470c2
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panicking.rs:448:17
[INFO] [stdout]   20:     0x61eb54b1534a - core::panicking::assert_failed::h52cff5bf65409058
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panicking.rs:403:5
[INFO] [stdout]   21:     0x61eb54aeb85e - pusher::check_prom::{{closure}}::h20cd5b748e45d71a
[INFO] [stdout]                                at /opt/rustwide/workdir/tests/pusher.rs:211:2
[INFO] [stdout]   22:     0x61eb54af19ef - pusher::normal_operation::{{closure}}::hb6385971bfa32e89
[INFO] [stdout]                                at /opt/rustwide/workdir/tests/pusher.rs:315:68
[INFO] [stdout]   23:     0x61eb54b0061f - <core::pin::Pin<P> as core::future::future::Future>::poll::h650ceb7d3b51289e
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/future/future.rs:133:9
[INFO] [stdout]   24:     0x61eb54b005bb - <core::pin::Pin<P> as core::future::future::Future>::poll::h31e94c7c00ec8dd2
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/future/future.rs:133:9
[INFO] [stdout]   25:     0x61eb54afafdb - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}::he84c29acdb9e1156
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:73
[INFO] [stdout]   26:     0x61eb54afacdb - tokio::runtime::coop::with_budget::ha6f3bfd26f29105a
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:107:5
[INFO] [stdout]   27:     0x61eb54afacdb - tokio::runtime::coop::budget::he46a0f93b9cf6e98
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:73:5
[INFO] [stdout]   28:     0x61eb54afacdb - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::h9f1b729527c498a5
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:25
[INFO] [stdout]   29:     0x61eb54af834d - tokio::runtime::scheduler::current_thread::Context::enter::hce8d29abd0490ed8
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:428:19
[INFO] [stdout]   30:     0x61eb54af97b0 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::h377e78b580fe515b
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:728:44
[INFO] [stdout]   31:     0x61eb54af940b - tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}::h166fadaf8f5a106d
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:68
[INFO] [stdout]   32:     0x61eb54aff98a - tokio::runtime::context::scoped::Scoped<T>::set::hf4fff412f9d5f180
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   33:     0x61eb54b1ba8a - tokio::runtime::context::set_scheduler::{{closure}}::hfb9f02022bbfeaf3
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:38
[INFO] [stdout]   34:     0x61eb54b18405 - std::thread::local::LocalKey<T>::try_with::h5c9d8abb74d83775
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/thread/local.rs:315:12
[INFO] [stdout]   35:     0x61eb54b17cbd - std::thread::local::LocalKey<T>::with::hd40ef0920ed93907
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/thread/local.rs:279:20
[INFO] [stdout]   36:     0x61eb54b1b9d1 - tokio::runtime::context::set_scheduler::hc50f435613b0578a
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:17
[INFO] [stdout]   37:     0x61eb54af916e - tokio::runtime::scheduler::current_thread::CoreGuard::enter::hd4f031627ff3a6d6
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:27
[INFO] [stdout]   38:     0x61eb54af94e0 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::hd284b997707908df
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:716:24
[INFO] [stdout]   39:     0x61eb54af7426 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}::h41d232a2bbf74cb2
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:196:33
[INFO] [stdout]   40:     0x61eb54b0464c - tokio::runtime::context::runtime::enter_runtime::hb4499f13ad207725
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   41:     0x61eb54af6e07 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::h121aa522ce0cc888
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:184:9
[INFO] [stdout]   42:     0x61eb54affc50 - tokio::runtime::runtime::Runtime::block_on_inner::h8090a04922f81bba
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:368:52
[INFO] [stdout]   43:     0x61eb54affe6f - tokio::runtime::runtime::Runtime::block_on::he505f8c11e57599a
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:342:18
[INFO] [stdout]   44:     0x61eb54af0703 - pusher::normal_operation::h82748bc43a9be088
[INFO] [stdout]                                at /opt/rustwide/workdir/tests/pusher.rs:317:4
[INFO] [stdout]   45:     0x61eb54af05a7 - pusher::normal_operation::{{closure}}::h31cd4d3012ca61b1
[INFO] [stdout]                                at /opt/rustwide/workdir/tests/pusher.rs:291:32
[INFO] [stdout]   46:     0x61eb54b1c446 - core::ops::function::FnOnce::call_once::h7c8508f48af19ee4
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/ops/function.rs:253:5
[INFO] [stdout]   47:     0x61eb54b5d4ab - core::ops::function::FnOnce::call_once::h1f9474f1347fff52
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/ops/function.rs:253:5
[INFO] [stdout]   48:     0x61eb54b5d4ab - test::__rust_begin_short_backtrace::ha52ab26e77157f03
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:648:18
[INFO] [stdout]   49:     0x61eb54b5c5e5 - test::run_test_in_process::{{closure}}::h229c1a11a50b261f
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:671:74
[INFO] [stdout]   50:     0x61eb54b5c5e5 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h78821afbf97b39a9
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   51:     0x61eb54b5c5e5 - std::panicking::catch_unwind::do_call::h53abf3ed5bd32ba3
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:589:40
[INFO] [stdout]   52:     0x61eb54b5c5e5 - std::panicking::catch_unwind::h6deb5fe1e91873c9
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:552:19
[INFO] [stdout]   53:     0x61eb54b5c5e5 - std::panic::catch_unwind::hea0829b6b565654b
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panic.rs:359:14
[INFO] [stdout]   54:     0x61eb54b5c5e5 - test::run_test_in_process::hdc44dfecea3db21b
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:671:27
[INFO] [stdout]   55:     0x61eb54b5c5e5 - test::run_test::{{closure}}::h0364ba59bf23f652
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:592:43
[INFO] [stdout]   56:     0x61eb54b201e4 - test::run_test::{{closure}}::h32730b304eec05da
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:622:41
[INFO] [stdout]   57:     0x61eb54b201e4 - std::sys::backtrace::__rust_begin_short_backtrace::h177cb952fc54ad2c
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/backtrace.rs:158:18
[INFO] [stdout]   58:     0x61eb54b23bba - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h119d903520007597
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/thread/mod.rs:559:17
[INFO] [stdout]   59:     0x61eb54b23bba - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hab88401d606702a6
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   60:     0x61eb54b23bba - std::panicking::catch_unwind::do_call::h163fe12cca9901c2
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:589:40
[INFO] [stdout]   61:     0x61eb54b23bba - std::panicking::catch_unwind::he81557d0e17cc1bc
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:552:19
[INFO] [stdout]   62:     0x61eb54b23bba - std::panic::catch_unwind::hc2b8e79c20593955
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panic.rs:359:14
[INFO] [stdout]   63:     0x61eb54b23bba - std::thread::Builder::spawn_unchecked_::{{closure}}::hfa6b4978e07815cc
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/thread/mod.rs:557:30
[INFO] [stdout]   64:     0x61eb54b23bba - core::ops::function::FnOnce::call_once{{vtable.shim}}::hddcd7484a13fd793
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/ops/function.rs:253:5
[INFO] [stdout]   65:     0x61eb552ebc0f - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h5924238c754de3b8
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/alloc/src/boxed.rs:1971:9
[INFO] [stdout]   66:     0x61eb552ebc0f - std::sys::pal::unix::thread::Thread::new::thread_start::hb6e99e73da4d28f8
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/pal/unix/thread.rs:107:17
[INFO] [stdout]   67:     0x76d4d38bfaa4 - <unknown>
[INFO] [stdout]   68:     0x76d4d394ca34 - clone
[INFO] [stdout]   69:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- many_devices stdout ----
[INFO] [stdout] OpenTelemetry metrics error occurred. Metrics error: reader is shut down or not registered
[INFO] [stdout] 
[INFO] [stdout] thread 'many_devices' panicked at tests/pusher.rs:211:5:
[INFO] [stdout] assertion `left == right` failed
[INFO] [stdout]   left: ""
[INFO] [stdout]  right: "# TYPE devices_total counter\ndevices_total{otel_scope_name=\"axum-opentelemetry\"} 12\n# TYPE http_requests_duration_seconds histogram\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"0\"} 0\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"5\"} 2\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"10\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"25\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"50\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"75\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"100\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"250\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"500\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"750\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"1000\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"2500\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"5000\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"7500\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"10000\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"+Inf\"} 3\nhttp_requests_duration_seconds_sum{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} FLOAT\nhttp_requests_duration_seconds_count{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} 3\n# TYPE http_requests_total counter\nhttp_requests_total{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} 3\n# TYPE notifications_total counter\nnotifications_total{otel_scope_name=\"axum-opentelemetry\"} 3\n# HELP otel_scope_info Instrumentation Scope metadata\n# TYPE otel_scope_info gauge\notel_scope_info{otel_scope_name=\"axum-opentelemetry\"} 1\n# HELP pushes_failed_total Failed pushes\n# TYPE pushes_failed_total counter\npushes_failed_total{device_type=\"AndroidLegacy\",otel_scope_name=\"axum-opentelemetry\"} 1\npushes_failed_total{device_type=\"None\",otel_scope_name=\"axum-opentelemetry\"} 1\n# HELP pushes_successful_total Successful pushes\n# TYPE pushes_successful_total counter\npushes_successful_total{device_type=\"Android\",otel_scope_name=\"axum-opentelemetry\"} 3\npushes_successful_total{device_type=\"AndroidLegacy\",otel_scope_name=\"axum-opentelemetry\"} 2\npushes_successful_total{device_type=\"Ios\",otel_scope_name=\"axum-opentelemetry\"} 3\npushes_successful_total{device_type=\"None\",otel_scope_name=\"axum-opentelemetry\"} 2\n# HELP target_info Target metadata\n# TYPE target_info gauge\ntarget_info{service_name=\"Hedwig\"} 1\n"
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x61eb552e6a12 - std::backtrace_rs::backtrace::libunwind::trace::h2d45396358f41939
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x61eb552e6a12 - std::backtrace_rs::backtrace::trace_unsynchronized::hffcefc0b67f1d6e2
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x61eb552e6a12 - std::sys::backtrace::_print_fmt::hd72f71d23b436b92
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/backtrace.rs:66:9
[INFO] [stdout]    3:     0x61eb552e6a12 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hdcfcb6d4c8489523
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/backtrace.rs:39:26
[INFO] [stdout]    4:     0x61eb5530f193 - core::fmt::rt::Argument::fmt::h2c56b3114963061a
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/fmt/rt.rs:173:76
[INFO] [stdout]    5:     0x61eb5530f193 - core::fmt::write::h8a494366950f23bb
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/fmt/mod.rs:1468:25
[INFO] [stdout]    6:     0x61eb552e3683 - std::io::default_write_fmt::h7b8824096454f323
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x61eb552e3683 - std::io::Write::write_fmt::h4e71294925c334d0
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/io/mod.rs:1954:13
[INFO] [stdout]    8:     0x61eb552e6862 - std::sys::backtrace::BacktraceLock::print::hb2a626a81e06b2dc
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/backtrace.rs:42:9
[INFO] [stdout]    9:     0x61eb552e7f99 - std::panicking::default_hook::{{closure}}::h4f78485264f12d10
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:300:27
[INFO] [stdout]   10:     0x61eb552e7dce - std::panicking::default_hook::h2c66fc99e962531d
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:324:9
[INFO] [stdout]   11:     0x61eb54b57c44 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hb6bd65493727d71a
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   12:     0x61eb54b57c44 - test::test_main_with_exit_callback::{{closure}}::h60217ec76a0ece4a
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x61eb552e8a6e - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h318e4efb8c1a5689
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/alloc/src/boxed.rs:1985:9
[INFO] [stdout]   14:     0x61eb552e8a6e - std::panicking::rust_panic_with_hook::h33ac55f64bbd807d
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:841:13
[INFO] [stdout]   15:     0x61eb552e873a - std::panicking::begin_panic_handler::{{closure}}::h30e7cb89678a57fe
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:706:13
[INFO] [stdout]   16:     0x61eb552e6f19 - std::sys::backtrace::__rust_end_short_backtrace::hed60f27456c16ced
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/backtrace.rs:174:18
[INFO] [stdout]   17:     0x61eb552e83cd - __rustc[de2ca18b4c54d5b8]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:697:5
[INFO] [stdout]   18:     0x61eb5530c3b0 - core::panicking::panic_fmt::h62f63d096dd276af
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panicking.rs:75:14
[INFO] [stdout]   19:     0x61eb5530c7bd - core::panicking::assert_failed_inner::h102b4539a88470c2
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panicking.rs:448:17
[INFO] [stdout]   20:     0x61eb54b1534a - core::panicking::assert_failed::h52cff5bf65409058
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panicking.rs:403:5
[INFO] [stdout]   21:     0x61eb54aeb85e - pusher::check_prom::{{closure}}::h20cd5b748e45d71a
[INFO] [stdout]                                at /opt/rustwide/workdir/tests/pusher.rs:211:2
[INFO] [stdout]   22:     0x61eb54af4466 - pusher::many_devices::{{closure}}::hde1990b42217d846
[INFO] [stdout]                                at /opt/rustwide/workdir/tests/pusher.rs:372:64
[INFO] [stdout]   23:     0x61eb54b0061f - <core::pin::Pin<P> as core::future::future::Future>::poll::h650ceb7d3b51289e
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/future/future.rs:133:9
[INFO] [stdout]   24:     0x61eb54b005bb - <core::pin::Pin<P> as core::future::future::Future>::poll::h31e94c7c00ec8dd2
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/future/future.rs:133:9
[INFO] [stdout]   25:     0x61eb54afafdb - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}::he84c29acdb9e1156
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:73
[INFO] [stdout]   26:     0x61eb54afacdb - tokio::runtime::coop::with_budget::ha6f3bfd26f29105a
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:107:5
[INFO] [stdout]   27:     0x61eb54afacdb - tokio::runtime::coop::budget::he46a0f93b9cf6e98
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:73:5
[INFO] [stdout]   28:     0x61eb54afacdb - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::h9f1b729527c498a5
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:25
[INFO] [stdout]   29:     0x61eb54af834d - tokio::runtime::scheduler::current_thread::Context::enter::hce8d29abd0490ed8
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:428:19
[INFO] [stdout]   30:     0x61eb54af97b0 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::h377e78b580fe515b
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:728:44
[INFO] [stdout]   31:     0x61eb54af940b - tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}::h166fadaf8f5a106d
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:68
[INFO] [stdout]   32:     0x61eb54aff98a - tokio::runtime::context::scoped::Scoped<T>::set::hf4fff412f9d5f180
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   33:     0x61eb54b1ba8a - tokio::runtime::context::set_scheduler::{{closure}}::hfb9f02022bbfeaf3
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:38
[INFO] [stdout]   34:     0x61eb54b18405 - std::thread::local::LocalKey<T>::try_with::h5c9d8abb74d83775
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/thread/local.rs:315:12
[INFO] [stdout]   35:     0x61eb54b17cbd - std::thread::local::LocalKey<T>::with::hd40ef0920ed93907
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/thread/local.rs:279:20
[INFO] [stdout]   36:     0x61eb54b1b9d1 - tokio::runtime::context::set_scheduler::hc50f435613b0578a
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:17
[INFO] [stdout]   37:     0x61eb54af916e - tokio::runtime::scheduler::current_thread::CoreGuard::enter::hd4f031627ff3a6d6
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:27
[INFO] [stdout]   38:     0x61eb54af94e0 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::hd284b997707908df
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:716:24
[INFO] [stdout]   39:     0x61eb54af7426 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}::h41d232a2bbf74cb2
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:196:33
[INFO] [stdout]   40:     0x61eb54b0464c - tokio::runtime::context::runtime::enter_runtime::hb4499f13ad207725
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   41:     0x61eb54af6e07 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::h121aa522ce0cc888
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:184:9
[INFO] [stdout]   42:     0x61eb54affc50 - tokio::runtime::runtime::Runtime::block_on_inner::h8090a04922f81bba
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:368:52
[INFO] [stdout]   43:     0x61eb54affe6f - tokio::runtime::runtime::Runtime::block_on::he505f8c11e57599a
[INFO] [stdout]                                at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:342:18
[INFO] [stdout]   44:     0x61eb54af2b33 - pusher::many_devices::h3c4d5f73fe646f5a
[INFO] [stdout]                                at /opt/rustwide/workdir/tests/pusher.rs:374:4
[INFO] [stdout]   45:     0x61eb54af29d7 - pusher::many_devices::{{closure}}::h4c9c7fe2ae4d8752
[INFO] [stdout]                                at /opt/rustwide/workdir/tests/pusher.rs:336:28
[INFO] [stdout]   46:     0x61eb54b1c376 - core::ops::function::FnOnce::call_once::h10de86d5f3e700e8
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/ops/function.rs:253:5
[INFO] [stdout]   47:     0x61eb54b5d4ab - core::ops::function::FnOnce::call_once::h1f9474f1347fff52
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/ops/function.rs:253:5
[INFO] [stdout]   48:     0x61eb54b5d4ab - test::__rust_begin_short_backtrace::ha52ab26e77157f03
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:648:18
[INFO] [stdout]   49:     0x61eb54b5c5e5 - test::run_test_in_process::{{closure}}::h229c1a11a50b261f
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:671:74
[INFO] [stdout]   50:     0x61eb54b5c5e5 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h78821afbf97b39a9
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   51:     0x61eb54b5c5e5 - std::panicking::catch_unwind::do_call::h53abf3ed5bd32ba3
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:589:40
[INFO] [stdout]   52:     0x61eb54b5c5e5 - std::panicking::catch_unwind::h6deb5fe1e91873c9
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:552:19
[INFO] [stdout]   53:     0x61eb54b5c5e5 - std::panic::catch_unwind::hea0829b6b565654b
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panic.rs:359:14
[INFO] [stdout]   54:     0x61eb54b5c5e5 - test::run_test_in_process::hdc44dfecea3db21b
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:671:27
[INFO] [stdout]   55:     0x61eb54b5c5e5 - test::run_test::{{closure}}::h0364ba59bf23f652
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:592:43
[INFO] [stdout]   56:     0x61eb54b201e4 - test::run_test::{{closure}}::h32730b304eec05da
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/test/src/lib.rs:622:41
[INFO] [stdout]   57:     0x61eb54b201e4 - std::sys::backtrace::__rust_begin_short_backtrace::h177cb952fc54ad2c
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/backtrace.rs:158:18
[INFO] [stdout]   58:     0x61eb54b23bba - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h119d903520007597
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/thread/mod.rs:559:17
[INFO] [stdout]   59:     0x61eb54b23bba - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hab88401d606702a6
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panic/unwind_safe.rs:272:9
[INFO] [stdout]   60:     0x61eb54b23bba - std::panicking::catch_unwind::do_call::h163fe12cca9901c2
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:589:40
[INFO] [stdout]   61:     0x61eb54b23bba - std::panicking::catch_unwind::he81557d0e17cc1bc
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:552:19
[INFO] [stdout]   62:     0x61eb54b23bba - std::panic::catch_unwind::hc2b8e79c20593955
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panic.rs:359:14
[INFO] [stdout]   63:     0x61eb54b23bba - std::thread::Builder::spawn_unchecked_::{{closure}}::hfa6b4978e07815cc
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/thread/mod.rs:557:30
[INFO] [stdout]   64:     0x61eb54b23bba - core::ops::function::FnOnce::call_once{{vtable.shim}}::hddcd7484a13fd793
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/ops/function.rs:253:5
[INFO] [stdout]   65:     0x61eb552ebc0f - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h5924238c754de3b8
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/alloc/src/boxed.rs:1971:9
[INFO] [stdout]   66:     0x61eb552ebc0f - std::sys::pal::unix::thread::Thread::new::thread_start::hb6e99e73da4d28f8
[INFO] [stdout]                                at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/pal/unix/thread.rs:107:17
[INFO] [stdout]   67:     0x76d4d38bfaa4 - <unknown>
[INFO] [stdout]   68:     0x76d4d394ca34 - clone
[INFO] [stdout]   69:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] failures:
[INFO] [stdout]     many_devices
[INFO] [stdout]     normal_operation
[INFO] [stdout] 
[INFO] [stdout] test result: FAILED. 4 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.52s
[INFO] [stdout] 
[INFO] [stderr] error: test failed, to rerun pass `--test pusher`
[INFO] running `Command { std: "docker" "inspect" "17a0606e883f3ce3cef9d2216ddd73c1ce84991f2b9bd01c32acce31a8134e9e", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "17a0606e883f3ce3cef9d2216ddd73c1ce84991f2b9bd01c32acce31a8134e9e", kill_on_drop: false }`
[INFO] [stdout] 17a0606e883f3ce3cef9d2216ddd73c1ce84991f2b9bd01c32acce31a8134e9e
