[INFO] cloning repository https://github.com/famedly/hedwig [INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/famedly/hedwig" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Ffamedly%2Fhedwig", kill_on_drop: false }` [INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Ffamedly%2Fhedwig'... [INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }` [INFO] [stdout] 672df0359f43560ac7c93ed3e228df75d309968f [INFO] testing famedly/hedwig against master#ad85bc524b1ad696e42061ad8338d382dffbdbe5 for pr-146237 [INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Ffamedly%2Fhedwig" "/workspace/builds/worker-3-tc1/source", kill_on_drop: false }` [INFO] [stderr] Cloning into '/workspace/builds/worker-3-tc1/source'... [INFO] [stderr] done. [INFO] started tweaking git repo https://github.com/famedly/hedwig [INFO] finished tweaking git repo https://github.com/famedly/hedwig [INFO] tweaked toml for git repo https://github.com/famedly/hedwig written to /workspace/builds/worker-3-tc1/source/Cargo.toml [INFO] validating manifest of git repo https://github.com/famedly/hedwig on toolchain ad85bc524b1ad696e42061ad8338d382dffbdbe5 [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+ad85bc524b1ad696e42061ad8338d382dffbdbe5" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }` [INFO] crate git repo https://github.com/famedly/hedwig already has a lockfile, it will not be regenerated [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+ad85bc524b1ad696e42061ad8338d382dffbdbe5" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }` [INFO] [stderr] Blocking waiting for file lock on package cache [INFO] [stderr] Updating crates.io index [INFO] [stderr] Updating git repository `https://github.com/famedly/axum-opentelemetry-middleware.git` [INFO] [stderr] Blocking waiting for file lock on package cache [INFO] [stderr] Downloading crates ... [INFO] [stderr] Downloaded gcp_auth v0.8.1 [INFO] [stderr] Downloaded vergen-gitcl v1.0.5 [INFO] [stderr] Downloaded cargo_metadata v0.19.1 [INFO] [stderr] Downloaded vergen-lib v0.1.6 [INFO] [stderr] Downloaded axum-extra v0.8.0 [INFO] [stderr] Downloaded opentelemetry v0.21.0 [INFO] [stderr] Downloaded opentelemetry-prometheus v0.14.1 [INFO] [stderr] Downloaded color-spantrace v0.2.1 [INFO] [stderr] Downloaded prometheus v0.13.4 [INFO] [stderr] Downloaded opentelemetry_sdk v0.21.2 [INFO] [stderr] Downloaded vergen v9.0.4 [INFO] [stderr] Downloaded color-eyre v0.6.3 [INFO] [stderr] Downloaded firebae-cm v0.1.0 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:7ad1b28ee6f5f7f699f6cf7015098d6ccdd96d6f2d78dd06228f5b4c9faf309c" "/opt/rustwide/cargo-home/bin/cargo" "+ad85bc524b1ad696e42061ad8338d382dffbdbe5" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }` [INFO] [stdout] 5f544ecd1eacf8cbd64f2e6d61a4e885af350b21e6539ccfbf98503f1b0c3492 [INFO] running `Command { std: "docker" "start" "-a" "5f544ecd1eacf8cbd64f2e6d61a4e885af350b21e6539ccfbf98503f1b0c3492", kill_on_drop: false }` [INFO] running `Command { std: "docker" "inspect" "5f544ecd1eacf8cbd64f2e6d61a4e885af350b21e6539ccfbf98503f1b0c3492", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "5f544ecd1eacf8cbd64f2e6d61a4e885af350b21e6539ccfbf98503f1b0c3492", kill_on_drop: false }` [INFO] [stdout] 5f544ecd1eacf8cbd64f2e6d61a4e885af350b21e6539ccfbf98503f1b0c3492 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:7ad1b28ee6f5f7f699f6cf7015098d6ccdd96d6f2d78dd06228f5b4c9faf309c" "/opt/rustwide/cargo-home/bin/cargo" "+ad85bc524b1ad696e42061ad8338d382dffbdbe5" "build" "--frozen" "--message-format=json", kill_on_drop: false }` [INFO] [stdout] c199f878be41f2b2878f4c09b573a6fa1e52920309bd3bdacc2f1fbe2d954e6a [INFO] running `Command { std: "docker" "start" "-a" "c199f878be41f2b2878f4c09b573a6fa1e52920309bd3bdacc2f1fbe2d954e6a", kill_on_drop: false }` [INFO] [stderr] Compiling syn v2.0.98 [INFO] [stderr] Compiling log v0.4.25 [INFO] [stderr] Compiling serde v1.0.217 [INFO] [stderr] Compiling parking_lot_core v0.9.10 [INFO] [stderr] Compiling crossbeam-utils v0.8.21 [INFO] [stderr] Compiling ring v0.17.8 [INFO] [stderr] Compiling http-body v0.4.6 [INFO] [stderr] Compiling serde_json v1.0.138 [INFO] [stderr] Compiling num-traits v0.2.19 [INFO] [stderr] Compiling semver v1.0.25 [INFO] [stderr] Compiling anyhow v1.0.95 [INFO] [stderr] Compiling rayon-core v1.12.1 [INFO] [stderr] Compiling camino v1.1.9 [INFO] [stderr] Compiling vergen-lib v0.1.6 [INFO] [stderr] Compiling aho-corasick v1.1.3 [INFO] [stderr] Compiling either v1.13.0 [INFO] [stderr] Compiling tracing-log v0.2.0 [INFO] [stderr] Compiling deranged v0.3.11 [INFO] [stderr] Compiling axum-core v0.3.4 [INFO] [stderr] Compiling vergen v9.0.4 [INFO] [stderr] Compiling rustls v0.21.12 [INFO] [stderr] Compiling num_threads v0.1.7 [INFO] [stderr] Compiling parking_lot v0.12.3 [INFO] [stderr] Compiling tracing-subscriber v0.3.19 [INFO] [stderr] Compiling ahash v0.7.8 [INFO] [stderr] Compiling headers-core v0.2.0 [INFO] [stderr] Compiling axum v0.6.20 [INFO] [stderr] Compiling vergen-gitcl v1.0.5 [INFO] [stderr] Compiling prometheus v0.13.4 [INFO] [stderr] Compiling rustix v0.38.44 [INFO] [stderr] Compiling headers v0.3.9 [INFO] [stderr] Compiling crossbeam-channel v0.5.14 [INFO] [stderr] Compiling hashbrown v0.12.3 [INFO] [stderr] Compiling crossbeam-epoch v0.9.18 [INFO] [stderr] Compiling backtrace v0.3.71 [INFO] [stderr] Compiling ring v0.16.20 [INFO] [stderr] Compiling glob v0.3.2 [INFO] [stderr] Compiling eyre v0.6.12 [INFO] [stderr] Compiling gimli v0.28.1 [INFO] [stderr] Compiling miniz_oxide v0.7.4 [INFO] [stderr] Compiling crossbeam-deque v0.8.6 [INFO] [stderr] Compiling rustls-native-certs v0.6.3 [INFO] [stderr] Compiling ordered-multimap v0.4.3 [INFO] [stderr] Compiling ordered-float v4.6.0 [INFO] [stderr] Compiling time v0.3.37 [INFO] [stderr] Compiling dirs-sys-next v0.1.2 [INFO] [stderr] Compiling object v0.32.2 [INFO] [stderr] Compiling indenter v0.3.3 [INFO] [stderr] Compiling home v0.5.11 [INFO] [stderr] Compiling owo-colors v3.5.0 [INFO] [stderr] Compiling regex-automata v0.4.9 [INFO] [stderr] Compiling rayon v1.10.0 [INFO] [stderr] Compiling rustc-demangle v0.1.24 [INFO] [stderr] Compiling nom v7.1.3 [INFO] [stderr] Compiling chrono v0.4.39 [INFO] [stderr] Compiling yaml-rust v0.4.5 [INFO] [stderr] Compiling dirs-next v2.0.0 [INFO] [stderr] Compiling rust-ini v0.18.0 [INFO] [stderr] Compiling pathdiff v0.2.3 [INFO] [stderr] Compiling which v4.4.2 [INFO] [stderr] Compiling addr2line v0.21.0 [INFO] [stderr] Compiling regex v1.11.1 [INFO] [stderr] Compiling sysinfo v0.33.1 [INFO] [stderr] Compiling synstructure v0.13.1 [INFO] [stderr] Compiling darling_core v0.20.10 [INFO] [stderr] Compiling sct v0.7.1 [INFO] [stderr] Compiling rustls-webpki v0.101.7 [INFO] [stderr] Compiling serde_derive v1.0.217 [INFO] [stderr] Compiling futures-macro v0.3.31 [INFO] [stderr] Compiling tokio-macros v2.5.0 [INFO] [stderr] Compiling tracing-attributes v0.1.28 [INFO] [stderr] Compiling zerovec-derive v0.10.3 [INFO] [stderr] Compiling displaydoc v0.2.5 [INFO] [stderr] Compiling thiserror-impl v2.0.11 [INFO] [stderr] Compiling icu_provider_macros v1.5.0 [INFO] [stderr] Compiling zerofrom-derive v0.1.5 [INFO] [stderr] Compiling yoke-derive v0.7.5 [INFO] [stderr] Compiling thiserror-impl v1.0.69 [INFO] [stderr] Compiling async-trait v0.1.86 [INFO] [stderr] Compiling tokio v1.43.0 [INFO] [stderr] Compiling pin-project-internal v1.1.9 [INFO] [stderr] Compiling zerocopy-derive v0.7.35 [INFO] [stderr] Compiling futures-util v0.3.31 [INFO] [stderr] Compiling axum-macros v0.3.8 [INFO] [stderr] Compiling tracing v0.1.41 [INFO] [stderr] Compiling thiserror v2.0.11 [INFO] [stderr] Compiling thiserror v1.0.69 [INFO] [stderr] Compiling pin-project v1.1.9 [INFO] [stderr] Compiling pest v2.7.15 [INFO] [stderr] Compiling opentelemetry v0.21.0 [INFO] [stderr] Compiling tracing-error v0.2.1 [INFO] [stderr] Compiling tracing-futures v0.2.5 [INFO] [stderr] Compiling zerofrom v0.1.5 [INFO] [stderr] Compiling zerocopy v0.7.35 [INFO] [stderr] Compiling yoke v0.7.5 [INFO] [stderr] Compiling color-spantrace v0.2.1 [INFO] [stderr] Compiling darling_macro v0.20.10 [INFO] [stderr] Compiling color-eyre v0.6.3 [INFO] [stderr] Compiling zerovec v0.10.4 [INFO] [stderr] Compiling darling v0.20.10 [INFO] [stderr] Compiling derive_builder_core v0.20.2 [INFO] [stderr] Compiling ppv-lite86 v0.2.20 [INFO] [stderr] Compiling pest_meta v2.7.15 [INFO] [stderr] Compiling derive_builder_macro v0.20.2 [INFO] [stderr] Compiling rand_chacha v0.3.1 [INFO] [stderr] Compiling tinystr v0.7.6 [INFO] [stderr] Compiling icu_collections v1.5.0 [INFO] [stderr] Compiling derive_builder v0.20.2 [INFO] [stderr] Compiling rand v0.8.5 [INFO] [stderr] Compiling icu_locid v1.5.0 [INFO] [stderr] Compiling pest_generator v2.7.15 [INFO] [stderr] Compiling icu_provider v1.5.0 [INFO] [stderr] Compiling icu_locid_transform v1.5.0 [INFO] [stderr] Compiling pest_derive v2.7.15 [INFO] [stderr] Compiling futures-executor v0.3.31 [INFO] [stderr] Compiling futures v0.3.31 [INFO] [stderr] Compiling icu_properties v1.5.1 [INFO] [stderr] Compiling serde_urlencoded v0.7.1 [INFO] [stderr] Compiling serde_path_to_error v0.1.16 [INFO] [stderr] Compiling ron v0.7.1 [INFO] [stderr] Compiling json5 v0.4.1 [INFO] [stderr] Compiling toml v0.5.11 [INFO] [stderr] Compiling tokio-util v0.7.13 [INFO] [stderr] Compiling tokio-stream v0.1.17 [INFO] [stderr] Compiling cargo-platform v0.1.9 [INFO] [stderr] Compiling tower v0.4.13 [INFO] [stderr] Compiling tokio-rustls v0.24.1 [INFO] [stderr] Compiling rustc_version v0.4.1 [INFO] [stderr] Compiling opentelemetry_sdk v0.21.2 [INFO] [stderr] Compiling h2 v0.3.26 [INFO] [stderr] Compiling config v0.13.4 [INFO] [stderr] Compiling cargo_metadata v0.19.1 [INFO] [stderr] Compiling icu_normalizer v1.5.0 [INFO] [stderr] Compiling opentelemetry-prometheus v0.14.1 [INFO] [stderr] Compiling idna_adapter v1.2.0 [INFO] [stderr] Compiling idna v1.0.3 [INFO] [stderr] Compiling tracing-appender v0.2.3 [INFO] [stderr] Compiling url v2.5.4 [INFO] [stderr] Compiling matrix-hedwig v2.0.0 (/opt/rustwide/workdir) [INFO] [stderr] Compiling hyper v0.14.32 [INFO] [stderr] Compiling hyper-rustls v0.24.2 [INFO] [stderr] Compiling reqwest v0.11.27 [INFO] [stderr] Compiling gcp_auth v0.8.1 [INFO] [stderr] Compiling firebae-cm v0.1.0 [INFO] [stderr] Compiling axum-opentelemetry-middleware v0.2.1 (https://github.com/famedly/axum-opentelemetry-middleware.git?rev=f8da4a9642fcc2a5671e4f2eaa5831b94c62464f#f8da4a96) [INFO] [stderr] Compiling axum-extra v0.8.0 [INFO] [stderr] Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 50s [INFO] running `Command { std: "docker" "inspect" "c199f878be41f2b2878f4c09b573a6fa1e52920309bd3bdacc2f1fbe2d954e6a", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "c199f878be41f2b2878f4c09b573a6fa1e52920309bd3bdacc2f1fbe2d954e6a", kill_on_drop: false }` [INFO] [stdout] c199f878be41f2b2878f4c09b573a6fa1e52920309bd3bdacc2f1fbe2d954e6a [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:7ad1b28ee6f5f7f699f6cf7015098d6ccdd96d6f2d78dd06228f5b4c9faf309c" "/opt/rustwide/cargo-home/bin/cargo" "+ad85bc524b1ad696e42061ad8338d382dffbdbe5" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }` [INFO] [stdout] 1994a6ecf166679f25f30cebcc00bfdc7ea4f56d00b35694709cbe1a3b38e311 [INFO] running `Command { std: "docker" "start" "-a" "1994a6ecf166679f25f30cebcc00bfdc7ea4f56d00b35694709cbe1a3b38e311", kill_on_drop: false }` [INFO] [stderr] Compiling vergen v9.0.4 [INFO] [stderr] Compiling vergen-gitcl v1.0.5 [INFO] [stderr] Compiling matrix-hedwig v2.0.0 (/opt/rustwide/workdir) [INFO] [stderr] Finished `test` profile [unoptimized + debuginfo] target(s) in 17.38s [INFO] running `Command { std: "docker" "inspect" "1994a6ecf166679f25f30cebcc00bfdc7ea4f56d00b35694709cbe1a3b38e311", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "1994a6ecf166679f25f30cebcc00bfdc7ea4f56d00b35694709cbe1a3b38e311", kill_on_drop: false }` [INFO] [stdout] 1994a6ecf166679f25f30cebcc00bfdc7ea4f56d00b35694709cbe1a3b38e311 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:7ad1b28ee6f5f7f699f6cf7015098d6ccdd96d6f2d78dd06228f5b4c9faf309c" "/opt/rustwide/cargo-home/bin/cargo" "+ad85bc524b1ad696e42061ad8338d382dffbdbe5" "test" "--frozen", kill_on_drop: false }` [INFO] [stdout] a77a1a8d8d5df8c6ce4540b30f4a3e0403d2099c4dbefd7fe870bc4ebc12d93d [INFO] running `Command { std: "docker" "start" "-a" "a77a1a8d8d5df8c6ce4540b30f4a3e0403d2099c4dbefd7fe870bc4ebc12d93d", kill_on_drop: false }` [INFO] [stderr] Finished `test` profile [unoptimized + debuginfo] target(s) in 0.32s [INFO] [stderr] Running unittests src/lib.rs (/opt/rustwide/target/debug/deps/matrix_hedwig-e2cf413ba9b0df88) [INFO] [stdout] [INFO] [stderr] Running unittests src/main.rs (/opt/rustwide/target/debug/deps/matrix_hedwig-6e54086d5a6d15ff) [INFO] [stdout] running 0 tests [INFO] [stderr] Running tests/pusher.rs (/opt/rustwide/target/debug/deps/pusher-3b767914102d311f) [INFO] [stdout] [INFO] [stdout] test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s [INFO] [stdout] [INFO] [stdout] [INFO] [stdout] running 0 tests [INFO] [stdout] [INFO] [stdout] test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s [INFO] [stdout] [INFO] [stdout] [INFO] [stdout] running 6 tests [INFO] [stdout] test bad_json ... ok [INFO] [stdout] test push_body_limit ... ok [INFO] [stdout] test many_requests ... ok [INFO] [stdout] test normal_operation ... FAILED [INFO] [stdout] test fcm_failure ... ok [INFO] [stdout] test many_devices ... FAILED [INFO] [stdout] [INFO] [stdout] failures: [INFO] [stdout] [INFO] [stdout] ---- normal_operation stdout ---- [INFO] [stdout] OpenTelemetry metrics error occurred. Metrics error: reader is shut down or not registered [INFO] [stdout] [INFO] [stdout] thread 'normal_operation' (30) panicked at tests/pusher.rs:211:5: [INFO] [stdout] assertion `left == right` failed [INFO] [stdout] left: "" [INFO] [stdout] right: "# TYPE devices_total counter\ndevices_total{otel_scope_name=\"axum-opentelemetry\"} 8\n# TYPE http_requests_duration_seconds histogram\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"0\"} 0\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"5\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"10\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"25\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"50\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"75\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"100\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"250\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"500\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"750\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"1000\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"2500\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"5000\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"7500\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"10000\"} 8\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"+Inf\"} 8\nhttp_requests_duration_seconds_sum{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} FLOAT\nhttp_requests_duration_seconds_count{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} 8\n# TYPE http_requests_total counter\nhttp_requests_total{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} 8\n# TYPE notifications_total counter\nnotifications_total{otel_scope_name=\"axum-opentelemetry\"} 8\n# HELP otel_scope_info Instrumentation Scope metadata\n# TYPE otel_scope_info gauge\notel_scope_info{otel_scope_name=\"axum-opentelemetry\"} 1\n# HELP pushes_successful_total Successful pushes\n# TYPE pushes_successful_total counter\npushes_successful_total{device_type=\"Android\",otel_scope_name=\"axum-opentelemetry\"} 2\npushes_successful_total{device_type=\"AndroidLegacy\",otel_scope_name=\"axum-opentelemetry\"} 2\npushes_successful_total{device_type=\"Ios\",otel_scope_name=\"axum-opentelemetry\"} 2\npushes_successful_total{device_type=\"None\",otel_scope_name=\"axum-opentelemetry\"} 2\n# HELP target_info Target metadata\n# TYPE target_info gauge\ntarget_info{service_name=\"Hedwig\"} 1\n" [INFO] [stdout] stack backtrace: [INFO] [stdout] 0: 0x647f8140b622 - std::backtrace_rs::backtrace::libunwind::trace::h52580dd202462214 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9 [INFO] [stdout] 1: 0x647f8140b622 - std::backtrace_rs::backtrace::trace_unsynchronized::hc969519abce0f52b [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14 [INFO] [stdout] 2: 0x647f8140b622 - std::sys::backtrace::_print_fmt::hfd5825900b6e0030 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/backtrace.rs:66:9 [INFO] [stdout] 3: 0x647f8140b622 - ::fmt::h427144ad75cfb218 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/backtrace.rs:39:26 [INFO] [stdout] 4: 0x647f8141ee7f - core::fmt::rt::Argument::fmt::hd5ccc9cf97cea7f7 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/fmt/rt.rs:173:76 [INFO] [stdout] 5: 0x647f8141ee7f - core::fmt::write::h593aaf5adf0f5dae [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/fmt/mod.rs:1468:25 [INFO] [stdout] 6: 0x647f813d81a1 - std::io::default_write_fmt::h305b5a0727055346 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/io/mod.rs:639:11 [INFO] [stdout] 7: 0x647f813d81a1 - std::io::Write::write_fmt::h76f14a8a88733dc5 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/io/mod.rs:1954:13 [INFO] [stdout] 8: 0x647f813e43a2 - std::sys::backtrace::BacktraceLock::print::h3953113552dca3ca [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/backtrace.rs:42:9 [INFO] [stdout] 9: 0x647f813e91cf - std::panicking::default_hook::{{closure}}::h486b96ad75eafbb8 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:301:27 [INFO] [stdout] 10: 0x647f813e9061 - std::panicking::default_hook::h99456317c5a1a20c [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:325:9 [INFO] [stdout] 11: 0x647f80c3c3fe - as core::ops::function::Fn>::call::h441e4551014e4cf8 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/alloc/src/boxed.rs:1999:9 [INFO] [stdout] 12: 0x647f80c3c3fe - test::test_main_with_exit_callback::{{closure}}::hdbca54e17f1a00e4 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:145:21 [INFO] [stdout] 13: 0x647f813e993f - as core::ops::function::Fn>::call::h1e57b72c4e1d3045 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/alloc/src/boxed.rs:1999:9 [INFO] [stdout] 14: 0x647f813e993f - std::panicking::panic_with_hook::h67ab0df20212e4ea [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:842:13 [INFO] [stdout] 15: 0x647f813e96ea - std::panicking::panic_handler::{{closure}}::hc9e4a933ae92e208 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:707:13 [INFO] [stdout] 16: 0x647f813e44e9 - std::sys::backtrace::__rust_end_short_backtrace::hbfa72df9b68c2d19 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/backtrace.rs:174:18 [INFO] [stdout] 17: 0x647f813cbc5d - __rustc[8cce077e14951490]::rust_begin_unwind [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:698:5 [INFO] [stdout] 18: 0x647f81427e80 - core::panicking::panic_fmt::h49931053d20abf41 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/panicking.rs:75:14 [INFO] [stdout] 19: 0x647f81427c33 - core::panicking::assert_failed_inner::h3eabd3c15a9e5a8c [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/panicking.rs:439:17 [INFO] [stdout] 20: 0x647f80c12260 - core::panicking::assert_failed::h5f65a5c29ea06f36 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/panicking.rs:394:5 [INFO] [stdout] 21: 0x647f80bec0e2 - pusher::check_prom::{{closure}}::hef21dd8b227ac4e3 [INFO] [stdout] at /opt/rustwide/workdir/tests/pusher.rs:211:2 [INFO] [stdout] 22: 0x647f80bf7c0b - pusher::normal_operation::{{closure}}::hed35e2ae28d6ca3f [INFO] [stdout] at /opt/rustwide/workdir/tests/pusher.rs:315:68 [INFO] [stdout] 23: 0x647f80c027af - as core::future::future::Future>::poll::hf86d1c0bfa71c252 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/future/future.rs:133:9 [INFO] [stdout] 24: 0x647f80c0268b - as core::future::future::Future>::poll::h256a8f5b48d29ea4 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/future/future.rs:133:9 [INFO] [stdout] 25: 0x647f80bfecbb - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}::h2e8bb79f65308330 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:73 [INFO] [stdout] 26: 0x647f80bfea2b - tokio::runtime::coop::with_budget::hb779138fedc439e4 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:107:5 [INFO] [stdout] 27: 0x647f80bfea2b - tokio::runtime::coop::budget::h52967e5077c441fb [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:73:5 [INFO] [stdout] 28: 0x647f80bfea2b - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::h4466e3c03afe4187 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:25 [INFO] [stdout] 29: 0x647f80bfbaad - tokio::runtime::scheduler::current_thread::Context::enter::h48287663cfe036f8 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:428:19 [INFO] [stdout] 30: 0x647f80bfd4f0 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::h69c3270dc3545338 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:728:44 [INFO] [stdout] 31: 0x647f80bfd14b - tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}::h126abddb9167d8c3 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:68 [INFO] [stdout] 32: 0x647f80c1309a - tokio::runtime::context::scoped::Scoped::set::h22ee5bb5e6f2ec1f [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/scoped.rs:40:9 [INFO] [stdout] 33: 0x647f80c0761a - tokio::runtime::context::set_scheduler::{{closure}}::h59dc71f6aa0d5f71 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:38 [INFO] [stdout] 34: 0x647f80bf9b45 - std::thread::local::LocalKey::try_with::h2dd0784d7d3eb36b [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/thread/local.rs:315:12 [INFO] [stdout] 35: 0x647f80bf951d - std::thread::local::LocalKey::with::h33a8bd1c68c97ff5 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/thread/local.rs:279:20 [INFO] [stdout] 36: 0x647f80c07511 - tokio::runtime::context::set_scheduler::h1a282db079d11e7d [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:17 [INFO] [stdout] 37: 0x647f80bfceae - tokio::runtime::scheduler::current_thread::CoreGuard::enter::hd6337026b1648158 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:27 [INFO] [stdout] 38: 0x647f80bfd220 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::h456307cd457a67a8 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:716:24 [INFO] [stdout] 39: 0x647f80bfadb6 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}::h02a439719dea7925 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:196:33 [INFO] [stdout] 40: 0x647f80c17bac - tokio::runtime::context::runtime::enter_runtime::h0a586887d3de3ffd [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/runtime.rs:65:16 [INFO] [stdout] 41: 0x647f80bfaa77 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::h7b6af901e9ff3bfc [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:184:9 [INFO] [stdout] 42: 0x647f80c0a980 - tokio::runtime::runtime::Runtime::block_on_inner::h0deb3ab368604cc0 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:368:52 [INFO] [stdout] 43: 0x647f80c0ad0f - tokio::runtime::runtime::Runtime::block_on::hc48a1ccc261d84b3 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:342:18 [INFO] [stdout] 44: 0x647f80bf68e3 - pusher::normal_operation::hf994a17e61c9bdc5 [INFO] [stdout] at /opt/rustwide/workdir/tests/pusher.rs:317:4 [INFO] [stdout] 45: 0x647f80bf69a7 - pusher::normal_operation::{{closure}}::hca5244e7d2ed7fcf [INFO] [stdout] at /opt/rustwide/workdir/tests/pusher.rs:291:32 [INFO] [stdout] 46: 0x647f80c19746 - core::ops::function::FnOnce::call_once::hcc8414a94d586881 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/ops/function.rs:253:5 [INFO] [stdout] 47: 0x647f80c3c1bb - core::ops::function::FnOnce::call_once::h5d9016d521f5aee4 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/ops/function.rs:253:5 [INFO] [stdout] 48: 0x647f80c3c1bb - test::__rust_begin_short_backtrace::h123fcee610330442 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:663:18 [INFO] [stdout] 49: 0x647f80c51ca5 - test::run_test_in_process::{{closure}}::h0442efda2e6359b2 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:686:74 [INFO] [stdout] 50: 0x647f80c51ca5 - as core::ops::function::FnOnce<()>>::call_once::hfe3528c8dd09682c [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/panic/unwind_safe.rs:274:9 [INFO] [stdout] 51: 0x647f80c51ca5 - std::panicking::catch_unwind::do_call::hb1659f7844fe8531 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:590:40 [INFO] [stdout] 52: 0x647f80c51ca5 - std::panicking::catch_unwind::hdc1f4be0019e72ad [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:553:19 [INFO] [stdout] 53: 0x647f80c51ca5 - std::panic::catch_unwind::hd6d7ccfa4513e770 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panic.rs:359:14 [INFO] [stdout] 54: 0x647f80c51ca5 - test::run_test_in_process::h6bbb839ed0802366 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:686:27 [INFO] [stdout] 55: 0x647f80c51ca5 - test::run_test::{{closure}}::he4fad7398c8acc11 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:607:43 [INFO] [stdout] 56: 0x647f80c28614 - test::run_test::{{closure}}::h41975797e198d26c [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:637:41 [INFO] [stdout] 57: 0x647f80c28614 - std::sys::backtrace::__rust_begin_short_backtrace::hfb46c397857336f1 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/backtrace.rs:158:18 [INFO] [stdout] 58: 0x647f80c2bfba - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hddf826a64f532bea [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/thread/mod.rs:559:17 [INFO] [stdout] 59: 0x647f80c2bfba - as core::ops::function::FnOnce<()>>::call_once::hc3518c3eab91aeb3 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/panic/unwind_safe.rs:274:9 [INFO] [stdout] 60: 0x647f80c2bfba - std::panicking::catch_unwind::do_call::h7bf3ad6dc012415a [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:590:40 [INFO] [stdout] 61: 0x647f80c2bfba - std::panicking::catch_unwind::h8b88f0375c767e36 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:553:19 [INFO] [stdout] 62: 0x647f80c2bfba - std::panic::catch_unwind::h66cb5d61a90f7bc5 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panic.rs:359:14 [INFO] [stdout] 63: 0x647f80c2bfba - std::thread::Builder::spawn_unchecked_::{{closure}}::h65bc9d15fb966bbd [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/thread/mod.rs:557:30 [INFO] [stdout] 64: 0x647f80c2bfba - core::ops::function::FnOnce::call_once{{vtable.shim}}::he8d2c9a8796342e5 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/ops/function.rs:253:5 [INFO] [stdout] 65: 0x647f813de60f - as core::ops::function::FnOnce>::call_once::hef12b47e5c033835 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/alloc/src/boxed.rs:1985:9 [INFO] [stdout] 66: 0x647f813de60f - std::sys::pal::unix::thread::Thread::new::thread_start::h135087e7c97657bc [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/pal/unix/thread.rs:118:17 [INFO] [stdout] 67: 0x7d2a242f3aa4 - [INFO] [stdout] 68: 0x7d2a24380a34 - clone [INFO] [stdout] 69: 0x0 - [INFO] [stdout] [INFO] [stdout] ---- many_devices stdout ---- [INFO] [stdout] OpenTelemetry metrics error occurred. Metrics error: reader is shut down or not registered [INFO] [stdout] [INFO] [stdout] thread 'many_devices' (28) panicked at tests/pusher.rs:211:5: [INFO] [stdout] assertion `left == right` failed [INFO] [stdout] left: "" [INFO] [stdout] right: "# TYPE devices_total counter\ndevices_total{otel_scope_name=\"axum-opentelemetry\"} 12\n# TYPE http_requests_duration_seconds histogram\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"0\"} 0\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"5\"} 2\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"10\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"25\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"50\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"75\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"100\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"250\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"500\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"750\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"1000\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"2500\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"5000\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"7500\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"10000\"} 3\nhttp_requests_duration_seconds_bucket{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\",le=\"+Inf\"} 3\nhttp_requests_duration_seconds_sum{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} FLOAT\nhttp_requests_duration_seconds_count{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} 3\n# TYPE http_requests_total counter\nhttp_requests_total{endpoint=\"/_matrix/push/v1/notify\",method=\"POST\",status=\"200\",otel_scope_name=\"axum-opentelemetry\"} 3\n# TYPE notifications_total counter\nnotifications_total{otel_scope_name=\"axum-opentelemetry\"} 3\n# HELP otel_scope_info Instrumentation Scope metadata\n# TYPE otel_scope_info gauge\notel_scope_info{otel_scope_name=\"axum-opentelemetry\"} 1\n# HELP pushes_failed_total Failed pushes\n# TYPE pushes_failed_total counter\npushes_failed_total{device_type=\"AndroidLegacy\",otel_scope_name=\"axum-opentelemetry\"} 1\npushes_failed_total{device_type=\"None\",otel_scope_name=\"axum-opentelemetry\"} 1\n# HELP pushes_successful_total Successful pushes\n# TYPE pushes_successful_total counter\npushes_successful_total{device_type=\"Android\",otel_scope_name=\"axum-opentelemetry\"} 3\npushes_successful_total{device_type=\"AndroidLegacy\",otel_scope_name=\"axum-opentelemetry\"} 2\npushes_successful_total{device_type=\"Ios\",otel_scope_name=\"axum-opentelemetry\"} 3\npushes_successful_total{device_type=\"None\",otel_scope_name=\"axum-opentelemetry\"} 2\n# HELP target_info Target metadata\n# TYPE target_info gauge\ntarget_info{service_name=\"Hedwig\"} 1\n" [INFO] [stdout] stack backtrace: [INFO] [stdout] 0: 0x647f8140b622 - std::backtrace_rs::backtrace::libunwind::trace::h52580dd202462214 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9 [INFO] [stdout] 1: 0x647f8140b622 - std::backtrace_rs::backtrace::trace_unsynchronized::hc969519abce0f52b [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14 [INFO] [stdout] 2: 0x647f8140b622 - std::sys::backtrace::_print_fmt::hfd5825900b6e0030 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/backtrace.rs:66:9 [INFO] [stdout] 3: 0x647f8140b622 - ::fmt::h427144ad75cfb218 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/backtrace.rs:39:26 [INFO] [stdout] 4: 0x647f8141ee7f - core::fmt::rt::Argument::fmt::hd5ccc9cf97cea7f7 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/fmt/rt.rs:173:76 [INFO] [stdout] 5: 0x647f8141ee7f - core::fmt::write::h593aaf5adf0f5dae [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/fmt/mod.rs:1468:25 [INFO] [stdout] 6: 0x647f813d81a1 - std::io::default_write_fmt::h305b5a0727055346 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/io/mod.rs:639:11 [INFO] [stdout] 7: 0x647f813d81a1 - std::io::Write::write_fmt::h76f14a8a88733dc5 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/io/mod.rs:1954:13 [INFO] [stdout] 8: 0x647f813e43a2 - std::sys::backtrace::BacktraceLock::print::h3953113552dca3ca [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/backtrace.rs:42:9 [INFO] [stdout] 9: 0x647f813e91cf - std::panicking::default_hook::{{closure}}::h486b96ad75eafbb8 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:301:27 [INFO] [stdout] 10: 0x647f813e9061 - std::panicking::default_hook::h99456317c5a1a20c [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:325:9 [INFO] [stdout] 11: 0x647f80c3c3fe - as core::ops::function::Fn>::call::h441e4551014e4cf8 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/alloc/src/boxed.rs:1999:9 [INFO] [stdout] 12: 0x647f80c3c3fe - test::test_main_with_exit_callback::{{closure}}::hdbca54e17f1a00e4 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:145:21 [INFO] [stdout] 13: 0x647f813e993f - as core::ops::function::Fn>::call::h1e57b72c4e1d3045 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/alloc/src/boxed.rs:1999:9 [INFO] [stdout] 14: 0x647f813e993f - std::panicking::panic_with_hook::h67ab0df20212e4ea [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:842:13 [INFO] [stdout] 15: 0x647f813e96ea - std::panicking::panic_handler::{{closure}}::hc9e4a933ae92e208 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:707:13 [INFO] [stdout] 16: 0x647f813e44e9 - std::sys::backtrace::__rust_end_short_backtrace::hbfa72df9b68c2d19 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/backtrace.rs:174:18 [INFO] [stdout] 17: 0x647f813cbc5d - __rustc[8cce077e14951490]::rust_begin_unwind [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:698:5 [INFO] [stdout] 18: 0x647f81427e80 - core::panicking::panic_fmt::h49931053d20abf41 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/panicking.rs:75:14 [INFO] [stdout] 19: 0x647f81427c33 - core::panicking::assert_failed_inner::h3eabd3c15a9e5a8c [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/panicking.rs:439:17 [INFO] [stdout] 20: 0x647f80c12260 - core::panicking::assert_failed::h5f65a5c29ea06f36 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/panicking.rs:394:5 [INFO] [stdout] 21: 0x647f80bec0e2 - pusher::check_prom::{{closure}}::hef21dd8b227ac4e3 [INFO] [stdout] at /opt/rustwide/workdir/tests/pusher.rs:211:2 [INFO] [stdout] 22: 0x647f80bf1ae2 - pusher::many_devices::{{closure}}::h223cd7f9f0364f1e [INFO] [stdout] at /opt/rustwide/workdir/tests/pusher.rs:372:64 [INFO] [stdout] 23: 0x647f80c027af - as core::future::future::Future>::poll::hf86d1c0bfa71c252 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/future/future.rs:133:9 [INFO] [stdout] 24: 0x647f80c0268b - as core::future::future::Future>::poll::h256a8f5b48d29ea4 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/future/future.rs:133:9 [INFO] [stdout] 25: 0x647f80bfecbb - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}::h2e8bb79f65308330 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:73 [INFO] [stdout] 26: 0x647f80bfea2b - tokio::runtime::coop::with_budget::hb779138fedc439e4 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:107:5 [INFO] [stdout] 27: 0x647f80bfea2b - tokio::runtime::coop::budget::h52967e5077c441fb [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:73:5 [INFO] [stdout] 28: 0x647f80bfea2b - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::h4466e3c03afe4187 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:25 [INFO] [stdout] 29: 0x647f80bfbaad - tokio::runtime::scheduler::current_thread::Context::enter::h48287663cfe036f8 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:428:19 [INFO] [stdout] 30: 0x647f80bfd4f0 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::h69c3270dc3545338 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:728:44 [INFO] [stdout] 31: 0x647f80bfd14b - tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}::h126abddb9167d8c3 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:68 [INFO] [stdout] 32: 0x647f80c1309a - tokio::runtime::context::scoped::Scoped::set::h22ee5bb5e6f2ec1f [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/scoped.rs:40:9 [INFO] [stdout] 33: 0x647f80c0761a - tokio::runtime::context::set_scheduler::{{closure}}::h59dc71f6aa0d5f71 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:38 [INFO] [stdout] 34: 0x647f80bf9b45 - std::thread::local::LocalKey::try_with::h2dd0784d7d3eb36b [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/thread/local.rs:315:12 [INFO] [stdout] 35: 0x647f80bf951d - std::thread::local::LocalKey::with::h33a8bd1c68c97ff5 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/thread/local.rs:279:20 [INFO] [stdout] 36: 0x647f80c07511 - tokio::runtime::context::set_scheduler::h1a282db079d11e7d [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:17 [INFO] [stdout] 37: 0x647f80bfceae - tokio::runtime::scheduler::current_thread::CoreGuard::enter::hd6337026b1648158 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:27 [INFO] [stdout] 38: 0x647f80bfd220 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::h456307cd457a67a8 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:716:24 [INFO] [stdout] 39: 0x647f80bfadb6 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}::h02a439719dea7925 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:196:33 [INFO] [stdout] 40: 0x647f80c17bac - tokio::runtime::context::runtime::enter_runtime::h0a586887d3de3ffd [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/runtime.rs:65:16 [INFO] [stdout] 41: 0x647f80bfaa77 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::h7b6af901e9ff3bfc [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:184:9 [INFO] [stdout] 42: 0x647f80c0a980 - tokio::runtime::runtime::Runtime::block_on_inner::h0deb3ab368604cc0 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:368:52 [INFO] [stdout] 43: 0x647f80c0ad0f - tokio::runtime::runtime::Runtime::block_on::hc48a1ccc261d84b3 [INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:342:18 [INFO] [stdout] 44: 0x647f80bf01a3 - pusher::many_devices::h7865b59b31206fff [INFO] [stdout] at /opt/rustwide/workdir/tests/pusher.rs:374:4 [INFO] [stdout] 45: 0x647f80bf1d47 - pusher::many_devices::{{closure}}::h5566921b1af79fbb [INFO] [stdout] at /opt/rustwide/workdir/tests/pusher.rs:336:28 [INFO] [stdout] 46: 0x647f80c19556 - core::ops::function::FnOnce::call_once::h3a1ace58b0b44910 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/ops/function.rs:253:5 [INFO] [stdout] 47: 0x647f80c3c1bb - core::ops::function::FnOnce::call_once::h5d9016d521f5aee4 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/ops/function.rs:253:5 [INFO] [stdout] 48: 0x647f80c3c1bb - test::__rust_begin_short_backtrace::h123fcee610330442 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:663:18 [INFO] [stdout] 49: 0x647f80c51ca5 - test::run_test_in_process::{{closure}}::h0442efda2e6359b2 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:686:74 [INFO] [stdout] 50: 0x647f80c51ca5 - as core::ops::function::FnOnce<()>>::call_once::hfe3528c8dd09682c [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/panic/unwind_safe.rs:274:9 [INFO] [stdout] 51: 0x647f80c51ca5 - std::panicking::catch_unwind::do_call::hb1659f7844fe8531 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:590:40 [INFO] [stdout] 52: 0x647f80c51ca5 - std::panicking::catch_unwind::hdc1f4be0019e72ad [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:553:19 [INFO] [stdout] 53: 0x647f80c51ca5 - std::panic::catch_unwind::hd6d7ccfa4513e770 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panic.rs:359:14 [INFO] [stdout] 54: 0x647f80c51ca5 - test::run_test_in_process::h6bbb839ed0802366 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:686:27 [INFO] [stdout] 55: 0x647f80c51ca5 - test::run_test::{{closure}}::he4fad7398c8acc11 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:607:43 [INFO] [stdout] 56: 0x647f80c28614 - test::run_test::{{closure}}::h41975797e198d26c [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/test/src/lib.rs:637:41 [INFO] [stdout] 57: 0x647f80c28614 - std::sys::backtrace::__rust_begin_short_backtrace::hfb46c397857336f1 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/backtrace.rs:158:18 [INFO] [stdout] 58: 0x647f80c2bfba - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hddf826a64f532bea [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/thread/mod.rs:559:17 [INFO] [stdout] 59: 0x647f80c2bfba - as core::ops::function::FnOnce<()>>::call_once::hc3518c3eab91aeb3 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/panic/unwind_safe.rs:274:9 [INFO] [stdout] 60: 0x647f80c2bfba - std::panicking::catch_unwind::do_call::h7bf3ad6dc012415a [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:590:40 [INFO] [stdout] 61: 0x647f80c2bfba - std::panicking::catch_unwind::h8b88f0375c767e36 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panicking.rs:553:19 [INFO] [stdout] 62: 0x647f80c2bfba - std::panic::catch_unwind::h66cb5d61a90f7bc5 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/panic.rs:359:14 [INFO] [stdout] 63: 0x647f80c2bfba - std::thread::Builder::spawn_unchecked_::{{closure}}::h65bc9d15fb966bbd [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/thread/mod.rs:557:30 [INFO] [stdout] 64: 0x647f80c2bfba - core::ops::function::FnOnce::call_once{{vtable.shim}}::he8d2c9a8796342e5 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/core/src/ops/function.rs:253:5 [INFO] [stdout] 65: 0x647f813de60f - as core::ops::function::FnOnce>::call_once::hef12b47e5c033835 [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/alloc/src/boxed.rs:1985:9 [INFO] [stdout] 66: 0x647f813de60f - std::sys::pal::unix::thread::Thread::new::thread_start::h135087e7c97657bc [INFO] [stdout] at /rustc/ad85bc524b1ad696e42061ad8338d382dffbdbe5/library/std/src/sys/pal/unix/thread.rs:118:17 [INFO] [stdout] 67: 0x7d2a242f3aa4 - [INFO] [stdout] 68: 0x7d2a24380a34 - clone [INFO] [stdout] 69: 0x0 - [INFO] [stdout] [INFO] [stdout] [INFO] [stdout] failures: [INFO] [stdout] many_devices [INFO] [stdout] normal_operation [INFO] [stdout] [INFO] [stdout] test result: FAILED. 4 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.52s [INFO] [stdout] [INFO] [stderr] error: test failed, to rerun pass `--test pusher` [INFO] running `Command { std: "docker" "inspect" "a77a1a8d8d5df8c6ce4540b30f4a3e0403d2099c4dbefd7fe870bc4ebc12d93d", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "a77a1a8d8d5df8c6ce4540b30f4a3e0403d2099c4dbefd7fe870bc4ebc12d93d", kill_on_drop: false }` [INFO] [stdout] a77a1a8d8d5df8c6ce4540b30f4a3e0403d2099c4dbefd7fe870bc4ebc12d93d