[INFO] cloning repository https://github.com/aopilatov/manifold
[INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/aopilatov/manifold" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Faopilatov%2Fmanifold", kill_on_drop: false }`
[INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Faopilatov%2Fmanifold'...
[INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }`
[INFO] [stdout] ba1fe719f42d6947949766f182d7a4fbb70e17e4
[INFO] testing aopilatov/manifold against 1.98.0-beta.1 for beta-1.98-1
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Faopilatov%2Fmanifold" "/workspace/builds/worker-1-tc2/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-1-tc2/source'...
[INFO] [stderr] done.
[INFO] started tweaking git repo https://github.com/aopilatov/manifold
[INFO] finished tweaking git repo https://github.com/aopilatov/manifold
[INFO] tweaked toml for git repo https://github.com/aopilatov/manifold written to /workspace/builds/worker-1-tc2/source/Cargo.toml
[INFO] validating manifest of git repo https://github.com/aopilatov/manifold on toolchain 1.98.0-beta.1
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.98.0-beta.1" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate git repo https://github.com/aopilatov/manifold already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.98.0-beta.1" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-m" "1610612736" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:3d5ced03c013a94a2f102a4510f48a6e9184255caf5fd8244f58017bde7f5210" "sleep" "infinity", kill_on_drop: false }`
[INFO] [stdout] 16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b
[INFO] running `Command { std: "docker" "start" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-w" "/opt/rustwide/workdir" "--user" "0:0" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b" "/opt/rustwide/cargo-home/bin/cargo" "+1.98.0-beta.1" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b" "/opt/rustwide/cargo-home/bin/cargo" "+1.98.0-beta.1" "build" "--frozen" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr]    Compiling quote v1.0.46
[INFO] [stderr]    Compiling smallvec v1.15.2
[INFO] [stderr]    Compiling bytes v1.12.0
[INFO] [stderr]    Compiling memchr v2.8.2
[INFO] [stderr]    Compiling log v0.4.33
[INFO] [stderr]    Compiling either v1.16.0
[INFO] [stderr]    Compiling mio v1.2.1
[INFO] [stderr]    Compiling getrandom v0.2.17
[INFO] [stderr]    Compiling anyhow v1.0.103
[INFO] [stderr]    Compiling zerocopy v0.8.52
[INFO] [stderr]    Compiling generic-array v0.14.7
[INFO] [stderr]    Compiling cc v1.2.65
[INFO] [stderr]    Compiling libc v0.2.186
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]    Compiling getrandom v0.4.3
[INFO] [stderr]    Compiling serde_json v1.0.150
[INFO] [stderr]    Compiling rustix v1.1.4
[INFO] [stderr]    Compiling parking_lot_core v0.9.12
[INFO] [stderr]    Compiling rand_core v0.6.4
[INFO] [stderr]    Compiling bitflags v2.13.0
[INFO] [stderr]    Compiling regex-syntax v0.8.11
[INFO] [stderr]    Compiling prettyplease v0.2.37
[INFO] [stderr]    Compiling linux-raw-sys v0.12.1
[INFO] [stderr]    Compiling syn v2.0.118
[INFO] [stderr]    Compiling parking_lot v0.12.5
[INFO] [stderr]    Compiling itertools v0.14.0
[INFO] [stderr]    Compiling fastrand v2.4.1
[INFO] [stderr]    Compiling http v1.4.2
[INFO] [stderr]    Compiling crypto-common v0.1.7
[INFO] [stderr]    Compiling block-buffer v0.10.4
[INFO] [stderr]    Compiling digest v0.10.7
[INFO] [stderr]    Compiling fixedbitset v0.5.7
[INFO] [stderr]    Compiling indexmap v1.9.3
[INFO] [stderr]    Compiling multimap v0.10.1
[INFO] [stderr]    Compiling utf-8 v0.7.6
[INFO] [stderr]    Compiling data-encoding v2.11.0
[INFO] [stderr]    Compiling num-integer v0.1.46
[INFO] [stderr]    Compiling sha1 v0.10.6
[INFO] [stderr]    Compiling protoc-bin-vendored-linux-x86_64 v3.2.0
[INFO] [stderr]    Compiling protoc-bin-vendored-macos-x86_64 v3.2.0
[INFO] [stderr]    Compiling petgraph v0.7.1
[INFO] [stderr]    Compiling protoc-bin-vendored-win32 v3.2.0
[INFO] [stderr]    Compiling protoc-bin-vendored-linux-aarch_64 v3.2.0
[INFO] [stderr]    Compiling protoc-bin-vendored-macos-aarch_64 v3.2.0
[INFO] [stderr]    Compiling hashbrown v0.12.3
[INFO] [stderr]    Compiling protoc-bin-vendored-linux-x86_32 v3.2.0
[INFO] [stderr]    Compiling protoc-bin-vendored-linux-s390_64 v3.2.0
[INFO] [stderr]    Compiling protoc-bin-vendored-linux-ppcle_64 v3.2.0
[INFO] [stderr]    Compiling zeroize v1.9.0
[INFO] [stderr]    Compiling num-bigint v0.4.6
[INFO] [stderr]    Compiling serde_path_to_error v0.1.20
[INFO] [stderr]    Compiling protoc-bin-vendored v3.2.0
[INFO] [stderr]    Compiling rustls-pki-types v1.14.1
[INFO] [stderr]    Compiling ring v0.17.14
[INFO] [stderr]    Compiling socket2 v0.5.10
[INFO] [stderr]    Compiling http-body v1.0.1
[INFO] [stderr]    Compiling http-body-util v0.1.3
[INFO] [stderr]    Compiling regex-automata v0.4.14
[INFO] [stderr]    Compiling num-conv v0.2.2
[INFO] [stderr]    Compiling time-core v0.1.9
[INFO] [stderr]    Compiling rustls v0.23.41
[INFO] [stderr]    Compiling deranged v0.5.8
[INFO] [stderr]    Compiling mime_guess v2.0.5
[INFO] [stderr]    Compiling itertools v0.13.0
[INFO] [stderr]    Compiling time-macros v0.2.30
[INFO] [stderr]    Compiling backon v1.6.0
[INFO] [stderr]    Compiling sha1_smol v1.0.1
[INFO] [stderr]    Compiling hashbrown v0.14.5
[INFO] [stderr]    Compiling webpki-roots v1.0.8
[INFO] [stderr]    Compiling humantime v2.3.0
[INFO] [stderr]    Compiling regex v1.12.4
[INFO] [stderr]    Compiling http-range-header v0.4.2
[INFO] [stderr]    Compiling nom v7.1.3
[INFO] [stderr]    Compiling crossbeam-channel v0.5.15
[INFO] [stderr]    Compiling tracing-log v0.2.0
[INFO] [stderr]    Compiling tempfile v3.27.0
[INFO] [stderr]    Compiling dashmap v6.2.1
[INFO] [stderr]    Compiling time v0.3.51
[INFO] [stderr]    Compiling hdrhistogram v7.5.4
[INFO] [stderr]    Compiling synstructure v0.13.2
[INFO] [stderr]    Compiling ppv-lite86 v0.2.21
[INFO] [stderr]    Compiling tokio-macros v2.7.0
[INFO] [stderr]    Compiling zerofrom-derive v0.1.7
[INFO] [stderr]    Compiling futures-macro v0.3.32
[INFO] [stderr]    Compiling tracing-attributes v0.1.31
[INFO] [stderr]    Compiling yoke-derive v0.8.2
[INFO] [stderr]    Compiling serde_derive v1.0.228
[INFO] [stderr]    Compiling zerovec-derive v0.11.3
[INFO] [stderr]    Compiling displaydoc v0.2.6
[INFO] [stderr]    Compiling prost-derive v0.13.5
[INFO] [stderr]    Compiling thiserror-impl v1.0.69
[INFO] [stderr]    Compiling rand_chacha v0.3.1
[INFO] [stderr]    Compiling async-trait v0.1.89
[INFO] [stderr]    Compiling futures-util v0.3.32
[INFO] [stderr]    Compiling rand v0.8.6
[INFO] [stderr]    Compiling pin-project-internal v1.1.13
[INFO] [stderr]    Compiling thiserror-impl v2.0.18
[INFO] [stderr]    Compiling tokio v1.52.3
[INFO] [stderr]    Compiling tracing v0.1.44
[INFO] [stderr]    Compiling async-stream-impl v0.3.6
[INFO] [stderr]    Compiling rustls-webpki v0.103.13
[INFO] [stderr]    Compiling thiserror v1.0.69
[INFO] [stderr]    Compiling matchers v0.2.0
[INFO] [stderr]    Compiling clap_derive v4.6.1
[INFO] [stderr]    Compiling async-stream v0.3.6
[INFO] [stderr]    Compiling tungstenite v0.24.0
[INFO] [stderr]    Compiling zerofrom v0.1.8
[INFO] [stderr]    Compiling prost v0.13.5
[INFO] [stderr]    Compiling yoke v0.8.3
[INFO] [stderr]    Compiling thiserror v2.0.18
[INFO] [stderr]    Compiling simple_asn1 v0.6.4
[INFO] [stderr]    Compiling zerovec v0.11.6
[INFO] [stderr]    Compiling zerotrie v0.2.4
[INFO] [stderr]    Compiling pin-project v1.1.13
[INFO] [stderr]    Compiling serde v1.0.228
[INFO] [stderr]    Compiling prost-types v0.13.5
[INFO] [stderr]    Compiling tinystr v0.8.3
[INFO] [stderr]    Compiling potential_utf v0.1.5
[INFO] [stderr]    Compiling icu_collections v2.2.0
[INFO] [stderr]    Compiling icu_locale_core v2.2.0
[INFO] [stderr]    Compiling clap v4.6.1
[INFO] [stderr]    Compiling serde_urlencoded v0.7.1
[INFO] [stderr]    Compiling toml_datetime v0.6.11
[INFO] [stderr]    Compiling serde_spanned v0.6.9
[INFO] [stderr]    Compiling jsonwebtoken v9.3.1
[INFO] [stderr]    Compiling tracing-serde v0.2.0
[INFO] [stderr]    Compiling humantime-serde v1.1.1
[INFO] [stderr]    Compiling tracing-subscriber v0.3.23
[INFO] [stderr]    Compiling toml_edit v0.22.27
[INFO] [stderr]    Compiling prost-build v0.13.5
[INFO] [stderr]    Compiling axum-core v0.4.5
[INFO] [stderr]    Compiling futures-executor v0.3.32
[INFO] [stderr]    Compiling futures v0.3.32
[INFO] [stderr]    Compiling tonic-build v0.12.3
[INFO] [stderr]    Compiling icu_provider v2.2.0
[INFO] [stderr]    Compiling icu_normalizer v2.2.0
[INFO] [stderr]    Compiling icu_properties v2.2.0
[INFO] [stderr]    Compiling manifold-protocol v0.1.0 (/opt/rustwide/workdir/crates/protocol)
[INFO] [stderr]    Compiling toml v0.8.23
[INFO] [stderr]    Compiling idna_adapter v1.2.2
[INFO] [stderr]    Compiling idna v1.1.0
[INFO] [stderr]    Compiling url v2.5.8
[INFO] [stderr]    Compiling tokio-util v0.7.18
[INFO] [stderr]    Compiling tower v0.5.3
[INFO] [stderr]    Compiling tokio-tungstenite v0.24.0
[INFO] [stderr]    Compiling tokio-stream v0.1.18
[INFO] [stderr]    Compiling tokio-rustls v0.26.4
[INFO] [stderr]    Compiling h2 v0.4.15
[INFO] [stderr]    Compiling tower v0.4.13
[INFO] [stderr]    Compiling combine v4.6.7
[INFO] [stderr]    Compiling tower-http v0.6.11
[INFO] [stderr]    Compiling redis v0.27.6
[INFO] [stderr]    Compiling hyper v1.10.1
[INFO] [stderr]    Compiling hyper-util v0.1.20
[INFO] [stderr]    Compiling hyper-timeout v0.5.2
[INFO] [stderr]    Compiling axum v0.7.9
[INFO] [stderr]    Compiling hyper-rustls v0.27.9
[INFO] [stderr]    Compiling reqwest v0.12.28
[INFO] [stderr]    Compiling tonic v0.12.3
[INFO] [stderr]    Compiling manifold-broker v0.1.0 (/opt/rustwide/workdir/crates/broker)
[INFO] [stderr]    Compiling manifold-loadtest v0.1.0 (/opt/rustwide/workdir/crates/loadtest)
[INFO] [stderr]    Compiling manifold-core v0.1.0 (/opt/rustwide/workdir/crates/core)
[INFO] [stderr]    Compiling manifold-server v0.1.0 (/opt/rustwide/workdir/crates/server)
[INFO] [stderr]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 39s
[INFO] running `Command { std: "docker" "inspect" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b" "/opt/rustwide/cargo-home/bin/cargo" "+1.98.0-beta.1" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr]    Compiling manifold-broker v0.1.0 (/opt/rustwide/workdir/crates/broker)
[INFO] [stderr]    Compiling manifold-core v0.1.0 (/opt/rustwide/workdir/crates/core)
[INFO] [stderr]    Compiling manifold-protocol v0.1.0 (/opt/rustwide/workdir/crates/protocol)
[INFO] [stderr]    Compiling manifold-loadtest v0.1.0 (/opt/rustwide/workdir/crates/loadtest)
[INFO] [stderr]    Compiling manifold-server v0.1.0 (/opt/rustwide/workdir/crates/server)
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 5.99s
[INFO] running `Command { std: "docker" "inspect" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-w" "/opt/rustwide/workdir" "--user" "0:0" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b" "/opt/rustwide/cargo-home/bin/cargo" "+1.98.0-beta.1" "test" "--frozen", kill_on_drop: false }`
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 0.41s
[INFO] [stderr]      Running unittests src/lib.rs (/opt/rustwide/target/debug/deps/manifold_broker-2dac9398e76f71c3)
[INFO] [stderr]      Running tests/redis_multinode.rs (/opt/rustwide/target/debug/deps/redis_multinode-8dfa20f49f0ed9dc)
[INFO] [stdout] 
[INFO] [stdout] running 0 tests
[INFO] [stdout] 
[INFO] [stdout] test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] running 3 tests
[INFO] [stdout] test cross_node_fanout has been running for over 60 seconds
[INFO] [stdout] test presence_shared_across_nodes has been running for over 60 seconds
[INFO] [stdout] test recovery_via_redis has been running for over 60 seconds
[ERROR] error running command: no output for 300 seconds
[INFO] running `Command { std: "docker" "inspect" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b", kill_on_drop: false }`
[INFO] [stdout] 16c21f2d68645950f770ca5241c7e83613addf4eff1621b076b044518e08e15b
