[INFO] cloning repository https://github.com/dommyrock/doc-rag-demo [INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/dommyrock/doc-rag-demo" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fdommyrock%2Fdoc-rag-demo", kill_on_drop: false }` [INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fdommyrock%2Fdoc-rag-demo'... [INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }` [INFO] [stdout] cd94d546c498bb0687e5098c7ddbd018cdb51ae1 [INFO] checking dommyrock/doc-rag-demo against try#f70ee28831acc074c109afab32d287057c3eca46 for pr-149389 [INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fdommyrock%2Fdoc-rag-demo" "/workspace/builds/worker-4-tc2/source", kill_on_drop: false }` [INFO] [stderr] Cloning into '/workspace/builds/worker-4-tc2/source'... [INFO] [stderr] done. [INFO] started tweaking git repo https://github.com/dommyrock/doc-rag-demo [INFO] finished tweaking git repo https://github.com/dommyrock/doc-rag-demo [INFO] tweaked toml for git repo https://github.com/dommyrock/doc-rag-demo written to /workspace/builds/worker-4-tc2/source/Cargo.toml [INFO] validating manifest of git repo https://github.com/dommyrock/doc-rag-demo on toolchain f70ee28831acc074c109afab32d287057c3eca46 [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+f70ee28831acc074c109afab32d287057c3eca46" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }` [INFO] crate git repo https://github.com/dommyrock/doc-rag-demo already has a lockfile, it will not be regenerated [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+f70ee28831acc074c109afab32d287057c3eca46" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }` [INFO] [stderr] Blocking waiting for file lock on package cache [INFO] [stderr] Updating git repository `https://github.com/huggingface/candle.git` [INFO] [stderr] Updating git submodule `https://github.com/NVIDIA/cutlass.git` [INFO] [stderr] Updating crates.io index [INFO] [stderr] Blocking waiting for file lock on package cache [ERROR] error running command: no output for 300 seconds [INFO] checking dommyrock/doc-rag-demo against try#f70ee28831acc074c109afab32d287057c3eca46 for pr-149389 [INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fdommyrock%2Fdoc-rag-demo" "/workspace/builds/worker-4-tc2/source", kill_on_drop: false }` [INFO] [stderr] Cloning into '/workspace/builds/worker-4-tc2/source'... [INFO] [stderr] done. [INFO] started tweaking git repo https://github.com/dommyrock/doc-rag-demo [INFO] finished tweaking git repo https://github.com/dommyrock/doc-rag-demo [INFO] tweaked toml for git repo https://github.com/dommyrock/doc-rag-demo written to /workspace/builds/worker-4-tc2/source/Cargo.toml [INFO] validating manifest of git repo https://github.com/dommyrock/doc-rag-demo on toolchain f70ee28831acc074c109afab32d287057c3eca46 [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+f70ee28831acc074c109afab32d287057c3eca46" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }` [INFO] crate git repo https://github.com/dommyrock/doc-rag-demo already has a lockfile, it will not be regenerated [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+f70ee28831acc074c109afab32d287057c3eca46" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }` [INFO] [stderr] Blocking waiting for file lock on package cache [INFO] [stderr] Blocking waiting for file lock on package cache [INFO] [stderr] Downloading crates ... [INFO] [stderr] Downloaded derive_builder_macro v0.20.0 [INFO] [stderr] Downloaded derive_builder v0.20.0 [INFO] [stderr] Downloaded monostate-impl v0.1.12 [INFO] [stderr] Downloaded yoke-derive v0.7.3 [INFO] [stderr] Downloaded darling_macro v0.20.8 [INFO] [stderr] Downloaded zerofrom v0.1.3 [INFO] [stderr] Downloaded monostate v0.1.12 [INFO] [stderr] Downloaded safetensors v0.4.3 [INFO] [stderr] Downloaded zerofrom-derive v0.1.3 [INFO] [stderr] Downloaded yoke v0.7.3 [INFO] [stderr] Downloaded derive_builder_core v0.20.0 [INFO] [stderr] Downloaded darling v0.20.8 [INFO] [stderr] Downloaded hf-hub v0.3.2 [INFO] [stderr] Downloaded darling_core v0.20.8 [INFO] [stderr] Downloaded pulp v0.18.10 [INFO] [stderr] Downloaded cc v1.0.95 [INFO] [stderr] Downloaded tokenizers v0.19.1 [INFO] [stderr] Downloaded cudarc v0.10.0 [INFO] [stderr] Downloaded h2 v0.4.4 [INFO] [stderr] Downloaded qdrant-client v1.8.0 [INFO] [stderr] Downloaded rustls v0.21.11 [INFO] [stderr] Downloaded rustix v0.38.33 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:94a0c148923f5b2b52a63ef0eeb1882ad339ab61bce784c8077cbe41c61feb6c" "/opt/rustwide/cargo-home/bin/cargo" "+f70ee28831acc074c109afab32d287057c3eca46" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }` [INFO] [stdout] 7f9a26064bf134c1d74a655b4e0ce6912d2d3046f5c3d4b60de6994dae3b9e48 [INFO] running `Command { std: "docker" "start" "-a" "7f9a26064bf134c1d74a655b4e0ce6912d2d3046f5c3d4b60de6994dae3b9e48", kill_on_drop: false }` [INFO] running `Command { std: "docker" "inspect" "7f9a26064bf134c1d74a655b4e0ce6912d2d3046f5c3d4b60de6994dae3b9e48", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "7f9a26064bf134c1d74a655b4e0ce6912d2d3046f5c3d4b60de6994dae3b9e48", kill_on_drop: false }` [INFO] [stdout] 7f9a26064bf134c1d74a655b4e0ce6912d2d3046f5c3d4b60de6994dae3b9e48 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:94a0c148923f5b2b52a63ef0eeb1882ad339ab61bce784c8077cbe41c61feb6c" "/opt/rustwide/cargo-home/bin/cargo" "+f70ee28831acc074c109afab32d287057c3eca46" "check" "--frozen" "--all" "--all-targets" "--message-format=json", kill_on_drop: false }` [INFO] [stdout] 0c077f39ecb6b76aa8913363447695c761bff46dbcc54149b028237a369647fe [INFO] running `Command { std: "docker" "start" "-a" "0c077f39ecb6b76aa8913363447695c761bff46dbcc54149b028237a369647fe", kill_on_drop: false }` [INFO] [stderr] Compiling proc-macro2 v1.0.81 [INFO] [stderr] Compiling autocfg v1.2.0 [INFO] [stderr] Compiling cc v1.0.95 [INFO] [stderr] Checking bytes v1.6.0 [INFO] [stderr] Checking tracing-core v0.1.32 [INFO] [stderr] Compiling serde v1.0.198 [INFO] [stderr] Checking getrandom v0.2.14 [INFO] [stderr] Checking parking_lot_core v0.9.9 [INFO] [stderr] Checking signal-hook-registry v1.4.2 [INFO] [stderr] Checking mio v0.8.11 [INFO] [stderr] Checking indexmap v2.2.6 [INFO] [stderr] Compiling either v1.11.0 [INFO] [stderr] Compiling serde_json v1.0.116 [INFO] [stderr] Checking bitflags v2.5.0 [INFO] [stderr] Checking reborrow v0.5.5 [INFO] [stderr] Checking rayon v1.10.0 [INFO] [stderr] Compiling rustversion v1.0.15 [INFO] [stderr] Checking rand_core v0.6.4 [INFO] [stderr] Compiling crossbeam-utils v0.8.19 [INFO] [stderr] Compiling rustls v0.21.11 [INFO] [stderr] Checking raw-cpuid v10.7.0 [INFO] [stderr] Compiling anyhow v1.0.82 [INFO] [stderr] Compiling seq-macro v0.3.5 [INFO] [stderr] Checking rustls-pki-types v1.4.1 [INFO] [stderr] Compiling lock_api v0.4.11 [INFO] [stderr] Compiling slab v0.4.9 [INFO] [stderr] Checking rand_chacha v0.3.1 [INFO] [stderr] Compiling num-traits v0.2.18 [INFO] [stderr] Compiling indexmap v1.9.3 [INFO] [stderr] Checking http v0.2.12 [INFO] [stderr] Compiling itertools v0.10.5 [INFO] [stderr] Compiling num_cpus v1.16.0 [INFO] [stderr] Checking rand v0.8.5 [INFO] [stderr] Compiling quote v1.0.36 [INFO] [stderr] Compiling syn v2.0.60 [INFO] [stderr] Checking parking_lot v0.12.1 [INFO] [stderr] Compiling syn v1.0.109 [INFO] [stderr] Compiling thiserror v1.0.59 [INFO] [stderr] Compiling crossbeam-epoch v0.9.18 [INFO] [stderr] Checking aho-corasick v1.1.3 [INFO] [stderr] Checking matchit v0.7.3 [INFO] [stderr] Checking stable_deref_trait v1.2.0 [INFO] [stderr] Compiling portable-atomic v1.6.0 [INFO] [stderr] Checking regex-syntax v0.8.3 [INFO] [stderr] Compiling cudarc v0.10.0 [INFO] [stderr] Compiling rustls v0.22.4 [INFO] [stderr] Checking anstream v0.6.13 [INFO] [stderr] Checking miniz_oxide v0.7.2 [INFO] [stderr] Checking unicode-width v0.1.11 [INFO] [stderr] Compiling axum-core v0.3.4 [INFO] [stderr] Checking http-body v0.4.6 [INFO] [stderr] Compiling axum v0.6.20 [INFO] [stderr] error: failed to run custom build command for `cudarc v0.10.0` [INFO] [stderr] note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation. [INFO] [stderr] [INFO] [stderr] Caused by: [INFO] [stderr] process didn't exit successfully: `/opt/rustwide/target/debug/build/cudarc-9c5d26f5357d9061/build-script-build` (exit status: 101) [INFO] [stderr] --- stdout [INFO] [stderr] cargo:rerun-if-changed=build.rs [INFO] [stderr] cargo:rerun-if-env-changed=CUDA_ROOT [INFO] [stderr] cargo:rerun-if-env-changed=CUDA_PATH [INFO] [stderr] cargo:rerun-if-env-changed=CUDA_TOOLKIT_ROOT_DIR [INFO] [stderr] [INFO] [stderr] --- stderr [INFO] [stderr] [INFO] [stderr] thread 'main' (879) panicked at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/cudarc-0.10.0/build.rs:21:13: [INFO] [stderr] Unable to find `include/cuda.h` under any of: ["/usr", "/usr/local/cuda", "/opt/cuda", "/usr/lib/cuda", "C:/Program Files/NVIDIA GPU Computing Toolkit", "C:/CUDA"]. Set the `CUDA_ROOT` environment variable to `$CUDA_ROOT/include/cuda.h` to override path. [INFO] [stderr] stack backtrace: [INFO] [stderr] 0: 0x60366b3fd112 - std[8a7011b5972ea393]::backtrace_rs::backtrace::libunwind::trace [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9 [INFO] [stderr] 1: 0x60366b3fd112 - std[8a7011b5972ea393]::backtrace_rs::backtrace::trace_unsynchronized:: [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14 [INFO] [stderr] 2: 0x60366b3fd112 - std[8a7011b5972ea393]::sys::backtrace::_print_fmt [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/sys/backtrace.rs:66:9 [INFO] [stderr] 3: 0x60366b3fd112 - <::print::DisplayBacktrace as core[b8facb68d32e370]::fmt::Display>::fmt [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/sys/backtrace.rs:39:26 [INFO] [stderr] 4: 0x60366b40df67 - ::fmt [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/core/src/fmt/rt.rs:152:76 [INFO] [stderr] 5: 0x60366b40df67 - core[b8facb68d32e370]::fmt::write [INFO] [stderr] 6: 0x60366b401286 - std[8a7011b5972ea393]::io::default_write_fmt:: [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/io/mod.rs:639:11 [INFO] [stderr] 7: 0x60366b401286 - ::write_fmt [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/io/mod.rs:1994:13 [INFO] [stderr] 8: 0x60366b3e7fcc - ::print [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/sys/backtrace.rs:42:9 [INFO] [stderr] 9: 0x60366b3e7fcc - std[8a7011b5972ea393]::panicking::default_hook::{closure#0} [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panicking.rs:293:27 [INFO] [stderr] 10: 0x60366b3f9421 - std[8a7011b5972ea393]::panicking::default_hook [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panicking.rs:320:9 [INFO] [stderr] 11: 0x60366b3f95db - std[8a7011b5972ea393]::panicking::panic_with_hook [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panicking.rs:826:13 [INFO] [stderr] 12: 0x60366b3e8088 - std[8a7011b5972ea393]::panicking::panic_handler::{closure#0} [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panicking.rs:699:13 [INFO] [stderr] 13: 0x60366b3dd519 - std[8a7011b5972ea393]::sys::backtrace::__rust_end_short_backtrace:: [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/sys/backtrace.rs:174:18 [INFO] [stderr] 14: 0x60366b3e89ed - __rustc[9de75a00f0b995ca]::rust_begin_unwind [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panicking.rs:690:5 [INFO] [stderr] 15: 0x60366b40e5ac - core[b8facb68d32e370]::panicking::panic_fmt [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/core/src/panicking.rs:80:14 [INFO] [stderr] 16: 0x60366b3cf376 - build_script_build[31d87b6d7f0f518c]::link_cuda::{closure#1} [INFO] [stderr] 17: 0x60366b3ccef9 - >::unwrap_or_else:: [INFO] [stderr] 18: 0x60366b3cf94d - build_script_build[31d87b6d7f0f518c]::link_cuda [INFO] [stderr] 19: 0x60366b3cf803 - build_script_build[31d87b6d7f0f518c]::main [INFO] [stderr] 20: 0x60366b3d1b93 - >::call_once [INFO] [stderr] 21: 0x60366b3cd8c6 - std[8a7011b5972ea393]::sys::backtrace::__rust_begin_short_backtrace:: [INFO] [stderr] 22: 0x60366b3ceb29 - std[8a7011b5972ea393]::rt::lang_start::<()>::{closure#0} [INFO] [stderr] 23: 0x60366b3f89aa - <&dyn core[b8facb68d32e370]::ops::function::Fn<(), Output = i32> + core[b8facb68d32e370]::panic::unwind_safe::RefUnwindSafe + core[b8facb68d32e370]::marker::Sync as core[b8facb68d32e370]::ops::function::FnOnce<()>>::call_once [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/core/src/ops/function.rs:287:21 [INFO] [stderr] 24: 0x60366b3f89aa - std[8a7011b5972ea393]::panicking::catch_unwind::do_call::<&dyn core[b8facb68d32e370]::ops::function::Fn<(), Output = i32> + core[b8facb68d32e370]::panic::unwind_safe::RefUnwindSafe + core[b8facb68d32e370]::marker::Sync, i32> [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panicking.rs:582:40 [INFO] [stderr] 25: 0x60366b3f89aa - std[8a7011b5972ea393]::panicking::catch_unwind:: + core[b8facb68d32e370]::panic::unwind_safe::RefUnwindSafe + core[b8facb68d32e370]::marker::Sync> [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panicking.rs:545:19 [INFO] [stderr] 26: 0x60366b3f89aa - std[8a7011b5972ea393]::panic::catch_unwind::<&dyn core[b8facb68d32e370]::ops::function::Fn<(), Output = i32> + core[b8facb68d32e370]::panic::unwind_safe::RefUnwindSafe + core[b8facb68d32e370]::marker::Sync, i32> [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panic.rs:359:14 [INFO] [stderr] 27: 0x60366b3f89aa - std[8a7011b5972ea393]::rt::lang_start_internal::{closure#0} [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/rt.rs:175:24 [INFO] [stderr] 28: 0x60366b3f89aa - std[8a7011b5972ea393]::panicking::catch_unwind::do_call:: [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panicking.rs:582:40 [INFO] [stderr] 29: 0x60366b3f89aa - std[8a7011b5972ea393]::panicking::catch_unwind:: [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panicking.rs:545:19 [INFO] [stderr] 30: 0x60366b3f89aa - std[8a7011b5972ea393]::panic::catch_unwind:: [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/panic.rs:359:14 [INFO] [stderr] 31: 0x60366b3f89aa - std[8a7011b5972ea393]::rt::lang_start_internal [INFO] [stderr] at /rustc/f70ee28831acc074c109afab32d287057c3eca46/library/std/src/rt.rs:171:5 [INFO] [stderr] 32: 0x60366b3cd841 - std[8a7011b5972ea393]::rt::lang_start::<()> [INFO] [stderr] 33: 0x60366b3d1d95 - main [INFO] [stderr] 34: 0x7e890529a1ca - [INFO] [stderr] 35: 0x7e890529a28b - __libc_start_main [INFO] [stderr] 36: 0x60366b3cccf5 - _start [INFO] [stderr] 37: 0x0 - [INFO] [stderr] warning: build failed, waiting for other jobs to finish... [INFO] running `Command { std: "docker" "inspect" "0c077f39ecb6b76aa8913363447695c761bff46dbcc54149b028237a369647fe", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "0c077f39ecb6b76aa8913363447695c761bff46dbcc54149b028237a369647fe", kill_on_drop: false }` [INFO] [stdout] 0c077f39ecb6b76aa8913363447695c761bff46dbcc54149b028237a369647fe