[INFO] cloning repository https://github.com/roG0d/gpt2_from_scratch_candle [INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/roG0d/gpt2_from_scratch_candle" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FroG0d%2Fgpt2_from_scratch_candle", kill_on_drop: false }` [INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FroG0d%2Fgpt2_from_scratch_candle'... [INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }` [INFO] [stdout] 8c412dd4ba6caa353c6a9ed0ee64dfe7372ea729 [INFO] checking roG0d/gpt2_from_scratch_candle against try#7d2afcda7a9c043d5175cbca3ab08f0acf02b13a for pr-140824 [INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FroG0d%2Fgpt2_from_scratch_candle" "/workspace/builds/worker-3-tc2/source", kill_on_drop: false }` [INFO] [stderr] Cloning into '/workspace/builds/worker-3-tc2/source'... [INFO] [stderr] done. [INFO] validating manifest of git repo https://github.com/roG0d/gpt2_from_scratch_candle on toolchain 7d2afcda7a9c043d5175cbca3ab08f0acf02b13a [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+7d2afcda7a9c043d5175cbca3ab08f0acf02b13a" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }` [INFO] started tweaking git repo https://github.com/roG0d/gpt2_from_scratch_candle [INFO] finished tweaking git repo https://github.com/roG0d/gpt2_from_scratch_candle [INFO] tweaked toml for git repo https://github.com/roG0d/gpt2_from_scratch_candle written to /workspace/builds/worker-3-tc2/source/Cargo.toml [INFO] crate git repo https://github.com/roG0d/gpt2_from_scratch_candle already has a lockfile, it will not be regenerated [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+7d2afcda7a9c043d5175cbca3ab08f0acf02b13a" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }` [INFO] [stderr] Blocking waiting for file lock on package cache [INFO] [stderr] Updating git repository `https://github.com/huggingface/candle.git` [INFO] [stderr] Updating git submodule `https://github.com/NVIDIA/cutlass.git` [INFO] [stderr] Updating crates.io index [INFO] [stderr] Blocking waiting for file lock on package cache [ERROR] error running command: no output for 300 seconds [INFO] checking roG0d/gpt2_from_scratch_candle against try#7d2afcda7a9c043d5175cbca3ab08f0acf02b13a for pr-140824 [INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FroG0d%2Fgpt2_from_scratch_candle" "/workspace/builds/worker-3-tc2/source", kill_on_drop: false }` [INFO] [stderr] Cloning into '/workspace/builds/worker-3-tc2/source'... [INFO] [stderr] done. [INFO] validating manifest of git repo https://github.com/roG0d/gpt2_from_scratch_candle on toolchain 7d2afcda7a9c043d5175cbca3ab08f0acf02b13a [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+7d2afcda7a9c043d5175cbca3ab08f0acf02b13a" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }` [INFO] started tweaking git repo https://github.com/roG0d/gpt2_from_scratch_candle [INFO] finished tweaking git repo https://github.com/roG0d/gpt2_from_scratch_candle [INFO] tweaked toml for git repo https://github.com/roG0d/gpt2_from_scratch_candle written to /workspace/builds/worker-3-tc2/source/Cargo.toml [INFO] crate git repo https://github.com/roG0d/gpt2_from_scratch_candle already has a lockfile, it will not be regenerated [INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+7d2afcda7a9c043d5175cbca3ab08f0acf02b13a" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }` [INFO] [stderr] Blocking waiting for file lock on package cache [INFO] [stderr] Blocking waiting for file lock on package cache [INFO] [stderr] Downloading crates ... [INFO] [stderr] Downloaded bindgen_cuda v0.1.4 [INFO] [stderr] Downloaded pulp v0.18.9 [INFO] [stderr] Downloaded safetensors v0.4.2 [INFO] [stderr] Downloaded cudarc v0.10.0 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:b0b074c097205a61b89e8ad263052f976b2b332c4dc5f02aef1fe52501660d6e" "/opt/rustwide/cargo-home/bin/cargo" "+7d2afcda7a9c043d5175cbca3ab08f0acf02b13a" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }` [INFO] [stdout] c3aa0d1180696f61c25e444e8a5e373cdb8156a7a511c5e30cc18696dcee7b84 [INFO] running `Command { std: "docker" "start" "-a" "c3aa0d1180696f61c25e444e8a5e373cdb8156a7a511c5e30cc18696dcee7b84", kill_on_drop: false }` [INFO] running `Command { std: "docker" "inspect" "c3aa0d1180696f61c25e444e8a5e373cdb8156a7a511c5e30cc18696dcee7b84", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "c3aa0d1180696f61c25e444e8a5e373cdb8156a7a511c5e30cc18696dcee7b84", kill_on_drop: false }` [INFO] [stdout] c3aa0d1180696f61c25e444e8a5e373cdb8156a7a511c5e30cc18696dcee7b84 [INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-3-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:b0b074c097205a61b89e8ad263052f976b2b332c4dc5f02aef1fe52501660d6e" "/opt/rustwide/cargo-home/bin/cargo" "+7d2afcda7a9c043d5175cbca3ab08f0acf02b13a" "check" "--frozen" "--all" "--all-targets" "--message-format=json", kill_on_drop: false }` [INFO] [stdout] 4b2a7c1bd930b7a1c62a42eb8b22d0d465dbc2a73e24b8aed804b5dd38bd0a8b [INFO] running `Command { std: "docker" "start" "-a" "4b2a7c1bd930b7a1c62a42eb8b22d0d465dbc2a73e24b8aed804b5dd38bd0a8b", kill_on_drop: false }` [INFO] [stderr] Compiling proc-macro2 v1.0.79 [INFO] [stderr] Compiling unicode-ident v1.0.12 [INFO] [stderr] Compiling libc v0.2.153 [INFO] [stderr] Compiling crossbeam-utils v0.8.19 [INFO] [stderr] Compiling autocfg v1.1.0 [INFO] [stderr] Compiling libm v0.2.8 [INFO] [stderr] Checking cfg-if v1.0.0 [INFO] [stderr] Compiling rayon-core v1.12.1 [INFO] [stderr] Checking ppv-lite86 v0.2.17 [INFO] [stderr] Checking reborrow v0.5.5 [INFO] [stderr] Compiling paste v1.0.14 [INFO] [stderr] Checking either v1.10.0 [INFO] [stderr] Checking bitflags v1.3.2 [INFO] [stderr] Compiling num-traits v0.2.18 [INFO] [stderr] Checking raw-cpuid v10.7.0 [INFO] [stderr] Checking crossbeam-epoch v0.9.18 [INFO] [stderr] Compiling seq-macro v0.3.5 [INFO] [stderr] Compiling quote v1.0.35 [INFO] [stderr] Checking crossbeam-deque v0.8.5 [INFO] [stderr] Checking once_cell v1.19.0 [INFO] [stderr] Checking getrandom v0.2.12 [INFO] [stderr] Compiling syn v2.0.53 [INFO] [stderr] Checking rand_core v0.6.4 [INFO] [stderr] Checking rand_chacha v0.3.1 [INFO] [stderr] Compiling serde v1.0.197 [INFO] [stderr] Checking rayon v1.9.0 [INFO] [stderr] Compiling glob v0.3.1 [INFO] [stderr] Checking rand v0.8.5 [INFO] [stderr] Compiling serde_json v1.0.114 [INFO] [stderr] Compiling crc32fast v1.4.0 [INFO] [stderr] Checking ryu v1.0.17 [INFO] [stderr] Checking stable_deref_trait v1.2.0 [INFO] [stderr] Compiling thiserror v1.0.58 [INFO] [stderr] Checking itoa v1.0.10 [INFO] [stderr] Compiling num_cpus v1.16.0 [INFO] [stderr] Compiling cudarc v0.10.0 [INFO] [stderr] Compiling matrixmultiply v0.3.8 [INFO] [stderr] Checking rand_distr v0.4.3 [INFO] [stderr] error: failed to run custom build command for `cudarc v0.10.0` [INFO] [stderr] note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation. [INFO] [stderr] [INFO] [stderr] Caused by: [INFO] [stderr] process didn't exit successfully: `/opt/rustwide/target/debug/build/cudarc-9c5d26f5357d9061/build-script-build` (exit status: 101) [INFO] [stderr] --- stdout [INFO] [stderr] cargo:rerun-if-changed=build.rs [INFO] [stderr] cargo:rerun-if-env-changed=CUDA_ROOT [INFO] [stderr] cargo:rerun-if-env-changed=CUDA_PATH [INFO] [stderr] cargo:rerun-if-env-changed=CUDA_TOOLKIT_ROOT_DIR [INFO] [stderr] [INFO] [stderr] --- stderr [INFO] [stderr] [INFO] [stderr] thread 'main' panicked at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/cudarc-0.10.0/build.rs:21:13: [INFO] [stderr] Unable to find `include/cuda.h` under any of: ["/usr", "/usr/local/cuda", "/opt/cuda", "/usr/lib/cuda", "C:/Program Files/NVIDIA GPU Computing Toolkit", "C:/CUDA"]. Set the `CUDA_ROOT` environment variable to `$CUDA_ROOT/include/cuda.h` to override path. [INFO] [stderr] stack backtrace: [INFO] [stderr] 0: 0x62a2e5544ee2 - std::backtrace_rs::backtrace::libunwind::trace::h0d87a033f18a7fae [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9 [INFO] [stderr] 1: 0x62a2e5544ee2 - std::backtrace_rs::backtrace::trace_unsynchronized::hf0f0ce7bb304c4c4 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14 [INFO] [stderr] 2: 0x62a2e5544ee2 - std::sys::backtrace::_print_fmt::h191fbaa49de2a93c [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/sys/backtrace.rs:66:9 [INFO] [stderr] 3: 0x62a2e5544ee2 - ::fmt::h2e21c62a24abbc15 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/sys/backtrace.rs:39:26 [INFO] [stderr] 4: 0x62a2e555f243 - core::fmt::rt::Argument::fmt::h6e5ee6192f2b2698 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/core/src/fmt/rt.rs:181:76 [INFO] [stderr] 5: 0x62a2e555f243 - core::fmt::write::he722e62da8c1e9e8 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/core/src/fmt/mod.rs:1446:25 [INFO] [stderr] 6: 0x62a2e55428d3 - std::io::default_write_fmt::ha5518580fd70e138 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/io/mod.rs:639:11 [INFO] [stderr] 7: 0x62a2e55428d3 - std::io::Write::write_fmt::hcb7e25f2599a2273 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/io/mod.rs:1914:13 [INFO] [stderr] 8: 0x62a2e5544d32 - std::sys::backtrace::BacktraceLock::print::h5bbc6bf022f4b4e4 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/sys/backtrace.rs:42:9 [INFO] [stderr] 9: 0x62a2e5545da2 - std::panicking::default_hook::{{closure}}::h1fedd83de2eb085e [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panicking.rs:300:22 [INFO] [stderr] 10: 0x62a2e5545ba5 - std::panicking::default_hook::h2b7ba9a6fb124ab7 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panicking.rs:327:9 [INFO] [stderr] 11: 0x62a2e5546672 - std::panicking::rust_panic_with_hook::h84a58634492b5ed6 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panicking.rs:833:13 [INFO] [stderr] 12: 0x62a2e554650a - std::panicking::begin_panic_handler::{{closure}}::h521406465ff7da15 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panicking.rs:706:13 [INFO] [stderr] 13: 0x62a2e55453d9 - std::sys::backtrace::__rust_end_short_backtrace::h020019c2da31dea5 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/sys/backtrace.rs:168:18 [INFO] [stderr] 14: 0x62a2e554619d - __rustc[dfb5731ed8a4fdce]::rust_begin_unwind [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panicking.rs:697:5 [INFO] [stderr] 15: 0x62a2e555e710 - core::panicking::panic_fmt::h23bc8c485ba3bdce [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/core/src/panicking.rs:75:14 [INFO] [stderr] 16: 0x62a2e5524c32 - build_script_build::link_cuda::{{closure}}::h3d5da6487706dc21 [INFO] [stderr] 17: 0x62a2e55231c5 - core::option::Option::unwrap_or_else::h046f0d2db5b676b5 [INFO] [stderr] 18: 0x62a2e5524795 - build_script_build::link_cuda::h1aa08c4033942cb9 [INFO] [stderr] 19: 0x62a2e5524655 - build_script_build::main::h03c9987685c1955d [INFO] [stderr] 20: 0x62a2e5521773 - core::ops::function::FnOnce::call_once::h5048ce1e3d316244 [INFO] [stderr] 21: 0x62a2e5520e56 - std::sys::backtrace::__rust_begin_short_backtrace::h89b360ad64cfe3f3 [INFO] [stderr] 22: 0x62a2e5520db9 - std::rt::lang_start::{{closure}}::h89bc577da26bf62f [INFO] [stderr] 23: 0x62a2e553fd24 - core::ops::function::impls:: for &F>::call_once::h83e03a0232dd0fd4 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/core/src/ops/function.rs:284:13 [INFO] [stderr] 24: 0x62a2e553fd24 - std::panicking::try::do_call::h4cb5a132c952b48a [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panicking.rs:589:40 [INFO] [stderr] 25: 0x62a2e553fd24 - std::panicking::try::hf636b8ab9f2350f5 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panicking.rs:552:19 [INFO] [stderr] 26: 0x62a2e553fd24 - std::panic::catch_unwind::h6bfb07e3b0bf4c17 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panic.rs:359:14 [INFO] [stderr] 27: 0x62a2e553fd24 - std::rt::lang_start_internal::{{closure}}::h76773210632dee74 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/rt.rs:168:24 [INFO] [stderr] 28: 0x62a2e553fd24 - std::panicking::try::do_call::hc8ad85365e92e0f3 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panicking.rs:589:40 [INFO] [stderr] 29: 0x62a2e553fd24 - std::panicking::try::h9336dcf83336494d [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panicking.rs:552:19 [INFO] [stderr] 30: 0x62a2e553fd24 - std::panic::catch_unwind::h67f7b2d9b6f50c76 [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/panic.rs:359:14 [INFO] [stderr] 31: 0x62a2e553fd24 - std::rt::lang_start_internal::h4a8cc11f0c67dc3a [INFO] [stderr] at /rustc/7d2afcda7a9c043d5175cbca3ab08f0acf02b13a/library/std/src/rt.rs:164:5 [INFO] [stderr] 32: 0x62a2e5520da1 - std::rt::lang_start::h185d38927f45798f [INFO] [stderr] 33: 0x62a2e5525025 - main [INFO] [stderr] 34: 0x702b50cb81ca - [INFO] [stderr] 35: 0x702b50cb828b - __libc_start_main [INFO] [stderr] 36: 0x62a2e551fbc5 - _start [INFO] [stderr] 37: 0x0 - [INFO] [stderr] warning: build failed, waiting for other jobs to finish... [INFO] running `Command { std: "docker" "inspect" "4b2a7c1bd930b7a1c62a42eb8b22d0d465dbc2a73e24b8aed804b5dd38bd0a8b", kill_on_drop: false }` [INFO] running `Command { std: "docker" "rm" "-f" "4b2a7c1bd930b7a1c62a42eb8b22d0d465dbc2a73e24b8aed804b5dd38bd0a8b", kill_on_drop: false }` [INFO] [stdout] 4b2a7c1bd930b7a1c62a42eb8b22d0d465dbc2a73e24b8aed804b5dd38bd0a8b