[INFO] cloning repository https://github.com/hezhaozhao-git/rust_flash_attention_demo
[INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/hezhaozhao-git/rust_flash_attention_demo" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fhezhaozhao-git%2Frust_flash_attention_demo", kill_on_drop: false }`
[INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fhezhaozhao-git%2Frust_flash_attention_demo'...
[INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }`
[INFO] [stdout] ccaab0bc84bbe140f41e6d1686549d31eb909bcf
[INFO] checking hezhaozhao-git/rust_flash_attention_demo against f4dd856795debe636f2ba39a900f162f5abeb7e8 for pr-160904
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fhezhaozhao-git%2Frust_flash_attention_demo" "/workspace/builds/worker-5-tc2/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-5-tc2/source'...
[INFO] [stderr] done.
[INFO] started tweaking git repo https://github.com/hezhaozhao-git/rust_flash_attention_demo
[INFO] finished tweaking git repo https://github.com/hezhaozhao-git/rust_flash_attention_demo
[INFO] tweaked toml for git repo https://github.com/hezhaozhao-git/rust_flash_attention_demo written to /workspace/builds/worker-5-tc2/source/Cargo.toml
[INFO] validating manifest of git repo https://github.com/hezhaozhao-git/rust_flash_attention_demo on toolchain f4dd856795debe636f2ba39a900f162f5abeb7e8
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+f4dd856795debe636f2ba39a900f162f5abeb7e8" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate git repo https://github.com/hezhaozhao-git/rust_flash_attention_demo already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+f4dd856795debe636f2ba39a900f162f5abeb7e8" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]     Updating crates.io index
[INFO] [stderr]     Updating git repository `https://github.com/huggingface/candle-flash-attn-v1`
[INFO] [stderr]     Blocking waiting for file lock on package cache
[ERROR] error running command: no output for 300 seconds
[INFO] checking hezhaozhao-git/rust_flash_attention_demo against f4dd856795debe636f2ba39a900f162f5abeb7e8 for pr-160904
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fhezhaozhao-git%2Frust_flash_attention_demo" "/workspace/builds/worker-5-tc2/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-5-tc2/source'...
[INFO] [stderr] done.
[INFO] started tweaking git repo https://github.com/hezhaozhao-git/rust_flash_attention_demo
[INFO] finished tweaking git repo https://github.com/hezhaozhao-git/rust_flash_attention_demo
[INFO] tweaked toml for git repo https://github.com/hezhaozhao-git/rust_flash_attention_demo written to /workspace/builds/worker-5-tc2/source/Cargo.toml
[INFO] validating manifest of git repo https://github.com/hezhaozhao-git/rust_flash_attention_demo on toolchain f4dd856795debe636f2ba39a900f162f5abeb7e8
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+f4dd856795debe636f2ba39a900f162f5abeb7e8" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate git repo https://github.com/hezhaozhao-git/rust_flash_attention_demo already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+f4dd856795debe636f2ba39a900f162f5abeb7e8" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded riff v1.0.1
[INFO] [stderr]   Downloaded serde_derive v1.0.192
[INFO] [stderr]   Downloaded gemm-f32 v0.16.15
[INFO] [stderr]   Downloaded zerofrom-derive v0.1.3
[INFO] [stderr]   Downloaded gemm-c64 v0.16.15
[INFO] [stderr]   Downloaded gemm v0.16.15
[INFO] [stderr]   Downloaded gemm-f16 v0.16.15
[INFO] [stderr]   Downloaded zerofrom v0.1.3
[INFO] [stderr]   Downloaded yoke-derive v0.7.2
[INFO] [stderr]   Downloaded gemm-f64 v0.16.15
[INFO] [stderr]   Downloaded gemm-c32 v0.16.15
[INFO] [stderr]   Downloaded candle-ext v0.1.5
[INFO] [stderr]   Downloaded serde v1.0.192
[INFO] [stderr]   Downloaded yoke v0.7.2
[INFO] [stderr]   Downloaded synstructure v0.13.0
[INFO] [stderr]   Downloaded gemm-common v0.16.15
[INFO] [stderr]   Downloaded candle-kernels v0.3.1
[INFO] [stderr]   Downloaded candle-nn v0.3.1
[INFO] [stderr]   Downloaded candle-transformers v0.3.1
[INFO] [stderr]   Downloaded half v2.3.1
[INFO] [stderr]   Downloaded pulp v0.18.4
[INFO] [stderr]   Downloaded cc v1.0.84
[INFO] [stderr]   Downloaded cudarc v0.9.15
[INFO] [stderr]   Downloaded candle-core v0.3.1
[INFO] [stderr]   Downloaded wav v1.0.0
[INFO] [stderr]   Downloaded candle-flash-attn v0.3.1
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-5-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-5-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-m" "1610612736" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:3111399a4047eeb3a02b7a90e478d715f38a8c6669b5c4b49d30a17385265909" "sleep" "infinity", kill_on_drop: false }`
[INFO] [stdout] 09f5bdc89f839233d35cd31b049c4aae40c431b27fece1ee2b5400e8afca1856
[INFO] running `Command { std: "docker" "start" "09f5bdc89f839233d35cd31b049c4aae40c431b27fece1ee2b5400e8afca1856", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "09f5bdc89f839233d35cd31b049c4aae40c431b27fece1ee2b5400e8afca1856", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-w" "/opt/rustwide/workdir" "--user" "0:0" "09f5bdc89f839233d35cd31b049c4aae40c431b27fece1ee2b5400e8afca1856" "/opt/rustwide/cargo-home/bin/cargo" "+f4dd856795debe636f2ba39a900f162f5abeb7e8" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "09f5bdc89f839233d35cd31b049c4aae40c431b27fece1ee2b5400e8afca1856", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-w" "/opt/rustwide/workdir" "--user" "0:0" "09f5bdc89f839233d35cd31b049c4aae40c431b27fece1ee2b5400e8afca1856" "/opt/rustwide/cargo-home/bin/cargo" "+f4dd856795debe636f2ba39a900f162f5abeb7e8" "check" "--frozen" "--all" "--all-targets" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr]    Compiling proc-macro2 v1.0.69
[INFO] [stderr]    Compiling libc v0.2.150
[INFO] [stderr]    Compiling libm v0.2.8
[INFO] [stderr]    Compiling num-traits v0.2.17
[INFO] [stderr]     Checking reborrow v0.5.5
[INFO] [stderr]     Checking once_cell v1.18.0
[INFO] [stderr]    Compiling adler v1.0.2
[INFO] [stderr]    Compiling memchr v2.6.4
[INFO] [stderr]    Compiling gimli v0.28.0
[INFO] [stderr]    Compiling memoffset v0.9.0
[INFO] [stderr]    Compiling crossbeam-utils v0.8.16
[INFO] [stderr]     Checking raw-cpuid v10.7.0
[INFO] [stderr]    Compiling anyhow v1.0.75
[INFO] [stderr]    Compiling rustc-demangle v0.1.23
[INFO] [stderr]    Compiling seq-macro v0.3.5
[INFO] [stderr]    Compiling either v1.9.0
[INFO] [stderr]    Compiling unicode-xid v0.2.4
[INFO] [stderr]    Compiling serde v1.0.192
[INFO] [stderr]    Compiling miniz_oxide v0.7.1
[INFO] [stderr]    Compiling serde_json v1.0.108
[INFO] [stderr]     Checking itoa v1.0.9
[INFO] [stderr]     Checking ryu v1.0.15
[INFO] [stderr]    Compiling crossbeam-epoch v0.9.15
[INFO] [stderr]    Compiling cudarc v0.9.15
[INFO] [stderr]     Checking zip v0.6.6
[INFO] [stderr]     Checking tracing-core v0.1.32
[INFO] [stderr]     Checking riff v1.0.1
[INFO] [stderr]     Checking wav v1.0.0
[INFO] [stderr]    Compiling object v0.32.1
[INFO] [stderr]     Checking getrandom v0.2.11
[INFO] [stderr]     Checking num_cpus v1.16.0
[INFO] [stderr]     Checking memmap2 v0.7.1
[INFO] [stderr] error: failed to run custom build command for `cudarc v0.9.15`
[INFO] [stderr] note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `/opt/rustwide/target/debug/build/cudarc/97dbc8f09a55dfa1/out/build_script_build` (exit status: 101)
[INFO] [stderr]   --- stdout
[INFO] [stderr]   cargo:rerun-if-changed=build.rs
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_ROOT
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_PATH
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_TOOLKIT_ROOT_DIR
[INFO] [stderr] 
[INFO] [stderr]   --- stderr
[INFO] [stderr] 
[INFO] [stderr]   thread 'main' (479) panicked at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/cudarc-0.9.15/build.rs:21:13:
[INFO] [stderr]   Unable to find `include/cuda.h` under any of: ["/usr", "/usr/local/cuda", "/opt/cuda", "/usr/lib/cuda", "C:/Program Files/NVIDIA GPU Computing Toolkit", "C:/CUDA"]. Set the `CUDA_ROOT` environment variable to `$CUDA_ROOT/include/cuda.h` to override path.
[INFO] [stderr]   stack backtrace:
[INFO] [stderr]      0:     0x60993ba47201 - std[18c5f2a687b463a0]::backtrace_rs::backtrace::libunwind::trace
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stderr]      1:     0x60993ba47201 - std[18c5f2a687b463a0]::backtrace_rs::backtrace::trace_unsynchronized::<std[18c5f2a687b463a0]::sys::backtrace::_print_fmt::{closure#1}>
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stderr]      2:     0x60993ba47201 - std[18c5f2a687b463a0]::sys::backtrace::_print_fmt
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/sys/backtrace.rs:74:9
[INFO] [stderr]      3:     0x60993ba47201 - <<std[18c5f2a687b463a0]::sys::backtrace::BacktraceLock>::print::DisplayBacktrace as core[ee3e5f980a4e04aa]::fmt::Display>::fmt
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/sys/backtrace.rs:44:26
[INFO] [stderr]      4:     0x60993ba57a7a - <core[ee3e5f980a4e04aa]::fmt::rt::Argument>::fmt
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/core/src/fmt/rt.rs:152:76
[INFO] [stderr]      5:     0x60993ba57a7a - core[ee3e5f980a4e04aa]::fmt::write
[INFO] [stderr]      6:     0x60993ba4a7cc - core[ee3e5f980a4e04aa]::io::write::default_write_fmt::<std[18c5f2a687b463a0]::sys::stdio::unix::Stderr>
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/core/src/io/write.rs:402:11
[INFO] [stderr]      7:     0x60993ba4a7cc - <std[18c5f2a687b463a0]::sys::stdio::unix::Stderr as core[ee3e5f980a4e04aa]::io::write::Write>::write_fmt
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/core/src/io/write.rs:335:13
[INFO] [stderr]      8:     0x60993ba30d96 - <std[18c5f2a687b463a0]::sys::backtrace::BacktraceLock>::print
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/sys/backtrace.rs:47:9
[INFO] [stderr]      9:     0x60993ba30d96 - std[18c5f2a687b463a0]::panicking::default_hook::{closure#0}
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panicking.rs:289:27
[INFO] [stderr]     10:     0x60993ba43791 - std[18c5f2a687b463a0]::panicking::default_hook
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panicking.rs:316:9
[INFO] [stderr]     11:     0x60993ba4394b - std[18c5f2a687b463a0]::panicking::panic_with_hook
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panicking.rs:811:13
[INFO] [stderr]     12:     0x60993ba30e42 - std[18c5f2a687b463a0]::panicking::panic_handler::{closure#0}
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panicking.rs:685:13
[INFO] [stderr]     13:     0x60993ba29b39 - std[18c5f2a687b463a0]::sys::backtrace::__rust_end_short_backtrace::<std[18c5f2a687b463a0]::panicking::panic_handler::{closure#0}, !>
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/sys/backtrace.rs:182:18
[INFO] [stderr]     14:     0x60993ba3160d - __rustc[62f6431ebd7834b7]::rust_begin_unwind
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panicking.rs:676:5
[INFO] [stderr]     15:     0x60993ba581ac - core[ee3e5f980a4e04aa]::panicking::panic_fmt
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/core/src/panicking.rs:80:14
[INFO] [stderr]     16:     0x60993ba1d6b6 - build_script_build[2678db42de4484e3]::link_cuda::{closure#1}
[INFO] [stderr]     17:     0x60993ba1b4e0 - <core[ee3e5f980a4e04aa]::option::Option<std[18c5f2a687b463a0]::path::PathBuf>>::unwrap_or_else::<build_script_build[2678db42de4484e3]::link_cuda::{closure#1}>
[INFO] [stderr]     18:     0x60993ba1dc8d - build_script_build[2678db42de4484e3]::link_cuda
[INFO] [stderr]     19:     0x60993ba1db43 - build_script_build[2678db42de4484e3]::main
[INFO] [stderr]     20:     0x60993ba1fc13 - <fn() as core[ee3e5f980a4e04aa]::ops::function::FnOnce<()>>::call_once
[INFO] [stderr]     21:     0x60993ba1be3e - std[18c5f2a687b463a0]::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()>
[INFO] [stderr]     22:     0x60993ba1cf39 - std[18c5f2a687b463a0]::rt::lang_start::<()>::{closure#0}
[INFO] [stderr]     23:     0x60993ba42f38 - <&dyn core[ee3e5f980a4e04aa]::ops::function::Fn<(), Output = i32> + core[ee3e5f980a4e04aa]::panic::unwind_safe::RefUnwindSafe + core[ee3e5f980a4e04aa]::marker::Sync as core[ee3e5f980a4e04aa]::ops::function::FnOnce<()>>::call_once
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/core/src/ops/function.rs:287:21
[INFO] [stderr]     24:     0x60993ba42f38 - std[18c5f2a687b463a0]::panicking::catch_unwind::do_call::<&dyn core[ee3e5f980a4e04aa]::ops::function::Fn<(), Output = i32> + core[ee3e5f980a4e04aa]::panic::unwind_safe::RefUnwindSafe + core[ee3e5f980a4e04aa]::marker::Sync, i32>
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panicking.rs:573:43
[INFO] [stderr]     25:     0x60993ba42f38 - std[18c5f2a687b463a0]::panicking::catch_unwind::<i32, &dyn core[ee3e5f980a4e04aa]::ops::function::Fn<(), Output = i32> + core[ee3e5f980a4e04aa]::panic::unwind_safe::RefUnwindSafe + core[ee3e5f980a4e04aa]::marker::Sync>
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panicking.rs:541:19
[INFO] [stderr]     26:     0x60993ba42f38 - std[18c5f2a687b463a0]::panic::catch_unwind::<&dyn core[ee3e5f980a4e04aa]::ops::function::Fn<(), Output = i32> + core[ee3e5f980a4e04aa]::panic::unwind_safe::RefUnwindSafe + core[ee3e5f980a4e04aa]::marker::Sync, i32>
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panic.rs:359:14
[INFO] [stderr]     27:     0x60993ba42f38 - std[18c5f2a687b463a0]::rt::lang_start_internal::{closure#0}
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/rt.rs:175:24
[INFO] [stderr]     28:     0x60993ba42f38 - std[18c5f2a687b463a0]::panicking::catch_unwind::do_call::<std[18c5f2a687b463a0]::rt::lang_start_internal::{closure#0}, isize>
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panicking.rs:573:43
[INFO] [stderr]     29:     0x60993ba42f38 - std[18c5f2a687b463a0]::panicking::catch_unwind::<isize, std[18c5f2a687b463a0]::rt::lang_start_internal::{closure#0}>
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panicking.rs:541:19
[INFO] [stderr]     30:     0x60993ba42f38 - std[18c5f2a687b463a0]::panic::catch_unwind::<std[18c5f2a687b463a0]::rt::lang_start_internal::{closure#0}, isize>
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/panic.rs:359:14
[INFO] [stderr]     31:     0x60993ba42f38 - std[18c5f2a687b463a0]::rt::lang_start_internal
[INFO] [stderr]                                  at /rustc/f4dd856795debe636f2ba39a900f162f5abeb7e8/library/std/src/rt.rs:171:5
[INFO] [stderr]     32:     0x60993ba1baa1 - std[18c5f2a687b463a0]::rt::lang_start::<()>
[INFO] [stderr]     33:     0x60993ba1fd95 - main
[INFO] [stderr]     34:     0x7339f8522601 - <unknown>
[INFO] [stderr]     35:     0x7339f8522718 - __libc_start_main
[INFO] [stderr]     36:     0x60993ba1b3b5 - _start
[INFO] [stderr]     37:                0x0 - <unknown>
[INFO] [stderr] warning: build failed, waiting for other jobs to finish...
[INFO] running `Command { std: "docker" "inspect" "09f5bdc89f839233d35cd31b049c4aae40c431b27fece1ee2b5400e8afca1856", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "09f5bdc89f839233d35cd31b049c4aae40c431b27fece1ee2b5400e8afca1856", kill_on_drop: false }`
[INFO] [stdout] 09f5bdc89f839233d35cd31b049c4aae40c431b27fece1ee2b5400e8afca1856
