[INFO] cloning repository https://github.com/vishpat/gemma-slurm
[INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/vishpat/gemma-slurm" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fvishpat%2Fgemma-slurm", kill_on_drop: false }`
[INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fvishpat%2Fgemma-slurm'...
[INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }`
[INFO] [stdout] 0cd32cfd0a2959c4878ba632de18d63fd0896cb1
[INFO] checking vishpat/gemma-slurm against try#54cf65b8d066c082e536455049b95943cf30378f for pr-151920
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fvishpat%2Fgemma-slurm" "/workspace/builds/worker-4-tc2/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-4-tc2/source'...
[INFO] [stderr] done.
[INFO] started tweaking git repo https://github.com/vishpat/gemma-slurm
[INFO] finished tweaking git repo https://github.com/vishpat/gemma-slurm
[INFO] tweaked toml for git repo https://github.com/vishpat/gemma-slurm written to /workspace/builds/worker-4-tc2/source/Cargo.toml
[INFO] validating manifest of git repo https://github.com/vishpat/gemma-slurm on toolchain 54cf65b8d066c082e536455049b95943cf30378f
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+54cf65b8d066c082e536455049b95943cf30378f" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate git repo https://github.com/vishpat/gemma-slurm already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+54cf65b8d066c082e536455049b95943cf30378f" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]     Updating git repository `https://github.com/huggingface/candle`
[INFO] [stderr]     Updating git submodule `https://github.com/NVIDIA/cutlass.git`
[INFO] [stderr]     Updating crates.io index
[INFO] [stderr]     Updating git repository `https://github.com/zackangelo/float8`
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded bitflags v2.9.3
[INFO] [stderr]   Downloaded rayon-cond v0.4.0
[INFO] [stderr]   Downloaded macro_rules_attribute v0.2.2
[INFO] [stderr]   Downloaded monostate-impl v0.1.14
[INFO] [stderr]   Downloaded bindgen_cuda v0.1.5
[INFO] [stderr]   Downloaded macro_rules_attribute-proc_macro v0.2.2
[INFO] [stderr]   Downloaded socks v0.3.4
[INFO] [stderr]   Downloaded monostate v0.1.14
[INFO] [stderr]   Downloaded ug-cuda v0.4.0
[INFO] [stderr]   Downloaded dyn-stack v0.13.0
[INFO] [stderr]   Downloaded dary_heap v0.3.7
[INFO] [stderr]   Downloaded hf-hub v0.4.3
[INFO] [stderr]   Downloaded unicode-normalization-alignments v0.1.12
[INFO] [stderr]   Downloaded indexmap v2.11.0
[INFO] [stderr]   Downloaded tokenizers v0.21.4
[INFO] [stderr]   Downloaded esaxx-rs v0.1.10
[INFO] [stderr]   Downloaded image v0.25.6
[INFO] [stderr]   Downloaded webpki-roots v1.0.0
[INFO] [stderr]   Downloaded cudarc v0.16.6
[INFO] [stderr]   Downloaded spm_precompiled v0.1.4
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:61361fe0aef631f17e9d025a70c5a647956f8c671dd02950a60ad3f5cc5526d7" "/opt/rustwide/cargo-home/bin/cargo" "+54cf65b8d066c082e536455049b95943cf30378f" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] [stdout] 43b2c69ee7a02749ab36704010429eeef192aff6826b234fecad26afbe71191d
[INFO] running `Command { std: "docker" "start" "-a" "43b2c69ee7a02749ab36704010429eeef192aff6826b234fecad26afbe71191d", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "43b2c69ee7a02749ab36704010429eeef192aff6826b234fecad26afbe71191d", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "43b2c69ee7a02749ab36704010429eeef192aff6826b234fecad26afbe71191d", kill_on_drop: false }`
[INFO] [stdout] 43b2c69ee7a02749ab36704010429eeef192aff6826b234fecad26afbe71191d
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:61361fe0aef631f17e9d025a70c5a647956f8c671dd02950a60ad3f5cc5526d7" "/opt/rustwide/cargo-home/bin/cargo" "+54cf65b8d066c082e536455049b95943cf30378f" "check" "--frozen" "--all" "--all-targets" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] 96f5e95277ba7287d28b540a7cc30c9ebdf69f6d3af8a9b92de69f32f8d68400
[INFO] running `Command { std: "docker" "start" "-a" "96f5e95277ba7287d28b540a7cc30c9ebdf69f6d3af8a9b92de69f32f8d68400", kill_on_drop: false }`
[INFO] [stderr]    Compiling crossbeam-utils v0.8.21
[INFO] [stderr]    Compiling libm v0.2.15
[INFO] [stderr]    Compiling rayon-core v1.13.0
[INFO] [stderr]     Checking memchr v2.7.5
[INFO] [stderr]     Checking bitflags v2.9.3
[INFO] [stderr]    Compiling proc-macro2 v1.0.101
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]     Checking either v1.15.0
[INFO] [stderr]     Checking reborrow v0.5.5
[INFO] [stderr]    Compiling serde v1.0.219
[INFO] [stderr]    Compiling seq-macro v0.3.6
[INFO] [stderr]    Compiling lock_api v0.4.13
[INFO] [stderr]     Checking rand_chacha v0.9.0
[INFO] [stderr]    Compiling pulp v0.21.5
[INFO] [stderr]     Checking parking_lot_core v0.9.11
[INFO] [stderr]    Compiling serde_json v1.0.143
[INFO] [stderr]    Compiling openssl-sys v0.9.109
[INFO] [stderr]     Checking mio v1.0.4
[INFO] [stderr]     Checking raw-cpuid v11.5.0
[INFO] [stderr]     Checking signal-hook-registry v1.4.6
[INFO] [stderr]     Checking socket2 v0.6.0
[INFO] [stderr]     Checking rand v0.9.2
[INFO] [stderr]    Compiling quote v1.0.40
[INFO] [stderr]    Compiling libc v0.2.175
[INFO] [stderr]     Checking hashbrown v0.15.5
[INFO] [stderr]     Checking raw-cpuid v10.7.0
[INFO] [stderr]    Compiling syn v2.0.106
[INFO] [stderr]    Compiling rustversion v1.0.22
[INFO] [stderr]    Compiling glob v0.3.3
[INFO] [stderr]     Checking parking_lot v0.12.4
[INFO] [stderr]    Compiling openssl v0.10.73
[INFO] [stderr]    Compiling crossbeam-epoch v0.9.18
[INFO] [stderr]    Compiling ring v0.17.14
[INFO] [stderr]    Compiling num_cpus v1.17.0
[INFO] [stderr]    Compiling winnow v0.7.13
[INFO] [stderr]    Compiling native-tls v0.2.14
[INFO] [stderr]    Compiling crossbeam-deque v0.8.6
[INFO] [stderr]     Checking indexmap v2.11.0
[INFO] [stderr]     Checking libloading v0.8.8
[INFO] [stderr]    Compiling cudarc v0.16.6
[INFO] [stderr]     Checking adler2 v2.0.1
[INFO] [stderr]     Checking getrandom v0.2.16
[INFO] [stderr]     Checking miniz_oxide v0.8.9
[INFO] [stderr]    Compiling object v0.36.7
[INFO] [stderr]     Checking aho-corasick v1.1.3
[INFO] [stderr] error: failed to run custom build command for `cudarc v0.16.6`
[INFO] [stderr] note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `/opt/rustwide/target/debug/build/cudarc-bd367dd8e93933f4/build-script-build` (exit status: 101)
[INFO] [stderr]   --- stdout
[INFO] [stderr]   cargo:rerun-if-changed=build.rs
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_ROOT
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_PATH
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_TOOLKIT_ROOT_DIR
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDARC_CUDA_VERSION
[INFO] [stderr] 
[INFO] [stderr]   --- stderr
[INFO] [stderr] 
[INFO] [stderr]   thread 'main' (707) panicked at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/cudarc-0.16.6/build.rs:98:10:
[INFO] [stderr]   Failed to execute `nvcc`: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[INFO] [stderr]   stack backtrace:
[INFO] [stderr]      0:     0x654b56809d5a - std[2b8fb8c27541df22]::backtrace_rs::backtrace::libunwind::trace
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stderr]      1:     0x654b56809d5a - std[2b8fb8c27541df22]::backtrace_rs::backtrace::trace_unsynchronized::<std[2b8fb8c27541df22]::sys::backtrace::_print_fmt::{closure#1}>
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stderr]      2:     0x654b56809d5a - std[2b8fb8c27541df22]::sys::backtrace::_print_fmt
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/sys/backtrace.rs:74:9
[INFO] [stderr]      3:     0x654b56809d5a - <<std[2b8fb8c27541df22]::sys::backtrace::BacktraceLock>::print::DisplayBacktrace as core[ce8ad42acf3c47be]::fmt::Display>::fmt
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/sys/backtrace.rs:44:26
[INFO] [stderr]      4:     0x654b5681b26a - <core[ce8ad42acf3c47be]::fmt::rt::Argument>::fmt
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/core/src/fmt/rt.rs:152:76
[INFO] [stderr]      5:     0x654b5681b26a - core[ce8ad42acf3c47be]::fmt::write
[INFO] [stderr]      6:     0x654b5680e082 - std[2b8fb8c27541df22]::io::default_write_fmt::<std[2b8fb8c27541df22]::sys::stdio::unix::Stderr>
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/io/mod.rs:639:11
[INFO] [stderr]      7:     0x654b5680e082 - <std[2b8fb8c27541df22]::sys::stdio::unix::Stderr as std[2b8fb8c27541df22]::io::Write>::write_fmt
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/io/mod.rs:1994:13
[INFO] [stderr]      8:     0x654b567ef5cf - <std[2b8fb8c27541df22]::sys::backtrace::BacktraceLock>::print
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/sys/backtrace.rs:47:9
[INFO] [stderr]      9:     0x654b567ef5cf - std[2b8fb8c27541df22]::panicking::default_hook::{closure#0}
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panicking.rs:292:27
[INFO] [stderr]     10:     0x654b568059c1 - std[2b8fb8c27541df22]::panicking::default_hook
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panicking.rs:319:9
[INFO] [stderr]     11:     0x654b56805b7b - std[2b8fb8c27541df22]::panicking::panic_with_hook
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panicking.rs:825:13
[INFO] [stderr]     12:     0x654b567ef688 - std[2b8fb8c27541df22]::panicking::panic_handler::{closure#0}
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panicking.rs:698:13
[INFO] [stderr]     13:     0x654b567e4a19 - std[2b8fb8c27541df22]::sys::backtrace::__rust_end_short_backtrace::<std[2b8fb8c27541df22]::panicking::panic_handler::{closure#0}, !>
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/sys/backtrace.rs:182:18
[INFO] [stderr]     14:     0x654b567efeed - __rustc[23a7528c93bdc9b2]::rust_begin_unwind
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panicking.rs:689:5
[INFO] [stderr]     15:     0x654b5681b8dc - core[ce8ad42acf3c47be]::panicking::panic_fmt
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/core/src/panicking.rs:80:14
[INFO] [stderr]     16:     0x654b5681b6b2 - core[ce8ad42acf3c47be]::result::unwrap_failed
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/core/src/result.rs:1867:5
[INFO] [stderr]     17:     0x654b567d1872 - <core[ce8ad42acf3c47be]::result::Result<std[2b8fb8c27541df22]::process::Output, std[2b8fb8c27541df22]::io::error::Error>>::expect
[INFO] [stderr]     18:     0x654b567d0566 - build_script_build[c1b8659d85251a99]::cuda_version_from_build_system
[INFO] [stderr]     19:     0x654b567d0f54 - build_script_build[c1b8659d85251a99]::main
[INFO] [stderr]     20:     0x654b567d8923 - <fn() as core[ce8ad42acf3c47be]::ops::function::FnOnce<()>>::call_once
[INFO] [stderr]     21:     0x654b567ceb16 - std[2b8fb8c27541df22]::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()>
[INFO] [stderr]     22:     0x654b567cf379 - std[2b8fb8c27541df22]::rt::lang_start::<()>::{closure#0}
[INFO] [stderr]     23:     0x654b56804c06 - <&dyn core[ce8ad42acf3c47be]::ops::function::Fn<(), Output = i32> + core[ce8ad42acf3c47be]::marker::Sync + core[ce8ad42acf3c47be]::panic::unwind_safe::RefUnwindSafe as core[ce8ad42acf3c47be]::ops::function::FnOnce<()>>::call_once
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/core/src/ops/function.rs:287:21
[INFO] [stderr]     24:     0x654b56804c06 - std[2b8fb8c27541df22]::panicking::catch_unwind::do_call::<&dyn core[ce8ad42acf3c47be]::ops::function::Fn<(), Output = i32> + core[ce8ad42acf3c47be]::marker::Sync + core[ce8ad42acf3c47be]::panic::unwind_safe::RefUnwindSafe, i32>
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panicking.rs:581:40
[INFO] [stderr]     25:     0x654b56804c06 - std[2b8fb8c27541df22]::panicking::catch_unwind::<i32, &dyn core[ce8ad42acf3c47be]::ops::function::Fn<(), Output = i32> + core[ce8ad42acf3c47be]::marker::Sync + core[ce8ad42acf3c47be]::panic::unwind_safe::RefUnwindSafe>
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panicking.rs:544:19
[INFO] [stderr]     26:     0x654b56804c06 - std[2b8fb8c27541df22]::panic::catch_unwind::<&dyn core[ce8ad42acf3c47be]::ops::function::Fn<(), Output = i32> + core[ce8ad42acf3c47be]::marker::Sync + core[ce8ad42acf3c47be]::panic::unwind_safe::RefUnwindSafe, i32>
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panic.rs:359:14
[INFO] [stderr]     27:     0x654b56804c06 - std[2b8fb8c27541df22]::rt::lang_start_internal::{closure#0}
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/rt.rs:175:24
[INFO] [stderr]     28:     0x654b56804c06 - std[2b8fb8c27541df22]::panicking::catch_unwind::do_call::<std[2b8fb8c27541df22]::rt::lang_start_internal::{closure#0}, isize>
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panicking.rs:581:40
[INFO] [stderr]     29:     0x654b56804c06 - std[2b8fb8c27541df22]::panicking::catch_unwind::<isize, std[2b8fb8c27541df22]::rt::lang_start_internal::{closure#0}>
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panicking.rs:544:19
[INFO] [stderr]     30:     0x654b56804c06 - std[2b8fb8c27541df22]::panic::catch_unwind::<std[2b8fb8c27541df22]::rt::lang_start_internal::{closure#0}, isize>
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/panic.rs:359:14
[INFO] [stderr]     31:     0x654b56804c06 - std[2b8fb8c27541df22]::rt::lang_start_internal
[INFO] [stderr]                                  at /rustc/54cf65b8d066c082e536455049b95943cf30378f/library/std/src/rt.rs:171:5
[INFO] [stderr]     32:     0x654b567ceb01 - std[2b8fb8c27541df22]::rt::lang_start::<()>
[INFO] [stderr]     33:     0x654b567d49a5 - main
[INFO] [stderr]     34:     0x758fc37da1ca - <unknown>
[INFO] [stderr]     35:     0x758fc37da28b - __libc_start_main
[INFO] [stderr]     36:     0x654b567cd8f5 - _start
[INFO] [stderr]     37:                0x0 - <unknown>
[INFO] [stderr] warning: build failed, waiting for other jobs to finish...
[INFO] running `Command { std: "docker" "inspect" "96f5e95277ba7287d28b540a7cc30c9ebdf69f6d3af8a9b92de69f32f8d68400", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "96f5e95277ba7287d28b540a7cc30c9ebdf69f6d3af8a9b92de69f32f8d68400", kill_on_drop: false }`
[INFO] [stdout] 96f5e95277ba7287d28b540a7cc30c9ebdf69f6d3af8a9b92de69f32f8d68400
