[INFO] cloning repository https://github.com/Googlefan256/faster-llm-training
[INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/Googlefan256/faster-llm-training" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FGooglefan256%2Ffaster-llm-training", kill_on_drop: false }`
[INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FGooglefan256%2Ffaster-llm-training'...
[INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }`
[INFO] [stdout] b368c8d0b04a33343b50752a9eecade20272a4d6
[INFO] checking Googlefan256/faster-llm-training against master#5a45ab9738330fb317d49e3594c2db5248b1e971 for pr-136303
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FGooglefan256%2Ffaster-llm-training" "/workspace/builds/worker-1-tc1/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-1-tc1/source'...
[INFO] [stderr] done.
[INFO] validating manifest of git repo https://github.com/Googlefan256/faster-llm-training on toolchain 5a45ab9738330fb317d49e3594c2db5248b1e971
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+5a45ab9738330fb317d49e3594c2db5248b1e971" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] started tweaking git repo https://github.com/Googlefan256/faster-llm-training
[INFO] finished tweaking git repo https://github.com/Googlefan256/faster-llm-training
[INFO] tweaked toml for git repo https://github.com/Googlefan256/faster-llm-training written to /workspace/builds/worker-1-tc1/source/Cargo.toml
[INFO] crate git repo https://github.com/Googlefan256/faster-llm-training already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+5a45ab9738330fb317d49e3594c2db5248b1e971" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]     Updating git repository `https://github.com/huggingface/candle.git`
[INFO] [stderr]     Updating git submodule `https://github.com/NVIDIA/cutlass.git`
[INFO] [stderr]     Updating crates.io index
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded monostate-impl v0.1.13
[INFO] [stderr]   Downloaded bindgen_cuda v0.1.5
[INFO] [stderr]   Downloaded macro_rules_attribute-proc_macro v0.2.0
[INFO] [stderr]   Downloaded ug v0.0.2
[INFO] [stderr]   Downloaded ug-cuda v0.0.2
[INFO] [stderr]   Downloaded tokenizers v0.21.0
[INFO] [stderr]   Downloaded macro_rules_attribute v0.2.0
[INFO] [stderr]   Downloaded monostate v0.1.13
[INFO] [stderr]   Downloaded ureq v2.10.1
[INFO] [stderr]   Downloaded rayon-cond v0.3.0
[INFO] [stderr]   Downloaded cudarc v0.12.1
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:c80049f3b88b82089a44e0f06d0d6029d44b96b7257e55a1cd63dbc9f4c33334" "/opt/rustwide/cargo-home/bin/cargo" "+5a45ab9738330fb317d49e3594c2db5248b1e971" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] [stdout] 1c9138da7301cbc5ab8a4e315a94d723a27f9f61928766359019c3f442ea004a
[INFO] running `Command { std: "docker" "start" "-a" "1c9138da7301cbc5ab8a4e315a94d723a27f9f61928766359019c3f442ea004a", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "1c9138da7301cbc5ab8a4e315a94d723a27f9f61928766359019c3f442ea004a", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "1c9138da7301cbc5ab8a4e315a94d723a27f9f61928766359019c3f442ea004a", kill_on_drop: false }`
[INFO] [stdout] 1c9138da7301cbc5ab8a4e315a94d723a27f9f61928766359019c3f442ea004a
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:c80049f3b88b82089a44e0f06d0d6029d44b96b7257e55a1cd63dbc9f4c33334" "/opt/rustwide/cargo-home/bin/cargo" "+5a45ab9738330fb317d49e3594c2db5248b1e971" "check" "--frozen" "--all" "--all-targets" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] aeb3953ad1969b4466c0d3ef24d7917c9f69cc1fae0e65fdee4786e3c8360dcb
[INFO] running `Command { std: "docker" "start" "-a" "aeb3953ad1969b4466c0d3ef24d7917c9f69cc1fae0e65fdee4786e3c8360dcb", kill_on_drop: false }`
[INFO] [stderr]    Compiling proc-macro2 v1.0.92
[INFO] [stderr]    Compiling unicode-ident v1.0.14
[INFO] [stderr]    Compiling libc v0.2.167
[INFO] [stderr]    Compiling autocfg v1.4.0
[INFO] [stderr]    Compiling libm v0.2.11
[INFO] [stderr]    Compiling shlex v1.3.0
[INFO] [stderr]     Checking once_cell v1.20.2
[INFO] [stderr]    Compiling crossbeam-utils v0.8.20
[INFO] [stderr]     Checking memchr v2.7.4
[INFO] [stderr]     Checking stable_deref_trait v1.2.0
[INFO] [stderr]    Compiling pkg-config v0.3.31
[INFO] [stderr]     Checking either v1.13.0
[INFO] [stderr]     Checking itoa v1.0.14
[INFO] [stderr]    Compiling serde v1.0.215
[INFO] [stderr]     Checking pin-project-lite v0.2.15
[INFO] [stderr]    Compiling vcpkg v0.2.15
[INFO] [stderr]    Compiling paste v1.0.15
[INFO] [stderr]    Compiling glob v0.3.1
[INFO] [stderr]     Checking writeable v0.5.5
[INFO] [stderr]     Checking litemap v0.7.4
[INFO] [stderr]    Compiling cc v1.2.2
[INFO] [stderr]     Checking reborrow v0.5.5
[INFO] [stderr]     Checking log v0.4.22
[INFO] [stderr]     Checking bytes v1.9.0
[INFO] [stderr]     Checking futures-sink v0.3.31
[INFO] [stderr]     Checking futures-core v0.3.31
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]    Compiling slab v0.4.9
[INFO] [stderr]    Compiling serde_json v1.0.133
[INFO] [stderr]     Checking bitflags v2.6.0
[INFO] [stderr]     Checking equivalent v1.0.1
[INFO] [stderr]     Checking hashbrown v0.15.2
[INFO] [stderr]     Checking icu_locid_transform_data v1.5.0
[INFO] [stderr]     Checking futures-channel v0.3.31
[INFO] [stderr]     Checking raw-cpuid v10.7.0
[INFO] [stderr]     Checking smallvec v1.13.2
[INFO] [stderr]    Compiling openssl v0.10.68
[INFO] [stderr]     Checking icu_properties_data v1.5.0
[INFO] [stderr]    Compiling thiserror v1.0.69
[INFO] [stderr]    Compiling seq-macro v0.3.5
[INFO] [stderr]     Checking futures-io v0.3.31
[INFO] [stderr]     Checking futures-task v0.3.31
[INFO] [stderr]     Checking foreign-types-shared v0.1.1
[INFO] [stderr]    Compiling crossbeam-epoch v0.9.18
[INFO] [stderr]     Checking foreign-types v0.3.2
[INFO] [stderr]     Checking libloading v0.8.6
[INFO] [stderr]    Compiling quote v1.0.37
[INFO] [stderr]     Checking utf16_iter v1.0.5
[INFO] [stderr]     Checking write16 v1.0.0
[INFO] [stderr]    Compiling syn v2.0.90
[INFO] [stderr]     Checking icu_normalizer_data v1.5.0
[INFO] [stderr]    Compiling native-tls v0.2.12
[INFO] [stderr]     Checking utf8_iter v1.0.4
[INFO] [stderr]    Compiling crossbeam-deque v0.8.5
[INFO] [stderr]     Checking tracing-core v0.1.33
[INFO] [stderr]     Checking http v0.2.12
[INFO] [stderr]     Checking crc32fast v1.4.2
[INFO] [stderr]    Compiling strsim v0.11.1
[INFO] [stderr]    Compiling fnv v1.0.7
[INFO] [stderr]     Checking minimal-lexical v0.2.1
[INFO] [stderr]    Compiling toml_datetime v0.6.8
[INFO] [stderr]    Compiling rayon-core v1.12.1
[INFO] [stderr]     Checking getrandom v0.2.15
[INFO] [stderr]     Checking mio v1.0.3
[INFO] [stderr]     Checking socket2 v0.5.8
[INFO] [stderr]    Compiling httparse v1.9.5
[INFO] [stderr]     Checking num-integer v0.1.46
[INFO] [stderr]     Checking rand_core v0.6.4
[INFO] [stderr]    Compiling ident_case v1.0.1
[INFO] [stderr]     Checking openssl-probe v0.1.5
[INFO] [stderr]    Compiling winnow v0.6.20
[INFO] [stderr]    Compiling rayon v1.10.0
[INFO] [stderr]     Checking indexmap v2.6.0
[INFO] [stderr]    Compiling num_cpus v1.16.0
[INFO] [stderr]     Checking tracing v0.1.41
[INFO] [stderr]     Checking form_urlencoded v1.2.1
[INFO] [stderr]     Checking num-bigint v0.4.6
[INFO] [stderr]     Checking nom v7.1.3
[INFO] [stderr]     Checking aho-corasick v1.1.3
[INFO] [stderr]     Checking regex-syntax v0.8.5
[INFO] [stderr]    Compiling anyhow v1.0.93
[INFO] [stderr]     Checking rustls-pki-types v1.10.0
[INFO] [stderr]     Checking untrusted v0.9.0
[INFO] [stderr]     Checking spin v0.9.8
[INFO] [stderr]     Checking try-lock v0.2.5
[INFO] [stderr]     Checking want v0.3.1
[INFO] [stderr]     Checking num-iter v0.1.45
[INFO] [stderr]    Compiling rustls v0.23.19
[INFO] [stderr]     Checking tower-service v0.3.3
[INFO] [stderr]    Compiling portable-atomic v1.10.0
[INFO] [stderr]     Checking httpdate v1.0.3
[INFO] [stderr]     Checking adler2 v2.0.0
[INFO] [stderr]    Compiling cudarc v0.12.1
[INFO] [stderr]     Checking http-body v0.4.6
[INFO] [stderr]     Checking miniz_oxide v0.8.0
[INFO] [stderr]     Checking itertools v0.12.1
[INFO] [stderr]    Compiling openssl-sys v0.9.104
[INFO] [stderr]    Compiling ring v0.17.8
[INFO] [stderr] error: failed to run custom build command for `cudarc v0.12.1`
[INFO] [stderr] note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `/opt/rustwide/target/debug/build/cudarc-8aa46bb1c04d3141/build-script-build` (exit status: 101)
[INFO] [stderr]   --- stdout
[INFO] [stderr]   cargo:rerun-if-changed=build.rs
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_ROOT
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_PATH
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_TOOLKIT_ROOT_DIR
[INFO] [stderr] 
[INFO] [stderr]   --- stderr
[INFO] [stderr] 
[INFO] [stderr]   thread 'main' panicked at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/cudarc-0.12.1/build.rs:57:10:
[INFO] [stderr]   Failed to execute `nvcc`: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[INFO] [stderr]   stack backtrace:
[INFO] [stderr]      0:     0x5b69f308d9b0 - std::backtrace_rs::backtrace::libunwind::trace::h00ed8f7a49b23656
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
[INFO] [stderr]      1:     0x5b69f308d9b0 - std::backtrace_rs::backtrace::trace_unsynchronized::hb9a616e2fc1d376f
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
[INFO] [stderr]      2:     0x5b69f308d9b0 - std::sys::backtrace::_print_fmt::h7db298e83a256d29
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/sys/backtrace.rs:66:9
[INFO] [stderr]      3:     0x5b69f308d9b0 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hd1ae340659526716
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/sys/backtrace.rs:39:26
[INFO] [stderr]      4:     0x5b69f30aecf3 - core::fmt::rt::Argument::fmt::hac01a348e20ee70d
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/core/src/fmt/rt.rs:177:76
[INFO] [stderr]      5:     0x5b69f30aecf3 - core::fmt::write::hc504a50407028ef6
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/core/src/fmt/mod.rs:1449:21
[INFO] [stderr]      6:     0x5b69f308aae3 - std::io::Write::write_fmt::ha00ae9415040e09a
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/io/mod.rs:1890:15
[INFO] [stderr]      7:     0x5b69f308d802 - std::sys::backtrace::BacktraceLock::print::h6f3c8a163b5a5ba5
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/sys/backtrace.rs:42:9
[INFO] [stderr]      8:     0x5b69f308e74f - std::panicking::default_hook::{{closure}}::h4661160df59af739
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panicking.rs:298:22
[INFO] [stderr]      9:     0x5b69f308e5ba - std::panicking::default_hook::h79b0b0cb5c9a45eb
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panicking.rs:325:9
[INFO] [stderr]     10:     0x5b69f308f022 - std::panicking::rust_panic_with_hook::he8f5885f4873cada
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panicking.rs:831:13
[INFO] [stderr]     11:     0x5b69f308eeba - std::panicking::begin_panic_handler::{{closure}}::hce5053f4756cff33
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panicking.rs:704:13
[INFO] [stderr]     12:     0x5b69f308deb9 - std::sys::backtrace::__rust_end_short_backtrace::h634718bdeca7b4fe
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/sys/backtrace.rs:168:18
[INFO] [stderr]     13:     0x5b69f308eb4d - rust_begin_unwind
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panicking.rs:695:5
[INFO] [stderr]     14:     0x5b69f30adf20 - core::panicking::panic_fmt::h0e90de0256284326
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/core/src/panicking.rs:75:14
[INFO] [stderr]     15:     0x5b69f30ae2c6 - core::result::unwrap_failed::h5fc58cd3e583ac28
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/core/src/result.rs:1704:5
[INFO] [stderr]     16:     0x5b69f3063ece - core::result::Result<T,E>::expect::hab35186372c430bd
[INFO] [stderr]     17:     0x5b69f3066b76 - build_script_build::cuda_version_from_build_system::hca20a570a1e85ea2
[INFO] [stderr]     18:     0x5b69f3066912 - build_script_build::main::h14f6a0b32300e4a1
[INFO] [stderr]     19:     0x5b69f30625a3 - core::ops::function::FnOnce::call_once::hd5c4a5c181af5bed
[INFO] [stderr]     20:     0x5b69f30622f6 - std::sys::backtrace::__rust_begin_short_backtrace::h3fd911e5547fe451
[INFO] [stderr]     21:     0x5b69f3064f09 - std::rt::lang_start::{{closure}}::hdc7974243411bb91
[INFO] [stderr]     22:     0x5b69f3087670 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h1b551c6ace84f634
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/core/src/ops/function.rs:284:13
[INFO] [stderr]     23:     0x5b69f3087670 - std::panicking::try::do_call::hcda773f48bd80f09
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panicking.rs:587:40
[INFO] [stderr]     24:     0x5b69f3087670 - std::panicking::try::h225c38e862182721
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panicking.rs:550:19
[INFO] [stderr]     25:     0x5b69f3087670 - std::panic::catch_unwind::h6eb4a99877840d20
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panic.rs:358:14
[INFO] [stderr]     26:     0x5b69f3087670 - std::rt::lang_start_internal::{{closure}}::h64597ebca62f6e02
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/rt.rs:168:24
[INFO] [stderr]     27:     0x5b69f3087670 - std::panicking::try::do_call::ha3630280e291428a
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panicking.rs:587:40
[INFO] [stderr]     28:     0x5b69f3087670 - std::panicking::try::h13d8cfc120ae8ff3
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panicking.rs:550:19
[INFO] [stderr]     29:     0x5b69f3087670 - std::panic::catch_unwind::h9b1225c928e48a40
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/panic.rs:358:14
[INFO] [stderr]     30:     0x5b69f3087670 - std::rt::lang_start_internal::h27f44511957e98ab
[INFO] [stderr]                                  at /rustc/5a45ab9738330fb317d49e3594c2db5248b1e971/library/std/src/rt.rs:164:5
[INFO] [stderr]     31:     0x5b69f3064ef1 - std::rt::lang_start::h5b2bfc63e5ec068b
[INFO] [stderr]     32:     0x5b69f3068b25 - main
[INFO] [stderr]     33:     0x7c3c8492cd90 - <unknown>
[INFO] [stderr]     34:     0x7c3c8492ce40 - __libc_start_main
[INFO] [stderr]     35:     0x5b69f305c5a5 - _start
[INFO] [stderr]     36:                0x0 - <unknown>
[INFO] [stderr] warning: build failed, waiting for other jobs to finish...
[INFO] running `Command { std: "docker" "inspect" "aeb3953ad1969b4466c0d3ef24d7917c9f69cc1fae0e65fdee4786e3c8360dcb", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "aeb3953ad1969b4466c0d3ef24d7917c9f69cc1fae0e65fdee4786e3c8360dcb", kill_on_drop: false }`
[INFO] [stdout] aeb3953ad1969b4466c0d3ef24d7917c9f69cc1fae0e65fdee4786e3c8360dcb
