[INFO] cloning repository https://github.com/Googlefan256/faster-llm-training
[INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/Googlefan256/faster-llm-training" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FGooglefan256%2Ffaster-llm-training", kill_on_drop: false }`
[INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FGooglefan256%2Ffaster-llm-training'...
[INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }`
[INFO] [stdout] b368c8d0b04a33343b50752a9eecade20272a4d6
[INFO] checking Googlefan256/faster-llm-training against try#2b39fe756f52bc2825bf5b2bfc385cab922dae10 for pr-143011
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2FGooglefan256%2Ffaster-llm-training" "/workspace/builds/worker-6-tc2/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-6-tc2/source'...
[INFO] [stderr] done.
[INFO] started tweaking git repo https://github.com/Googlefan256/faster-llm-training
[INFO] finished tweaking git repo https://github.com/Googlefan256/faster-llm-training
[INFO] tweaked toml for git repo https://github.com/Googlefan256/faster-llm-training written to /workspace/builds/worker-6-tc2/source/Cargo.toml
[INFO] validating manifest of git repo https://github.com/Googlefan256/faster-llm-training on toolchain 2b39fe756f52bc2825bf5b2bfc385cab922dae10
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+2b39fe756f52bc2825bf5b2bfc385cab922dae10" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate git repo https://github.com/Googlefan256/faster-llm-training already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+2b39fe756f52bc2825bf5b2bfc385cab922dae10" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:90999bfc7ae267e83380e433d8e61a7c072ca6729e92edbae886d3423b3a6f4c" "/opt/rustwide/cargo-home/bin/cargo" "+2b39fe756f52bc2825bf5b2bfc385cab922dae10" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] [stdout] 49faedeb72573c1e850cc8d15cf5a6fb73d9499096bbdff68e6d6d38f7c77c46
[INFO] running `Command { std: "docker" "start" "-a" "49faedeb72573c1e850cc8d15cf5a6fb73d9499096bbdff68e6d6d38f7c77c46", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "49faedeb72573c1e850cc8d15cf5a6fb73d9499096bbdff68e6d6d38f7c77c46", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "49faedeb72573c1e850cc8d15cf5a6fb73d9499096bbdff68e6d6d38f7c77c46", kill_on_drop: false }`
[INFO] [stdout] 49faedeb72573c1e850cc8d15cf5a6fb73d9499096bbdff68e6d6d38f7c77c46
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:90999bfc7ae267e83380e433d8e61a7c072ca6729e92edbae886d3423b3a6f4c" "/opt/rustwide/cargo-home/bin/cargo" "+2b39fe756f52bc2825bf5b2bfc385cab922dae10" "check" "--frozen" "--all" "--all-targets" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] 45cddfc739ebd5e99360f971683bd9e2b8acb121ed628720dca715c2c24cebdd
[INFO] running `Command { std: "docker" "start" "-a" "45cddfc739ebd5e99360f971683bd9e2b8acb121ed628720dca715c2c24cebdd", kill_on_drop: false }`
[INFO] [stderr]    Compiling quote v1.0.37
[INFO] [stderr]    Compiling libc v0.2.167
[INFO] [stderr]    Compiling libm v0.2.11
[INFO] [stderr]    Compiling crossbeam-utils v0.8.20
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]    Compiling cc v1.2.2
[INFO] [stderr]     Checking stable_deref_trait v1.2.0
[INFO] [stderr]    Compiling pkg-config v0.3.31
[INFO] [stderr]     Checking either v1.13.0
[INFO] [stderr]    Compiling serde v1.0.215
[INFO] [stderr]     Checking pin-project-lite v0.2.15
[INFO] [stderr]    Compiling syn v2.0.90
[INFO] [stderr]    Compiling paste v1.0.15
[INFO] [stderr]    Compiling vcpkg v0.2.15
[INFO] [stderr]    Compiling glob v0.3.1
[INFO] [stderr]     Checking litemap v0.7.4
[INFO] [stderr]     Checking bitflags v1.3.2
[INFO] [stderr]     Checking crossbeam-epoch v0.9.18
[INFO] [stderr]     Checking reborrow v0.5.5
[INFO] [stderr]     Checking log v0.4.22
[INFO] [stderr]     Checking bytes v1.9.0
[INFO] [stderr]     Checking crossbeam-deque v0.8.5
[INFO] [stderr]     Checking getrandom v0.2.15
[INFO] [stderr]    Compiling serde_json v1.0.133
[INFO] [stderr]     Checking rayon-core v1.12.1
[INFO] [stderr]     Checking icu_locid_transform_data v1.5.0
[INFO] [stderr]     Checking rand_core v0.6.4
[INFO] [stderr]     Checking equivalent v1.0.1
[INFO] [stderr]     Checking indexmap v2.6.0
[INFO] [stderr]     Checking futures-channel v0.3.31
[INFO] [stderr]     Checking socket2 v0.5.8
[INFO] [stderr]     Checking mio v1.0.3
[INFO] [stderr]     Checking raw-cpuid v10.7.0
[INFO] [stderr]     Checking rayon v1.10.0
[INFO] [stderr]     Checking foreign-types-shared v0.1.1
[INFO] [stderr]    Compiling openssl-sys v0.9.104
[INFO] [stderr]     Checking futures-io v0.3.31
[INFO] [stderr]    Compiling openssl v0.10.68
[INFO] [stderr]     Checking smallvec v1.13.2
[INFO] [stderr]     Checking icu_properties_data v1.5.0
[INFO] [stderr]    Compiling seq-macro v0.3.5
[INFO] [stderr]    Compiling ring v0.17.8
[INFO] [stderr]     Checking foreign-types v0.3.2
[INFO] [stderr]     Checking num-integer v0.1.46
[INFO] [stderr]     Checking libloading v0.8.6
[INFO] [stderr]    Compiling hashbrown v0.15.2
[INFO] [stderr]    Compiling native-tls v0.2.12
[INFO] [stderr]     Checking icu_normalizer_data v1.5.0
[INFO] [stderr]     Checking http v0.2.12
[INFO] [stderr]     Checking tracing-core v0.1.33
[INFO] [stderr]    Compiling fnv v1.0.7
[INFO] [stderr]    Compiling httparse v1.9.5
[INFO] [stderr]    Compiling toml_datetime v0.6.8
[INFO] [stderr]     Checking openssl-probe v0.1.5
[INFO] [stderr]    Compiling ident_case v1.0.1
[INFO] [stderr]    Compiling winnow v0.6.20
[INFO] [stderr]     Checking minimal-lexical v0.2.1
[INFO] [stderr]    Compiling strsim v0.11.1
[INFO] [stderr]     Checking tracing v0.1.41
[INFO] [stderr]     Checking nom v7.1.3
[INFO] [stderr]    Compiling num_cpus v1.16.0
[INFO] [stderr]     Checking num-bigint v0.4.6
[INFO] [stderr]     Checking aho-corasick v1.1.3
[INFO] [stderr]     Checking regex-syntax v0.8.5
[INFO] [stderr]     Checking rustls-pki-types v1.10.0
[INFO] [stderr]    Compiling synstructure v0.13.1
[INFO] [stderr]    Compiling darling_core v0.20.10
[INFO] [stderr]     Checking spin v0.9.8
[INFO] [stderr]     Checking untrusted v0.9.0
[INFO] [stderr]    Compiling anyhow v1.0.93
[INFO] [stderr]    Compiling toml_edit v0.22.22
[INFO] [stderr]     Checking num-rational v0.4.2
[INFO] [stderr]     Checking http-body v0.4.6
[INFO] [stderr]     Checking num-iter v0.1.45
[INFO] [stderr]     Checking regex-automata v0.4.9
[INFO] [stderr]    Compiling rustls v0.23.19
[INFO] [stderr]    Compiling cudarc v0.12.1
[INFO] [stderr]     Checking miniz_oxide v0.8.0
[INFO] [stderr]    Compiling bindgen_cuda v0.1.5
[INFO] [stderr] error: failed to run custom build command for `cudarc v0.12.1`
[INFO] [stderr] note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `/opt/rustwide/target/debug/build/cudarc-8aa46bb1c04d3141/build-script-build` (exit status: 101)
[INFO] [stderr]   --- stdout
[INFO] [stderr]   cargo:rerun-if-changed=build.rs
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_ROOT
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_PATH
[INFO] [stderr]   cargo:rerun-if-env-changed=CUDA_TOOLKIT_ROOT_DIR
[INFO] [stderr] 
[INFO] [stderr]   --- stderr
[INFO] [stderr] 
[INFO] [stderr]   thread 'main' panicked at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/cudarc-0.12.1/build.rs:57:10:
[INFO] [stderr]   Failed to execute `nvcc`: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[INFO] [stderr]   stack backtrace:
[INFO] [stderr]      0:     0x5eaf03bf8ba2 - std::backtrace_rs::backtrace::libunwind::trace::h7832c52827c08170
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stderr]      1:     0x5eaf03bf8ba2 - std::backtrace_rs::backtrace::trace_unsynchronized::he6d92e6eb24a4073
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stderr]      2:     0x5eaf03bf8ba2 - std::sys::backtrace::_print_fmt::hf139cc888d064b09
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/sys/backtrace.rs:66:9
[INFO] [stderr]      3:     0x5eaf03bf8ba2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h7083fbd4e20a000f
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/sys/backtrace.rs:39:26
[INFO] [stderr]      4:     0x5eaf03c18483 - core::fmt::rt::Argument::fmt::h9d8c637e6bfd5ccf
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/core/src/fmt/rt.rs:173:76
[INFO] [stderr]      5:     0x5eaf03c18483 - core::fmt::write::hdf3c6ae14c3e8125
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/core/src/fmt/mod.rs:1465:25
[INFO] [stderr]      6:     0x5eaf03bf6193 - std::io::default_write_fmt::h15936264287de7bf
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/io/mod.rs:639:11
[INFO] [stderr]      7:     0x5eaf03bf6193 - std::io::Write::write_fmt::h84df75adb307f7e6
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/io/mod.rs:1954:13
[INFO] [stderr]      8:     0x5eaf03bf89f2 - std::sys::backtrace::BacktraceLock::print::h30a2059d03e705e4
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/sys/backtrace.rs:42:9
[INFO] [stderr]      9:     0x5eaf03bf9a66 - std::panicking::default_hook::{{closure}}::hf489e9f54ce49c27
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panicking.rs:300:27
[INFO] [stderr]     10:     0x5eaf03bf9869 - std::panicking::default_hook::h20d961ad9a0e12f0
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panicking.rs:327:9
[INFO] [stderr]     11:     0x5eaf03bfa332 - std::panicking::rust_panic_with_hook::h306db6d00921085e
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panicking.rs:833:13
[INFO] [stderr]     12:     0x5eaf03bfa1ca - std::panicking::begin_panic_handler::{{closure}}::h19782e9851ecb030
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panicking.rs:706:13
[INFO] [stderr]     13:     0x5eaf03bf9099 - std::sys::backtrace::__rust_end_short_backtrace::h475e7ebac10ba75b
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/sys/backtrace.rs:168:18
[INFO] [stderr]     14:     0x5eaf03bf9e5d - __rustc[3b00513142ba1c9a]::rust_begin_unwind
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panicking.rs:697:5
[INFO] [stderr]     15:     0x5eaf03c17770 - core::panicking::panic_fmt::h6940cb0a7d852762
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/core/src/panicking.rs:75:14
[INFO] [stderr]     16:     0x5eaf03c17ae6 - core::result::unwrap_failed::h9414f4e046da6e5a
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/core/src/result.rs:1761:5
[INFO] [stderr]     17:     0x5eaf03bc8202 - core::result::Result<T,E>::expect::h5bd701c39920068b
[INFO] [stderr]     18:     0x5eaf03bcdfc6 - build_script_build::cuda_version_from_build_system::h9835869f340396c3
[INFO] [stderr]     19:     0x5eaf03bcdd42 - build_script_build::main::hedb0e79d06e09725
[INFO] [stderr]     20:     0x5eaf03bc9f23 - core::ops::function::FnOnce::call_once::hf586c14b623a3e29
[INFO] [stderr]     21:     0x5eaf03bd07e6 - std::sys::backtrace::__rust_begin_short_backtrace::hf4018f317dcf5516
[INFO] [stderr]     22:     0x5eaf03bc9cd9 - std::rt::lang_start::{{closure}}::h1bd8cdd37d78d636
[INFO] [stderr]     23:     0x5eaf03bf2e10 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h0021e0fe3c5bf8c3
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/core/src/ops/function.rs:284:21
[INFO] [stderr]     24:     0x5eaf03bf2e10 - std::panicking::catch_unwind::do_call::hfef49f2cd13a22dd
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panicking.rs:589:40
[INFO] [stderr]     25:     0x5eaf03bf2e10 - std::panicking::catch_unwind::h34968869e68aaad1
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panicking.rs:552:19
[INFO] [stderr]     26:     0x5eaf03bf2e10 - std::panic::catch_unwind::h5235c74f5d682697
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panic.rs:359:14
[INFO] [stderr]     27:     0x5eaf03bf2e10 - std::rt::lang_start_internal::{{closure}}::hcee20ca47ef7f51b
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/rt.rs:175:24
[INFO] [stderr]     28:     0x5eaf03bf2e10 - std::panicking::catch_unwind::do_call::h359adcb901a84505
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panicking.rs:589:40
[INFO] [stderr]     29:     0x5eaf03bf2e10 - std::panicking::catch_unwind::h8f4f1e91d2aabe23
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panicking.rs:552:19
[INFO] [stderr]     30:     0x5eaf03bf2e10 - std::panic::catch_unwind::h73a78f1a5b0a6154
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/panic.rs:359:14
[INFO] [stderr]     31:     0x5eaf03bf2e10 - std::rt::lang_start_internal::he64187ddd3d90437
[INFO] [stderr]                                  at /rustc/2b39fe756f52bc2825bf5b2bfc385cab922dae10/library/std/src/rt.rs:171:5
[INFO] [stderr]     32:     0x5eaf03bc9cc1 - std::rt::lang_start::hfcdd3a8dad0d0857
[INFO] [stderr]     33:     0x5eaf03bd0035 - main
[INFO] [stderr]     34:     0x7414962b51ca - <unknown>
[INFO] [stderr]     35:     0x7414962b528b - __libc_start_main
[INFO] [stderr]     36:     0x5eaf03bc5b15 - _start
[INFO] [stderr]     37:                0x0 - <unknown>
[INFO] [stderr] warning: build failed, waiting for other jobs to finish...
[INFO] running `Command { std: "docker" "inspect" "45cddfc739ebd5e99360f971683bd9e2b8acb121ed628720dca715c2c24cebdd", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "45cddfc739ebd5e99360f971683bd9e2b8acb121ed628720dca715c2c24cebdd", kill_on_drop: false }`
[INFO] [stdout] 45cddfc739ebd5e99360f971683bd9e2b8acb121ed628720dca715c2c24cebdd
