[INFO] fetching crate ai-workbench-lib 0.4.0...
[INFO] testing ai-workbench-lib-0.4.0 against 1.93.0 for beta-1.94-1
[INFO] extracting crate ai-workbench-lib 0.4.0 into /workspace/builds/worker-6-tc1/source
[INFO] started tweaking crates.io crate ai-workbench-lib 0.4.0
[INFO] finished tweaking crates.io crate ai-workbench-lib 0.4.0
[INFO] tweaked toml for crates.io crate ai-workbench-lib 0.4.0 written to /workspace/builds/worker-6-tc1/source/Cargo.toml
[INFO] validating manifest of crates.io crate ai-workbench-lib 0.4.0 on toolchain 1.93.0
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.93.0" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate crates.io crate ai-workbench-lib 0.4.0 already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+1.93.0" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr]     Updating crates.io index
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded nu-ansi-term v0.50.1
[INFO] [stderr]   Downloaded aws-smithy-query v0.60.7
[INFO] [stderr]   Downloaded aws-types v1.3.8
[INFO] [stderr]   Downloaded aws-smithy-observability v0.1.3
[INFO] [stderr]   Downloaded wit-bindgen v0.45.0
[INFO] [stderr]   Downloaded base64-simd v0.8.0
[INFO] [stderr]   Downloaded aws-smithy-checksums v0.63.8
[INFO] [stderr]   Downloaded aws-smithy-eventstream v0.60.11
[INFO] [stderr]   Downloaded xmlparser v0.13.6
[INFO] [stderr]   Downloaded crc-fast v1.3.0
[INFO] [stderr]   Downloaded aws-sdk-sso v1.83.0
[INFO] [stderr]   Downloaded aws-sdk-ssooidc v1.84.0
[INFO] [stderr]   Downloaded indexmap v2.11.0
[INFO] [stderr]   Downloaded aws-smithy-runtime v1.9.1
[INFO] [stderr]   Downloaded regex v1.11.2
[INFO] [stderr]   Downloaded aws-sdk-sts v1.85.0
[INFO] [stderr]   Downloaded bindgen v0.69.5
[INFO] [stderr]   Downloaded aws-lc-rs v1.13.3
[INFO] [stderr]   Downloaded aws-config v1.8.6
[INFO] [stderr]   Downloaded aws-sdk-bedrockruntime v1.105.0
[INFO] [stderr]   Downloaded markdown v1.0.0
[INFO] [stderr]   Downloaded time v0.3.42
[INFO] [stderr]   Downloaded security-framework v3.3.0
[INFO] [stderr]   Downloaded cc v1.2.35
[INFO] [stderr]   Downloaded wasi v0.14.3+wasi-0.2.4
[INFO] [stderr]   Downloaded aws-runtime v1.5.10
[INFO] [stderr]   Downloaded p256 v0.11.1
[INFO] [stderr]   Downloaded aws-sigv4 v1.3.4
[INFO] [stderr]   Downloaded prettyplease v0.2.37
[INFO] [stderr]   Downloaded aws-smithy-runtime-api v1.9.0
[INFO] [stderr]   Downloaded aws-smithy-http-client v1.1.1
[INFO] [stderr]   Downloaded aws-credential-types v1.2.6
[INFO] [stderr]   Downloaded aws-smithy-types v1.3.2
[INFO] [stderr]   Downloaded regex-lite v0.1.7
[INFO] [stderr]   Downloaded cmake v0.1.54
[INFO] [stderr]   Downloaded find-msvc-tools v0.1.0
[INFO] [stderr]   Downloaded bytes-utils v0.1.4
[INFO] [stderr]   Downloaded deranged v0.5.3
[INFO] [stderr]   Downloaded time-macros v0.2.23
[INFO] [stderr]   Downloaded aws-smithy-json v0.61.5
[INFO] [stderr]   Downloaded aws-smithy-http v0.62.3
[INFO] [stderr]   Downloaded aws-smithy-async v1.2.5
[INFO] [stderr]   Downloaded time-core v0.1.5
[INFO] [stderr]   Downloaded aws-smithy-xml v0.60.10
[INFO] [stderr]   Downloaded unicode-id v0.3.5
[INFO] [stderr]   Downloaded aws-sdk-s3 v1.104.0
[INFO] [stderr]   Downloaded aws-lc-sys v0.30.0
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:98afbf2d16093dec2546ff8915fddc74e65664aa03fc460b3712b1c2c54a33e4" "/opt/rustwide/cargo-home/bin/cargo" "+1.93.0" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] [stdout] 35b74b9c2bda8fa68b01146aeee4f8118c21f4cfe4da062218d7d30b58d00696
[INFO] running `Command { std: "docker" "start" "-a" "35b74b9c2bda8fa68b01146aeee4f8118c21f4cfe4da062218d7d30b58d00696", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "35b74b9c2bda8fa68b01146aeee4f8118c21f4cfe4da062218d7d30b58d00696", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "35b74b9c2bda8fa68b01146aeee4f8118c21f4cfe4da062218d7d30b58d00696", kill_on_drop: false }`
[INFO] [stdout] 35b74b9c2bda8fa68b01146aeee4f8118c21f4cfe4da062218d7d30b58d00696
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:98afbf2d16093dec2546ff8915fddc74e65664aa03fc460b3712b1c2c54a33e4" "/opt/rustwide/cargo-home/bin/cargo" "+1.93.0" "build" "--frozen" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] e5b6f3c2e7ba6e47016c8c7272351cab4802fa7eb578ccbde8ea333d0b66f2a8
[INFO] running `Command { std: "docker" "start" "-a" "e5b6f3c2e7ba6e47016c8c7272351cab4802fa7eb578ccbde8ea333d0b66f2a8", kill_on_drop: false }`
[INFO] [stderr]    Compiling libc v0.2.175
[INFO] [stderr]    Compiling cfg-if v1.0.3
[INFO] [stderr]    Compiling smallvec v1.15.1
[INFO] [stderr]    Compiling bytes v1.10.1
[INFO] [stderr]    Compiling parking_lot_core v0.9.11
[INFO] [stderr]    Compiling memchr v2.7.5
[INFO] [stderr]    Compiling zeroize v1.8.1
[INFO] [stderr]    Compiling proc-macro2 v1.0.101
[INFO] [stderr]    Compiling lock_api v0.4.13
[INFO] [stderr]    Compiling find-msvc-tools v0.1.0
[INFO] [stderr]    Compiling subtle v2.6.1
[INFO] [stderr]    Compiling typenum v1.18.0
[INFO] [stderr]    Compiling tracing-core v0.1.34
[INFO] [stderr]    Compiling futures-channel v0.3.31
[INFO] [stderr]    Compiling num-traits v0.2.19
[INFO] [stderr]    Compiling ryu v1.0.20
[INFO] [stderr]    Compiling deranged v0.5.3
[INFO] [stderr]    Compiling outref v0.5.2
[INFO] [stderr]    Compiling time-core v0.1.5
[INFO] [stderr]    Compiling vsimd v0.8.0
[INFO] [stderr]    Compiling fs_extra v1.3.0
[INFO] [stderr]    Compiling dunce v1.0.5
[INFO] [stderr]    Compiling allocator-api2 v0.2.21
[INFO] [stderr]    Compiling aws-lc-rs v1.13.3
[INFO] [stderr]    Compiling try-lock v0.2.5
[INFO] [stderr]    Compiling const-oid v0.9.6
[INFO] [stderr]    Compiling getrandom v0.3.3
[INFO] [stderr]    Compiling quote v1.0.40
[INFO] [stderr]    Compiling stable_deref_trait v1.2.0
[INFO] [stderr]    Compiling want v0.3.1
[INFO] [stderr]    Compiling der v0.6.1
[INFO] [stderr]    Compiling rustls-pki-types v1.12.0
[INFO] [stderr]    Compiling syn v2.0.106
[INFO] [stderr]    Compiling tower-service v0.3.3
[INFO] [stderr]    Compiling log v0.4.27
[INFO] [stderr]    Compiling base64ct v1.8.0
[INFO] [stderr]    Compiling base64-simd v0.8.0
[INFO] [stderr]    Compiling rustls v0.21.12
[INFO] [stderr]    Compiling semver v1.0.26
[INFO] [stderr]    Compiling rustls v0.23.31
[INFO] [stderr]    Compiling crc32fast v1.5.0
[INFO] [stderr]    Compiling base16ct v0.1.1
[INFO] [stderr]    Compiling base64 v0.21.7
[INFO] [stderr]    Compiling time v0.3.42
[INFO] [stderr]    Compiling hashbrown v0.15.5
[INFO] [stderr]    Compiling ipnet v2.11.0
[INFO] [stderr]    Compiling rustls-native-certs v0.8.1
[INFO] [stderr]    Compiling tower-layer v0.3.3
[INFO] [stderr]    Compiling mio v1.0.4
[INFO] [stderr]    Compiling signal-hook-registry v1.4.6
[INFO] [stderr]    Compiling socket2 v0.6.0
[INFO] [stderr]    Compiling jobserver v0.1.34
[INFO] [stderr]    Compiling http v1.3.1
[INFO] [stderr]    Compiling getrandom v0.2.16
[INFO] [stderr]    Compiling http v0.2.12
[INFO] [stderr]    Compiling cc v1.2.35
[INFO] [stderr]    Compiling bytes-utils v0.1.4
[INFO] [stderr]    Compiling parking_lot v0.12.4
[INFO] [stderr]    Compiling generic-array v0.14.7
[INFO] [stderr]    Compiling num-integer v0.1.46
[INFO] [stderr]    Compiling rand_core v0.6.4
[INFO] [stderr]    Compiling indexmap v2.11.0
[INFO] [stderr]    Compiling spki v0.6.0
[INFO] [stderr]    Compiling rustls-pemfile v1.0.4
[INFO] [stderr]    Compiling pkcs8 v0.9.0
[INFO] [stderr]    Compiling ff v0.12.1
[INFO] [stderr]    Compiling socket2 v0.5.10
[INFO] [stderr]    Compiling rustls-native-certs v0.6.3
[INFO] [stderr]    Compiling tower v0.5.2
[INFO] [stderr]    Compiling litemap v0.8.0
[INFO] [stderr]    Compiling hex v0.4.3
[INFO] [stderr]    Compiling writeable v0.6.1
[INFO] [stderr]    Compiling group v0.12.1
[INFO] [stderr]    Compiling rustc_version v0.4.1
[INFO] [stderr]    Compiling crypto-bigint v0.5.5
[INFO] [stderr]    Compiling icu_normalizer_data v2.0.0
[INFO] [stderr]    Compiling zerocopy v0.8.26
[INFO] [stderr]    Compiling icu_properties_data v2.0.1
[INFO] [stderr]    Compiling http-body v1.0.1
[INFO] [stderr]    Compiling http-body v0.4.6
[INFO] [stderr]    Compiling block-buffer v0.10.4
[INFO] [stderr]    Compiling http-body-util v0.1.3
[INFO] [stderr]    Compiling crypto-common v0.1.6
[INFO] [stderr]    Compiling crypto-bigint v0.4.9
[INFO] [stderr]    Compiling cmake v0.1.54
[INFO] [stderr]    Compiling sec1 v0.3.0
[INFO] [stderr]    Compiling digest v0.10.7
[INFO] [stderr]    Compiling aws-types v1.3.8
[INFO] [stderr]    Compiling uuid v1.18.1
[INFO] [stderr]    Compiling regex-lite v0.1.7
[INFO] [stderr]    Compiling rand_core v0.9.3
[INFO] [stderr]    Compiling hmac v0.12.1
[INFO] [stderr]    Compiling signature v1.6.4
[INFO] [stderr]    Compiling sha2 v0.10.9
[INFO] [stderr]    Compiling aho-corasick v1.1.3
[INFO] [stderr]    Compiling regex-syntax v0.8.6
[INFO] [stderr]    Compiling crc-catalog v2.4.0
[INFO] [stderr]    Compiling xmlparser v0.13.6
[INFO] [stderr]    Compiling crc v3.3.0
[INFO] [stderr]    Compiling urlencoding v2.1.3
[INFO] [stderr]    Compiling elliptic-curve v0.12.3
[INFO] [stderr]    Compiling rfc6979 v0.3.1
[INFO] [stderr]    Compiling sha1 v0.10.6
[INFO] [stderr]    Compiling md-5 v0.10.6
[INFO] [stderr]    Compiling serde_json v1.0.143
[INFO] [stderr]    Compiling anyhow v1.0.99
[INFO] [stderr]    Compiling lru v0.12.5
[INFO] [stderr]    Compiling tracing-log v0.2.0
[INFO] [stderr]    Compiling csv-core v0.1.12
[INFO] [stderr]    Compiling ecdsa v0.14.8
[INFO] [stderr]    Compiling thread_local v1.1.9
[INFO] [stderr]    Compiling aws-smithy-xml v0.60.10
[INFO] [stderr]    Compiling nu-ansi-term v0.50.1
[INFO] [stderr]    Compiling unicode-id v0.3.5
[INFO] [stderr]    Compiling aws-lc-sys v0.30.0
[INFO] [stderr]    Compiling ring v0.17.14
[INFO] [stderr]    Compiling markdown v1.0.0
[INFO] [stderr]    Compiling p256 v0.11.1
[INFO] [stderr]    Compiling tracing-subscriber v0.3.20
[INFO] [stderr]    Compiling ppv-lite86 v0.2.21
[INFO] [stderr]    Compiling regex-automata v0.4.10
[INFO] [stderr]    Compiling rand_chacha v0.9.0
[INFO] [stderr]    Compiling rand v0.9.2
[INFO] [stderr]    Compiling synstructure v0.13.2
[INFO] [stderr]    Compiling tokio-macros v2.5.0
[INFO] [stderr]    Compiling futures-macro v0.3.31
[INFO] [stderr]    Compiling tracing-attributes v0.1.30
[INFO] [stderr]    Compiling zerofrom-derive v0.1.6
[INFO] [stderr]    Compiling yoke-derive v0.8.0
[INFO] [stderr]    Compiling zerovec-derive v0.11.1
[INFO] [stderr]    Compiling tokio v1.47.1
[INFO] [stderr]    Compiling rustls-webpki v0.101.7
[INFO] [stderr]    Compiling futures-util v0.3.31
[INFO] [stderr]    Compiling sct v0.7.1
[INFO] [stderr]    Compiling displaydoc v0.2.5
[INFO] [stderr]    Compiling zerofrom v0.1.6
[INFO] [stderr]    Compiling serde_derive v1.0.219
[INFO] [stderr]    Compiling yoke v0.8.0
[INFO] [stderr]    Compiling tracing v0.1.41
[INFO] [stderr]    Compiling zerovec v0.11.4
[INFO] [stderr]    Compiling zerotrie v0.2.2
[INFO] [stderr]    Compiling regex v1.11.2
[INFO] [stderr]    Compiling tinystr v0.8.1
[INFO] [stderr]    Compiling potential_utf v0.1.3
[INFO] [stderr]    Compiling icu_collections v2.0.0
[INFO] [stderr]    Compiling serde v1.0.219
[INFO] [stderr]    Compiling icu_locale_core v2.0.0
[INFO] [stderr]    Compiling crc-fast v1.3.0
[INFO] [stderr]    Compiling futures-executor v0.3.31
[INFO] [stderr]    Compiling icu_provider v2.0.0
[INFO] [stderr]    Compiling futures v0.3.31
[INFO] [stderr]    Compiling icu_properties v2.0.1
[INFO] [stderr]    Compiling icu_normalizer v2.0.0
[INFO] [stderr]    Compiling idna_adapter v1.2.1
[INFO] [stderr]    Compiling idna v1.1.0
[INFO] [stderr]    Compiling csv v1.3.1
[INFO] [stderr]    Compiling tokio-util v0.7.16
[INFO] [stderr]    Compiling aws-smithy-async v1.2.5
[INFO] [stderr]    Compiling tokio-rustls v0.24.1
[INFO] [stderr]    Compiling url v2.5.7
[INFO] [stderr]    Compiling aws-smithy-types v1.3.2
[INFO] [stderr]    Compiling h2 v0.4.12
[INFO] [stderr]    Compiling h2 v0.3.27
[INFO] [stderr]    Compiling aws-smithy-runtime-api v1.9.0
[INFO] [stderr]    Compiling aws-smithy-eventstream v0.60.11
[INFO] [stderr]    Compiling aws-smithy-json v0.61.5
[INFO] [stderr]    Compiling aws-smithy-query v0.60.7
[INFO] [stderr]    Compiling rustls-webpki v0.103.4
[INFO] [stderr]    Compiling aws-smithy-http v0.62.3
[INFO] [stderr]    Compiling aws-credential-types v1.2.6
[INFO] [stderr]    Compiling aws-smithy-observability v0.1.3
[INFO] [stderr]    Compiling aws-smithy-checksums v0.63.8
[INFO] [stderr]    Compiling aws-sigv4 v1.3.4
[INFO] [stderr]    Compiling hyper v0.14.32
[INFO] [stderr]    Compiling hyper v1.7.0
[INFO] [stderr]    Compiling hyper-util v0.1.16
[INFO] [stderr]    Compiling hyper-rustls v0.24.2
[INFO] [stderr]    Compiling tokio-rustls v0.26.2
[INFO] [stderr]    Compiling hyper-rustls v0.27.7
[INFO] [stderr]    Compiling aws-smithy-http-client v1.1.1
[INFO] [stderr]    Compiling aws-smithy-runtime v1.9.1
[INFO] [stderr]    Compiling aws-runtime v1.5.10
[INFO] [stderr]    Compiling aws-sdk-sts v1.85.0
[INFO] [stderr]    Compiling aws-sdk-sso v1.83.0
[INFO] [stderr]    Compiling aws-sdk-ssooidc v1.84.0
[INFO] [stderr]    Compiling aws-sdk-bedrockruntime v1.105.0
[INFO] [stderr]    Compiling aws-sdk-s3 v1.104.0
[INFO] [stderr]    Compiling aws-config v1.8.6
[INFO] [stderr]    Compiling ai-workbench-lib v0.4.0 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 45s
[INFO] running `Command { std: "docker" "inspect" "e5b6f3c2e7ba6e47016c8c7272351cab4802fa7eb578ccbde8ea333d0b66f2a8", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "e5b6f3c2e7ba6e47016c8c7272351cab4802fa7eb578ccbde8ea333d0b66f2a8", kill_on_drop: false }`
[INFO] [stdout] e5b6f3c2e7ba6e47016c8c7272351cab4802fa7eb578ccbde8ea333d0b66f2a8
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:98afbf2d16093dec2546ff8915fddc74e65664aa03fc460b3712b1c2c54a33e4" "/opt/rustwide/cargo-home/bin/cargo" "+1.93.0" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] 71c2dbfdb4d93166243da2062bd52b9f7b30c45fede05dfca28374610556abcd
[INFO] running `Command { std: "docker" "start" "-a" "71c2dbfdb4d93166243da2062bd52b9f7b30c45fede05dfca28374610556abcd", kill_on_drop: false }`
[INFO] [stderr]    Compiling ai-workbench-lib v0.4.0 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 8.82s
[INFO] running `Command { std: "docker" "inspect" "71c2dbfdb4d93166243da2062bd52b9f7b30c45fede05dfca28374610556abcd", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "71c2dbfdb4d93166243da2062bd52b9f7b30c45fede05dfca28374610556abcd", kill_on_drop: false }`
[INFO] [stdout] 71c2dbfdb4d93166243da2062bd52b9f7b30c45fede05dfca28374610556abcd
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-6-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=warn" "-e" "RUSTDOCFLAGS=--cap-lints=warn" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:98afbf2d16093dec2546ff8915fddc74e65664aa03fc460b3712b1c2c54a33e4" "/opt/rustwide/cargo-home/bin/cargo" "+1.93.0" "test" "--frozen", kill_on_drop: false }`
[INFO] [stdout] 96dcab67665824a2e8d6f90a16ff8d31ea96a71ebfe8530db10a3750526fdef0
[INFO] running `Command { std: "docker" "start" "-a" "96dcab67665824a2e8d6f90a16ff8d31ea96a71ebfe8530db10a3750526fdef0", kill_on_drop: false }`
[INFO] [stderr]     Finished `test` profile [unoptimized + debuginfo] target(s) in 0.47s
[INFO] [stderr]      Running unittests src/lib.rs (/opt/rustwide/target/debug/deps/ai_workbench_lib-de16b73fc674d5db)
[INFO] [stdout] 
[INFO] [stdout] running 37 tests
[INFO] [stdout] test modules::file_discovery::file_discovery_tests::test_file_info_structure ... ok
[INFO] [stdout] test modules::file_splitter::splitter::integration_tests::test_large_csv_file_splitting ... ok
[INFO] [stdout] test modules::file_splitter::splitter::tests::test_json_file_splitting ... ok
[INFO] [stdout] test modules::file_splitter::splitter::tests::test_empty_file ... ok
[INFO] [stdout] test modules::file_splitter::splitter::tests::test_invalid_csv_fallback ... ok
[INFO] [stdout] test modules::file_splitter::splitter::tests::test_file_type_detection_by_extension ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::code_splitter::tests::test_large_file_splitting ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::code_splitter::tests::test_simple_code_splitting ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::csv_splitter::tests::test_csv_format_validation ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::csv_splitter::tests::test_csv_single_chunk ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::csv_splitter::tests::test_csv_multiple_chunks_with_headers ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::csv_splitter::tests::test_record_boundary_preservation ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::csv_splitter::tests::test_tsv_format ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::json_splitter::tests::test_json_lines_detection ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::json_splitter::tests::test_json_lines_splitting ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::text_splitter::tests::test_line_boundary_preservation ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::text_splitter::tests::test_large_file_multiple_chunks ... ok
[INFO] [stdout] test modules::file_splitter::splitter_types::text_splitter::tests::test_small_file_single_chunk ... ok
[INFO] [stdout] test modules::file_splitter::splitter::integration_tests::test_large_text_file_splitting ... ok
[INFO] [stdout] test modules::file_splitter::splitter::integration_tests::test_mixed_file_types ... FAILED
[INFO] [stdout] test modules::file_splitter::splitter::tests::test_binary_file_rejection ... FAILED
[INFO] [stdout] test modules::file_splitter::splitter_types::json_splitter::tests::test_json_array_detection ... FAILED
[INFO] [stdout] test modules::file_splitter::splitter_types::json_splitter::tests::test_standard_json_detection ... FAILED
[INFO] [stdout] test modules::file_splitter::splitter_types::json_splitter::tests::test_invalid_json_rejection ... FAILED
[INFO] [stdout] test modules::file_splitter::splitter::integration_tests::test_edge_cases ... FAILED
[INFO] [stdout] [2m2026-01-28T04:46:36.501104Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.501166Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.501177Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.501188Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected folder input (ends with '/') - processing as folder
[INFO] [stdout] [2m2026-01-28T04:46:36.501209Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_folder DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.501219Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing folder: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.501227Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/
[INFO] [stdout] [2m2026-01-28T04:46:36.501238Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Making S3 ListObjects request #1
[INFO] [stdout] [2m2026-01-28T04:46:36.501252Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending ListObjectsV2 request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.501418Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.504698Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'test-data/'
[INFO] [stdout] [2m2026-01-28T04:46:36.504729Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.504744Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected folder input (ends with '/') - processing as folder
[INFO] [stdout] [2m2026-01-28T04:46:36.504758Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_folder DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.504768Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing folder: 'test-data/'
[INFO] [stdout] [2m2026-01-28T04:46:36.504777Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/
[INFO] [stdout] [2m2026-01-28T04:46:36.504790Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Making S3 ListObjects request #1
[INFO] [stdout] [2m2026-01-28T04:46:36.504806Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending ListObjectsV2 request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.505258Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.505288Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'mixed-files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.505296Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.505308Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected folder input (ends with '/') - processing as folder
[INFO] [stdout] [2m2026-01-28T04:46:36.505321Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_folder DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.505330Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing folder: 'mixed-files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.505339Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/mixed-files/
[INFO] [stdout] [2m2026-01-28T04:46:36.505350Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Making S3 ListObjects request #1
[INFO] [stdout] [2m2026-01-28T04:46:36.505362Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending ListObjectsV2 request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.506232Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Starting job execution for job_id: test-folder-b9529451-b884-45a5-99ba-915617036348
[INFO] [stdout] [2m2026-01-28T04:46:36.506279Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.506292Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Input specification: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.506300Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.506310Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Full S3 path would be: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/
[INFO] [stdout] [2m2026-01-28T04:46:36.506321Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m FileDiscovery instance created successfully
[INFO] [stdout] [2m2026-01-28T04:46:36.506329Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Attempting to discover files...
[INFO] [stdout] [2m2026-01-28T04:46:36.506359Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.506368Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.506375Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.506384Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected folder input (ends with '/') - processing as folder
[INFO] [stdout] [2m2026-01-28T04:46:36.506436Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_folder DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.506445Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing folder: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.506454Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/
[INFO] [stdout] [2m2026-01-28T04:46:36.506464Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Making S3 ListObjects request #1
[INFO] [stdout] [2m2026-01-28T04:46:36.506475Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending ListObjectsV2 request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.508546Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Starting job execution for job_id: test-large-72368795-f29e-4ff8-bd5a-323cfe59c369
[INFO] [stdout] [2m2026-01-28T04:46:36.508590Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.508602Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Input specification: 'large_files/large_document.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.508613Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.508624Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Full S3 path would be: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/large_files/large_document.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.508639Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m FileDiscovery instance created successfully
[INFO] [stdout] [2m2026-01-28T04:46:36.508649Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Attempting to discover files...
[INFO] [stdout] [2m2026-01-28T04:46:36.508707Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.508719Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'large_files/large_document.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.508731Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.508746Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.508792Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.508806Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'large_files/large_document.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.508817Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/large_files/large_document.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.508831Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.508874Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.508892Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'large_files/large_document.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.508904Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.508916Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/large_files/large_document.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.508929Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.511167Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 ListObjectsV2 request failed: dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.511200Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.511230Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.511238Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. Folder/prefix 'text_files/' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.511250Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No list permissions for this bucket/prefix
[INFO] [stdout] [2m2026-01-28T04:46:36.511272Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.511281Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2' does not exist
[INFO] [stdout] [2m2026-01-28T04:46:36.511298Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.511388Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to list objects in S3
[INFO] [stdout] [2m2026-01-28T04:46:36.511416Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.511478Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'test-data/sample.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.511489Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.511513Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.511525Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.511542Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'test-data/sample.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.511553Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.511564Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.511574Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.511592Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'test-data/sample.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.511602Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.511613Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.511643Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.503458Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.512696Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'large-dataset/'
[INFO] [stdout] [2m2026-01-28T04:46:36.512711Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.512721Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected folder input (ends with '/') - processing as folder
[INFO] [stdout] [2m2026-01-28T04:46:36.512733Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_folder DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.512741Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing folder: 'large-dataset/'
[INFO] [stdout] [2m2026-01-28T04:46:36.512749Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/large-dataset/
[INFO] [stdout] [2m2026-01-28T04:46:36.512760Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Making S3 ListObjects request #1
[INFO] [stdout] [2m2026-01-28T04:46:36.512772Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending ListObjectsV2 request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.514378Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'large_files/large_document.txt': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.514416Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.514446Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.514457Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'large_files/large_document.txt' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.514472Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.514482Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.514493Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.514502Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.516487Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.516533Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Input specification: 'text_files/sample.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.516543Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.516554Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Full S3 path would be: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.516568Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m FileDiscovery instance created successfully
[INFO] [stdout] [2m2026-01-28T04:46:36.516579Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Attempting to discover files...
[INFO] [stdout] [2m2026-01-28T04:46:36.516590Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.516610Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'text_files/sample.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.516620Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.516631Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.516643Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.516652Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'text_files/sample.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.516682Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.516693Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.516719Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.516729Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'text_files/sample.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.516738Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.516748Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.516757Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.511425Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to list objects in S3
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     S3 ListObjectsV2 failed for folder 'text_files/': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_folder::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:157:32
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:31:45
[INFO] [stdout]    3: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::JobProcessor::run::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:57:43
[INFO] [stdout]    5: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_folder_end_to_end::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:787:31
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    8: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    9: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]   10: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   15: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   16: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   18: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   19: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   23: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   26: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   27: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_folder_end_to_end
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:787:37
[INFO] [stdout]   28: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_folder_end_to_end::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:759:52
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   31: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   32: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   33: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   34: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   35: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   36: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   37: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   39: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   40: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   41: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   42: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   43: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   44: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   45: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   46: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   47: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   48: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   49: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   50: <unknown>
[INFO] [stdout]   51: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.517263Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.517286Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m File discovery failed with error: Failed to list objects in S3
[INFO] [stdout] [2m2026-01-28T04:46:36.517475Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.517509Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'test-data/sample.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.517518Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.517530Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.517543Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.517551Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'test-data/sample.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.517560Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.517571Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.517582Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.517591Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'test-data/sample.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.517601Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.517610Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.517632Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.517303Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Error context: Failed to list objects in S3
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     S3 ListObjectsV2 failed for folder 'text_files/': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_folder::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:157:32
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:31:45
[INFO] [stdout]    3: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::JobProcessor::run::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:57:43
[INFO] [stdout]    5: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_folder_end_to_end::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:787:31
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    8: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    9: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]   10: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   15: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   16: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   18: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   19: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   23: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   26: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   27: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_folder_end_to_end
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:787:37
[INFO] [stdout]   28: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_folder_end_to_end::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:759:52
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   31: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   32: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   33: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   34: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   35: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   36: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   37: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   39: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   40: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   41: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   42: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   43: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   44: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   45: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   46: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   47: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   48: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   49: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   50: <unknown>
[INFO] [stdout]   51: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.517837Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG END (FAILED) ===
[INFO] [stdout] [2m2026-01-28T04:46:36.518950Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'large_files/large_document.txt': Failed to get metadata for large_files/large_document.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.518977Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.518987Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/large_files/large_document.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.519001Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.519010Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.519019Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.519030Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: large_files/large_document.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.519042Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: large_files/large_document.txt
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for large_files/large_document.txt
[INFO] [stdout]     1: S3 HEAD request failed for 'large_files/large_document.txt': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    5: ai_workbench_lib::modules::job_processor::core::JobProcessor::run::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:57:43
[INFO] [stdout]    6: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_with_large_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:882:31
[INFO] [stdout]    7: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    8: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    9: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]   10: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]   11: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   15: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   16: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   17: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   18: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   19: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   20: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   24: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   25: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   26: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   27: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   28: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_with_large_file
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:882:37
[INFO] [stdout]   29: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_with_large_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:854:50
[INFO] [stdout]   30: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   31: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   32: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   33: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   34: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   35: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   36: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   37: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   38: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   39: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   40: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   41: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   42: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   43: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   44: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   45: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   46: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   47: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   48: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   49: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   50: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   51: <unknown>
[INFO] [stdout]   52: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.520031Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.520049Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m File discovery failed with error: Failed to get metadata for single file: large_files/large_document.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.520064Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Error context: Failed to get metadata for single file: large_files/large_document.txt
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for large_files/large_document.txt
[INFO] [stdout]     1: S3 HEAD request failed for 'large_files/large_document.txt': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    5: ai_workbench_lib::modules::job_processor::core::JobProcessor::run::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:57:43
[INFO] [stdout]    6: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_with_large_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:882:31
[INFO] [stdout]    7: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    8: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    9: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]   10: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]   11: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   15: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   16: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   17: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   18: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   19: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   20: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   24: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   25: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   26: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   27: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   28: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_with_large_file
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:882:37
[INFO] [stdout]   29: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_with_large_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:854:50
[INFO] [stdout]   30: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   31: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   32: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   33: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   34: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   35: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   36: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   37: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   38: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   39: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   40: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   41: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   42: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   43: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   44: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   45: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   46: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   47: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   48: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   49: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   50: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   51: <unknown>
[INFO] [stdout]   52: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.520584Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG END (FAILED) ===
[INFO] [stdout] [2m2026-01-28T04:46:36.521596Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'test-data/sample.txt': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.521639Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.521687Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.521697Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'test-data/sample.txt' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.521709Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.521717Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.521726Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.521734Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.521803Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'test-data/sample.txt': Failed to get metadata for test-data/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.521819Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.521827Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.521839Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.521848Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.521856Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.521867Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: test-data/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.523000Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.523030Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Input specification: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.523041Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.523053Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Full S3 path would be: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/
[INFO] [stdout] [2m2026-01-28T04:46:36.523067Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m FileDiscovery instance created successfully
[INFO] [stdout] [2m2026-01-28T04:46:36.523076Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Attempting to discover files...
[INFO] [stdout] [2m2026-01-28T04:46:36.523086Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.523097Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.523107Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.523120Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected folder input (ends with '/') - processing as folder
[INFO] [stdout] [2m2026-01-28T04:46:36.523133Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_folder DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.523143Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing folder: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.523152Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/
[INFO] [stdout] [2m2026-01-28T04:46:36.523165Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Making S3 ListObjects request #1
[INFO] [stdout] [2m2026-01-28T04:46:36.523178Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending ListObjectsV2 request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.523455Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.523499Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Input specification: 'text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.523510Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.523519Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Full S3 path would be: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.523533Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m FileDiscovery instance created successfully
[INFO] [stdout] [2m2026-01-28T04:46:36.523542Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Attempting to discover files...
[INFO] [stdout] [2m2026-01-28T04:46:36.523553Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.523562Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.523571Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.523582Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.523594Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.523603Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.523613Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.523625Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.523673Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.523687Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt'
[INFO] [stdout] [2m2026-01-28T04:46:36.523700Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.523709Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.523721Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.521879Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: test-data/sample.txt
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for test-data/sample.txt
[INFO] [stdout]     1: S3 HEAD request failed for 'test-data/sample.txt': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_single_file_discovery::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:441:62
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_single_file_discovery
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:443:22
[INFO] [stdout]   27: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_single_file_discovery::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:433:42
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.530875Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.531541Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'test-data/sample.txt': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.531576Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.531601Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.531609Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'test-data/sample.txt' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.531630Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.531638Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.531645Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.531652Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.531733Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'test-data/sample.txt': Failed to get metadata for test-data/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.531749Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.531756Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.531767Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.531775Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.531782Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.531793Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: test-data/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.531803Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: test-data/sample.txt
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for test-data/sample.txt
[INFO] [stdout]     1: S3 HEAD request failed for 'test-data/sample.txt': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:516:61
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:515:36
[INFO] [stdout]   27: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:498:40
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.532432Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.532460Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.532469Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'test-data/data.csv'
[INFO] [stdout] [2m2026-01-28T04:46:36.532477Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.532488Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.532499Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.532507Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'test-data/data.csv'
[INFO] [stdout] [2m2026-01-28T04:46:36.532516Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/data.csv
[INFO] [stdout] [2m2026-01-28T04:46:36.532526Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.532536Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.532544Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'test-data/data.csv'
[INFO] [stdout] [2m2026-01-28T04:46:36.532552Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.532561Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/data.csv
[INFO] [stdout] [2m2026-01-28T04:46:36.532571Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.533248Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 ListObjectsV2 request failed: dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.533268Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.533292Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.533300Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. Folder/prefix 'large-dataset/' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.533312Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No list permissions for this bucket/prefix
[INFO] [stdout] [2m2026-01-28T04:46:36.533320Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.533328Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2' does not exist
[INFO] [stdout] [2m2026-01-28T04:46:36.533338Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.533396Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to list objects in S3
[INFO] [stdout] [2m2026-01-28T04:46:36.533409Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to list objects in S3
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     S3 ListObjectsV2 failed for folder 'large-dataset/': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_folder::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:157:32
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:31:45
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_large_folder_discovery::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:551:61
[INFO] [stdout]    4: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    7: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    8: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]    9: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   13: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   14: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   15: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   17: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   18: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   21: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   23: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   25: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_large_folder_discovery
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:553:22
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_large_folder_discovery::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:543:43
[INFO] [stdout]   27: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   30: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   31: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   32: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   33: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   34: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   35: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   36: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   38: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   39: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   40: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   41: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   42: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   43: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   44: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   45: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   46: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   47: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   48: <unknown>
[INFO] [stdout]   49: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.533998Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.525560Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 ListObjectsV2 request failed: dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.535744Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.535807Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.535816Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. Folder/prefix 'test-data/' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.535830Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No list permissions for this bucket/prefix
[INFO] [stdout] [2m2026-01-28T04:46:36.535838Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.535846Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2' does not exist
[INFO] [stdout] [2m2026-01-28T04:46:36.535856Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.535941Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to list objects in S3
[INFO] [stdout] [2m2026-01-28T04:46:36.535958Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to list objects in S3
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     S3 ListObjectsV2 failed for folder 'test-data/': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_folder::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:157:32
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:31:45
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_processing_summary::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:621:60
[INFO] [stdout]    4: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    7: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    8: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]    9: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   13: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   14: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   15: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   17: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   18: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   21: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   23: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   25: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_processing_summary
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:623:22
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_processing_summary::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:614:39
[INFO] [stdout]   27: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   30: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   31: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   32: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   33: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   34: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   35: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   36: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   38: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   39: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   40: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   41: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   42: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   43: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   44: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   45: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   46: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   47: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   48: <unknown>
[INFO] [stdout]   49: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.536739Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.537296Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'test-data/data.csv': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.537322Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.537346Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.537354Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'test-data/data.csv' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.537366Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.537375Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.537384Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.537401Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.537463Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'test-data/data.csv': Failed to get metadata for test-data/data.csv
[INFO] [stdout] [2m2026-01-28T04:46:36.537477Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.537484Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/data.csv
[INFO] [stdout] [2m2026-01-28T04:46:36.537496Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.537504Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.537510Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.537520Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: test-data/data.csv
[INFO] [stdout] [2m2026-01-28T04:46:36.538021Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'text_files/sample.txt': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.538067Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.538099Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.537530Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: test-data/data.csv
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for test-data/data.csv
[INFO] [stdout]     1: S3 HEAD request failed for 'test-data/data.csv': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:516:61
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:515:36
[INFO] [stdout]   27: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:498:40
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.538149Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.528200Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 ListObjectsV2 request failed: dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.538176Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.538187Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'test-data/config.json'
[INFO] [stdout] [2m2026-01-28T04:46:36.538195Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.538206Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.538217Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.538226Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'test-data/config.json'
[INFO] [stdout] [2m2026-01-28T04:46:36.538234Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/config.json
[INFO] [stdout] [2m2026-01-28T04:46:36.538244Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.538209Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.538253Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.538261Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.538262Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'test-data/config.json'
[INFO] [stdout] [2m2026-01-28T04:46:36.538273Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.538272Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. Folder/prefix 'text_files/' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.538283Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/config.json
[INFO] [stdout] [2m2026-01-28T04:46:36.538286Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No list permissions for this bucket/prefix
[INFO] [stdout] [2m2026-01-28T04:46:36.538294Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.538296Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.538309Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2' does not exist
[INFO] [stdout] [2m2026-01-28T04:46:36.538319Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.538399Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to list objects in S3
[INFO] [stdout] [2m2026-01-28T04:46:36.525920Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 ListObjectsV2 request failed: dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.538705Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.538739Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.538748Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. Folder/prefix 'text_files/' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.538761Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No list permissions for this bucket/prefix
[INFO] [stdout] [2m2026-01-28T04:46:36.538770Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.538779Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2' does not exist
[INFO] [stdout] [2m2026-01-28T04:46:36.538789Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.538852Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to list objects in S3
[INFO] [stdout] [2m2026-01-28T04:46:36.538411Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to list objects in S3
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     S3 ListObjectsV2 failed for folder 'text_files/': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_folder::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:157:32
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:31:45
[INFO] [stdout]    3: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_file_discovery::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:955:42
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_file_discovery
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:955:48
[INFO] [stdout]   27: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_file_discovery::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:930:49
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.539140Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.529011Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 ListObjectsV2 request failed: dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.541751Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m File discovery failed with error: Failed to list objects in S3
[INFO] [stdout] [2m2026-01-28T04:46:36.538864Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to list objects in S3
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     S3 ListObjectsV2 failed for folder 'text_files/': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_folder::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:157:32
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:31:45
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_folder_discovery::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:470:60
[INFO] [stdout]    4: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    7: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    8: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]    9: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   13: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   14: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   15: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   17: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   18: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   21: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   23: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   25: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_folder_discovery
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:472:22
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_folder_discovery::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:462:37
[INFO] [stdout]   27: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   30: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   31: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   32: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   33: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   34: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   35: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   36: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   38: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   39: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   40: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   41: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   42: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   43: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   44: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   45: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   46: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   47: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   48: <unknown>
[INFO] [stdout]   49: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.541800Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.541772Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Error context: Failed to list objects in S3
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     S3 ListObjectsV2 failed for folder 'text_files/': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_folder::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:157:32
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:31:45
[INFO] [stdout]    3: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_file_discovery::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:955:42
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_file_discovery
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:955:48
[INFO] [stdout]   27: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_file_discovery::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:930:49
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.542294Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG END (FAILED) ===
[INFO] [stdout] [2m2026-01-28T04:46:36.538108Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'text_files/sample.txt' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.542677Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.542698Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.542708Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.542717Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.542797Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'text_files/sample.txt': Failed to get metadata for text_files/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.542813Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.542822Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.542835Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.542843Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.542852Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.542863Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: text_files/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.542876Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: text_files/sample.txt
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for text_files/sample.txt
[INFO] [stdout]     1: S3 HEAD request failed for 'text_files/sample.txt': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    5: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:1024:46
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    8: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    9: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]   10: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   15: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   16: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   18: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   19: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   23: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   26: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   27: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:1017:31
[INFO] [stdout]   28: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:980:52
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   31: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   32: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   33: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   34: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   35: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   36: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   37: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   39: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   40: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   41: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   42: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   43: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   44: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   45: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   46: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   47: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   48: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   49: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   50: <unknown>
[INFO] [stdout]   51: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.543573Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.543589Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m File discovery failed with error: Failed to get metadata for single file: text_files/sample.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.543602Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Error context: Failed to get metadata for single file: text_files/sample.txt
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for text_files/sample.txt
[INFO] [stdout]     1: S3 HEAD request failed for 'text_files/sample.txt': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    5: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:1024:46
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    8: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    9: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]   10: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   15: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   16: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   18: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   19: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   23: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   26: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   27: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:1017:31
[INFO] [stdout]   28: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:980:52
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   31: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   32: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   33: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   34: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   35: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   36: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   37: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   39: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   40: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   41: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   42: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   43: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   44: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   45: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   46: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   47: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   48: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   49: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   50: <unknown>
[INFO] [stdout]   51: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.544120Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG END (FAILED) ===
[INFO] [stdout] [2m2026-01-28T04:46:36.544194Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.544203Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Input specification: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.544211Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.544221Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Full S3 path would be: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/
[INFO] [stdout] [2m2026-01-28T04:46:36.544231Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m FileDiscovery instance created successfully
[INFO] [stdout] [2m2026-01-28T04:46:36.544238Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Attempting to discover files...
[INFO] [stdout] [2m2026-01-28T04:46:36.544248Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.544256Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.544263Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.544274Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected folder input (ends with '/') - processing as folder
[INFO] [stdout] [2m2026-01-28T04:46:36.544286Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_folder DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.544295Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing folder: 'text_files/'
[INFO] [stdout] [2m2026-01-28T04:46:36.544303Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/
[INFO] [stdout] [2m2026-01-28T04:46:36.544315Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Making S3 ListObjects request #1
[INFO] [stdout] [2m2026-01-28T04:46:36.544328Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending ListObjectsV2 request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.541542Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'test-data/config.json': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.545695Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.545727Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.545734Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'test-data/config.json' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.545746Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.545754Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.545762Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.545770Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.545838Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'test-data/config.json': Failed to get metadata for test-data/config.json
[INFO] [stdout] [2m2026-01-28T04:46:36.545854Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.545875Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/config.json
[INFO] [stdout] [2m2026-01-28T04:46:36.545887Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.545896Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.545903Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.545914Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: test-data/config.json
[INFO] [stdout] [2m2026-01-28T04:46:36.546233Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.546280Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.546310Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.546319Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.546334Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.546342Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.546362Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.546371Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.546449Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt': Failed to get metadata for text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.546470Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.546479Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.546492Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.546500Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.546508Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.546520Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.545925Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: test-data/config.json
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for test-data/config.json
[INFO] [stdout]     1: S3 HEAD request failed for 'test-data/config.json': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:516:61
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:515:36
[INFO] [stdout]   27: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:498:40
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.546612Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.546643Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.546652Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'test-data/document.md'
[INFO] [stdout] [2m2026-01-28T04:46:36.546679Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.546690Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.546702Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.546710Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'test-data/document.md'
[INFO] [stdout] [2m2026-01-28T04:46:36.546719Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/document.md
[INFO] [stdout] [2m2026-01-28T04:46:36.546730Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.546741Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.546749Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'test-data/document.md'
[INFO] [stdout] [2m2026-01-28T04:46:36.546758Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.546768Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/document.md
[INFO] [stdout] [2m2026-01-28T04:46:36.546778Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.549431Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'test-data/document.md': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.549481Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.549508Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.549516Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'test-data/document.md' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.549535Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.549544Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.549568Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.549575Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.549622Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.549681Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.549691Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. Folder/prefix 'mixed-files/' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.549710Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No list permissions for this bucket/prefix
[INFO] [stdout] [2m2026-01-28T04:46:36.549719Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.549739Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2' does not exist
[INFO] [stdout] [2m2026-01-28T04:46:36.549750Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.551064Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 ListObjectsV2 request failed: dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.551098Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.551120Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.551128Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. Folder/prefix 'text_files/' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.551147Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No list permissions for this bucket/prefix
[INFO] [stdout] [2m2026-01-28T04:46:36.551156Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.551176Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2' does not exist
[INFO] [stdout] [2m2026-01-28T04:46:36.551189Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.551257Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to list objects in S3
[INFO] [stdout] [2m2026-01-28T04:46:36.546533Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout]     1: S3 HEAD request failed for 'text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    5: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_single_file_end_to_end::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:673:55
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    8: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    9: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]   10: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   15: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   16: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   18: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   19: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   23: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   26: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   27: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_single_file_end_to_end
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:710:37
[INFO] [stdout]   28: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_single_file_end_to_end::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:638:57
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   31: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   32: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   33: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   34: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   35: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   36: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   37: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   39: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   40: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   41: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   42: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   43: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   44: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   45: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   46: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   47: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   48: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   49: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   50: <unknown>
[INFO] [stdout]   51: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.551440Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.551474Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m File discovery failed with error: Failed to get metadata for single file: text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout] [2m2026-01-28T04:46:36.551278Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to list objects in S3
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     S3 ListObjectsV2 failed for folder 'text_files/': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_folder::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:157:32
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:31:45
[INFO] [stdout]    3: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:1024:46
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:1017:31
[INFO] [stdout]   27: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:980:52
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.551955Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.551970Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m File discovery failed with error: Failed to list objects in S3
[INFO] [stdout] [2m2026-01-28T04:46:36.551502Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Error context: Failed to get metadata for single file: text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt
[INFO] [stdout]     1: S3 HEAD request failed for 'text_files/0c472776-7c22-464b-93bf-3714fd229b01.txt': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    5: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_single_file_end_to_end::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:673:55
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    8: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    9: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]   10: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   15: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   16: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   18: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   19: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   23: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   26: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   27: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_single_file_end_to_end
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:710:37
[INFO] [stdout]   28: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_single_file_end_to_end::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:638:57
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   31: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   32: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   33: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   34: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   35: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   36: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   37: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   39: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   40: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   41: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   42: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   43: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   44: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   45: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   46: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   47: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   48: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   49: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   50: <unknown>
[INFO] [stdout]   51: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.552017Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG END (FAILED) ===
[INFO] [stdout] [2m2026-01-28T04:46:36.551983Z[0m [31mERROR[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m Error context: Failed to list objects in S3
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     S3 ListObjectsV2 failed for folder 'text_files/': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_folder::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:157:32
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:31:45
[INFO] [stdout]    3: ai_workbench_lib::modules::job_processor::core::JobProcessor::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:102:83
[INFO] [stdout]    4: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:1024:46
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:1017:31
[INFO] [stdout]   27: ai_workbench_lib::modules::job_processor::core::job_processor_tests::test_job_processor_config_validation::{{closure}}
[INFO] [stdout]              at ./src/../modules/job_processor/core.rs:980:52
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.552471Z[0m [32m INFO[0m [2mai_workbench_lib::modules::job_processor::core[0m[2m:[0m === FILE DISCOVERY DEBUG END (FAILED) ===
[INFO] [stdout] [2m2026-01-28T04:46:36.552788Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to list objects in S3
[INFO] [stdout] [2m2026-01-28T04:46:36.552819Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to list objects in S3
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     S3 ListObjectsV2 failed for folder 'mixed-files/': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_folder::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:157:32
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:31:45
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_filtering::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:588:61
[INFO] [stdout]    4: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    7: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    8: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]    9: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   13: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   14: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   15: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   17: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   18: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   21: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   23: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   25: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_filtering
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:590:22
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_filtering::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:580:35
[INFO] [stdout]   27: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   30: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   31: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   32: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   33: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   34: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   35: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   36: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   38: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   39: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   40: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   41: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   42: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   43: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   44: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   45: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   46: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   47: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   48: <unknown>
[INFO] [stdout]   49: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.553426Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.554065Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'test-data/document.md': Failed to get metadata for test-data/document.md
[INFO] [stdout] [2m2026-01-28T04:46:36.554099Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.554107Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/document.md
[INFO] [stdout] [2m2026-01-28T04:46:36.554130Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.554138Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.554153Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.554163Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: test-data/document.md
[INFO] [stdout] [2m2026-01-28T04:46:36.554184Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: test-data/document.md
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for test-data/document.md
[INFO] [stdout]     1: S3 HEAD request failed for 'test-data/document.md': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:516:61
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:515:36
[INFO] [stdout]   27: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:498:40
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.554795Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.554823Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.554841Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'test-data/script.py'
[INFO] [stdout] [2m2026-01-28T04:46:36.554849Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.554870Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.554882Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.554898Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'test-data/script.py'
[INFO] [stdout] [2m2026-01-28T04:46:36.554907Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/script.py
[INFO] [stdout] [2m2026-01-28T04:46:36.554925Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.554935Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.554951Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'test-data/script.py'
[INFO] [stdout] [2m2026-01-28T04:46:36.554960Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.554977Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/script.py
[INFO] [stdout] [2m2026-01-28T04:46:36.554988Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.558150Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'test-data/script.py': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.558188Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.558214Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.558222Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'test-data/script.py' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.558235Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.558243Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.558252Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.558259Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.558314Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'test-data/script.py': Failed to get metadata for test-data/script.py
[INFO] [stdout] [2m2026-01-28T04:46:36.558336Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.558345Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/script.py
[INFO] [stdout] [2m2026-01-28T04:46:36.558357Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.558366Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.558374Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.558384Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: test-data/script.py
[INFO] [stdout] [2m2026-01-28T04:46:36.558395Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: test-data/script.py
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for test-data/script.py
[INFO] [stdout]     1: S3 HEAD request failed for 'test-data/script.py': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:516:61
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:515:36
[INFO] [stdout]   27: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:498:40
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.559011Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.559041Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.559052Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'test-data/code.rs'
[INFO] [stdout] [2m2026-01-28T04:46:36.559060Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.559071Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.559082Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.559091Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'test-data/code.rs'
[INFO] [stdout] [2m2026-01-28T04:46:36.559099Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/code.rs
[INFO] [stdout] [2m2026-01-28T04:46:36.559122Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.559132Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.559141Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'test-data/code.rs'
[INFO] [stdout] [2m2026-01-28T04:46:36.559149Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.559159Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/code.rs
[INFO] [stdout] [2m2026-01-28T04:46:36.559170Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.561602Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'test-data/code.rs': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.561636Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.561684Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.561703Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'test-data/code.rs' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.561717Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.561725Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.561740Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.561747Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.561799Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'test-data/code.rs': Failed to get metadata for test-data/code.rs
[INFO] [stdout] [2m2026-01-28T04:46:36.561819Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.561826Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/code.rs
[INFO] [stdout] [2m2026-01-28T04:46:36.561837Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.561844Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.561851Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.561860Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: test-data/code.rs
[INFO] [stdout] [2m2026-01-28T04:46:36.561869Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: test-data/code.rs
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for test-data/code.rs
[INFO] [stdout]     1: S3 HEAD request failed for 'test-data/code.rs': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:516:61
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:515:36
[INFO] [stdout]   27: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:498:40
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.562469Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.562493Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.562502Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'test-data/styles.css'
[INFO] [stdout] [2m2026-01-28T04:46:36.562511Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.562521Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.562533Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.562541Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'test-data/styles.css'
[INFO] [stdout] [2m2026-01-28T04:46:36.562550Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/styles.css
[INFO] [stdout] [2m2026-01-28T04:46:36.562561Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.562570Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.562578Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'test-data/styles.css'
[INFO] [stdout] [2m2026-01-28T04:46:36.562587Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.562597Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/styles.css
[INFO] [stdout] [2m2026-01-28T04:46:36.562607Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.565052Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'test-data/styles.css': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.565087Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.565109Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.565117Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'test-data/styles.css' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.565128Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.565137Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.565159Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.565167Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.565215Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'test-data/styles.css': Failed to get metadata for test-data/styles.css
[INFO] [stdout] [2m2026-01-28T04:46:36.565239Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.565247Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/styles.css
[INFO] [stdout] [2m2026-01-28T04:46:36.565259Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.565269Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.565277Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.565300Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: test-data/styles.css
[INFO] [stdout] [2m2026-01-28T04:46:36.565312Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: test-data/styles.css
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for test-data/styles.css
[INFO] [stdout]     1: S3 HEAD request failed for 'test-data/styles.css': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:516:61
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:515:36
[INFO] [stdout]   27: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:498:40
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.566050Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] [2m2026-01-28T04:46:36.566076Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.566086Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Input spec: 'test-data/markup.html'
[INFO] [stdout] [2m2026-01-28T04:46:36.566095Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Workspace bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.566116Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Detected single file input (no trailing '/') - processing as single file
[INFO] [stdout] [2m2026-01-28T04:46:36.566128Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === process_single_file DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.566145Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Processing single file: 'test-data/markup.html'
[INFO] [stdout] [2m2026-01-28T04:46:36.566153Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/markup.html
[INFO] [stdout] [2m2026-01-28T04:46:36.566173Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Attempting to get file metadata...
[INFO] [stdout] [2m2026-01-28T04:46:36.566183Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === get_file_metadata DEBUG START ===
[INFO] [stdout] [2m2026-01-28T04:46:36.566191Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Getting metadata for file: 'test-data/markup.html'
[INFO] [stdout] [2m2026-01-28T04:46:36.566211Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Bucket: 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.566221Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Full S3 path: s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/markup.html
[INFO] [stdout] [2m2026-01-28T04:46:36.566240Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Sending HEAD request to S3...
[INFO] [stdout] [2m2026-01-28T04:46:36.568403Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m S3 HEAD request failed for 'test-data/markup.html': dispatch failure
[INFO] [stdout] [2m2026-01-28T04:46:36.574062Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } })
[INFO] [stdout] [2m2026-01-28T04:46:36.574106Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Possible causes:
[INFO] [stdout] [2m2026-01-28T04:46:36.574115Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File 'test-data/markup.html' does not exist in bucket 'ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2'
[INFO] [stdout] [2m2026-01-28T04:46:36.574128Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No read permissions for this file/bucket
[INFO] [stdout] [2m2026-01-28T04:46:36.574137Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. AWS credentials are invalid or expired
[INFO] [stdout] [2m2026-01-28T04:46:36.574146Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Network connectivity issues
[INFO] [stdout] [2m2026-01-28T04:46:36.574154Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   5. S3 service is temporarily unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.574239Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Failed to get metadata for file 'test-data/markup.html': Failed to get metadata for test-data/markup.html
[INFO] [stdout] [2m2026-01-28T04:46:36.574257Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m This usually means:
[INFO] [stdout] [2m2026-01-28T04:46:36.574266Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   1. File does not exist at s3://ai-workbench-6c9c43db-7fe6-42f1-8b11-8f82323f83f0-eu-west-2/test-data/markup.html
[INFO] [stdout] [2m2026-01-28T04:46:36.574278Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   2. No permissions to access the file
[INFO] [stdout] [2m2026-01-28T04:46:36.574287Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   3. S3 service is unavailable
[INFO] [stdout] [2m2026-01-28T04:46:36.574295Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m   4. Incorrect bucket name or file path
[INFO] [stdout] [2m2026-01-28T04:46:36.574305Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m discover_files failed: Failed to get metadata for single file: test-data/markup.html
[INFO] [stdout] [2m2026-01-28T04:46:36.574316Z[0m [31mERROR[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m Error details: Failed to get metadata for single file: test-data/markup.html
[INFO] [stdout] 
[INFO] [stdout] Caused by:
[INFO] [stdout]     0: Failed to get metadata for test-data/markup.html
[INFO] [stdout]     1: S3 HEAD request failed for 'test-data/markup.html': dispatch failure
[INFO] [stdout] 
[INFO] [stdout] Stack backtrace:
[INFO] [stdout]    0: anyhow::error::<impl anyhow::Error>::msg
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.99/src/backtrace.rs:27:14
[INFO] [stdout]    1: ai_workbench_lib::modules::file_discovery::FileDiscovery::get_file_metadata::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:267:28
[INFO] [stdout]    2: ai_workbench_lib::modules::file_discovery::FileDiscovery::process_single_file::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:59:64
[INFO] [stdout]    3: ai_workbench_lib::modules::file_discovery::FileDiscovery::discover_files::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:35:50
[INFO] [stdout]    4: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:516:61
[INFO] [stdout]    5: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    6: <core::pin::Pin<P> as core::future::future::Future>::poll
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/future/future.rs:133:9
[INFO] [stdout]    7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:70
[INFO] [stdout]    8: tokio::task::coop::with_budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
[INFO] [stdout]    9: tokio::task::coop::budget
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
[INFO] [stdout]   10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:742:25
[INFO] [stdout]   11: tokio::runtime::scheduler::current_thread::Context::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:432:19
[INFO] [stdout]   12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:741:44
[INFO] [stdout]   13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:68
[INFO] [stdout]   14: tokio::runtime::context::scoped::Scoped<T>::set
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
[INFO] [stdout]   15: tokio::runtime::context::set_scheduler::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
[INFO] [stdout]   16: std::thread::local::LocalKey<T>::try_with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:513:12
[INFO] [stdout]   17: std::thread::local::LocalKey<T>::with
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/local.rs:477:20
[INFO] [stdout]   18: tokio::runtime::context::set_scheduler
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
[INFO] [stdout]   19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:829:27
[INFO] [stdout]   20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:729:24
[INFO] [stdout]   21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:200:33
[INFO] [stdout]   22: tokio::runtime::context::runtime::enter_runtime
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
[INFO] [stdout]   23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/current_thread/mod.rs:188:9
[INFO] [stdout]   24: tokio::runtime::runtime::Runtime::block_on_inner
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:356:52
[INFO] [stdout]   25: tokio::runtime::runtime::Runtime::block_on
[INFO] [stdout]              at /opt/rustwide/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
[INFO] [stdout]   26: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:515:36
[INFO] [stdout]   27: ai_workbench_lib::modules::file_discovery::file_discovery_tests::test_file_type_detection::{{closure}}
[INFO] [stdout]              at ./src/../modules/file_discovery.rs:498:40
[INFO] [stdout]   28: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   29: core::ops::function::FnOnce::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   30: test::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   31: test::run_test_in_process::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   32: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   33: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   34: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   35: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   36: test::run_test_in_process
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   37: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   38: test::run_test::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   39: std::sys::backtrace::__rust_begin_short_backtrace
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   40: std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   42: std::panicking::catch_unwind::do_call
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   43: std::panicking::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   44: std::panic::catch_unwind
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   45: std::thread::lifecycle::spawn_unchecked::{{closure}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   46: core::ops::function::FnOnce::call_once{{vtable.shim}}
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   47: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   48: std::sys::thread::unix::Thread::new::thread_start
[INFO] [stdout]              at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   49: <unknown>
[INFO] [stdout]   50: clone
[INFO] [stdout] [2m2026-01-28T04:46:36.574929Z[0m [32m INFO[0m [2mai_workbench_lib::modules::file_discovery[0m[2m:[0m === FileDiscovery::discover_files DEBUG END ===
[INFO] [stdout] test modules::job_processor::core::job_processor_tests::test_job_processor_folder_end_to_end ... ok
[INFO] [stdout] test modules::job_processor::core::job_processor_tests::test_job_processor_with_large_file ... ok
[INFO] [stdout] test modules::file_discovery::file_discovery_tests::test_single_file_discovery ... ok
[INFO] [stdout] test modules::file_discovery::file_discovery_tests::test_large_folder_discovery ... ok
[INFO] [stdout] test modules::file_discovery::file_discovery_tests::test_processing_summary ... ok
[INFO] [stdout] test modules::file_discovery::file_discovery_tests::test_folder_discovery ... ok
[INFO] [stdout] test modules::job_processor::core::job_processor_tests::test_job_processor_file_discovery ... ok
[INFO] [stdout] test modules::file_discovery::file_discovery_tests::test_file_filtering ... ok
[INFO] [stdout] test modules::job_processor::core::job_processor_tests::test_job_processor_config_validation ... ok
[INFO] [stdout] test modules::job_processor::core::job_processor_tests::test_job_processor_single_file_end_to_end ... ok
[INFO] [stdout] test modules::file_discovery::file_discovery_tests::test_file_type_detection ... ok
[INFO] [stdout] test modules::file_splitter::splitter::integration_tests::test_performance_characteristics ... FAILED
[INFO] [stdout] 
[INFO] [stdout] failures:
[INFO] [stdout] 
[INFO] [stdout] ---- modules::file_splitter::splitter::integration_tests::test_mixed_file_types stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'modules::file_splitter::splitter::integration_tests::test_mixed_file_types' (35) panicked at src/../modules/file_splitter/splitter.rs:245:13:
[INFO] [stdout] assertion `left == right` failed
[INFO] [stdout]   left: Text
[INFO] [stdout]  right: Xml
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::libunwind::trace::h90e89fac33585832
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::trace_unsynchronized::heebcf2380a8e7359
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5d69c4873bd2 - std::sys::backtrace::_print_fmt::hdf3e2a22d2f810db
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:68:9
[INFO] [stdout]    3:     0x5d69c4873bd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hac1d885928ba8582
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:38:26
[INFO] [stdout]    4:     0x5d69c4887167 - core::fmt::rt::Argument::fmt::h6901eb7c19abebe8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x5d69c4887167 - core::fmt::write::h83ebb4d32483be9e
[INFO] [stdout]    6:     0x5d69c483f2a6 - std::io::default_write_fmt::h751a79cdae495f10
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5d69c483f2a6 - std::io::Write::write_fmt::h9ef514f8700ff7b2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:1994:13
[INFO] [stdout]    8:     0x5d69c4851939 - std::sys::backtrace::BacktraceLock::print::hcc2e1a3ac996c911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:41:9
[INFO] [stdout]    9:     0x5d69c4851939 - std::panicking::default_hook::{{closure}}::h8e9c4d1276f0925f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x5d69c48517d1 - std::panicking::default_hook::h2b2078d38b534dfb
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x5d69c35d294e - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h39d051e7dedac6a7
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   12:     0x5d69c35d294e - test::test_main_with_exit_callback::{{closure}}::h4b005fe75814c302
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5d69c4851c62 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hd426070722c13cbf
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   14:     0x5d69c4851c62 - std::panicking::panic_with_hook::h39b739724e701bfd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:833:13
[INFO] [stdout]   15:     0x5d69c48519f8 - std::panicking::panic_handler::{{closure}}::he540c4833054e458
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:698:13
[INFO] [stdout]   16:     0x5d69c484bee9 - std::sys::backtrace::__rust_end_short_backtrace::hfa179d89deec8aed
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:176:18
[INFO] [stdout]   17:     0x5d69c483338d - __rustc[d131491b17107b07]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:689:5
[INFO] [stdout]   18:     0x5d69c4891a2c - core::panicking::panic_fmt::ha564519d657d9c46
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x5d69c48918b3 - core::panicking::assert_failed_inner::h6fc38e43be639d33
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:439:17
[INFO] [stdout]   20:     0x5d69c3522145 - core::panicking::assert_failed::h832048c24f1b0b7d
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:394:5
[INFO] [stdout]   21:     0x5d69c3559661 - ai_workbench_lib::modules::file_splitter::splitter::integration_tests::test_mixed_file_types::h89ea6974cc76cd5a
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter.rs:245:13
[INFO] [stdout]   22:     0x5d69c3559727 - ai_workbench_lib::modules::file_splitter::splitter::integration_tests::test_mixed_file_types::{{closure}}::hd22fd42ac002f75d
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter.rs:231:31
[INFO] [stdout]   23:     0x5d69c3541a56 - core::ops::function::FnOnce::call_once::ha47e2ff8c32caa1a
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5d69c35d277b - core::ops::function::FnOnce::call_once::h0217b30cfdee2e67
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   25:     0x5d69c35d277b - test::__rust_begin_short_backtrace::h6d98714a1e28f556
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   26:     0x5d69c35e610a - test::run_test_in_process::{{closure}}::he2c70d2634d3a3cc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   27:     0x5d69c35e610a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h40a0a825bcf17fdd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   28:     0x5d69c35e610a - std::panicking::catch_unwind::do_call::h6c8eb5dd6e7b5975
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   29:     0x5d69c35e610a - std::panicking::catch_unwind::h55455eac2972ad1e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   30:     0x5d69c35e610a - std::panic::catch_unwind::h0025465150a5e558
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   31:     0x5d69c35e610a - test::run_test_in_process::hb61e682f9b630fbc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   32:     0x5d69c35e610a - test::run_test::{{closure}}::hc2c2a7a38ab176e2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   33:     0x5d69c35c03a4 - test::run_test::{{closure}}::ha91867b85cd2a911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   34:     0x5d69c35c03a4 - std::sys::backtrace::__rust_begin_short_backtrace::ha00a9fe02049c339
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   35:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}::hd629fba9153dfb55
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   36:     0x5d69c35c3cd2 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4189018816a5c5d9
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   37:     0x5d69c35c3cd2 - std::panicking::catch_unwind::do_call::hf9fcdbca3c2e9877
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   38:     0x5d69c35c3cd2 - std::panicking::catch_unwind::h97b92d88631c7c64
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   39:     0x5d69c35c3cd2 - std::panic::catch_unwind::ha77cdbb68e725e5e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   40:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::h511f05bc1a132253
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   41:     0x5d69c35c3cd2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6b68b31e9b685cb8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   42:     0x5d69c484748f - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h22a27d165e4e871d
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   43:     0x5d69c484748f - std::sys::thread::unix::Thread::new::thread_start::h45cc87bb053add0f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   44:     0x7d9fba878aa4 - <unknown>
[INFO] [stdout]   45:     0x7d9fba905a64 - clone
[INFO] [stdout]   46:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- modules::file_splitter::splitter::tests::test_binary_file_rejection stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'modules::file_splitter::splitter::tests::test_binary_file_rejection' (37) panicked at src/../modules/file_splitter/splitter.rs:137:9:
[INFO] [stdout] assertion failed: result.is_err()
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::libunwind::trace::h90e89fac33585832
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::trace_unsynchronized::heebcf2380a8e7359
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5d69c4873bd2 - std::sys::backtrace::_print_fmt::hdf3e2a22d2f810db
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:68:9
[INFO] [stdout]    3:     0x5d69c4873bd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hac1d885928ba8582
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:38:26
[INFO] [stdout]    4:     0x5d69c4887167 - core::fmt::rt::Argument::fmt::h6901eb7c19abebe8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x5d69c4887167 - core::fmt::write::h83ebb4d32483be9e
[INFO] [stdout]    6:     0x5d69c483f2a6 - std::io::default_write_fmt::h751a79cdae495f10
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5d69c483f2a6 - std::io::Write::write_fmt::h9ef514f8700ff7b2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:1994:13
[INFO] [stdout]    8:     0x5d69c4851939 - std::sys::backtrace::BacktraceLock::print::hcc2e1a3ac996c911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:41:9
[INFO] [stdout]    9:     0x5d69c4851939 - std::panicking::default_hook::{{closure}}::h8e9c4d1276f0925f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x5d69c48517d1 - std::panicking::default_hook::h2b2078d38b534dfb
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x5d69c35d294e - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h39d051e7dedac6a7
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   12:     0x5d69c35d294e - test::test_main_with_exit_callback::{{closure}}::h4b005fe75814c302
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5d69c4851c62 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hd426070722c13cbf
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   14:     0x5d69c4851c62 - std::panicking::panic_with_hook::h39b739724e701bfd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:833:13
[INFO] [stdout]   15:     0x5d69c4851a2a - std::panicking::panic_handler::{{closure}}::he540c4833054e458
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:691:13
[INFO] [stdout]   16:     0x5d69c484bee9 - std::sys::backtrace::__rust_end_short_backtrace::hfa179d89deec8aed
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:176:18
[INFO] [stdout]   17:     0x5d69c483338d - __rustc[d131491b17107b07]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:689:5
[INFO] [stdout]   18:     0x5d69c4891a2c - core::panicking::panic_fmt::ha564519d657d9c46
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x5d69c48919f2 - core::panicking::panic::h40aea00c5b72b770
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:150:5
[INFO] [stdout]   20:     0x5d69c355ba10 - ai_workbench_lib::modules::file_splitter::splitter::tests::test_binary_file_rejection::h27e1cf6a6eed2776
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter.rs:137:9
[INFO] [stdout]   21:     0x5d69c355ba57 - ai_workbench_lib::modules::file_splitter::splitter::tests::test_binary_file_rejection::{{closure}}::h43064abdc03b9c72
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter.rs:130:36
[INFO] [stdout]   22:     0x5d69c35416e6 - core::ops::function::FnOnce::call_once::h750879556a8bf20a
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   23:     0x5d69c35d277b - core::ops::function::FnOnce::call_once::h0217b30cfdee2e67
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5d69c35d277b - test::__rust_begin_short_backtrace::h6d98714a1e28f556
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   25:     0x5d69c35e610a - test::run_test_in_process::{{closure}}::he2c70d2634d3a3cc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   26:     0x5d69c35e610a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h40a0a825bcf17fdd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   27:     0x5d69c35e610a - std::panicking::catch_unwind::do_call::h6c8eb5dd6e7b5975
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   28:     0x5d69c35e610a - std::panicking::catch_unwind::h55455eac2972ad1e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   29:     0x5d69c35e610a - std::panic::catch_unwind::h0025465150a5e558
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   30:     0x5d69c35e610a - test::run_test_in_process::hb61e682f9b630fbc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   31:     0x5d69c35e610a - test::run_test::{{closure}}::hc2c2a7a38ab176e2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   32:     0x5d69c35c03a4 - test::run_test::{{closure}}::ha91867b85cd2a911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   33:     0x5d69c35c03a4 - std::sys::backtrace::__rust_begin_short_backtrace::ha00a9fe02049c339
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   34:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}::hd629fba9153dfb55
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   35:     0x5d69c35c3cd2 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4189018816a5c5d9
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   36:     0x5d69c35c3cd2 - std::panicking::catch_unwind::do_call::hf9fcdbca3c2e9877
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   37:     0x5d69c35c3cd2 - std::panicking::catch_unwind::h97b92d88631c7c64
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   38:     0x5d69c35c3cd2 - std::panic::catch_unwind::ha77cdbb68e725e5e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   39:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::h511f05bc1a132253
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   40:     0x5d69c35c3cd2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6b68b31e9b685cb8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   41:     0x5d69c484748f - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h22a27d165e4e871d
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   42:     0x5d69c484748f - std::sys::thread::unix::Thread::new::thread_start::h45cc87bb053add0f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   43:     0x7d9fba878aa4 - <unknown>
[INFO] [stdout]   44:     0x7d9fba905a64 - clone
[INFO] [stdout]   45:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- modules::file_splitter::splitter_types::json_splitter::tests::test_json_array_detection stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'modules::file_splitter::splitter_types::json_splitter::tests::test_json_array_detection' (50) panicked at src/../modules/file_splitter/splitter_types/json_splitter.rs:364:9:
[INFO] [stdout] assertion `left == right` failed
[INFO] [stdout]   left: JsonLines
[INFO] [stdout]  right: Standard
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::libunwind::trace::h90e89fac33585832
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::trace_unsynchronized::heebcf2380a8e7359
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5d69c4873bd2 - std::sys::backtrace::_print_fmt::hdf3e2a22d2f810db
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:68:9
[INFO] [stdout]    3:     0x5d69c4873bd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hac1d885928ba8582
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:38:26
[INFO] [stdout]    4:     0x5d69c4887167 - core::fmt::rt::Argument::fmt::h6901eb7c19abebe8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x5d69c4887167 - core::fmt::write::h83ebb4d32483be9e
[INFO] [stdout]    6:     0x5d69c483f2a6 - std::io::default_write_fmt::h751a79cdae495f10
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5d69c483f2a6 - std::io::Write::write_fmt::h9ef514f8700ff7b2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:1994:13
[INFO] [stdout]    8:     0x5d69c4851939 - std::sys::backtrace::BacktraceLock::print::hcc2e1a3ac996c911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:41:9
[INFO] [stdout]    9:     0x5d69c4851939 - std::panicking::default_hook::{{closure}}::h8e9c4d1276f0925f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x5d69c48517d1 - std::panicking::default_hook::h2b2078d38b534dfb
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x5d69c35d294e - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h39d051e7dedac6a7
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   12:     0x5d69c35d294e - test::test_main_with_exit_callback::{{closure}}::h4b005fe75814c302
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5d69c4851c62 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hd426070722c13cbf
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   14:     0x5d69c4851c62 - std::panicking::panic_with_hook::h39b739724e701bfd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:833:13
[INFO] [stdout]   15:     0x5d69c48519f8 - std::panicking::panic_handler::{{closure}}::he540c4833054e458
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:698:13
[INFO] [stdout]   16:     0x5d69c484bee9 - std::sys::backtrace::__rust_end_short_backtrace::hfa179d89deec8aed
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:176:18
[INFO] [stdout]   17:     0x5d69c483338d - __rustc[d131491b17107b07]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:689:5
[INFO] [stdout]   18:     0x5d69c4891a2c - core::panicking::panic_fmt::ha564519d657d9c46
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x5d69c48918b3 - core::panicking::assert_failed_inner::h6fc38e43be639d33
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:439:17
[INFO] [stdout]   20:     0x5d69c35220a3 - core::panicking::assert_failed::h6209512cb17b85ae
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:394:5
[INFO] [stdout]   21:     0x5d69c35561d4 - ai_workbench_lib::modules::file_splitter::splitter_types::json_splitter::tests::test_json_array_detection::h8961b71c1fe876a5
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter_types/json_splitter.rs:364:9
[INFO] [stdout]   22:     0x5d69c35561f7 - ai_workbench_lib::modules::file_splitter::splitter_types::json_splitter::tests::test_json_array_detection::{{closure}}::hbbe28280cc07a27c
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter_types/json_splitter.rs:358:35
[INFO] [stdout]   23:     0x5d69c3541236 - core::ops::function::FnOnce::call_once::h300ba3b8810fbefe
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5d69c35d277b - core::ops::function::FnOnce::call_once::h0217b30cfdee2e67
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   25:     0x5d69c35d277b - test::__rust_begin_short_backtrace::h6d98714a1e28f556
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   26:     0x5d69c35e610a - test::run_test_in_process::{{closure}}::he2c70d2634d3a3cc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   27:     0x5d69c35e610a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h40a0a825bcf17fdd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   28:     0x5d69c35e610a - std::panicking::catch_unwind::do_call::h6c8eb5dd6e7b5975
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   29:     0x5d69c35e610a - std::panicking::catch_unwind::h55455eac2972ad1e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   30:     0x5d69c35e610a - std::panic::catch_unwind::h0025465150a5e558
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   31:     0x5d69c35e610a - test::run_test_in_process::hb61e682f9b630fbc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   32:     0x5d69c35e610a - test::run_test::{{closure}}::hc2c2a7a38ab176e2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   33:     0x5d69c35c03a4 - test::run_test::{{closure}}::ha91867b85cd2a911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   34:     0x5d69c35c03a4 - std::sys::backtrace::__rust_begin_short_backtrace::ha00a9fe02049c339
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   35:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}::hd629fba9153dfb55
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   36:     0x5d69c35c3cd2 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4189018816a5c5d9
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   37:     0x5d69c35c3cd2 - std::panicking::catch_unwind::do_call::hf9fcdbca3c2e9877
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   38:     0x5d69c35c3cd2 - std::panicking::catch_unwind::h97b92d88631c7c64
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   39:     0x5d69c35c3cd2 - std::panic::catch_unwind::ha77cdbb68e725e5e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   40:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::h511f05bc1a132253
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   41:     0x5d69c35c3cd2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6b68b31e9b685cb8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   42:     0x5d69c484748f - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h22a27d165e4e871d
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   43:     0x5d69c484748f - std::sys::thread::unix::Thread::new::thread_start::h45cc87bb053add0f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   44:     0x7d9fba878aa4 - <unknown>
[INFO] [stdout]   45:     0x7d9fba905a64 - clone
[INFO] [stdout]   46:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- modules::file_splitter::splitter_types::json_splitter::tests::test_standard_json_detection stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'modules::file_splitter::splitter_types::json_splitter::tests::test_standard_json_detection' (53) panicked at src/../modules/file_splitter/splitter_types/json_splitter.rs:354:9:
[INFO] [stdout] assertion `left == right` failed
[INFO] [stdout]   left: JsonLines
[INFO] [stdout]  right: Standard
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::libunwind::trace::h90e89fac33585832
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::trace_unsynchronized::heebcf2380a8e7359
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5d69c4873bd2 - std::sys::backtrace::_print_fmt::hdf3e2a22d2f810db
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:68:9
[INFO] [stdout]    3:     0x5d69c4873bd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hac1d885928ba8582
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:38:26
[INFO] [stdout]    4:     0x5d69c4887167 - core::fmt::rt::Argument::fmt::h6901eb7c19abebe8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x5d69c4887167 - core::fmt::write::h83ebb4d32483be9e
[INFO] [stdout]    6:     0x5d69c483f2a6 - std::io::default_write_fmt::h751a79cdae495f10
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5d69c483f2a6 - std::io::Write::write_fmt::h9ef514f8700ff7b2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:1994:13
[INFO] [stdout]    8:     0x5d69c4851939 - std::sys::backtrace::BacktraceLock::print::hcc2e1a3ac996c911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:41:9
[INFO] [stdout]    9:     0x5d69c4851939 - std::panicking::default_hook::{{closure}}::h8e9c4d1276f0925f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x5d69c48517d1 - std::panicking::default_hook::h2b2078d38b534dfb
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x5d69c35d294e - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h39d051e7dedac6a7
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   12:     0x5d69c35d294e - test::test_main_with_exit_callback::{{closure}}::h4b005fe75814c302
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5d69c4851c62 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hd426070722c13cbf
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   14:     0x5d69c4851c62 - std::panicking::panic_with_hook::h39b739724e701bfd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:833:13
[INFO] [stdout]   15:     0x5d69c48519f8 - std::panicking::panic_handler::{{closure}}::he540c4833054e458
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:698:13
[INFO] [stdout]   16:     0x5d69c484bee9 - std::sys::backtrace::__rust_end_short_backtrace::hfa179d89deec8aed
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:176:18
[INFO] [stdout]   17:     0x5d69c483338d - __rustc[d131491b17107b07]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:689:5
[INFO] [stdout]   18:     0x5d69c4891a2c - core::panicking::panic_fmt::ha564519d657d9c46
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x5d69c48918b3 - core::panicking::assert_failed_inner::h6fc38e43be639d33
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:439:17
[INFO] [stdout]   20:     0x5d69c35220a3 - core::panicking::assert_failed::h6209512cb17b85ae
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:394:5
[INFO] [stdout]   21:     0x5d69c3556c94 - ai_workbench_lib::modules::file_splitter::splitter_types::json_splitter::tests::test_standard_json_detection::h5d416bf1d2e7e5bd
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter_types/json_splitter.rs:354:9
[INFO] [stdout]   22:     0x5d69c3556cb7 - ai_workbench_lib::modules::file_splitter::splitter_types::json_splitter::tests::test_standard_json_detection::{{closure}}::h10c5f9d9a5b83d82
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter_types/json_splitter.rs:348:38
[INFO] [stdout]   23:     0x5d69c3541276 - core::ops::function::FnOnce::call_once::h4049c1ed99cd1ed7
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5d69c35d277b - core::ops::function::FnOnce::call_once::h0217b30cfdee2e67
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   25:     0x5d69c35d277b - test::__rust_begin_short_backtrace::h6d98714a1e28f556
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   26:     0x5d69c35e610a - test::run_test_in_process::{{closure}}::he2c70d2634d3a3cc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   27:     0x5d69c35e610a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h40a0a825bcf17fdd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   28:     0x5d69c35e610a - std::panicking::catch_unwind::do_call::h6c8eb5dd6e7b5975
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   29:     0x5d69c35e610a - std::panicking::catch_unwind::h55455eac2972ad1e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   30:     0x5d69c35e610a - std::panic::catch_unwind::h0025465150a5e558
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   31:     0x5d69c35e610a - test::run_test_in_process::hb61e682f9b630fbc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   32:     0x5d69c35e610a - test::run_test::{{closure}}::hc2c2a7a38ab176e2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   33:     0x5d69c35c03a4 - test::run_test::{{closure}}::ha91867b85cd2a911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   34:     0x5d69c35c03a4 - std::sys::backtrace::__rust_begin_short_backtrace::ha00a9fe02049c339
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   35:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}::hd629fba9153dfb55
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   36:     0x5d69c35c3cd2 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4189018816a5c5d9
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   37:     0x5d69c35c3cd2 - std::panicking::catch_unwind::do_call::hf9fcdbca3c2e9877
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   38:     0x5d69c35c3cd2 - std::panicking::catch_unwind::h97b92d88631c7c64
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   39:     0x5d69c35c3cd2 - std::panic::catch_unwind::ha77cdbb68e725e5e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   40:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::h511f05bc1a132253
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   41:     0x5d69c35c3cd2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6b68b31e9b685cb8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   42:     0x5d69c484748f - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h22a27d165e4e871d
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   43:     0x5d69c484748f - std::sys::thread::unix::Thread::new::thread_start::h45cc87bb053add0f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   44:     0x7d9fba878aa4 - <unknown>
[INFO] [stdout]   45:     0x7d9fba905a64 - clone
[INFO] [stdout]   46:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- modules::file_splitter::splitter_types::json_splitter::tests::test_invalid_json_rejection stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'modules::file_splitter::splitter_types::json_splitter::tests::test_invalid_json_rejection' (49) panicked at src/../modules/file_splitter/splitter_types/json_splitter.rs:400:9:
[INFO] [stdout] assertion `left == right` failed
[INFO] [stdout]   left: JsonLines
[INFO] [stdout]  right: Invalid
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::libunwind::trace::h90e89fac33585832
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::trace_unsynchronized::heebcf2380a8e7359
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5d69c4873bd2 - std::sys::backtrace::_print_fmt::hdf3e2a22d2f810db
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:68:9
[INFO] [stdout]    3:     0x5d69c4873bd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hac1d885928ba8582
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:38:26
[INFO] [stdout]    4:     0x5d69c4887167 - core::fmt::rt::Argument::fmt::h6901eb7c19abebe8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x5d69c4887167 - core::fmt::write::h83ebb4d32483be9e
[INFO] [stdout]    6:     0x5d69c483f2a6 - std::io::default_write_fmt::h751a79cdae495f10
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5d69c483f2a6 - std::io::Write::write_fmt::h9ef514f8700ff7b2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:1994:13
[INFO] [stdout]    8:     0x5d69c4851939 - std::sys::backtrace::BacktraceLock::print::hcc2e1a3ac996c911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:41:9
[INFO] [stdout]    9:     0x5d69c4851939 - std::panicking::default_hook::{{closure}}::h8e9c4d1276f0925f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x5d69c48517d1 - std::panicking::default_hook::h2b2078d38b534dfb
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x5d69c35d294e - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h39d051e7dedac6a7
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   12:     0x5d69c35d294e - test::test_main_with_exit_callback::{{closure}}::h4b005fe75814c302
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5d69c4851c62 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hd426070722c13cbf
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   14:     0x5d69c4851c62 - std::panicking::panic_with_hook::h39b739724e701bfd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:833:13
[INFO] [stdout]   15:     0x5d69c48519f8 - std::panicking::panic_handler::{{closure}}::he540c4833054e458
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:698:13
[INFO] [stdout]   16:     0x5d69c484bee9 - std::sys::backtrace::__rust_end_short_backtrace::hfa179d89deec8aed
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:176:18
[INFO] [stdout]   17:     0x5d69c483338d - __rustc[d131491b17107b07]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:689:5
[INFO] [stdout]   18:     0x5d69c4891a2c - core::panicking::panic_fmt::ha564519d657d9c46
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x5d69c48918b3 - core::panicking::assert_failed_inner::h6fc38e43be639d33
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:439:17
[INFO] [stdout]   20:     0x5d69c35220a3 - core::panicking::assert_failed::h6209512cb17b85ae
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:394:5
[INFO] [stdout]   21:     0x5d69c3556b44 - ai_workbench_lib::modules::file_splitter::splitter_types::json_splitter::tests::test_invalid_json_rejection::he6dc1609fef9c495
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter_types/json_splitter.rs:400:9
[INFO] [stdout]   22:     0x5d69c3556b67 - ai_workbench_lib::modules::file_splitter::splitter_types::json_splitter::tests::test_invalid_json_rejection::{{closure}}::h959bc5d4404cccdd
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter_types/json_splitter.rs:395:37
[INFO] [stdout]   23:     0x5d69c35415d6 - core::ops::function::FnOnce::call_once::h646b29ae3ad6efed
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5d69c35d277b - core::ops::function::FnOnce::call_once::h0217b30cfdee2e67
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   25:     0x5d69c35d277b - test::__rust_begin_short_backtrace::h6d98714a1e28f556
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   26:     0x5d69c35e610a - test::run_test_in_process::{{closure}}::he2c70d2634d3a3cc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   27:     0x5d69c35e610a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h40a0a825bcf17fdd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   28:     0x5d69c35e610a - std::panicking::catch_unwind::do_call::h6c8eb5dd6e7b5975
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   29:     0x5d69c35e610a - std::panicking::catch_unwind::h55455eac2972ad1e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   30:     0x5d69c35e610a - std::panic::catch_unwind::h0025465150a5e558
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   31:     0x5d69c35e610a - test::run_test_in_process::hb61e682f9b630fbc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   32:     0x5d69c35e610a - test::run_test::{{closure}}::hc2c2a7a38ab176e2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   33:     0x5d69c35c03a4 - test::run_test::{{closure}}::ha91867b85cd2a911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   34:     0x5d69c35c03a4 - std::sys::backtrace::__rust_begin_short_backtrace::ha00a9fe02049c339
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   35:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}::hd629fba9153dfb55
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   36:     0x5d69c35c3cd2 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4189018816a5c5d9
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   37:     0x5d69c35c3cd2 - std::panicking::catch_unwind::do_call::hf9fcdbca3c2e9877
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   38:     0x5d69c35c3cd2 - std::panicking::catch_unwind::h97b92d88631c7c64
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   39:     0x5d69c35c3cd2 - std::panic::catch_unwind::ha77cdbb68e725e5e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   40:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::h511f05bc1a132253
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   41:     0x5d69c35c3cd2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6b68b31e9b685cb8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   42:     0x5d69c484748f - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h22a27d165e4e871d
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   43:     0x5d69c484748f - std::sys::thread::unix::Thread::new::thread_start::h45cc87bb053add0f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   44:     0x7d9fba878aa4 - <unknown>
[INFO] [stdout]   45:     0x7d9fba905a64 - clone
[INFO] [stdout]   46:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- modules::file_splitter::splitter::integration_tests::test_edge_cases stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'modules::file_splitter::splitter::integration_tests::test_edge_cases' (32) panicked at src/../modules/file_splitter/splitter.rs:263:9:
[INFO] [stdout] assertion failed: chunks[0].metadata.has_headers
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::libunwind::trace::h90e89fac33585832
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::trace_unsynchronized::heebcf2380a8e7359
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5d69c4873bd2 - std::sys::backtrace::_print_fmt::hdf3e2a22d2f810db
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:68:9
[INFO] [stdout]    3:     0x5d69c4873bd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hac1d885928ba8582
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:38:26
[INFO] [stdout]    4:     0x5d69c4887167 - core::fmt::rt::Argument::fmt::h6901eb7c19abebe8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x5d69c4887167 - core::fmt::write::h83ebb4d32483be9e
[INFO] [stdout]    6:     0x5d69c483f2a6 - std::io::default_write_fmt::h751a79cdae495f10
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5d69c483f2a6 - std::io::Write::write_fmt::h9ef514f8700ff7b2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:1994:13
[INFO] [stdout]    8:     0x5d69c4851939 - std::sys::backtrace::BacktraceLock::print::hcc2e1a3ac996c911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:41:9
[INFO] [stdout]    9:     0x5d69c4851939 - std::panicking::default_hook::{{closure}}::h8e9c4d1276f0925f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x5d69c48517d1 - std::panicking::default_hook::h2b2078d38b534dfb
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x5d69c35d294e - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h39d051e7dedac6a7
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   12:     0x5d69c35d294e - test::test_main_with_exit_callback::{{closure}}::h4b005fe75814c302
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5d69c4851c62 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hd426070722c13cbf
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   14:     0x5d69c4851c62 - std::panicking::panic_with_hook::h39b739724e701bfd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:833:13
[INFO] [stdout]   15:     0x5d69c4851a2a - std::panicking::panic_handler::{{closure}}::he540c4833054e458
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:691:13
[INFO] [stdout]   16:     0x5d69c484bee9 - std::sys::backtrace::__rust_end_short_backtrace::hfa179d89deec8aed
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:176:18
[INFO] [stdout]   17:     0x5d69c483338d - __rustc[d131491b17107b07]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:689:5
[INFO] [stdout]   18:     0x5d69c4891a2c - core::panicking::panic_fmt::ha564519d657d9c46
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x5d69c48919f2 - core::panicking::panic::h40aea00c5b72b770
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:150:5
[INFO] [stdout]   20:     0x5d69c3558d1e - ai_workbench_lib::modules::file_splitter::splitter::integration_tests::test_edge_cases::hf41a7065a9af5cd7
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter.rs:263:9
[INFO] [stdout]   21:     0x5d69c3559007 - ai_workbench_lib::modules::file_splitter::splitter::integration_tests::test_edge_cases::{{closure}}::h30402f80b8aa5bfc
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter.rs:251:25
[INFO] [stdout]   22:     0x5d69c35411f6 - core::ops::function::FnOnce::call_once::h2a63859c42e8c1c3
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   23:     0x5d69c35d277b - core::ops::function::FnOnce::call_once::h0217b30cfdee2e67
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5d69c35d277b - test::__rust_begin_short_backtrace::h6d98714a1e28f556
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   25:     0x5d69c35e610a - test::run_test_in_process::{{closure}}::he2c70d2634d3a3cc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   26:     0x5d69c35e610a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h40a0a825bcf17fdd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   27:     0x5d69c35e610a - std::panicking::catch_unwind::do_call::h6c8eb5dd6e7b5975
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   28:     0x5d69c35e610a - std::panicking::catch_unwind::h55455eac2972ad1e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   29:     0x5d69c35e610a - std::panic::catch_unwind::h0025465150a5e558
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   30:     0x5d69c35e610a - test::run_test_in_process::hb61e682f9b630fbc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   31:     0x5d69c35e610a - test::run_test::{{closure}}::hc2c2a7a38ab176e2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   32:     0x5d69c35c03a4 - test::run_test::{{closure}}::ha91867b85cd2a911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   33:     0x5d69c35c03a4 - std::sys::backtrace::__rust_begin_short_backtrace::ha00a9fe02049c339
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   34:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}::hd629fba9153dfb55
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   35:     0x5d69c35c3cd2 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4189018816a5c5d9
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   36:     0x5d69c35c3cd2 - std::panicking::catch_unwind::do_call::hf9fcdbca3c2e9877
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   37:     0x5d69c35c3cd2 - std::panicking::catch_unwind::h97b92d88631c7c64
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   38:     0x5d69c35c3cd2 - std::panic::catch_unwind::ha77cdbb68e725e5e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   39:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::h511f05bc1a132253
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   40:     0x5d69c35c3cd2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6b68b31e9b685cb8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   41:     0x5d69c484748f - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h22a27d165e4e871d
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   42:     0x5d69c484748f - std::sys::thread::unix::Thread::new::thread_start::h45cc87bb053add0f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   43:     0x7d9fba878aa4 - <unknown>
[INFO] [stdout]   44:     0x7d9fba905a64 - clone
[INFO] [stdout]   45:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] ---- modules::file_splitter::splitter::integration_tests::test_performance_characteristics stdout ----
[INFO] [stdout] 
[INFO] [stdout] thread 'modules::file_splitter::splitter::integration_tests::test_performance_characteristics' (36) panicked at src/../modules/file_splitter/splitter.rs:288:9:
[INFO] [stdout] assertion failed: chunks.len() > 1
[INFO] [stdout] stack backtrace:
[INFO] [stdout]    0:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::libunwind::trace::h90e89fac33585832
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
[INFO] [stdout]    1:     0x5d69c4873bd2 - std::backtrace_rs::backtrace::trace_unsynchronized::heebcf2380a8e7359
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
[INFO] [stdout]    2:     0x5d69c4873bd2 - std::sys::backtrace::_print_fmt::hdf3e2a22d2f810db
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:68:9
[INFO] [stdout]    3:     0x5d69c4873bd2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hac1d885928ba8582
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:38:26
[INFO] [stdout]    4:     0x5d69c4887167 - core::fmt::rt::Argument::fmt::h6901eb7c19abebe8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/fmt/rt.rs:152:76
[INFO] [stdout]    5:     0x5d69c4887167 - core::fmt::write::h83ebb4d32483be9e
[INFO] [stdout]    6:     0x5d69c483f2a6 - std::io::default_write_fmt::h751a79cdae495f10
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:639:11
[INFO] [stdout]    7:     0x5d69c483f2a6 - std::io::Write::write_fmt::h9ef514f8700ff7b2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/io/mod.rs:1994:13
[INFO] [stdout]    8:     0x5d69c4851939 - std::sys::backtrace::BacktraceLock::print::hcc2e1a3ac996c911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:41:9
[INFO] [stdout]    9:     0x5d69c4851939 - std::panicking::default_hook::{{closure}}::h8e9c4d1276f0925f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:292:27
[INFO] [stdout]   10:     0x5d69c48517d1 - std::panicking::default_hook::h2b2078d38b534dfb
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:316:9
[INFO] [stdout]   11:     0x5d69c35d294e - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h39d051e7dedac6a7
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   12:     0x5d69c35d294e - test::test_main_with_exit_callback::{{closure}}::h4b005fe75814c302
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:145:21
[INFO] [stdout]   13:     0x5d69c4851c62 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hd426070722c13cbf
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
[INFO] [stdout]   14:     0x5d69c4851c62 - std::panicking::panic_with_hook::h39b739724e701bfd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:833:13
[INFO] [stdout]   15:     0x5d69c4851a2a - std::panicking::panic_handler::{{closure}}::he540c4833054e458
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:691:13
[INFO] [stdout]   16:     0x5d69c484bee9 - std::sys::backtrace::__rust_end_short_backtrace::hfa179d89deec8aed
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:176:18
[INFO] [stdout]   17:     0x5d69c483338d - __rustc[d131491b17107b07]::rust_begin_unwind
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:689:5
[INFO] [stdout]   18:     0x5d69c4891a2c - core::panicking::panic_fmt::ha564519d657d9c46
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:80:14
[INFO] [stdout]   19:     0x5d69c48919f2 - core::panicking::panic::h40aea00c5b72b770
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:150:5
[INFO] [stdout]   20:     0x5d69c355aece - ai_workbench_lib::modules::file_splitter::splitter::integration_tests::test_performance_characteristics::hf2943b7e0cd2c9fe
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter.rs:288:9
[INFO] [stdout]   21:     0x5d69c355b117 - ai_workbench_lib::modules::file_splitter::splitter::integration_tests::test_performance_characteristics::{{closure}}::hb7fe1f536bba3ed3
[INFO] [stdout]                                at /opt/rustwide/workdir/src/../modules/file_splitter/splitter.rs:273:42
[INFO] [stdout]   22:     0x5d69c3541ba6 - core::ops::function::FnOnce::call_once::hb9ceed80ef10c83d
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   23:     0x5d69c35d277b - core::ops::function::FnOnce::call_once::h0217b30cfdee2e67
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   24:     0x5d69c35d277b - test::__rust_begin_short_backtrace::h6d98714a1e28f556
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:663:18
[INFO] [stdout]   25:     0x5d69c35e610a - test::run_test_in_process::{{closure}}::he2c70d2634d3a3cc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:74
[INFO] [stdout]   26:     0x5d69c35e610a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h40a0a825bcf17fdd
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   27:     0x5d69c35e610a - std::panicking::catch_unwind::do_call::h6c8eb5dd6e7b5975
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   28:     0x5d69c35e610a - std::panicking::catch_unwind::h55455eac2972ad1e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   29:     0x5d69c35e610a - std::panic::catch_unwind::h0025465150a5e558
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   30:     0x5d69c35e610a - test::run_test_in_process::hb61e682f9b630fbc
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:686:27
[INFO] [stdout]   31:     0x5d69c35e610a - test::run_test::{{closure}}::hc2c2a7a38ab176e2
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:607:43
[INFO] [stdout]   32:     0x5d69c35c03a4 - test::run_test::{{closure}}::ha91867b85cd2a911
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/test/src/lib.rs:637:41
[INFO] [stdout]   33:     0x5d69c35c03a4 - std::sys::backtrace::__rust_begin_short_backtrace::ha00a9fe02049c339
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:160:18
[INFO] [stdout]   34:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::{{closure}}::hd629fba9153dfb55
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:92:13
[INFO] [stdout]   35:     0x5d69c35c3cd2 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4189018816a5c5d9
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panic/unwind_safe.rs:274:9
[INFO] [stdout]   36:     0x5d69c35c3cd2 - std::panicking::catch_unwind::do_call::hf9fcdbca3c2e9877
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:581:40
[INFO] [stdout]   37:     0x5d69c35c3cd2 - std::panicking::catch_unwind::h97b92d88631c7c64
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:544:19
[INFO] [stdout]   38:     0x5d69c35c3cd2 - std::panic::catch_unwind::ha77cdbb68e725e5e
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panic.rs:359:14
[INFO] [stdout]   39:     0x5d69c35c3cd2 - std::thread::lifecycle::spawn_unchecked::{{closure}}::h511f05bc1a132253
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/thread/lifecycle.rs:90:26
[INFO] [stdout]   40:     0x5d69c35c3cd2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6b68b31e9b685cb8
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/ops/function.rs:250:5
[INFO] [stdout]   41:     0x5d69c484748f - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h22a27d165e4e871d
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2206:9
[INFO] [stdout]   42:     0x5d69c484748f - std::sys::thread::unix::Thread::new::thread_start::h45cc87bb053add0f
[INFO] [stdout]                                at /rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/thread/unix.rs:118:17
[INFO] [stdout]   43:     0x7d9fba878aa4 - <unknown>
[INFO] [stdout]   44:     0x7d9fba905a64 - clone
[INFO] [stdout]   45:                0x0 - <unknown>
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] failures:
[INFO] [stdout]     modules::file_splitter::splitter::integration_tests::test_edge_cases
[INFO] [stdout]     modules::file_splitter::splitter::integration_tests::test_mixed_file_types
[INFO] [stdout]     modules::file_splitter::splitter::integration_tests::test_performance_characteristics
[INFO] [stdout]     modules::file_splitter::splitter::tests::test_binary_file_rejection
[INFO] [stdout]     modules::file_splitter::splitter_types::json_splitter::tests::test_invalid_json_rejection
[INFO] [stdout]     modules::file_splitter::splitter_types::json_splitter::tests::test_json_array_detection
[INFO] [stdout]     modules::file_splitter::splitter_types::json_splitter::tests::test_standard_json_detection
[INFO] [stdout] 
[INFO] [stdout] test result: FAILED. 30 passed; 7 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.43s
[INFO] [stdout] 
[INFO] [stderr] error: test failed, to rerun pass `--lib`
[INFO] running `Command { std: "docker" "inspect" "96dcab67665824a2e8d6f90a16ff8d31ea96a71ebfe8530db10a3750526fdef0", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "96dcab67665824a2e8d6f90a16ff8d31ea96a71ebfe8530db10a3750526fdef0", kill_on_drop: false }`
[INFO] [stdout] 96dcab67665824a2e8d6f90a16ff8d31ea96a71ebfe8530db10a3750526fdef0
