build error

Job failed with exit code: 1. Reason: cache miss: [11/16] WORKDIR /workspace cache miss: [ 4/16] RUN pip install --upgrade pip setuptools wheel && pip install maturin huggingface_hub datasets requests pyarrow rustbpe pandas tiktoken pydantic ninja packaging einops cache miss: [12/16] COPY overlay /workspace/feather cache miss: [ 9/16] RUN pip install tilelang || echo "[dockerfile] tilelang optional install failed — continuing" cache miss: [16/16] RUN export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 && echo "building htm_rust GPU kernels for HTM_CUDA_ARCH=sm_86 TORCH_CUDA_ARCH_LIST=8.6" && maturin build --release --features gpu --manifest-path htm_rust/Cargo.toml && pip install htm_rust/target/wheels/htm_rust-*.whl cache miss: [15/16] RUN python -m py_compile hydra/training.py prepare.py train.py && bash -n scripts/run_domain_expanded_pretrain.sh cache miss: [ 8/16] RUN SITE=/opt/conda/lib/python3.11/site-packages/mamba_ssm && test -f "$SITE/modules/mamba3.py" && test -f "$SITE/ops/triton/mamba3/mamba3_siso_combined.py" && test -s "$SITE/__init__.py" && echo "mamba3 graft + __init__ override verified" cache miss: [ 7/16] COPY mamba_ssm_init.py /opt/conda/lib/python3.11/site-packages/mamba_ssm/__init__.py cache miss: [13/16] COPY entrypoint.py /app/entrypoint.py cache miss: [ 2/16] RUN apt-get update && apt-get install -y --no-install-recommends git curl ca-certificates build-essential pkg-config libssl-dev && rm -rf /var/lib/apt/lists/* cache miss: [10/16] RUN pip install --force-reinstall --no-deps 'triton==3.4.0' && python -c "import triton; from triton import language as tl; assert hasattr(triton, 'set_allocator'), 'missing triton.set_allocator'; assert hasattr(tl, 'make_tensor_descriptor'), 'missing tl.make_tensor_descriptor'; print(f'triton={triton.__version__} set_allocator+make_tensor_descriptor OK, AttrsDescriptor shimmed in mamba_ssm/__init__.py')" cache miss: [ 5/16] RUN pip install 'https://github.com/Dao-AILab/causal-conv1d/releases/download/v1.6.0/causal_conv1d-1.6.0+cu12torch2.5cxx11abiFALSE-cp311-cp311-linux_x86_64.whl' 'https://github.com/state-spaces/mamba/releases/download/v2.3.0/mamba_ssm-2.3.0+cu12torch2.5cxx11abiFALSE-cp311-cp311-linux_x86_64.whl' && python -c "import importlib.metadata as m; print('installed mamba_ssm=' + m.version('mamba_ssm') + ' causal_conv1d=' + m.version('causal_conv1d'))" cache miss: [ 3/16] RUN curl https://sh.rustup.rs -sSf | bash -s -- -y --profile minimal --default-toolchain stable cache miss: [14/16] WORKDIR /workspace/feather cache miss: [ 6/16] RUN SITE=/opt/conda/lib/python3.11/site-packages/mamba_ssm && BASE=https://raw.githubusercontent.com/state-spaces/mamba/main && curl -fsSL "$BASE/mamba_ssm/modules/mamba3.py" -o "$SITE/modules/mamba3.py" && mkdir -p "$SITE/ops/triton/mamba3" && for f in __init__.py angle_dt.py mamba3_mimo_rotary_step.py mamba3_mimo_utils.py mamba3_siso_bwd.py mamba3_siso_combined.py mamba3_siso_fwd.py mamba3_siso_step.py utils.py; do curl -fsSL "$BASE/mamba_ssm/ops/triton/mamba3/$f" -o "$SITE/ops/triton/mamba3/$f"; done {"total":21,"completed":20,"user_total":16,"user_cached":0,"user_completed":15,"user_cacheable":15,"from":1,"miss":15,"client_duration_ms":263770}

Build logs: