Disable Docker layers cache #23

New Issue

antux18 · 2024-07-24T11:48:23+02:00

antux18 commented

2024-07-24 11:48:23 +02:00

Looks like Docker is caching layers of a Dockerfile:

#6 [ 2/11] WORKDIR /root
#6 CACHED

#7 [ 6/11] RUN conda install python~=3.10 pip cmake && conda clean --all
#7 CACHED

#8 [ 4/11] RUN apt-get update && apt-get install -y --no-install-recommends   build-essential git wget   libgtest-dev libprotobuf-dev protobuf-compiler libgflags-dev libsqlite3-dev llvm-dev   && apt-get clean autoclean && rm -rf /var/lib/apt/lists/{apt,dpkg,cache,log} /tmp/* /var/tmp/*
#8 CACHED

#9 [ 5/11] RUN wget https://repo.anaconda.com/miniconda/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh -O install_miniconda.sh &&   bash install_miniconda.sh -b -p /opt/conda && rm install_miniconda.sh
#9 CACHED

#10 [ 8/11] RUN git clone https://github.com/nox-410/tvm --recursive -b welder   && mkdir tvm/build && cd tvm/build && cp ../cmake/config.cmake .   && echo "set(USE_LLVM ON)" >> config.cmake && echo "set(USE_CUDA ON)" >> config.cmake   && cmake .. && make -j
#10 CACHED

#11 [ 9/11] RUN git clone https://github.com/nox-410/nnfusion -b welder   && mkdir nnfusion/build   && cd nnfusion/build && cmake .. && make -j
#11 CACHED

#12 [ 7/11] RUN pip install --no-cache-dir --default-timeout=1000 torch==1.12 torchvision==0.13 timm==0.5.4 einops     onnx==1.12 onnxruntime-gpu==1.12 onnxconverter_common==1.12     attrs cloudpickle decorator psutil synr tornado xgboost==1.5.0 regex pandas pytest     && rm -rf ~/.cache/pip
#12 CACHED

#13 [10/11] RUN git clone https://github.com/nox-410/cutlass -b welder
#13 CACHED

#14 [ 3/11] RUN echo "LC_ALL=en_US.UTF-8" >> /etc/environment
#14 CACHED

#15 [11/11] COPY . welder/
#15 CACHED

We need to disable this to ensure that an artifact is really reproducible at the time it is tested.

Looks like Docker is caching layers of a Dockerfile: ``` #6 [ 2/11] WORKDIR /root #6 CACHED #7 [ 6/11] RUN conda install python~=3.10 pip cmake && conda clean --all #7 CACHED #8 [ 4/11] RUN apt-get update && apt-get install -y --no-install-recommends build-essential git wget libgtest-dev libprotobuf-dev protobuf-compiler libgflags-dev libsqlite3-dev llvm-dev && apt-get clean autoclean && rm -rf /var/lib/apt/lists/{apt,dpkg,cache,log} /tmp/* /var/tmp/* #8 CACHED #9 [ 5/11] RUN wget https://repo.anaconda.com/miniconda/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh -O install_miniconda.sh && bash install_miniconda.sh -b -p /opt/conda && rm install_miniconda.sh #9 CACHED #10 [ 8/11] RUN git clone https://github.com/nox-410/tvm --recursive -b welder && mkdir tvm/build && cd tvm/build && cp ../cmake/config.cmake . && echo "set(USE_LLVM ON)" >> config.cmake && echo "set(USE_CUDA ON)" >> config.cmake && cmake .. && make -j #10 CACHED #11 [ 9/11] RUN git clone https://github.com/nox-410/nnfusion -b welder && mkdir nnfusion/build && cd nnfusion/build && cmake .. && make -j #11 CACHED #12 [ 7/11] RUN pip install --no-cache-dir --default-timeout=1000 torch==1.12 torchvision==0.13 timm==0.5.4 einops onnx==1.12 onnxruntime-gpu==1.12 onnxconverter_common==1.12 attrs cloudpickle decorator psutil synr tornado xgboost==1.5.0 regex pandas pytest && rm -rf ~/.cache/pip #12 CACHED #13 [10/11] RUN git clone https://github.com/nox-410/cutlass -b welder #13 CACHED #14 [ 3/11] RUN echo "LC_ALL=en_US.UTF-8" >> /etc/environment #14 CACHED #15 [11/11] COPY . welder/ #15 CACHED ``` We need to disable this to ensure that an artifact is really reproducible at the time it is tested.

GuilloteauQ commented

2024-07-24 14:04:55 +02:00

https://stackoverflow.com/a/35595021

antux18 self-assigned this 2024-07-24 14:52:28 +02:00

antux18 referenced this issue from a commit

2024-07-24 18:11:54 +02:00

Added initial support for Python venvs, close #21. Disabled the Docker build cache by default, and added an option to reenable it, close #23. Removed conditions on the existance of git_packages and misc_packages, because the workflow enforces their existance with the Nickel artifact contract.

antux18 closed this issue

2024-07-24 18:11:54 +02:00

Sign in to join this conversation.