I was about to start my blog with a bit different post, however during preparation of continuous deployment for this site a strange error occurred.

CI Setup

First of all, few words regarding my setup. I intensively use Gitlab and Gitlab CI together with Gitlab-Runner for all kinds of automation. I mainly use my private instance of Gitlab CE. Currently I try to run everything in Docker containers. This way it's easy to setup environment per project without having to install all the dependencies on host machine. There's one special gitlab-runner that is set up to do all the deployments and that one is with a shell executor.

403 Forbidden

CI setup for this site is quite simple. First job generates the page and creates an archive out of it. That archive is then cached between stages using Minio server. That stage is run inside a python:alpine Docker container. Everything was passing as it should. Second stage is the deploy stage. That one is run not on my servers, but on my private machine, for security reasons.

I rather don't push code to my repositories while I'm not using my main machine, thus I decided to go the path with running gitlab-runner locally that will hold credentials to servers. And this was working fine until today.

I was trying to deploy this blog using CI, but I was getting a FATAL: Received: 403 Forbiden, when Gitlab-Runner tried to update cache. It was 100% reproducible. Build stage was able to upload cache to Minio server, but deploy stage was unable to fetch it.

Cache settings and credentials were identical on all gitlab-runners. Internet was also not very helpful with this error. I switched local gitlab-runner to debug mode, since in logs in the Gitlab UI there is not much information about what happened during cache fetching. Unfortunately there is not much more information in gitlab-runners debug output.

Since there was no information about that error in the Internet, it had to be something with my setup. Only additional information I had was entry from Minio log:

{"cause":"Signature does not match","level":"error",...}

The Solution

It occurs that you can't perform HTTPS handshake if your local time is screwed. And mine was screwed - I was like two hours in the future. That could be due to hibernation of my PC. For some reason it didn't synchronize time using NTP protocol, although ntp daemon is set up and running. Maybe it was because recent problems with internal battery, that is keeping time in BIOS while PC is off line. Nevertheless, setting time properly using ntp-client was a way to go. After that gitlab-runner was able to update cache from Minio server without any problems.