Nix / NixOS

Hi! I would like to host a transparent proxy for cache.nixos.org on my local kubernetes cluster.

I took the following NGINX config https://nixos.wiki/wiki/FAQ/Private_Cache_Proxy and created all the folders on the mounted storage.

This is the kubernetes deployment:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nix-cache-volume
spec:
  capacity:
    storage: 500Gi
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/k8s/nix-cache" # Needs exists before PV is created!
  persistentVolumeReclaimPolicy: Retain
***
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nix-cache-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 500Gi
***
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nix-cache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nix-cache
  template:
    metadata:
      labels:
        app: nix-cache
        name: nix-cache
    spec:
      volumes:
        - name: nix-cache-storage
          persistentVolumeClaim:
            claimName: nix-cache-pvc
        - name: nix-cache-config
          configMap:
            name: nix-cache-config
      containers:
        - name: nix-cache
          image: nginx:1.27.0 
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nix-cache-storage
              mountPath: /data
            - name: nix-cache-config
              mountPath: /etc/nginx/sites-available/default
          resources:
            limits:
              memory: "512Mi"
              cpu: "300m"
            requests:
              memory: "256Mi"
              cpu: "200m"
***
apiVersion: v1
kind: Service
metadata:
  name: nix-cache
spec:
  selector:
    app: nix-cache
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
***
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nix-cache-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  rules:
    - host: "nix-cache.raspi.home"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: nix-cache
                port:
                  number: 80
  tls:
    - secretName: nix-cache-raspi-home-tls
      hosts:
        - "nix-cache.raspi.home"
***
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nix-cache.raspi.home
spec:
  commonName: nix-cache.raspi.home
  dnsNames:
    - "nix-cache.raspi.home"
  secretName: nix-cache-raspi-home-tls
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
***
apiVersion: v1
kind: ConfigMap
metadata:
  name: nix-cache-config
data:
  nginx.conf: |
    server {
      listen 80;
      server_name nix-cache.raspi.home;

      location ~ ^/nix-cache-info {
        proxy_store        on;
        proxy_store_access user:rw group:rw all:r;
        proxy_temp_path    /data/nginx/nix-cache-info/temp;
        root               /data/nginx/nix-cache-info/store;

        proxy_set_header Host "cache.nixos.org";
        proxy_pass https://cache.nixos.org;
      }

      location ~^/nar/.+$ {
        proxy_store        on;
        proxy_store_access user:rw group:rw all:r;
        proxy_temp_path    /data/nginx/nar/temp;
        root               /data/nginx/nar/store;

        proxy_set_header Host "cache.nixos.org";
        proxy_pass https://cache.nixos.org;
      }
    }

To use the cache I added it to the substituters.

  nix.settings.substituters = [
    "https://nix-cache.raspi.home/"
  ];

But when I try to use it, get the error:

# Trigger a download
nix develop nixpkgs#just
# Error message
warning: 'https://nix-cache.raspi.home' does not appear to be a binary cache

In the logs of the NGINX I see the following error:

2024/08/03 12:09:30 [error] 31#31: *3 open() "/usr/share/nginx/html/nix-cache-info" failed (2: No such file or directory), client: 10.42.2.7, server: localhost, request: "GET /nix-cache-info HTTP/1 │
│ 10.42.2.7 - - [03/Aug/2024:12:09:30 +0000] "GET /nix-cache-info HTTP/1.1" 404 153 "-" "curl/8.8.0 Nix/2.18.5" "10.42.2.1"                                                                             │
│ 10.42.2.7 - - [03/Aug/2024:12:09:30 +0000] "PUT /nix-cache-info HTTP/1.1" 405 157 "-" "curl/8.8.0 Nix/2.18.5" "10.42.2.1"    

Any ideas whats wrong? I'm neither an nix nor an nginx expert, so maybe it is something really simple but I cannot figure it out.