Custom configurations for ingress-nginx with kustomize? Here's how without headaches!How to: Custom configurations for ingress-nginx with kustomize

Ingress-nginx is one of the most popular ingress controllers for Kubernetes. In this blog post, we show how one can adjust the ingress-nginx K8s resources and easily persist them using kustomize. This makes installation and updates simpler and less error-prone.

1. Problem

Many ingress-nginx installations require adjustments. Whether it's a custom error backend, where Command-Args and the ConfigMap need to be modified, or additional ports are needed, or K8s service Annotations must be matched with a Load Balancer. These adjustments can be made manually, but this increases the risk during installation and updates of overlooking something or configuring incorrectly. Moreover, each update becomes challenging and requires additional documentation to ensure nothing is forgotten.

The question arises: how can these adjustments be persisted so that they are automatically applied and during updates only the version number needs to be changed?

2. Approach

kustomize to the rescue! The requirements are clear: resource adjustments in K8s should be stored in code and automatically rolled out during installation and updates. kustomize allows specifying the K8s yaml manifests of ingress-nginx as a resource and applying the necessary patches. We have stored the following kustomization.yaml in ./ base/ingress-nginx:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - github.com/kubernetes/ingress-nginx/deploy/static/provider/cloud?ref=controller-v1.12.3

patches:
  - path: patch-configmap.yaml
  - path: patch-deployment.yaml
  - path: patch-service.yaml

As resources the K8s manifests in version 1.12.3 are specified. Additionally, we have three patches for the ConfigMap, Deployment, and Service. We will look at these in more detail in the next sections.

3. Adjusting ConfigMap

The patch for the ConfigMap must be in the same directory as the kustomization.yaml, as specified there. Here is the content of patch-configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  proxy-buffer-number: "4"
  proxy-buffer-size: 128k
  proxy-busy-buffers-size: 256k
  custom-http-errors: 404,503,502,504

In the ConfigMap, a few proxy-values are set, as well as the HTTP status codes specified for which a response from the Custom Errors Backend should be changed.

4. Adjusting Command

To adjust the command, the Deployment must be patched. patch-deployment.yaml looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  template:
    spec:
      containers:
        - name: controller
          args:
          - /nginx-ingress-controller
          - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
          - --election-id=ingress-nginx-leader
          - --controller-class=k8s.io/ingress-nginx
          - --ingress-class=nginx
          - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
          - --validating-webhook=:8443
          - --validating-webhook-certificate=/usr/local/certificates/cert
          - --validating-webhook-key=/usr/local/certificates/key
          - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
          - --default-backend-service=default/nginx-errors
          ports:
            - containerPort: 21
              name: ftp
              protocol: TCP

In this example, the args are overwritten to set --default-backend-service to default/nginx-errors. Additionally, port 21 is specified to enable FTP requests.

5. Adjusting the Service

The service usually contains annotations that are relevant for the load balancer. Ingress-nginx also offers K8s manifests that are already optimized for AWS, GKE, or Azure. If one wants to supplement these or chooses a different cloud provider, this can also be solved via a patch. Here is the corresponding patch-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  annotations:
    load-balancer.hetzner.cloud/location: fsn1
    load-balancer.hetzner.cloud/name: # ...
spec:
  ports:
    - name: ftp
      port: 21
      protocol: TCP
      targetPort: ftp

It contains two exemplary annotations that are relevant for Hetzner. One to specify the location of the load balancer and the other for the name of the load balancer. Furthermore, port 21 is specified on the service so that FTP requests reach the deployment.

6. Deployment and Updates

The yaml examples from the previous sections are already everything needed for deployment and updates. When deploying manually with kubectl, the following command is used: kubectl apply -k ./base/ingress-nginx.

For updates, only the version in ./base/ingress-nginx/kustomization.yaml needs to be adjusted and the kubectl command from above applied. Before each deployment or update, it's recommended to render the resources using kubectl kustomize ./base/ingress-nginx to verify that everything is correct.

7. Conclusion

As you hopefully saw, it's simple to customize ingress-nginx using kustomize. The code examples in this blog post are already everything needed. The content naturally needs to be adapted case by case. Similarly, it might be necessary to use multiple overlays because, for example, in staging and production, different annotations might be needed. But this doesn't make it more complex. The changes are clear and cleanly stored in the repository and thus documented. For updates, only the version needs to be changed - it can't get simpler.

From our perspective, there's no reason why ingress-nginx adjustments shouldn't be handled via kustomize. Do you have a counterargument? Then please let us know in the comments.

Frequently Asked Questions

1. Why should I kustomize ingress-nginx at all?

Many setups require custom configurations - such as a custom error backend, additional ports, or special load balancer bindings. Without customization, important features often remain missing.

2. What advantages does kustomize offer compared to a Helm Chart?

Kustomize works declaratively and directly on Kubernetes-YAMLs. You don't need template rendering and can specifically patch individual resources - simple, readable, and Git-friendly.

3. Can I combine kustomize with Helm?

Yes, but this blog post deliberately uses only kustomize to directly patch the ingress-nginx standard resources. Anyone using Helm can combine helm templates and kustomize.

4. What risks exist during updates?

When upstream resources change significantly, patches might fail. Therefore: Check carefully before each update. kubectl kustomize and examine whether everything renders cleanly.

5. What if I have multiple environments like Staging and Production?

kustomize with supported Overlays. You can use a common base directory and define specific patches per environment in separate Overlays.

6. Do I have to adjust all patches with every new ingress-nginx version?

Not necessarily. As long as names, structures, and resource paths do not change, patches often remain stable. Nevertheless, you should briefly review them during version changes.