How to use minikube + qemu2 + ingress-dns on macOS

macos Feb 29, 2024

This is a guide on how to use minikube and qemu2 on macOS to run Kubernetes locally on your machine. The guide covers the installation of required packages using brew and provides step-by-step instructions to help you get started. If you are interested in running Kubernetes on your macOS and prefer to use open-source solutions, this guide is for you.

My personal requirements:

  • Install packages using brew: https://brew.sh/
  • I don't want to use Docker or Podman
  • I use open-source solutions, if possible

My setup:

  • Laptop model: MacBook Pro 2019, Intel Core i9
  • OS: macOS Sonoma, version 14.3.1

Notes:

  • I don't know if this works on MacBooks with a M chip.

So, what are those components?

  • QEMU: An open-source virtualization software that enables emulation of various architectures and devices, allowing users to run virtual machines on their systems.
  • socket_vmnet: socket_vmnet provides vmnet.framework support for QEMU. Learn more about it at https://github.com/lima-vm/socket_vmnet and https://developer.apple.com/documentation/vmnet
  • minikube: A tool that allows you to run Kubernetes on your local machine, making it easy to develop and test your applications locally.
  • dnsmasq: DNS service for ingress controllers running on your minikube server.

Install qemu as our driver

To install qemu, follow the guide available at:
https://www.qemu.org/download/#macos

It will probably ask you to use brew, with the following command:

brew install qemu

Install qemu

Now, you will need to install the socket_vmnet network support. Follow the installation instructions available at:
https://minikube.sigs.k8s.io/docs/drivers/qemu/#networking

If nothing has changed since I wrote this post, run the following commands:

brew install socket_vmnet
brew tap homebrew/services
HOMEBREW=$(which brew) && sudo ${HOMEBREW} services restart socket_vmnet

Install socket_vmnet

Create your minikube cluster

With all the packages installed, let's start the minikube cluster. The following command will use qemu, use the socket_vmnet project for networking and install the default nginx ingress after that:

minikube start --addons=ingress,ingress-dns --driver=qemu --network=socket_vmnet

create a minkube cluster

When it finishes, you should be able to access the endpoint and get a 404 return code:

curl $(minikube ip)

curl the minikube ip

And that's it. Now, you should have everything you need to start developing on your macOS!

But man, you should just use Linux instead... kind works with a single command and doesn't need all this mess...

How to have local domain for development?

Let's assume you want to test your application using custom domains, like my-app.exodus.test. For that, you will need to have a wildcard that forwards the traffic to your minikube cluster. We will use the ingress-dns add-on for that. We previously installed it, when we start the minikube cluster. In other to make our computer to read information from it, we must create a proper resolver.

Now, let's create the resolver to use it:

# Create the resolver directory
sudo mkdir -v /etc/resolver

# Add your nameserver to resolvers
sudo bash -c 'echo "nameserver "$(minikube ip) > /etc/resolver/test'
sudo bash -c 'echo "domain test" >> /etc/resolver/test'
sudo bash -c 'echo "search_order 1" >> /etc/resolver/test'
sudo bash -c 'echo "timeout 5" >> /etc/resolver/test'

# Reload the mDNS
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist

# If you want to check the configuration, run
scutil --dns

Now, let's test it:

# Just to make sure, let's clean your DNS local cache
dscacheutil -q host -a name cafe.test

# Now, let's curl cafe.test
curl cafe.test
#> <html>
#> <head><title>404 Not Found</title></head>
#> <body>
#> <center><h1>404 Not Found</h1></center>
#> <hr><center>nginx</center>
#> </body>
#> </html>

If everything worked, now you are ready to create your first application! 😄

Create my first deployment + ingress

Finally, if you want to create a simple default app

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  ports:
  - name: http
    port: 80
  selector:
    app: web
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - image: docker.io/nginx
        name: nginx
        ports:
        - containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
spec:
  rules:
  - host: cafe.test
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-service
            port:
              number: 80
EOF

Now, you should be able to go your browser and open the http://cafe.test page!

Troubleshooting

I faced multiple issues I couldn't reliably reproduce. Examples:

  1. ISSUE: after restarting the machine, can't start the service
Error: Failure while executing; /bin/launchctl bootstrap system /Library/LaunchDaemons/homebrew.mxcl.socket_vmnet.plist exited with 5.
$ HOMEBREW=$(which brew) && sudo ${HOMEBREW} services start socket_vmnet
#> Bootstrap failed: 5: Input/output error
#> Error: Failure while executing; `/bin/launchctl bootstrap system /Library/LaunchDaemons/homebrew.mxcl.socket_vmnet.plist` exited with 5.

# Solution
# simply restart the service
$ HOMEBREW=$(which brew) && sudo ${HOMEBREW} services restart socket_vmnet

# then start minikube again
minikube start --addons=ingress --driver=qemu --network=socket_vmnet

References

Tags

Luiz Costa

I am a software engineer at Red Hat / Ansible. I love automation tools, games, and coffee. I am also an active contributor to open-source projects on GitHub.