How to use minikube + qemu2 + ingress-dns on macOS
This is a guide on how to use minikube and qemu2 on macOS to run Kubernetes locally on your machine. The guide covers the installation of required packages using brew and provides step-by-step instructions to help you get started. If you are interested in running Kubernetes on your macOS and prefer to use open-source solutions, this guide is for you.
My personal requirements:
- Install packages using
brew: https://brew.sh/ - I don't want to use Docker or Podman
- I use open-source solutions, if possible
My setup:
- Laptop model: MacBook Pro 2019, Intel Core i9
- OS: macOS Sonoma, version 14.3.1
Notes:
- I don't know if this works on MacBooks with a M chip.
So, what are those components?
QEMU: An open-source virtualization software that enables emulation of various architectures and devices, allowing users to run virtual machines on their systems.socket_vmnet: socket_vmnet provides vmnet.framework support for QEMU. Learn more about it at https://github.com/lima-vm/socket_vmnet and https://developer.apple.com/documentation/vmnetminikube: A tool that allows you to run Kubernetes on your local machine, making it easy to develop and test your applications locally.dnsmasq: DNS service for ingress controllers running on your minikube server.
Install qemu as our driver
To install qemu, follow the guide available at:
https://www.qemu.org/download/#macos
It will probably ask you to use brew, with the following command:
brew install qemuInstall qemu
Now, you will need to install the socket_vmnet network support. Follow the installation instructions available at:
https://minikube.sigs.k8s.io/docs/drivers/qemu/#networking
If nothing has changed since I wrote this post, run the following commands:
brew install socket_vmnet
brew tap homebrew/services
HOMEBREW=$(which brew) && sudo ${HOMEBREW} services restart socket_vmnetInstall socket_vmnet
Create your minikube cluster
With all the packages installed, let's start the minikube cluster. The following command will use qemu, use the socket_vmnet project for networking and install the default nginx ingress after that:
minikube start --addons=ingress,ingress-dns --driver=qemu --network=socket_vmnetcreate a minkube cluster
When it finishes, you should be able to access the endpoint and get a 404 return code:
curl $(minikube ip)curl the minikube ip
And that's it. Now, you should have everything you need to start developing on your macOS!
But man, you should just use Linux instead... kind works with a single command and doesn't need all this mess...How to have local domain for development?
Let's assume you want to test your application using custom domains, like my-app.exodus.test. For that, you will need to have a wildcard that forwards the traffic to your minikube cluster. We will use the ingress-dns add-on for that. We previously installed it, when we start the minikube cluster. In other to make our computer to read information from it, we must create a proper resolver.
Now, let's create the resolver to use it:
# Create the resolver directory
sudo mkdir -v /etc/resolver
# Add your nameserver to resolvers
sudo bash -c 'echo "nameserver "$(minikube ip) > /etc/resolver/test'
sudo bash -c 'echo "domain test" >> /etc/resolver/test'
sudo bash -c 'echo "search_order 1" >> /etc/resolver/test'
sudo bash -c 'echo "timeout 5" >> /etc/resolver/test'
# Reload the mDNS
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
# If you want to check the configuration, run
scutil --dnsNow, let's test it:
# Just to make sure, let's clean your DNS local cache
dscacheutil -q host -a name cafe.test
# Now, let's curl cafe.test
curl cafe.test
#> <html>
#> <head><title>404 Not Found</title></head>
#> <body>
#> <center><h1>404 Not Found</h1></center>
#> <hr><center>nginx</center>
#> </body>
#> </html>If everything worked, now you are ready to create your first application! 😄
Create my first deployment + ingress
Finally, if you want to create a simple default app
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
name: web-service
spec:
ports:
- name: http
port: 80
selector:
app: web
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-deployment
spec:
replicas: 2
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- image: docker.io/nginx
name: nginx
ports:
- containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web-ingress
spec:
rules:
- host: cafe.test
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web-service
port:
number: 80
EOFNow, you should be able to go your browser and open the http://cafe.test page!
Troubleshooting
I faced multiple issues I couldn't reliably reproduce. Examples:
- ISSUE: after restarting the machine, can't start the service
Error: Failure while executing; /bin/launchctl bootstrap system /Library/LaunchDaemons/homebrew.mxcl.socket_vmnet.plist exited with 5.$ HOMEBREW=$(which brew) && sudo ${HOMEBREW} services start socket_vmnet
#> Bootstrap failed: 5: Input/output error
#> Error: Failure while executing; `/bin/launchctl bootstrap system /Library/LaunchDaemons/homebrew.mxcl.socket_vmnet.plist` exited with 5.
# Solution
# simply restart the service
$ HOMEBREW=$(which brew) && sudo ${HOMEBREW} services restart socket_vmnet
# then start minikube again
minikube start --addons=ingress --driver=qemu --network=socket_vmnet