Kubeeye - Tool To Find Various Problems On Kubernetes, Such As Application Misconfiguration, Unhealthy Cluster Components And Node Problems
KubeEye is an inspection tool for Kubernetes to discover Kubernetes resources (by OPA ), cluster components, cluster nodes (by Node-Problem-Detector) and other configurations are meeting with best practices, and giving suggestions for modification.
KubeEye supports custom inspection rules and plugins installation. Through KubeEye Operator, you can view the inspection results and modification suggestions by the graphical display on the web page.
Architecture
KubeEye get cluster resource details by the Kubernetes API, inspect the resource configurations by inspection rules and plugins, and generate inspection results. See Architecture for details.
How to use
-
Install KubeEye on your machine
-
Download pre built executables from Releases.
-
Or you can build from source code
Note: make install will create kubeeye in /usr/local/bin/ on your machine.
git clone https://github.com/kubesphere/kubeeye.gitcd kubeeyemake installke
-
-
[Optional] Install Node-problem-Detector
Note: This will install npd on your cluster, only required if you want detailed report.
kubeeye install npd
- Run KubeEye
Note: The results of kubeeye sort by resource kind.
kubeeye auditKIND NAMESPACE NAME REASON LEVEL MESSAGENode docker-desktop kubelet has no sufficient memory available warning KubeletHasNoSufficientMemoryNode docker-desktop kubelet has no sufficient PID available warning KubeletHasNoSufficientPIDNode docker-desktop kubelet has disk pressure warning KubeletHasDiskPressureDeployment default testkubeeye NoCPULimitsDeployment default testkubeeye NoReadinessProbeDeployment default testkubeeye NotRunAsNonRootDeployment kube-system coredns NoCPULimitsDeployment kube-system coredns ImagePullPolicyNotAlwaysDeployment kube-system coredns NotRunAsNonRootDeployment kubeeye-system kubeeye-controller-manager ImagePullPolicyNotAlwaysDeployment kubeeye-system kubeeye-controller-manager NotRunAsNonRootDaemonSet kube-system kube-proxy NoCPULimitsDaemonSet k ube-system kube-proxy NotRunAsNonRootEvent kube-system coredns-558bd4d5db-c26j8.16d5fa3ddf56675f Unhealthy warning Readiness probe failed: Get "http://10.1.0.87:8181/ready": dial tcp 10.1.0.87:8181: connect: connection refusedEvent kube-system coredns-558bd4d5db-c26j8.16d5fa3fbdc834c9 Unhealthy warning Readiness probe failed: HTTP probe failed with statuscode: 503Event kube-system vpnkit-controller.16d5ac2b2b4fa1eb BackOff warning Back-off restarting failed containerEvent kube-system vpnkit-controller.16d5fa44d0502641 BackOff warning Back-off restarting failed containerEvent kubeeye-system kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f5fc3229c Failed warning Failed to pull image "controller:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for controller, repository does not exist or may require 'docker login': denied: requested access to the resource is deniedEvent kubeeye-system kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f61b28527 Failed warning Error: ImagePullBackOffRole kubeeye-system kubeeye-leader-election-role CanDeleteResourcesClusterRole kubeeye-manager-role CanDeleteResourcesClusterRole kubeeye-manager-role CanModifyWorkloadsClusterRole vpnkit-controller CanImpersonateUserClusterRole vpnkit-controller CanDeleteResources
What KubeEye can do
- KubeEye inspects cluster resources according with Kubernetes best practices, to make cluster stable.
- KubeEye can find problems of your cluster control plane, including kube-apiserver/kube-controller-manager/etcd, etc.
- KubeEye helps you detect all kinds of cluster nodes problems, including memory/cpu/disk pressure, unexpected kernel error logs, etc.
Checklist
YES/NO | CHECK ITEM | Description | Level |
---|---|---|---|
✅ | PrivilegeEscalationAllowed | Privilege escalation is allowed | danger |
✅ | CanImpersonateUser | The role/clusterrole can impersonate other user | warning |
✅ | CanModifyResources | The role/clusterrole can delete kubernetes resources | warning |
✅ | CanModifyWorkloads | The role/clusterrole can modify kubernetes workloads | warning |
✅ | NoCPULimits | The resource does not set limits of CPU in containers.resources | danger |
✅ | NoCPURequests | The resource does not set requests of CPU in containers.resources | danger |
✅ | HighRiskCapabilities | Have high-Risk options in capabilities such as ALL/SYS_ADMIN/NET_ADMIN | danger |
✅ | HostIPCAllowed | HostIPC Set to true | danger |
✅ | HostNetworkAllowed | HostNetwork Set to true | danger |
✅ | HostPIDAllowed | HostPID Set to true | danger |
✅ | HostPortAllowed | HostPort Set to true | danger |
✅ | ImagePullPolicyNotAlways | Image pull policy not always | warning |
✅ | ImageTagIsLatest | The image tag is latest | warning |
✅ | ImageTagMiss | The image tag do not declare | danger |
✅ | InsecureCapabilities | Have insecure options in capabilities such as KILL/SYS_CHROOT/CHOWN | danger |
✅ | NoLivenessProbe | The resource does not set livenessProbe | warning |
✅ | NoMemoryLimits | The resource does not set limits of memory in containers.resources | danger |
✅ | NoMemoryRequests | The resource does not set requests of memory in containers.resources | danger |
✅ | NoPriorityClassName | The resource does not set priorityClassName | ignore |
✅ | PrivilegedAllowed | Running a pod in a privileged mode means that the pod can access the host’s resources and kernel capabilities | danger |
✅ | NoReadinessProbe | The resource does not set readinessProbe | warning |
✅ | NotReadOnlyRootFilesystem | The resource does not set readOnlyRootFilesystem to true | warning |
✅ | NotRunAsNonRoot | The resource does not set runAsNonRoot to true, maybe executed run as a root account | warning |
✅ | CertificateExpiredPeriod | Certificate expiration date less than 30 days | danger |
✅ | EventAudit | Event audit | warning |
✅ | NodeStatus | node status audit | warning |
✅ | DockerStatus | docker status audit | warning |
✅ | KubeletStatus | kubelet status audit | warning |
Add your own inspection rules
Add custom OPA rules
- create a directory for OPA rules
mkdir opa
- Add custom OPA rules files
Note: the OPA rule for workloads, package name must be kubeeye_workloads_rego for RBAC, package name must be kubeeye_RBAC_rego for nodes, package name must be kubeeye_nodes_rego
- Save the following rules to rule file such as imageRegistryRule.rego to check the image registry address complies with rules.
package kubeeye_workloads_regodeny[msg] { resource := input type := resource.Object.kind resourcename := resource.Object.metadata.name resourcenamespace := resource.Object.metadata.namespace workloadsType := {"Deployment","ReplicaSet","DaemonSet","StatefulSet","Job"} workloadsType[type] not workloadsImageRegistryRule(resource) msg := { "Name": sprintf("%v", [resourcename]), "Namespace": sprintf("%v", [resourcenamespace]), "Type": sprintf("%v", [type]), "Message": "ImageRegistryNotmyregistry" }}workloadsImageRegistryRule(resource) { regex.match("^myregistry.public.kubesphere/basic/.+", resource.Object.spec.template.spec.containers[_].image)}
- Run KubeEye with custom rules
Note: Specify the path then Kubeeye will read all files in the directory that end with .rego.
root:# kubeeye audit -p ./opaNAMESPACE NAME KIND MESSAGEdefault nginx1 Deployment [ImageRegistryNotmyregistry NotReadOnlyRootFilesystem NotRunAsNonRoot]default nginx11 Deployment [ImageRegistryNotmyregistry PrivilegeEscalationAllowed HighRiskCapabilities HostIPCAllowed HostPortAllowed ImagePullPolicyNotAlways ImageTagIsLatest InsecureCapabilities NoPriorityClassName PrivilegedAllowed NotReadOnlyRootFilesystem NotRunAsNonRoot]default nginx111 Deployment [ImageRegistryNotmyregistry NoCPULimits NoCPURequests ImageTagMiss NoLivenessProbe NoMemoryLimits NoMemoryRequests NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot]
Add custom NPD rules
- edit configmap
kubectl edit ConfigMap node-problem-detector-config -n kube-system
- restart NPD deployment
kubectl rollout restart DaemonSet node-problem-detector -n kube-system
KubeEye Operator
What is KubeEye Operator
KubeEye Operator is an inspection platform for Kubernetes, manage KubeEye by operator and generate inspection result.
What KubeEye Operator can do
- KubeEye Operator provides management functions through web page.
- KubeEye Operator recode inspection results by CR, can view and compare cluster inspection results by web page.
- KubeEye Operator provides more plugins.
- KubeEye Operator provides more detailed modification suggestions.
deploy Kubeeye
kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye.yamlkubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye_insights.yaml
get the inspection results
kubectl get clusterinsight -o yaml
apiVersion: v1items:- apiVersion: kubeeye.kubesphere.io/v1alpha1 kind: ClusterInsight metadata: name: clusterinsight-sample namespace: default spec: auditPeriod: 24h status: auditResults: auditResults: - resourcesType: Node resultInfos: - namespace: "" resourceInfos: - items: - level: warning message: KubeletHasNoSufficientMemory reason: kubelet has no sufficient memory available - level: warning message: KubeletHasNoSufficientPID reason: kubelet has no sufficient PID available - level: warning message: KubeletHasDiskPressure reason: kubelet has disk pressure name: kubeeyeNode
Documents
Source: www.kitploit.com