Comment by maccard

As the other commentor said, you can deploy Prometheus/grafana into the k8s cluster and it pretty much does what you want it to do.