Remove Loki (or fix the compactor crashloop)

[bwplotka]

Compactor is crashlooping for Loki

Seems like a configuration issue, but I wonder if we are using Loki for Prombench use cases. We run in GKE and logging is already OOTB in, so it feels we could save some cost by removing Loki for our use case:

Disclaimer: I am working for Google Cloud, proposing to remove Grafana Loki instance from prombench, so second opinion from non-Googler Prometheus team member would be nice here (((((:

cc @bboreham @kakkoyun are you using Loki?

[bboreham]

I fixed up the config. #781

I find Google Cloud logging terrible to use compared to Loki.
Maybe it's a case of what you are used to.

E.g. it decorates everything with an angry red mark

[bboreham]

More generally, the test-infra repo is set up to be usable in different situations - GCP, AWS, Kind.
It would simplify quite a few things if we hard-coded to work only at Google.