Add debug logs to monitor retry in waitForEmptyEsNode

[hooseins]

One-line summary

Issue : #57 (related to some extent)

Adds some debug logs to the draining logic in order to understand a recent issue leading to a stuck scaling operation at draining of a pod.

Description

Increase the visibility of the retrues in order to understand issues with es-operator scale down in light of recent anomalies. During pod draining, es shards are queried to retrieve the shard count per node. There's a retry until the shards are zero on the pod so that the pod can be drained. However it is getting stuck there as per logs. Logs stop printing after a few retries pointing out better logging required at a few places where there's a retry due to errors like json unmarshaling or pod ip exclusion. A debug log would be sufficient which can be turned on only if needed.

Types of Changes

• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Configuration change
  [x] Refactor/improvements
• Documentation / non-code

Tasks

• Add debug logging in case of retries wherever missing in the call to get shards (includes one each for json unmarshalling and ip exclusion issues and one generic that prints the body and response of the get shards call)
• The existing es_client_test has been verified.

Review

• Tests
• Documentation
• CHANGELOG

Deployment Notes

These should highlight any db migrations, feature toggles, etc.

[girishc13]

👍

[otrosien]

👍

[girishc13]

@mikkeloscar Can you add your +1. We need to merge this ASAP and determine a fix.

[mikkeloscar]

👍

[otrosien]

👍