How can we reduce the storage costs of SkyWalking in a public cloud environment?

[linshenfanying]

Currently, our company primarily uses Java as the development language, and all services are deployed on the public cloud. We use SkyWalking (version 8.9.0) for tracing, with Elasticsearch (directly procured cloud service) as the storage. The required trace sampling rate is 100%. Currently, the data volume in Elasticsearch reaches 2TB per day, and storing 30 days of data has become a significant cost. Is there any way to reduce the data volume while minimizing the impact on the accuracy of the data in Elasticsearch? Alternatively, are there any other solutions to reduce storage costs?

[wu-sheng]

Elasticsearch, no, there's not a good way.
To reduce that, we introduce Banyandb(now is 0.7), it focused on lower resource costs.

[wu-sheng]

Read this for more details, https://skywalking.apache.org/blog/2024-06-04-banyandb-0.6-release/

[linshenfanying]

Are there any other ways, such as by reducing the data volume? We have observed that a lot of the data consists of entries like Thread/org.apache.skywalking.apm.toolkit.trace.RunnableWrapper/run and Kafka/Producer/Callback. For example, we don't need to collect all entries for Kafka/Producer/Callback.

[wu-sheng]

Unless you don't need the whole plugin, you can't do that.