Improved Spike Protection

[Dhrumil-Sentry]

We’ve often heard stories of incidents that led to sudden spikes in errors that deplete monthly Sentry quotas within minutes before our users can even respond and take action. Code can break and an errant line in a loop can deplete your event quota within minutes - we understand how frustrating this problem can be.

Based on user feedback, we identified two main problems with the current spike protection:

• It “adjusts” too quickly causing correctly identified spikes that last for a while to be misclassified as non-spikes
• The spike threshold is simply too high in some cases, resulting in spikes not being caught

The challenge for Sentry is to find the balance between preventing your quota from being depleted rapidly and ensuring we don’t indiscriminately drop events that may be very important for your monitoring needs (For e.g. events that you send to Sentry after a successful launch of a feature)

Based on user feedback, we’ve built a more reliable solution and will be rolling this out to beta customers over the coming weeks.

Summary of Improvements

Here’s an overview of the improvements:

1. Spike protection will be enforced on a per-project level, rather than across the whole organization
   1. This gives you more granular insights into which projects suffer from spikes frequently for you to be able to take preventive measures
   2. This will also help you ensure that a new project which inadvertently generates more events than anticipated does not blow through your quota
2. Spike protection will now work for transactions and attachments so you’ll be protected from transaction spikes or attachment spikes as well
3. Here’s how we have improved the spike detection algorithm:
   1. It considers hourly ingestion rates for the last 7 days instead of one day to determine baselines - this helps ensure the baseline is closer to your typical ingestion rate.
   2. It considers your quotas to determine spike thresholds - this helps avoid false positives for large orgs and new projects that may not have 7 days of historical data. It also ensures that spike protection helps your organization avoid unintended on-demand usage.
   3. The new algorithm accounts for daily and weekly seasonality - it is less likely to misclassify high volume at peak hours as a spike.
   4. Spike protection now accounts for historical variance. If your project is typically very stable in event volume, spike protection will be more aggressive in its detection. Conversely, if event volume typically varies quite a bit over time, detection will be more conservative to avoid false positives.

We have a worked example in our docs explaining how this algorithm would work.

View Spike Thresholds for Your Projects

We’ve also built an in-app user interface that lets you view previous spike information and trends:

1. Go to Stats
2. Select a Project from the selector above the widget showing your aggregated total
3. You can view the spike protection thresholds that were used in the past
4. You can also view details about the spikes Sentry detected - You can see the threshold, duration, and the count of events that we dropped to protect you from the spike

Notifications for Spike Protection

While we understand that no one wants more emails and slack messages, we’d like to ensure your team does not miss any notifications about a spike that could deplete your quotas.

• Sentry will send an email/slack alert to users with an Owner/Billing/Manager role on spike protection activation and deactivation
• If you wish to disable notifications for a project then you can go to Organization Settings > Spike protection >(Select Project)> Delete.
  
• You can also add an action to send spike protection notifications for a project to Slack or PagerDuty
  
• To turn off notifications only for yourself, go to User Settings > Notifications > Spike Protection. You can choose to turn off notifications for all projects, or customize notifications on a per-project basis.

Join the Beta List

The feature is currently in Beta and we’re working towards making this generally available. If you would like to have this feature enabled for your Org, drop us a note at [email protected]

We want to hear from you!

This is a new algorithm and we’d like feedback on how this is working for you. Specifically, we’d like to know:

1. Is the spike protection threshold for your project too high or low?
2. When notified of a spike, what are your usual next steps? Is there any additional information you’d like to see?
3. Do you know about the various tools such as Client-side filtering, Server-side Inbound filters, Rate-Limiting, and Client-side Event sampling to help you filter the events you send to Sentry?
4. Would you like to see any changes in how you are notified for spike protection?

Please feel free to share any feedback in the discussion below

[liadtaranis]

Thanks for the information @Dhrumil-Sentry
In addition to what you have already implemented, it would be great if you could allow sending spike protection alerts per Project/Team (currently they are sent to the billing owner). Such alerts would be much more effective and actionable.

[Dhrumil-Sentry]

@liadtaranis Thanks a lot for the feedback.

Do you wish to send these alerts to a specific Slack channel/Pagerduty service for each project?

[liadtaranis]

Exactly. Or a different email address. We have several teams, and each team owns several projects (services) - therefore it would be helpful only for the owning team to get the alert.

…

On Mon, Feb 13, 2023 at 2:23 AM Dhrumil ***@***.***> wrote: @liadtaranis <https://github.com/liadtaranis> Thanks a lot for the feedback. Do you wish to send these alerts to a specific Slack channel/Pagerduty service for each project? — Reply to this email directly, view it on GitHub <#44155 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A24WAU66YBZS6DRZXX67NC3WXF5H5ANCNFSM6AAAAAAUSD2RPU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *Liad Barkan* QA Team Lead Taranis | Innovation Center: Tel Aviv, Israel +972-546-345-378 www.taranis.com <https://www.linkedin.com/company/5283228/admin/> <https://www.youtube.com/channel/UCt-0bs-qsyk-27FbFGVc6lw/featured> <https://www.facebook.com/taranisvisual> <https://twitter.com/TaranisAg> <https://taranis.ag/>

[realkosty]

feedback from one prospective customer (not related to these improvements per se but general spike protection feature request): #47516

[Dhrumil-Sentry]

We have made improved spike protection available for all our customers. Based on customer feedback we've also made notification improvements that allow users to send a project's spike notifications to a Slack channel/Pagerduty. Please see the discussion above for more details.

Please see this blog post to learn how customers are using Spend Allocation and improved Spike Protection to better manage their quotas

[antonpodolsky]

Is/can the spike protection feature be available in https://github.com/getsentry/self-hosted ?

[chadwhitacre]

Let's use getsentry/self-hosted#2122 to track.

This keeps coming up (rolling features out to self-hosted) and we need a better system for it. In the meantime, we schlep.

[antonpodolsky]

@chadwhitacre Thanks, this will help us prevent meltdowns when one project starts spamming errors.

[marcosrouse]

The account managed by my team has multiple sentry organizations and quite a number of projects, and we want to setup notifications for every project where we have enabled spike protection. I was looking at the API docs and terraform provider, but it does not look like there is a way to programmatically configure the notifications for all the projects I have on my account. Can you point me to documentation that I might have missed or advise if it is actually possible to programmatically configure notifications for each spike protected project?

[Dhrumil-Sentry]

@marcosrouse We currently don't have this API published yet. We plan to publish these endpoints soon

Meanwhile, You can try the following unpublished API endpoint and payload (This endpoint may change in the future but we don't have any plans yet):

Post /organizations/{org-slug}/notifications/actions/
Content-Type: application/json

Payload for adding the email notification for Billing/Owner/Manager:
{ projects: ["<project_slug>"], serviceType: "sentry_notification", targetDisplay: "default", targetIdentifier: "default", targetType: "specific", triggerType: "spike-protection" }

Payload for adding a slack notification:

{ integrationId: <integration_id> # From the URL for your slack integration, integrationName: "<workspace_name>", projects: ["<project_slug>"], serviceType: "slack", targetDisplay: "test" # channel name, targetType: "specific", triggerType: "spike-protection" }

[marcosrouse]

Thanks, I will give it a try!

[saban90]

@Dhrumil-Sentry thank you on this.

Can you please also share other methods for spike endpoint?
For our use case, where we have many projects owned by many teams and we centrally set slack notification, but due to changes we need to check and update the channels at weekly basis.
With POST request we can just add the new notification channel but we also want to remove the old one.

Thank you.

[Dhrumil-Sentry]

@saban90 thanks for this feedback- we're currently working on publishing these APIs and will call out the API to remove a notification action as well

@sentaur-athena @schew2381 This is some helpful feedback for the spike protection notification APIs we want to publish