Elevated API Errors
Incident Report for Branch
Postmortem

On October 20, 2017 2:50PM PT our v1/url API endpoint began having issues handling HTTP GET requests. This resulted in an increase error rate to about 1% across our API. ~~Note, none of our other API endpoints were impacted.~~

UPDATE 6:23PM PT: upon closer investigation, we've determined this issue impacted several GET HTTP endpoints: v1/url, v1/deepview, v1/credits, v1/credithistory, v1/profile, v1/export, v2/export. While we're still investigating the root cause, we wanted to provide this update for visibility.

We identified the malfunctioning component, which was related to the SDK token and authentication service on that endpoint, and rolled back the change at 5:10PM which immediately rectified the issue. During that time window all GET requests the v1/url endpoint would have failed. This wasn't immediately detected because the overall GET request volume to that specific endpoint is relatively low compared to other methods and endpoints.

We're investigating the root cause and will carefully roll forward only once we're confident this won't happen again.

Posted Oct 20, 2017 - 17:33 PDT

Resolved
We have performed a rollback on our API, and error rates have returned to normal.
Posted Oct 20, 2017 - 17:32 PDT
Investigating
Our team is investigating a 1% increase in HTTP 400 error rate.
Posted Oct 20, 2017 - 17:10 PDT