Branch API Issue / Reporting Backfill
Incident Report for Branch
Resolved
Attribution processing and sending of webhooks are complete. Re-posting of updated Daily Export Reports for February 5 & 6 UTC will be posted by February 12, 8pm PT and available through February 19, 4pm PT. Data is immediately available via Dashboard and Custom Export API.
Posted Feb 12, 2020 - 10:48 PST
Update
Attribution processing is still in progress, with webhooks sending as information is processed. Current time estimate is that processing will be complete, and webhooks sent, by February 12, 10am PT. Subsequently, updated Daily Export reports for February 5-6 UTC will be posted at February 12, 8pm PT. Once posted, Daily Export reports will be available by Daily Export API and Custom Export API for 7 days.
Posted Feb 11, 2020 - 17:54 PST
Update
Data processing continues to progress and partial backfill data has started to appear within reports. Webhooks have begun sending at this time. The process of data population and webhook sending is estimated to continue for the next few days. We will continue providing daily updates through this process.
Posted Feb 10, 2020 - 17:33 PST
Update
Data processing for the affected time period is underway, and will appear in reporting after verification. Webhooks will be sent subsequent to verification as well. Current time estimate remains that the reporting backfill may take several days. We will continue to provide daily updates.
Posted Feb 09, 2020 - 18:53 PST
Update
We are continuing work on populating reporting data for the affected time period, along with sending of the respective webhooks/postbacks. Current time estimation remains that the reporting backfill may take several days. We will continue to provide daily updates.
Posted Feb 08, 2020 - 14:16 PST
Update
We are continuing work on populating reporting data for the affected time period, along with sending of the respective webhooks/postbacks. Time estimation is that the reporting backfill may take several days. We will continue to provide daily updates.
Posted Feb 07, 2020 - 16:32 PST
Update
We are continuing work on populating reporting data for the affected time period, along with sending of the respective webhooks/postbacks. Current time estimation is that the reporting backfill may take several days. We will continue to narrow this estimate as possible, and will provide daily updates.
Posted Feb 06, 2020 - 15:57 PST
Update
Processing systems remain stable as of 7:47pm PT February 5. We are continuing work to process data from the gap period. We will provide the next update prior to 4pm PT.
Posted Feb 06, 2020 - 08:54 PST
Monitoring
As of 7:47pm PT all processing systems are back to normal and processing realtime events. Our team is currently working to restore reporting data and webhooks that were not processed during this time period. We will provide the next update prior to 9am PT.
Posted Feb 05, 2020 - 21:03 PST
Update
We have identified several failing nodes in event processing pipelines. We manually halted event processing at 12pm PST as we triaged, investigated and worked to restore service. We are currently finishing the final testing and deployments to restore realtime processing, and are aiming to bring processing systems for Analytics, Attribution, and Webhooks back online as soon as possible. This will restore realtime services and processing. Following realtime processing, we will be immediately working to backfill and restore data and webhooks halted during the incident window.
Posted Feb 05, 2020 - 18:54 PST
Update
We are continuing work in restoring all Webhooks and Reporting service.
Posted Feb 05, 2020 - 17:10 PST
Identified
As of 3pm PST, API functionality began recovering and appears to have stabilized. We're continuing to monitor, and our engineering team is still working on restoring all services to 100%. We are continuing work to restore the publishing of data via Webhooks and to Analytics reporting.
Posted Feb 05, 2020 - 15:35 PST
Investigating
Starting around 12:00pm PST we are encountering issues in Branch API activity, specifically for Open, Install, Click and URL endpoints. This is resulting in a halt to Webhook traffic and new Analytics posting.
V2 Events and TUNE activity are not affected.
We are investigating and will continue to post updates.
Posted Feb 05, 2020 - 14:12 PST
This incident affected: API, Dashboard, and Webhooks.