As of 10PM PST on 7/2, V2 events resumed publishing as normal and without delay. All previous data is now republished for 7/1-7/2, except for 8h of data loss from approximately 8am-4pm PST on 7/1. This data cannot be recovered, and applies only to V2 commerce, user lifecycle, content, and custom events.
The root cause for this incident was a crash in our kafka publishing job and subsequent failures in alerting. We apologize for this issue, and are taking steps to prevent this from recurring in future. This includes auditing alerts, adding more monitoring and alerting, and fixing the root cause of the crash. Please reach out to your account manager or support with any questions.
Posted Jul 03, 2018 - 11:25 PDT
Identified
Starting at 07/01 at 10AM PST to present, V2 Event (custom, commerce, content, and user lifecycle events) stopped publishing, followed by slow ramp up. This means that webhooks did not fire for any V2 events and data is missing from dashboard and exports. We're now republishing, which means some webhooks will fire with a delay. We are focused on restoring service as quickly as possible, and we apologize for the inconvenience.