Wednesday, January 18, 2023, 6:30 PM IST (Irish Standard Time)

The Collaborate team was alerted to an issue with the Whiteboard Recording feature that impacted a subset of Collaborate Recordings during the incident timeframe.  The incident began at approximately 11:15 PM UTC on January 17th as a result of a misconfiguration during a non-impacting fix applied to the Recordings service.  This led to the inability of Whiteboard or Shared Files to be included in Collaborate Recordings, which were instead replaced with a “block color” in the video.

The incident was resolved on January 18th at approximately 11:15 AM UTC.    Affected Recordings cannot be reprocessed to include the Whiteboard or Shared Files data.  Recordings that did not use the Whiteboard or Shared Files feature were unaffected.   All services are fully operational.

The Collaborate team takes today’s service disruption very seriously and has taken immediate steps on corrective actions to ensure this incident does not reoccur.  Testing of the Whiteboard Recording feature was included in the plans that were executed following the application of the fix to the Recordings service.  However, those tests did not properly account for the full set of prerequisites which led to a successful result that was inaccurate.  Additionally, the signature of this incident did not sufficiently get picked up by Collaborate’s robust monitoring and alerting system which led to delays in response time.  Both of these scenarios have been corrected.