Time-based triggers not executing

Incident Report for Peripass

Postmortem

Summary

On Monday 8 June 2026 between 08:30 and 18:00 CEST, time-based triggers were not executed. These triggers are used for background processes such as clean-up actions and, in some cases, synchronizations with external systems.

A shorter interruption occurred on Tuesday 9 June 2026 between 09:50 and 10:20 CEST after a second rollback.

Impact

During these windows, scheduled background processes were delayed. This may have affected clean-up actions and some synchronizations with external systems.

Root Cause

The issue was caused by an incompatibility introduced during the automated rollback after the 1.66 release.

The rollback moved the platform back to an older version, but some time-based trigger state had already been created by the newer version. After the rollback to 1.65, the platform was unable to process part of that persisted state correctly. As a result, time-based triggers stopped executing.

Follow-up

We have added alerting to detect when time-based trigger execution drops significantly. We will also improve rollback testing, so incompatibilities between versions can be detected before they affect production.

Posted Jun 12, 2026 - 13:41 CEST

Resolved

An incompatibility was introduced during the automated rollback, which resulted in time-based triggers no longer being executed between 08:30 and 18:00 on Monday. These triggers are used for background processes such as clean-up actions and, in some cases, synchronizations with external systems.

After identifying that time-based triggers were no longer being executed, we released a patch on Monday evening to fix the incompatibility introduced during the initial rollback together with a fix for the performance issues.

Unfortunately, this didn’t resolve the performance issues completely and we decided to perform a second rollback on Tuesday morning to minimize customer impact. This rollback resulted in a short interruption for time-based triggers between 9:50 and 10:20 on Tuesday.

Posted Jun 08, 2026 - 08:30 CEST