Management API outage

The Management API has recovered and is operating normally again after rescheduling affected internal Retail services to healthy infrastructure. This was correlated with ongoing compute instability at our infrastructure provider (Exoscale, CH-DK-2): https://exoscalestatus.com/incidents/217987#activity-id-461560

We’ll continue monitoring and will provide updates if the issue recurs.

We are reopening this incident due to renewed intermittent errors on the Management API (500 responses/timeouts). This correlates with an ongoing incident at our infrastructure provider (Exoscale) impacting compute in CH-DK-2, causing instability/unavailability of some of our instances.
We are actively rescheduling affected services onto healthy infrastructure.
Next update in 30 minutes or sooner if fully restored.

The incident has been mitigated and services are operational.
As part of the mitigation, we updated the DNS record for api.whalebone.io to point to a new Elastic IP.

Important (IP allowlisting/firewalls): api.whalebone.io now resolves to 159.100.248.48.
If your environment uses IP allowlisting, please allowlist 159.100.248.48 to restore/maintain access.

Our infrastructure provider (Exoscale) is investigating a compute partial outage (their status page), which is impacting one of our host. While Exoscale works on host recovery, we are executing a mitigation plan to restore the affected component on an alternate healthy host to unblock authentication and recover the Management API (provisioning + device management).
Next update in 30 minutes or earlier if service is restored.

We have detected an outage affecting the Management API, which is used for provisioning new retail subscribers and for device management. The API is currently returning the following error:
"error":"invalid combination of access key and secret key".

As a result, the following actions may fail or be delayed:

• Creating new subscribers
• Creating new devices
• Updating filtering policies

The Retail API is primarily used by telco operators and larger ISPs offering B2C services.

At this time, the issue appears to be related to a hardware failure at our infrastructure provider. We are actively working on a fix and will provide updates as soon as more information becomes available.

We apologize for any inconvenience this may cause and appreciate your patience.