In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 02, 2025 - 08:00 UTC
Apr 02, 2025 - 08:00 UTC
Scheduled - During this maintenance window, we will:
- Switch to a new DDoS provider (the main driver for the extended duration). We have tested it in non-production environments and expect no downtime, but IP addresses will change. Please update firewalls to use DNS names. We will trigger Agent-Cloud link reconnection but it should be transparent for end users.
- Upgrade Elasticsearch, which forms the backbone of our Event Feed. We do expect Events tab to not work properly.
Apr 2, 2025 08:00-10:00 UTC
- Switch to a new DDoS provider (the main driver for the extended duration). We have tested it in non-production environments and expect no downtime, but IP addresses will change. Please update firewalls to use DNS names. We will trigger Agent-Cloud link reconnection but it should be transparent for end users.
- Upgrade Elasticsearch, which forms the backbone of our Event Feed. We do expect Events tab to not work properly.
Apr 2, 2025 08:00-10:00 UTC
Investigating - We have found that a recent change in the nightly static builds of Netdata Agent causes meta data on the Agent to be overwritten. Specifically the sqlite3 database that keeps meta data on which timeseries stored in dbengine correspond with which metrics, and the information on the Agent's "machine GUID" are overwritten with the same copy in the build package.
Not affected are:
- All stable releases
- Native packages (.deb and .rpm)
Affected are all nightly static builds with the following version numbers:
- 2.3.0-50-nightly
- 2.3.0-60-nightly
- 2.3.0-72-nightly
- 2.3.0-78-nightly
- 2.3.0-87-nightly
The initial impact is that all affected Agent installs, even though they still have the timeseries data stored on disk, have lost all meta data associated with it, so these timeseries become inaccessable. This is unrecoverable. Additionally, the main form of identification is overwritten, too.
We are assessing what the impact is for users of Netdata Cloud, and will update this incident with more information when the investigation is completed.
The bug itself has been fixed and merged. We will issue a new nightly build shortly.
Apr 01, 2025 - 09:00 UTC
Not affected are:
- All stable releases
- Native packages (.deb and .rpm)
Affected are all nightly static builds with the following version numbers:
- 2.3.0-50-nightly
- 2.3.0-60-nightly
- 2.3.0-72-nightly
- 2.3.0-78-nightly
- 2.3.0-87-nightly
The initial impact is that all affected Agent installs, even though they still have the timeseries data stored on disk, have lost all meta data associated with it, so these timeseries become inaccessable. This is unrecoverable. Additionally, the main form of identification is overwritten, too.
We are assessing what the impact is for users of Netdata Cloud, and will update this incident with more information when the investigation is completed.
The bug itself has been fixed and merged. We will issue a new nightly build shortly.
Apr 01, 2025 - 09:00 UTC
Cloud Web UI
Operational
Cloud Alerts
Operational
Cloud Charts
Operational
Cloud Event Feed
Under Maintenance
Cloud Payment
Operational
Agent - Cloud Connection (ACLK)
Under Maintenance
Agent (all platforms)
Partial Outage
Agent Repositories
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.