Wikipedia:Wikipedia Signpost/2025-02-27/Technology report
Hear that? The wikis go silent twice a year
- Benoît Evellin is a Senior Movement Communications Specialist (Product & Tech), and Ana Eira is a Movement Communications Specialist, for the Wikimedia Foundation.
When you click the "edit" button on a wiki, you’re likely focused on improving the content. The process feels seamless: edit, save, repeat. From patrolling new edits to uploading photos or joining a campaign, you can count on the Wikimedia platform to be up and running—in your language, anywhere in the world. That is, except for a couple of minutes during the equinoxes.
Twice a year, around the equinoxes, the Wikimedia Foundation's Site Reliability Engineering (SRE) team performs a datacenter server switchover, redirecting all traffic from one primary server to another—essentially a backup. But why? In case of a crisis we can rely on the other datacenter.
The scheduled switchover process allows for essential maintenance and improvements at the backup center. It also allows the team to test their procedures, minimize the impact of the read-only time, and work on the overall reliability of the sites.
Thanks to Listen to Wikipedia, a playful tool that turns each edit into a sound and visualizes it as a floating bubble in real-time, you can actually hear the switchover take place. Before the switchover starts, you will hear the steady stream of edit sounds. But then—about 2 minutes in—the sound stops, which means the system has entered the read-only phase. And when the sounds start up again? That’s the moment engineers can finally breathe—it’s the clear signal that the toughest part of the switchover is done and edits are flowing again. Watch this video to experience that extraordinary moment.
This rare interruption happens because all Wikimedia wikis rely on a server designed specifically for their needs and managed by the Wikimedia Foundation. This setup allows us to remain independent, while delivering a reliable experience for users around the world.
The SRE team oversees a global network of datacenters, seven in total, spanning the United States, Singapore, the Netherlands, France, and, most recently, Brazil.These datacenters allow articles and other content to load quickly, securely, and privately, to be accessed anywhere and anytime.
At the heart of this network are the two application server groups which host the live copies of the projects. Having two application server groups is necessary to keep all the wikis we host alive: if one server fails, the second one can take over, and vice-versa. With an estimated 342 edits per minute happening on Wikipedia alone, having a backup server is a must.
While the switch over may sound simple, the reality is that the process has evolved significantly over the years.
Every equinox, it's switchover time
Since its inception, the datacenter switchover has been refined and is now largely automated. What once took nearly an hour of read-only downtime now takes between 2 and 3 minutes–a significant leap forward in efficiency. But how did we get here?
It all started in 2015, when an increase in donations allowed the Foundation to allocate a larger budget to establish a second datacenter capable of hosting the core of our ecosystem, using MediaWiki, and all other services that make it work. To prove the new datacenter’s capabilities, the SRE team had to test that all operations could be fully served from this new location.
The first switchover, in March 2016, was a big undertaking. It took six months of preparation and the effort of ten engineers. Beyond the technical groundwork, the Foundation shared the switchover timeline through banners, village pumps, and other outreach channels, keeping everyone informed.
During the process, a technical limitation in MediaWiki forced the SRE team to set the wikis to read-only mode for 45 minutes, followed by two hours of somewhat degraded performance. The wikis remained accessible, but no one could edit them. Shifting operations from a "warm" datacenter — already handling significant traffic — to a "cold" datacenter had noticeable impacts. The key takeaway? Regular switchover practice was essential.
By deciding to schedule switchovers regularly, the team wanted to guarantee that the backup center is always prepared to take over full operations in the event of an emergency. After each switchover, the new primary data center handles all traffic for a week, allowing time for essential maintenance and improvements at the secondary center. It also allows the team to test their procedures, diminish the impact of the read-only time, and work on the reliability of the sites.
Why the equinox?
Picking something memorable that does not particularly change across cultures, countries, hemispheres, jurisdictions, et cetera, allows more people to remember it and relate to it, making it more fitting to our global movement. Human-made things tend to vary a lot and have different connotations (including bad ones) across cultures, so we settled on something that has been close to a constant for humankind since times immemorial. That's an astronomical event that has been quite predictable by humankind for millennia: the solar equinox.
Keen-eyed Wikimedians will certainly note the traffic is not redirected precisely on the day of the equinox but on the Wednesday the week it happens, as it requires some preparation work. But it’s easy for everyone to remember, whether it’s editors, affiliates hosting events, or developers improving the platform’s codebase.
The decision to align the system switchover with the equinoxes reflects both practicality and a nod to Principal Site Reliability Engineer Alexandros Kosiaris’ passion for astronomy. The equinoxes aren’t just a functional choice — they’re an irresistible one, as they mark a natural rhythm that resonates with the team. As they like to say, edits "fly north in the spring and south in the fall", mirroring the migratory path of birds between datacenters.
To build resilience into this process and make sure all the right people know how to run it, the newest team member manages the switchover. A rite of passage that guarantees the process is thoroughly documented, easy to follow, and provides hands-on experience on one of the most critical operations. It’s also a reminder that behind the editing experience of the Wikimedia projects is a team of dedicated engineers, constantly learning and improving.
So the next time you click "edit", know that behind that simple action lies a carefully maintained network that makes your contribution part of the world’s shared knowledge — a collaboration of people and technology that makes the whole system work, unseen but essential.
If you want to keep an ear out for the next server switchover, listen to the wikis on Wednesday, March 19th 2025, at 14:00 UTC.
Discuss this story