Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

A studio rebuilt its cluster in 14 hours after a Yandex Cloud outage ahead of migrating 4 million players

A day before a 4-million-player migration, an attack on Yandex Cloud took down the zone hosting the studio's Russian infrastructure. With Claude Code, the team described the infrastructure for another cloud within hours and brought up a Kubernetes cluster 14 hours before launch. A week earlier, the same agent had found a 599-row table that the database read 310 million times. The lesson: prepare a backup cloud in advance rather than postponing it.

AI-processed from Habr AI; edited by Hamidun News
A studio rebuilt its cluster in 14 hours after a Yandex Cloud outage ahead of migrating 4 million players
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Game4.2 million accounts, about 80,000 daily active players
Key metric #2Claude Code found a full scan of a 599-row table run 310 million times
Key metric #3Buckets and CDN stayed in Yandex and moved two days later

Imagine this: you spend a year preparing to move a mobile game to a new version, and the evening before launch, the entire Russian part of your infrastructure disappears. That is what happened to one game studio. An attack on Yandex Cloud made the zone unavailable, and it hosted the Kubernetes cluster, buckets and CDN. About 4 million players had to be moved, that is 4.2 million accounts, of which roughly 80,000 log in per day (this is the DAU, the number of daily active users). The infrastructure team lead described how the team prepared, where Claude Code helped and where the team itself made mistakes.

What was moving and why

The old version of the game was a PHP monolith with many cron jobs and a Unity client. The data lived in MySQL, Redis and object storage, and everything ran on Hetzner servers. There were many reasons to change the stack. Unity was going to charge for every install, the architecture and design were outdated, and getting a new physical server meant waiting hours or days.

The new version is built on a different stack: a Flutter client, several dozen Go microservices, and Postgres, Valkey, Redpanda, Cassandra and OpenSearch. Most of it was placed in DigitalOcean, and the Russian part stayed in Yandex Cloud. About 20 people worked on the game, and six worked on the migration itself.

Three approaches to preparation

The first approach was load testing with Grafana k6. Comparing the new version with the old one did not work. One request from the old client and one from the new client are not comparable: the new client makes more calls to the backend, and each of those triggers a batch of internal calls between microservices. There was one useful lesson, though: hard CPU limits in Kubernetes throttle a container even when the server's processor is idle. The team removed the limits.

The second approach was changing the foundation on the fly. Three months before the migration, ScyllaDB discontinued its free version, so the team moved to Cassandra. Ingress-nginx was declared a finished project, so they chose Gateway API on Envoy. The single-threaded Redis was replaced with Valkey. Around the same time, they moved the first 10,000 players and decided to keep extra resources in reserve.

The third approach was Claude Code. The team connected the agent via MCP to repositories, metrics, logs and clusters, and gave it the main brief: the next wave would be 200,000 players. After that wave, it looked for bottlenecks for the future 4 million. Here is what it found:

  • a query against a 599-row table scanned the whole table each time and ran 310 million times, which kept Postgres's processor busy;
  • metrics in the form of histograms with many buckets threatened to choke Prometheus, whose memory was already approaching its 8 GB limit;
  • Redpanda brokers were 60–65% loaded versus 25–28% for the others, and in Cassandra the data share per node differed by a factor of two;
  • the rule of "no more than one pod of a service per server" would not let services scale to 20–30 copies: new pods would hang waiting for placement.

The agent also checked the people's hypotheses. "8,000 errors" turned out to be two IDs stuck in an endless retry loop. However, the team double-checked the conclusions about load balancers and certificates by hand, because the agent states both correct and incorrect conclusions with equal confidence.

The outage a day before and migration day

A week before this, the team had discussed an old debt: the entire Russian part sat in a single availability zone of Yandex Cloud. They decided to deal with it later. A week later, that zone failed, and resources could not be created in neighboring zones. The team began an emergency move to a less popular cloud. Claude Code helped describe the infrastructure as code within a few hours, and the cluster was brought up 14 hours before launch. Buckets and CDN stayed in Yandex and moved two days later. Most players did not notice the outage: the old version ran in two data centers.

Right before the migration, one more issue surfaced: some players with old clients were hitting the main domain, and their requests had to be routed to the old application. Such rules cannot be configured through the CDN, so the routing on Envoy was assembled in a rush together with Claude. The author admits that at one point he stopped keeping the traffic scheme in his head.

What happened next

According to the author, the downtime fit within the planned two hours. Then traffic on Envoy rose from 600–700 to 3,500 requests per second, and by evening to 5,100: the new client calls the API 6–7 times more often than the old one. Read latency in Cassandra went up from 1 to 4–5 ms, and expanding the cluster from 9 to 12 nodes helped. Prometheus did run out of memory anyway.

A few days later, an overloaded CoreDNS was discovered: because of the ndots: 5 setting, every name was first walked through the cluster's internal domains. After switching to ndots: 2, the number of DNS queries per day fell to 51 million from 114 million, and backend errors dropped to 16 from roughly 200.

The result: DAU the next day was 86,000 versus 73,000 on the old version, and infrastructure costs rose by 20%. Part of the growth came from marketing and new features, and the author calls revenue over three weeks comparable to before, so it is too early to draw final conclusions.

The practical lesson for teams: an AI agent is good at finding problems in data it has access to, but it does not replace a backup cloud, and it cannot see the risks that people have chosen to postpone.

Story

Key Questions Answered

What was moving and why?

The old version of the game was a PHP monolith with many cron jobs and a Unity client. The data lived in MySQL, Redis and object storage, and everything ran on Hetzner servers. There were many reasons to change the stack. Unity was going to charge for every install, the architecture and design were outdated, and getting a new physical server meant waiting hours or days.

What are the key details on Three approaches to preparation?

The first approach was load testing with Grafana k6. Comparing the new version with the old one did not work. One request from the old client and one from the new client are not comparable: the new client makes more calls to the backend, and each of those triggers a batch of internal calls between microservices. There was one useful lesson, though: hard CPU limits in Kubernetes throttle a container even when the server's processor is idle. The team removed the limits.

What are the key details on The outage a day before and migration day?

A week before this, the team had discussed an old debt: the entire Russian part sat in a single availability zone of Yandex Cloud. They decided to deal with it later. A week later, that zone failed, and resources could not be created in neighboring zones. The team began an emergency move to a less popular cloud. Claude Code helped describe the infrastructure as code within a few hours, and the cluster was brought up 14 hours before launch. Buckets and CDN stayed in Yandex and moved two days later.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…