How CodinGame survived a Reddit Hug of Death

5          
Last Monday afternoon, a link to CodinGame was posted on reddit. It gained a lot of success and resulted in what is called a hug of death: our platform went down for 2 hours because of the overwhelming amount of new visitors.

Internet is beautiful

This was so far a normal day. We were working on the last preparations for the next multiplayer coding contest Hypersonic to be held this Saturday. Around 2pm, Nico, our CTO, shouted that we were suddenly receiving a lot of traffic.
A link to our starting page had been posted on the subreddit /r/InternetIsBeautiful under the title “Learn to code writing a game”. The traffic was already 10 times higher than usual and we were rejoicing.
After a few minutes watching the incoming traffic grow more and more, we began to notice some lags…

When will it stop? Can we support the load?

From excitement to panic

After some quick monitoring, we realized the CPU of the database server was already capped. We decided to multiply the capacity of our RDS (Amazon Relational Database Service) by 4 on AWS. It took 30 minutes at the end of which it interrupted the service completely. We had no choice.
After that change, the server was able to take in a lot more load, lags came back again pretty soon, when some other point of the architecture started to fail too. Now the front-end servers were failing.
The CodinGame platform normally uses two application servers hosted on EC2, behind a load balancer. So we tried scaling up by doubling that number (we eventually reached 6 servers in the end), but some of them started to crash. We tried relaunching them, to see them fail again after a short time.
The platform was becoming barely usable. What a waste, all this traffic to a dead/lagging site...

Reddit had put CodinGame to its knees.


Crisis management

While activity on our social media channels started to increase, the CodinGame chat was buzzing with questions. The community regulars were doing their best welcoming newcomers.
The chat server was also under heavy load and wouldn’t accept new users (more on this later). Forum didn’t last long either. We took the opportunity to offer an AMA (ask me anything) in the comments of the reddit thread.
At the same time, a Twitch streamer was desperately waiting to do a Clash of Code session. Things were looking bad and everyone was preoccupied. Reddit finally tagged the post with “Hug of Death” so visitors didn’t end on a dead site.

The end of the tunnel

After two long hours, we managed to get the servers back and running, and we asked the reddit moderators to bring back the thread. Traffic came back as a wave as the thread was reposted on Hacker News and other tech news sites. And lags again. Something was taking down our servers and we couldn’t find what. Finally we put in place a script that would reboot a server each time it failed so we maintain the service over the night.

On Tuesday afternoon, when things had calmed down, we took the time to come back on what had happened. Traffic had been crazy: we got as many new users in one day as during the last two months. There had been technical failures. Understandable failures but to be taken care of.

Post-Mortem - What went wrong

1) The main bottleneck of the CodinGame platform is the RDS database. Every information on CodinGame is centralized there, all tangled. We cannot distribute the database without setting up separate servers splitting users, for instance by region (America, Europe, Asia). This is the major scaling issue.

Solution: the short term action we’ll take is to investigate caching solutions, distributed caches between our application servers.

2) The second issue we’re currently following up is a memory leak on the application servers. This bug occurs when a server is under heavy load and eventually leads to a crash.

Solution: track the bug, kill the bug.

3) The CodinGame forum is hosted on a really small machine, and is poorly integrated on the platform through an iframe so that every user on any page of the site sends an SSO authentication request to the forum. The number of queries that were sent to this instance capped the CPU, and just like that, the forum was no more.

Solution: Better integrate SSO between the website and the forum to avoid unnecessary requests. Migrate the forum on a tougher machine.

4) This very blog crashed, simply because it was… hosted on the same machine as the forum, that maxed out the CPU.

Solution: Separate applications in different containers/machines.

5) The CodinGame chat server did exactly what it was expected to do: fail under heavy load. It uses XMPP over WebSocket, with a custom client built on top of stanza.io, and a back-end built with Prosody. Prosody is a really cool back-end built with lua, but it has the slightest problem: it is single-threaded and non-distributable. So, as expected, the server process quickly reached 100% of a single CPU core and started to lag badly and act erratically. It didn’t crash, but wouldn’t take in new users.

Solution: Switch our back-end solution to a scalable one, such as eJabberd or MongooseIM.

6) Last point of failure in this real world scalability test: the push server. We use a websocket connection to push data to the client, triggering diverse events (notifications, mostly). This server had a limitation of open file descriptors, arbitrarily set to 10K. We set it to 100K and it started working again, accepting up to 60K simultaneous connections (this is the best estimation we have of the number of open tabs at that moment) before it went night-night.

Solution: This is an Embarrassingly Parallel Problem, and it should have been a scalable pool of servers behind a load balancer from the beginning.


The Reddit effect
We took a blow, but we're back on our feet, stronger than ever. We'll be happy to answer any question you might have about this special event. Don't hesitate!
Welcome to all the new users!

Dev Diary: how the Hypersonic contest was built

3          
There are 16 days left before the beginning of the next contest Hypersonic. We will celebrate the second anniversary* of the platform and we’re really excited to share it with you. We hope you're ready!
Here’s Hypersonic diary, from its origins:

Day minus 117, Conception

Romain, Art Director: “It’s a girl, we need to find her a name”


Romain working on the main illustration
This is the time to work on the main illustration. A vote is done in the development team to help Arthur and Aude find the prizes. "What would you like to win?"

Another vote is conducted to find the name of the contest. Not an easy task! 

Contest name: results of the vote
All of this should be done really early. Indeed at the end of a contest, the registration page of the next one should be updated. So everything needs to be ready at the end of Codebusters.

Day minus 85, Hacking

Alexis, Corporate Relations Hacker: “Hey, the contest has to be crazy good, because another company is ready to work with us”

Some developers register for fun and glory. Others want to get noticed by companies. Our goal is thus to showcase lots of interesting job opportunities to them.

Obviously this isn’t a one-day job. As of today there are 19 sponsors for Hypersonic.

Day minus 54, Brainstorm

The whole dev team:
“_ We could do a game like StarCraft?
_ Too complex.
_ Something chess-like ?
_ Too classic.
_ A game with bluff like poker?
_ Too random
...”

At some point, the type of game should be decided. Either we stay very close to an existing game (Puyo-Puyo for Smash the Code) or we build our own game (like for Codebusters). Indeed we try as much as possible to provide you with something reminiscent of the illustration, but it’s not always easy to do. 

The dev team also begins to discuss the possible rules for each league.

Background of the game

Day minus 50, Building

Julien, Game Builder: “The status for Hypersonic, well, it’s ongoing.”

Each game is mainly built by one developer only. Of course, all the team can help for anything: testing, code review…

It usually takes 2 full weeks until a prototype can be tested. At the same time, the design team (Romain and Malia) starts to work on the assets of the game.

Asset of the game

Day minus 36, Testing

Jerome, Product Manager: “Yeees, I beat the boss”

Follows a series of iterations between the game builder and the testers. Usually all developers try the game and give some feedback. Rules are tweaked, the statement is improved, art assets are added to the game...

One important concern, as you might guess, is the difficulty of the game.

Day minus 24, Promotion

Thibaud, community manager: “Already 5 CodingHubs!”

The more, the merrier. A contest is a programming competition, but it’s also a fun event where a lot of developers discuss and share their passion for coding. For example, in a CodingHub.

This is the time we start to communicate more about the contest, like with this blog post (blogception!).

Hour minus 1, Excitation

It will seem like a normal day at work although it'll be Saturday afternoon. Most of the team will gather to ensure a smooth launch of the contest.


We look forward to seeing you on Saturday 24th of September at 12:00pm EST for the opening of Hypersonic!


*CodinGame isn’t 2 years old, the platform was just really different before.

CodingHubs: Meet, Code, Enjoy!

4          
In less than 4 weeks now, several thousands of programmers will doggedly try to code the best bot they can in the Hypersonic contest. Competing in such a challenge is exhilarating, and the only thing you want to do (apart from coding) is discuss with others about the problem at hand. We're giving you the opportunity to do it, we're indeed launching the CodingHubs!

CodingWhat?

CodinGame does not only happen online! A CodingHub is a place to meet with other CodinGamers, code together around a game and enjoy the fun of coding.
Meet, code, enjoy!

CodingHubs are organized by voluntary CodinGamers usually for an evening at a private place, may it be at their university, at their company or at home. (provided they get the authorisation to do it).

And to add more fun to the event, we'll be sending CodinGame goodies to each Hub!

Great, where do I sign?

You can find all information (date, address, contact email...) about CodinHubs on the map below. [if the map doesn't show for you, trying clearing your cookies] They are indicated by a yellow wifi icon:



If you're interested in joining an existing CodingHub, you have to contact the organizer to register to it. Obviously the number of available seats is limited.

I can't find any Hub near me :(

No worries, there might not yet be a Hub close to you, but there may be other enthusiastic CodinGamers not too far. They are indicated by the green laptop icon on the map:

Fill this form to signal fellow CodinGamers you'd be ready to join a Hub.

I should update the map within 24h. One day it'll be dynamic, I promise. 

Let's be crazy, I'm up to the task

Interested in hosting your own hub? Congrats!

You can fill this slightly longer form and we'll get in touch with you really soon to help you organize it.

Same, I'll update the map as soon as I can, but at night, I sleep. 

Is there something else I should know?

Once you’ve registered, you can talk about it with your friends and colleagues. The more, the merrier!
And don't forget your laptop! ;)

If you've got any question, don't hesitate to comment below. Else I'm always reachable at thibaud@codingame.com

Hub for Codebusters in Lyon
Hub for Codebusters in Montpellier
Hub for Codebusters in Paris

How to build your own programming puzzle

1          
After having solved many programming puzzles, have you ever thought about creating your own puzzle? You’re not sure where to start? Or do you think you’re not up to the task? Here are 5 easy steps that should help you create your own coding puzzle.

1/ Pick a theme

You create a puzzle for other developers to solve it. They solve puzzles for several reasons, one of them being entertainment. Your first job is to entertain the solver. The theme won’t make it all but without a theme you surely won’t have a great puzzle.
Preferably choose a theme you like. It will be easier to create a top-notch puzzle.
Now that you have a theme, you can find an idea of puzzle more easily, or adapt the idea you already had to the theme.
You should also think about the originality of your puzzle/theme. Try to avoid classic topics of programming puzzles (like Ascii Art).

Ex: I’m fond of tennis, the goal of my puzzle will be to compute a tennis score from the chronological list of points.

2/ Write tests

You’ve got the idea of the puzzle: there should be a goal and some rules to achieve it. Make sure to define clear rules.
How will you check that a code has solved your problem? If it passes the tests you’ve created for it.
Simply write all the conditions a solution should respect to be valid and make a test for each one of them*.
If it’s a scaling problem, you’ll have to write tests of simple scenarios and at least one test for a larger scenario, so a brute force solution would fail this test (ie timeout).

Ex: My tests should verify that a solution correctly computes the score of a game and set, handles deuce and tie break...

3/ Write the solution of your puzzle

This part helps you assess the difficulty of your problem. A too hard puzzle might deter some developers from solving it. On the other hand a too easy puzzle won’t be challenging for advanced developers. You can tweak the rules (and obviously modify the corresponding tests and validators) to modify the difficulty.
Even if you know the solution to your puzzle**, it might be a good idea to check on the internet for similar problems. Maybe you’ve missed a trivial solution or forgotten a specific test case.

Ex: I found a StackExchange topic about my problem which makes me realize my puzzle may be too simple. To make it more tricky, I could decide to add incomplete inputs…

4/ Present your puzzle

This could be a boring part but this is highly important. The problem/puzzle might be familiar to you, but developers will just discover it.
A complete statement usually contains: a small story, the description of the problem (goal and rules), the description of the inputs and expected output along with an example. All of it must be very clear and simple to understand, else developers might not even try to solve it. The story shouldn’t be too long. It is used to create a fun atmosphere around the programming puzzle.
Title is also important, make it sexy! However it should not misleading.

Ex: Instead of "tennis scoring system", I could call the puzzle "Game, Set & Match". To be sure the description is clear, I would choose wordings from the related wikipedia article.

5/ Ask friends for review

You could think your job is finished at this step. It is not. You need to have someone review your puzzle. Don’t explain anything about your puzzle, just give to your reviewers the puzzle and ask them to solve it. There will be remarks for sure. Or proposals for better wordings. Maybe you made a typo. Or even a mistake in the test cases.
In any case, the quality of your contribution can just increase after this process.

The tennis puzzle is obviously incomplete but you can still give your opinion on it. If the global feedback is positive, I might even take the time to really create the puzzle :)


If you’ve followed these 5 steps thoroughly, you’re ready to share your puzzle with the world and become famous. This is what happened to SamSi whose puzzle “Heart of the City” got featured as Puzzle of the Week on CodinGame. 


Here is his story:

Careful, if you have not solved his puzzle yet, there are spoilers below

“My college is in a military area, so we have sort of pine trees all over. My friend and I were just roaming around. I was wondering if there existed any sort of pattern as to how many trees are visible if we are surrounded by them and they have equal spacing.

At first I thought that there was a pattern to it, I derived an equation from it and checked it against my bruteforce solution. It failed.

Later on, I realized that this problem is nothing but counting the number of co-prime pairs so I tried to find some related algorithms and I discovered Euler totient theorem.

So I decided to create a puzzle. I did several mistakes but my friend and the CodinGame community (MadKnight, Kyran…) were very helpful and told me what could be improved.

My piece of advice for creating a nice puzzle would be the following: 

  • it should be easy to understand
  • it should be challenging
  • it’s better if it’s related to nature or things people are familiar with”
Thank you SamSi and congratulations for your puzzle!

* On CodinGame, the tests will be visible, so you’ll have to write another set of tests (validators) to prevent hard-coded solutions. These validators need to validate a code the same way tests do.
** On CodinGame, your own solution is a proof that your puzzle is valid, it is now mandatory.

Artificial Intelligence: dive right in and give it a shot

3          
You know that fun is at the heart of the platform. As much as we want players to improve their programming skills, we want them to have fun doing so. We've discussed with Mikkanu from Illinois in the US and he agreed to share his experience on AI programming for fun.


CG: Hi Mikkanu, first tell us a bit about yourself :)


Mikkanu: I've been programming one way or another for more than 20 years. I've started on a Commodore with a BASIC interpreter. Now I work with big data for Amazon AWS. A few years ago I ran a start-up company with a focus on helping less software-oriented companies. 


Programming is for me both a job and a hobby. I enjoy spending my free time on platforms like CodinGame, HackerRank, or answering questions on StackOverflow because it's fun and it helps me flex muscles I don't get to use in my day to day work.



CG: You've tried AI games on CodinGame, how was your experience?

Mikkanu: I've had a great experience on CodinGame - right from the beginning. I think the first game I played was the drone multi-player game and then Tron. And after that it was likely the Terminator single player game. It was not my first time trying AI programming. Before starting my professional career I spent a few years working on games. 
But that was many years ago and since then I recall having tried some platforms such as .NET Terrarium (around 2004) and Fighting Code but those other platforms didn't captivate my attention.

CG: To what extend did playing AI games on CodinGame help you improve your bot programming skills?

Mikkanu: Playing games on CG is not just fun. It's helped me keep my skills sharp working with graphs and trees. Building and traversing graphs is a pretty much required skill to solve most of the CG games. Path finding algorithms like BFS, DFS, A* search or Dijkstra are also very useful. 
I can't say that I've learned new algorithms from other players on CG but I've definitely sharpened my skills and yes, I've benefited from studying strategies of other players on multi-player games. 

As a professional with a family I wish I had more time to dedicate to multi-player games, that's why I particularly enjoyed solo contests.

CG: What kind of tips would you give to beginners in AI who hesitate to try? 

Mikkanu: I've actually encouraged a few of my beginner friends to get in on CodinGame. My advise is to dive right in and give it a shot. I think the practice area is a pretty good way to get started but I also feel that it was better before when one could study other players' code if one got stuck. 

I would recommend submitting code often. That is one mistake that people tend to make, especially with multi-player contests in my opinion. I think the sooner you get your code submitted, the sooner you can find out how you're doing and then try to improve your solution.

CodinGame has had a lot of appeal to me because of the solo contests and because of the built-in online platform to write/debug and submit the code. I think it has a lot of potential.



CG: Thank you for the kind words and for your time Mikkanu!

Community Spotlight #3

0          

CodinGame would not be what it is without its community. The platform is yours, and your passion for it makes it alive and helpful. A lot of CodinGamers are presenting great content around CodinGame and we thought it would be very interesting to put them under the spotlight. 
You can find the previous episode here.


Get on TV with Matheu Plouffe and AutomateAllTheThings

If you're lacking determination to finish a puzzle which is giving you a hard time, I invite you to check Matheu Plouffe stream on Twitch: the3rdHunter. The amount of determination he shows to land on Mars using C# is incredible. Definitely inspirational.

"I am a huge fan of the platform you've developed! I'm currently in the middle of doing an IT degree that has a very strong emphasis on job readiness, and doesn't do nearly enough with algorithms and things like that, so CodinGame gives me a chance to work on those skills in a super fun way"

Thank you Matheu Plouffe!

Link to the Twitter of Matheu Plouffe
Link to the Twitch tv of The3rdHunter
Link to the replay of his last stream


AutomateAllTheThings has been streaming his Clash of Code sessions on Twitch for about a month. He has been playing private clashes with his audience for hours. And even sometimes, here at CodinGame we wanted to be part of it so we joined some clashes... It has been a blast! Really fun!

[CG]VonRickroll reaction on one of our Slack channels (after submitting his solution indeed)
"Totally in love with codingame, #gamification of #coding is wonderful"

Thank you AutomateAllTheThings!

Link to the Twitter of AutomateAllTheThings
Link to the Twitch tv of AutomateAllTheThings
Link to the last session of Clash of Code

Optimize your rank with Royale's and Magus' tools

Ever wanted to know on which game you should focus to quickly improve your multiplayer ranking? This is now possible thanks to Royale who presented his tool on the forum last month. Yes you have most probably heard the name before: he is the one with Magus competing for the top place in the multiplayer ranking.
With your profile handle (the code that you find in the url on your profile page after "https://www.codingame.com/profile/"), you can discover the amount of coding points you have for each multiplayer and optimization games and how much you can get.

Thank you Royale!

Link to the forum topic
Link to the tool results with my profile as an example (yes there is room for improvement :p )


Once you're in an arena game, you probably spend a lot of times on the tab "Last Battles" to analyse how your AI behaves. Magus has made your life easier by creating a tool to get statistics on your fights. He shared it last month on the forum.
You just have to input your username, choose the game on which you want to have some statistics. It will give you the win/defeat ratio of your AI against specific players. It can help you to identify bugs or specific flaws in your strategy.

Thank you Magus!

Link to the forum topic
Link to the tool results with my bot in Codebusters as an example (yes I'm not Gold... yet ;) )
Link to the GitHub project

Watch ElanVB Codebusters time-lapse

There are almost no words to describe what ElanVB did, it's just nice. 
So he managed to get in gold league of Codebusters contest, and then decided to start again and do a time-lapse of the process. At the end he reached the 290th place!
It's really interesting to see him comment properly his code, use a separate document to type out his ideas and remarks and finally see him use the CodinGame beta sync app. Enjoy!

"I just wanted to start off with thanks for the great site and competitions. I have gotten a lot of joy out of this community. That said I would like to try give back a bit. I don’t have much to offer but I did take a time lapse of how I programmed and tested my code for the CodeBusters competition."

Thank you ElanVB!

Link to the CodinGame beta sync app

Anyone can be a hero

You too are doing something special around CodinGame and would like to share it with the community? Don't hesitate to share your story with us at coders@codingame.com!