Node.js w/1M concurrent connections!

I’ve decided to ramp up the Node.js experiments, and pass the 1 million concurrent connections milestone. It worked, using a swarm of 500 Amazon EC2 test clients, each establishing ~2000 active long-poll COMET connections to a single 15GB rackspace cloud server.

This isn’t landing the mars rover, or curing cancer. It’s just a pretty cool milestone IMO, which may help a few people who want to use Node.js for a large number of concurrent connections. So, hopefully it’s of some benefit to a few Node developers who can use these settings as a starting point in their own projects.

Here’s the connection count as displayed on the sprite’s page:

Here’s a sysctl dumping the number of open file handles (sockets are file handles):

Here’s the view of “top” showing system resources in use:

I think it’s pretty reasonable for 1M connections to consume 16GB of memory, but it could probably be trimmed down quite a bit. I haven’t spent any time optimizing that. I’ll leave that for another day.

Here’s a latency test run against the comet URL:

The new tweaks, placed in /etc/sysctl.conf (CentOS) and then reloaded with “sysctl -p” :

net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 16384 33554432
net.ipv4.tcp_wmem = 4096 16384 33554432
net.ipv4.tcp_mem = 786432 1048576 26777216
net.ipv4.tcp_max_tw_buckets = 360000
net.core.netdev_max_backlog = 2500
vm.min_free_kbytes = 65536
vm.swappiness = 0
net.ipv4.ip_local_port_range = 1024 65535

Other than that, the steps were identical to the steps described in my previous blog posts, except this time using Node.js version 0.8.3.

Here is the server source code, so you can get a sense of the complexity level. Each connected client is actively sending messages, for the purpose of verifying the connections are alive. I haven’t pushed that throughput yet, to see what data rate can be sent. Since the modest 16GB memory was consumed, this would have likely caused swapping and meant little. I’ll give it a shot with a higher memory server next time.

Escape the 1.4GB V8 heap limit in Node.jsSprites Project

autoblitz + source code

Since some folks have expressed interest in trying this bot for themselves, I decided to share the source code and binary 🙂

It’s not perfect, but you can tweak the source code (for example, changing how long rp_thread::sleep() is called, and where), and probably get some better performance. If you’re up for a challenge, you can also try to improve the code that decides on the next move! (if you do, it’d be great if you shared the changes!).

Anyway, here is the zip file. [ ]

The binaries are in the export directory. You will be asked for a “hWnd” (window handle), which can be found using the program “Spyxx” which comes with visual studio.

Game playing AI bot

Every once in a while a silly idea grips you and you decide “Screw it, I’m going to do it”.

That happened with me with a game called Bejeweled Blitz on Facebook. My friends online have been competing for high score for the past few weeks. I got to thinking…”How hard would it be to write a program to automate playing this game..?”.

This is the result of that question. It took a few days at a few hours per day. Probably about 6 hours combined time. Right now the bot is able to play pretty well. It can do better than I can most the time (although my high score currently beats the bot’s high score). With some improvements, I expect that will dramatically change.

Anyway, here’s a video of the program in action.


The autoblitz bot just scored 258,400.

Cxbx Progress

There was some amount of activity in a private branch in the Cxbx project recently..

I have been talking to a developer, Martin, who has spent some of his extra time fiddling with Cxbx. He was able to get some teaser screenshots from Battlestar Galactica. The game displays the menu, and even some in-game. I won’t get into too many details, but here are a couple screenshots.

Battlestar Galactica - menu 2 Battlestar Galactica - menu 3

Battlestar Galactica - in-game (fullscreen) Battlestar Galactica - in-game (windowed)

I have not started to work on Cxbx on a regular basis — but I think this progress is very motivating, and it is tempting me to boot back up my development setup and take another look after a very long absence.

Many thanks to Martin for the contributions and for allowing me to post about his progress. Cheers.