I’ve decided to ramp up the Node.js experiments, and pass the 1 million concurrent connections milestone. It worked, using a swarm of 500 Amazon EC2 test clients, each establishing ~2000 active long-poll COMET connections to a single 15GB rackspace cloud server.
This isn’t landing the mars rover, or curing cancer. It’s just a pretty cool milestone IMO, which may help a few people who want to use Node.js for a large number of concurrent connections. So, hopefully it’s of some benefit to a few Node developers who can use these settings as a starting point in their own projects.
Here’s the connection count as displayed on the sprite’s page:
Here’s a sysctl dumping the number of open file handles (sockets are file handles):
Here’s the view of “top” showing system resources in use:
I think it’s pretty reasonable for 1M connections to consume 16GB of memory, but it could probably be trimmed down quite a bit. I haven’t spent any time optimizing that. I’ll leave that for another day.
Here’s a latency test run against the comet URL:
The new tweaks, placed in /etc/sysctl.conf (CentOS) and then reloaded with “sysctl -p” :
net.core.rmem_max = 33554432 net.core.wmem_max = 33554432 net.ipv4.tcp_rmem = 4096 16384 33554432 net.ipv4.tcp_wmem = 4096 16384 33554432 net.ipv4.tcp_mem = 786432 1048576 26777216 net.ipv4.tcp_max_tw_buckets = 360000 net.core.netdev_max_backlog = 2500 vm.min_free_kbytes = 65536 vm.swappiness = 0 net.ipv4.ip_local_port_range = 1024 65535
Other than that, the steps were identical to the steps described in my previous blog posts, except this time using Node.js version 0.8.3.
Here is the server source code, so you can get a sense of the complexity level. Each connected client is actively sending messages, for the purpose of verifying the connections are alive. I haven’t pushed that throughput yet, to see what data rate can be sent. Since the modest 16GB memory was consumed, this would have likely caused swapping and meant little. I’ll give it a shot with a higher memory server next time.
71 thoughts on “Node.js w/1M concurrent connections!”
1 question I really really want to ask is:
How many CPU are you running your 1M connections on?
Since socket.io does heartbeat, looping 1M times for heartbeat is very intensive it seems to me.
my disconnect problem kinda solved..i was firing more than 50 connections each second…not really sure how long it took you to launch 1M connections if you fire up 2 connections each seconds….
How many concurrent connections can you have with a static html serving server and no tweaks?
Can you example a web site who is using node.js ?