Escape the 1.4GB V8 heap limit in Node.js!

(continued from the sprites scalability experiments [250k] [100k])

Well, finally found a work around for the 1.4GB limit imposed on the V8 heap. This limitation was at first a “hard” limit, and after some tweaks it became a “soft” limit. Now, the combined tweaks listed below have removed the limit entirely! YESSSSSS!!

The first two were already mentioned in my previous few blog articles, and the new ones (#3 and #4) finally open up the possibility of utilizing the real capacity of your server hardware.

1) ulimit -n 999999

This effectively increases the number of sockets Node.js can have open. You won’t get far without it, as the default tends to be around 1024.

2) –nouse-idle-notification

This is a command line parameter you can pass to node. It gets passed along to the V8 engine, and will prevent it from constantly running that darn garbage collector. IMO, you can’t do a real-time server with constant ~4 second latency hiccups every 30 seconds or so.

You might want to also use the flag “–expose-gc”, which will enable the “gc();” function in your server JavaScript code. You can then tie this to an admin mechanism, so you will retain the power to trigger garbage collection at any time you want, without having to restart the server. For the most part, if you don’t leak Objects all over the place, you won’t really need to do this often, or at all. Still, it’s useful to have the capability.

3) –max-old-space-size=8192

This you can tweak to fit your particular server, but the value is in MB. I chose 8GB because my expectation is that 4GB is going to be plenty, and 8GB was just for good measure. You may also consider using “–max-new-space-size=2048” (measured in KB, as opposed to the other). I don’t believe that one is nearly as critical, though.

4) Compile the latest V8 source code, with two modifications

The current node distribution isn’t using the new V8 engine which has significant improvements to garbage collection and performance in general. It’s worth upgrading, if you need more memory and better performance. Luckily, upgrading is really easy.

cd node-v0.6.14/deps/v8
rm -rf *
svn export .

You can replace the version numbers as appropriate. There may be new versions of either node and/or V8 at the time you are reading this.

Next, you’ll need to add a single line addition to “SConstruct” inside the V8 directory.

    'CPPPATH': [src_dir],
    'CPPDEFINES': ['V8_MAX_SEMISPACE_SIZE=536870912'],

The second line above is new. Basically, you just need to set that V8_MAX_SEMISPACE definition. It will otherwise default to a much lower value, causing frequent garbage collection to trigger, depending on the memory characteristics of your server JS.

Next, comment out the call to “CollectAllGarbage” in “V8/heap-inl.h”:

    if (amount_since_last_global_gc > external_allocation_limit_) {
      //CollectAllGarbage(kNoGCFlags, "external memory allocation limit reached");

This was done “just in case” because it costs me money each time I run a scaling test. I wanted to be damn sure it was not going to trigger the external allocation limit GC!

That’s it! You will now want to rebuild both V8 and node, I would recommend doing a clean build for both, since replacing the entire V8 source may otherwise fail to take effect.

While writing this, I’ve got a server log open with 2281 MB of V8 heap usage. That’s far beyond the normal 1.4 GB limitation. The garbage collection is behaving, the server remains VERY responsive and low latency. In addition to the 250k concurrent and active connections, each of those nodes is also sending a sprite every 5 seconds.

There is still CPU to spare, the only limitation keeping me from trying 500k concurrent is my Amazon EC2 quota, which I already had to request an upgrade to do 250k.

OK – Now, go eat up all the memory you want on your Node servers 🙂

Node.js w/250k concurrent connections!

Okay, I figured it must be possible, so it had to be done..

The sprites server has smashed through the 250k barrier with 250,001 concurrent, active connections. In addition to the stuff talked about in my previous two blog entries, it took a few more optimizations.

1) Latest tagged revision of V8, which appears to perform a little better

The 250k limit is right on the fringe of what this server can pull off without violating the 1.4GB heap limitation in V8. It’s not clear to me why this limitation hasn’t bubbled up in priority enough to be taken care of yet. Just look at those free CPU cycles and unused memory! V8 is a complete bad-ass, and it’s just this one limitation that is holding it back from some really extreme capabilities.

I had tried 250k a few times, and couldn’t get it stable until after upgrading the /deps/v8/ directory of Node.js with the latest version tagged in SVN. I was really hoping the 1.4GB limit had been removed, but alas. Instead, just had to settle with some significant improvements to garbage collection and performance in general.

2) Workers via “cluster” module

I figured the onslaught of HTTP GET requests the 100 EC2 servers were unleashing at a rate of 100,000 JSON packets per second was contributing both to CPU and memory consumption, which was agitating the garbage collector enough to resist the 250k mark.

So, to reduce the overhead of these transient requests, which are in addition to the 250k concurrent connections (which really leaves us on average at about 350k connections at any given second, 100k of which are transients), I decided to leverage the cluster module.

The master performs the exact same tasks it did before. Except, now there are workers spawned, one for each CPU on the system, and they listen on a separate port from the master process. This port is used by the clients for these transient requests, and as they arrive they are parsed and forwarded to the master using the Node.js send() function.

The only reason this was a critical adjustment is, again, the 1.4GB heap limitation in V8. Keeping all the resources associated with those requests off the master process saves just enough memory to get past the 250k milestone.

TL;DR – V8, you’re so good, but it’s time for you to support more memory!