(continued from the sprites scalability experiments [250k] [100k])
Well, finally found a work around for the 1.4GB limit imposed on the V8 heap. This limitation was at first a “hard” limit, and after some tweaks it became a “soft” limit. Now, the combined tweaks listed below have removed the limit entirely! YESSSSSS!!
The first two were already mentioned in my previous few blog articles, and the new ones (#3 and #4) finally open up the possibility of utilizing the real capacity of your server hardware.
1) ulimit -n 999999
This effectively increases the number of sockets Node.js can have open. You won’t get far without it, as the default tends to be around 1024.
2) –nouse-idle-notification
This is a command line parameter you can pass to node. It gets passed along to the V8 engine, and will prevent it from constantly running that darn garbage collector. IMO, you can’t do a real-time server with constant ~4 second latency hiccups every 30 seconds or so.
You might want to also use the flag “–expose-gc”, which will enable the “gc();” function in your server JavaScript code. You can then tie this to an admin mechanism, so you will retain the power to trigger garbage collection at any time you want, without having to restart the server. For the most part, if you don’t leak Objects all over the place, you won’t really need to do this often, or at all. Still, it’s useful to have the capability.
3) –max-old-space-size=8192
This you can tweak to fit your particular server, but the value is in MB. I chose 8GB because my expectation is that 4GB is going to be plenty, and 8GB was just for good measure. You may also consider using “–max-new-space-size=2048” (measured in KB, as opposed to the other). I don’t believe that one is nearly as critical, though.
4) Compile the latest V8 source code, with two modifications
The current node distribution isn’t using the new V8 engine which has significant improvements to garbage collection and performance in general. It’s worth upgrading, if you need more memory and better performance. Luckily, upgrading is really easy.
cd node-v0.6.14/deps/v8 rm -rf * svn export http://v8.googlecode.com/svn/tags/3.10.0.5 .
You can replace the version numbers as appropriate. There may be new versions of either node and/or V8 at the time you are reading this.
Next, you’ll need to add a single line addition to “SConstruct” inside the V8 directory.
'CPPPATH': [src_dir],
'CPPDEFINES': ['V8_MAX_SEMISPACE_SIZE=536870912'],
The second line above is new. Basically, you just need to set that V8_MAX_SEMISPACE definition. It will otherwise default to a much lower value, causing frequent garbage collection to trigger, depending on the memory characteristics of your server JS.
Next, comment out the call to “CollectAllGarbage” in “V8/heap-inl.h”:
if (amount_since_last_global_gc > external_allocation_limit_) {
//CollectAllGarbage(kNoGCFlags, "external memory allocation limit reached");
}
This was done “just in case” because it costs me money each time I run a scaling test. I wanted to be damn sure it was not going to trigger the external allocation limit GC!
That’s it! You will now want to rebuild both V8 and node, I would recommend doing a clean build for both, since replacing the entire V8 source may otherwise fail to take effect.
While writing this, I’ve got a server log open with 2281 MB of V8 heap usage. That’s far beyond the normal 1.4 GB limitation. The garbage collection is behaving, the server remains VERY responsive and low latency. In addition to the 250k concurrent and active connections, each of those nodes is also sending a sprite every 5 seconds.
There is still CPU to spare, the only limitation keeping me from trying 500k concurrent is my Amazon EC2 quota, which I already had to request an upgrade to do 250k.
OK – Now, go eat up all the memory you want on your Node servers 🙂
caustik's blog 
Nice post man!
I didn’t know about change V8 manually for the latest version from google svn.
LikeLike
[quote]
There is still CPU to spare, the only limitation keeping me from trying 500k concurrent is my Amazon EC2 quota, which I already had to request an upgrade to do 250k.
[/quote]
Was this some custom request? Can you elaborate a little more
LikeLike
Amazon has a page to request quota increase. I made the mistake of only requesting 100. Did not anticipate node could be quite as scalable as it is 😛
LikeLike
Really I’m clapping my hands !! You finally make it. Your the guy to follow 🙂
Thank’s
LikeLike
That change with V8_MAX_SEMISPACE_SIZE=536870912 is very strange. Yes, it will reduce the number of GCs, but the pauses will be much longer when they happen. The semispace GCs are supposed to be frequent, but very fast: Normally around 0-2ms, rarely more than 5ms.
LikeLike
Erik, but the function it’s calling is named “CollectAllGarbage” – it takes as much as 10 seconds!
I started a thread on the V8 dev groups which talks about why it seems my densely connected graph structure is so slow to garbage collect for. My preference has been to keep GC enabled, and somehow improve how it performs with that sort of data structure involved in your program, but the thread dead ended and I found no easy way to hack it up.
I think the ideal solution would be to provide a mechanism to exclude “static” Objects, of the JS devs choosing, from garbage collection. Consider how the “gc();” function exists now. It’s an optional switch that gives you a little more power of garbage collection. The only addition I want is a new function (lets call it “gcExclude”) that can be enabled, and lets you hand it Object handles which will then be moved out of the garbage collector’s path of traversal. You can then later add them back, or they automatically add back in emergency low memory situations or program exit.
Don’t you figure, with node.js’s popularity, there are quite a few devs who are desperate for a way to mitigate the quality of service problems associated with stop-the-world garbage collection? This would seem a sensible solution which wouldn’t compromise V8’s design, would only impact devs who need it, and solves the problem with a simple function call .
LikeLike
oh, of course you know of that V8 dev thread — you had responded to it 😛
LikeLike
CollectAllGarbage called from AdjustAmountOfExternalAllocatedMemory has nothing to do with semispace size. Semispace size controls size of young generation. Young generation is managed by a copying collector — so if you make it too large that would cause very-very long pauses if survival rate is high (imagine copying 100 megabytes back and worth every now and then).
BTW I think we figured out why you were seeing very frequent GCs when approaching 1.4gb limit: apparently you were overflowing int used as counter external allocated memory. Erik committed a change to fix that: r11265.
Theoretically AdjustAmountOfExternalAllocatedMemory should start an incremental GC cycle instead of causing full collection but we are still discussing how to ensure progress of incremental marking if most allocations are happening externally.
LikeLike
Caustic your answer seems to relate to the external memory related GC triggering code, but my comment was related to the semispace change. The semispace GCs are always stop-the-world so if you want to avoid pauses you should actually be making the semispaces smaller. A 1Mbyte limit helps keep semispace-related pauses in the 1ms range.
LikeLike
Erik – the reason for the change I made with V8_MAX_SEMISPACE_SIZE, was to indirectly affect the external allocated memory trigger. There is a bit of code which bases that threshold off the semispace size. I believe they’re meant to be in proportion to each other or something?
Sounds like that brings with it some collateral damage to performance, though it must not have been significant enough of a hit to prevent 250k concurrent. Maybe if I use a cleaner technique to prevent that external alloc triggered GC, performance would be good enough to handle 500k :]
LikeLike
Well, 250 concurrent req/sec sounds very good, what is configuration of your server?
LikeLike
This was on a rackspace cloud server — I think it was on the 8GB size.
LikeLike
Hello, thanks for you share.
I insepect the v8 source code, find that (Heap.cc)Heap class,
intptr_t max_old_generation_size_; –this is the v8 max_old_space_size
so,the max size is 2GB because the type is int, and testify by like this ” node –max_old_space_size=1919 **.js”
the max size is 1919 and larger then 1919 is failed.
so you set –max-old-space-size=8192 is it success ？？？
LikeLike
The code you mention isn’t there in the version I used for this article (3.10.0).
Also, the command line parameter for old space is measured in MB, so 8GB actually requires only 16 bits (0x2000, 8192). I haven’t checked the latest code, so this could be wrong if it is per-multiplied to bytes before being placed in that variable. Sorry that I don’t have the time to keep this blog up to date with the latest versions of V8.
LikeLike
May be when you have time, could you test the same thing with Node.js 0.8?
And possibly your 500K test 😀
LikeLike
There was a interesting bit about IE10 latest JS Engine, being the world first Non Stop the World Concurrent GC, If V8 ever have that in near future it could boost the Node.Js to another level.
Anyway i though of something you could try to beat 😀 , 2M concurrent connection.
http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/
LikeLike
ulimit -n 999999
I get the error bash: ulimit: open files: cannot modify limit: Operation not permitted, when setting a limit above 1024.
Can I get away with using soft limits instead? ulimit -s 9999?
LikeLike
Ash, You’ll need root access
LikeLike
On a micro instance of EC2 I never get past say 1500 concurrent connections of node.js. I tried spawning 2 instances and having HAProxy in front of them but still it doesnt work beyond 1500. Till 1000 it looks almost fine but socket time out errors start to come on client side after that.
Please suggest what can be the issue
LikeLike
On which line ‘CPPPATH’: [src_dir],
‘CPPDEFINES’: [‘V8_MAX_SEMISPACE_SIZE=536870912’], should be place in SConstruct
LikeLike
how to clean build V8 and node ? I am new to nodejs
LikeLike
For anyone interested, one of the best ways to avoid the Heap, is to use Buffers. Buffers exist outside of the v8 heap, and are faster to access. Please watch whenever you convert to and from a buffer, as you are in affect creating an alloc on the heap. For high throughput applications always use streaming conventions, and learn how to work with buffers.
LikeLike
I still need to do the step 4 on node js 0.13pre version?
I´m needing 5 millions cluster node concurrent connection
LikeLike
Does it require to rebuild V8 & node when using NodeJS v0.10.35? Will just passing the args while starting node help?
LikeLike