caustik's blog

programming and music

Escape the 1.4GB V8 heap limit in Node.js!

with 23 comments

(continued from the sprites scalability experiments [250k] [100k])

Well, finally found a work around for the 1.4GB limit imposed on the V8 heap. This limitation was at first a “hard” limit, and after some tweaks it became a “soft” limit. Now, the combined tweaks listed below have removed the limit entirely! YESSSSSS!!

The first two were already mentioned in my previous few blog articles, and the new ones (#3 and #4) finally open up the possibility of utilizing the real capacity of your server hardware.

1) ulimit -n 999999

This effectively increases the number of sockets Node.js can have open. You won’t get far without it, as the default tends to be around 1024.

2) –nouse-idle-notification

This is a command line parameter you can pass to node. It gets passed along to the V8 engine, and will prevent it from constantly running that darn garbage collector. IMO, you can’t do a real-time server with constant ~4 second latency hiccups every 30 seconds or so.

You might want to also use the flag “–expose-gc”, which will enable the “gc();” function in your server JavaScript code. You can then tie this to an admin mechanism, so you will retain the power to trigger garbage collection at any time you want, without having to restart the server. For the most part, if you don’t leak Objects all over the place, you won’t really need to do this often, or at all. Still, it’s useful to have the capability.

3) –max-old-space-size=8192

This you can tweak to fit your particular server, but the value is in MB. I chose 8GB because my expectation is that 4GB is going to be plenty, and 8GB was just for good measure. You may also consider using “–max-new-space-size=2048″ (measured in KB, as opposed to the other). I don’t believe that one is nearly as critical, though.

4) Compile the latest V8 source code, with two modifications

The current node distribution isn’t using the new V8 engine which has significant improvements to garbage collection and performance in general. It’s worth upgrading, if you need more memory and better performance. Luckily, upgrading is really easy.

cd node-v0.6.14/deps/v8
rm -rf *
svn export http://v8.googlecode.com/svn/tags/3.10.0.5 .

You can replace the version numbers as appropriate. There may be new versions of either node and/or V8 at the time you are reading this.

Next, you’ll need to add a single line addition to “SConstruct” inside the V8 directory.

    'CPPPATH': [src_dir],
    'CPPDEFINES': ['V8_MAX_SEMISPACE_SIZE=536870912'],

The second line above is new. Basically, you just need to set that V8_MAX_SEMISPACE definition. It will otherwise default to a much lower value, causing frequent garbage collection to trigger, depending on the memory characteristics of your server JS.

Next, comment out the call to “CollectAllGarbage” in “V8/heap-inl.h”:

    if (amount_since_last_global_gc > external_allocation_limit_) {
      //CollectAllGarbage(kNoGCFlags, "external memory allocation limit reached");
    }

This was done “just in case” because it costs me money each time I run a scaling test. I wanted to be damn sure it was not going to trigger the external allocation limit GC!

That’s it! You will now want to rebuild both V8 and node, I would recommend doing a clean build for both, since replacing the entire V8 source may otherwise fail to take effect.

While writing this, I’ve got a server log open with 2281 MB of V8 heap usage. That’s far beyond the normal 1.4 GB limitation. The garbage collection is behaving, the server remains VERY responsive and low latency. In addition to the 250k concurrent and active connections, each of those nodes is also sending a sprite every 5 seconds.

There is still CPU to spare, the only limitation keeping me from trying 500k concurrent is my Amazon EC2 quota, which I already had to request an upgrade to do 250k.

OK – Now, go eat up all the memory you want on your Node servers :)

Written by caustik

April 11th, 2012 at 2:43 am

23 Responses to 'Escape the 1.4GB V8 heap limit in Node.js!'

Subscribe to comments with RSS or TrackBack to 'Escape the 1.4GB V8 heap limit in Node.js!'.

  1. Nice post man!

    I didn’t know about change V8 manually for the latest version from google svn.

  2. [quote]
    There is still CPU to spare, the only limitation keeping me from trying 500k concurrent is my Amazon EC2 quota, which I already had to request an upgrade to do 250k.
    [/quote]

    Was this some custom request? Can you elaborate a little more

    Matt Freeman

    11 Apr 12 at 5:14 am

  3. Amazon has a page to request quota increase. I made the mistake of only requesting 100. Did not anticipate node could be quite as scalable as it is :P

    caustik

    11 Apr 12 at 5:31 am

  4. Really I’m clapping my hands !! You finally make it. Your the guy to follow :)

    Thank’s

    Nodejs-news.com

    11 Apr 12 at 8:01 am

  5. That change with V8_MAX_SEMISPACE_SIZE=536870912 is very strange. Yes, it will reduce the number of GCs, but the pauses will be much longer when they happen. The semispace GCs are supposed to be frequent, but very fast: Normally around 0-2ms, rarely more than 5ms.

    Erik Corry

    11 Apr 12 at 8:58 am

  6. Erik, but the function it’s calling is named “CollectAllGarbage” – it takes as much as 10 seconds!

    I started a thread on the V8 dev groups which talks about why it seems my densely connected graph structure is so slow to garbage collect for. My preference has been to keep GC enabled, and somehow improve how it performs with that sort of data structure involved in your program, but the thread dead ended and I found no easy way to hack it up.

    I think the ideal solution would be to provide a mechanism to exclude “static” Objects, of the JS devs choosing, from garbage collection. Consider how the “gc();” function exists now. It’s an optional switch that gives you a little more power of garbage collection. The only addition I want is a new function (lets call it “gcExclude”) that can be enabled, and lets you hand it Object handles which will then be moved out of the garbage collector’s path of traversal. You can then later add them back, or they automatically add back in emergency low memory situations or program exit.

    Don’t you figure, with node.js’s popularity, there are quite a few devs who are desperate for a way to mitigate the quality of service problems associated with stop-the-world garbage collection? This would seem a sensible solution which wouldn’t compromise V8′s design, would only impact devs who need it, and solves the problem with a simple function call .

    caustik

    11 Apr 12 at 9:10 am

  7. oh, of course you know of that V8 dev thread — you had responded to it :P

    caustik

    11 Apr 12 at 9:16 am

  8. CollectAllGarbage called from AdjustAmountOfExternalAllocatedMemory has nothing to do with semispace size. Semispace size controls size of young generation. Young generation is managed by a copying collector — so if you make it too large that would cause very-very long pauses if survival rate is high (imagine copying 100 megabytes back and worth every now and then).

    BTW I think we figured out why you were seeing very frequent GCs when approaching 1.4gb limit: apparently you were overflowing int used as counter external allocated memory. Erik committed a change to fix that: r11265.

    Theoretically AdjustAmountOfExternalAllocatedMemory should start an incremental GC cycle instead of causing full collection but we are still discussing how to ensure progress of incremental marking if most allocations are happening externally.

  9. [...] Read full article Tagged as: JavaScript, Node.JS, NodeJS Comments Off Comments (0) Trackbacks (0) ( subscribe to comments on this post ) [...]

  10. Caustic your answer seems to relate to the external memory related GC triggering code, but my comment was related to the semispace change. The semispace GCs are always stop-the-world so if you want to avoid pauses you should actually be making the semispaces smaller. A 1Mbyte limit helps keep semispace-related pauses in the 1ms range.

    Erik Corry

    12 Apr 12 at 8:21 am

  11. Erik – the reason for the change I made with V8_MAX_SEMISPACE_SIZE, was to indirectly affect the external allocated memory trigger. There is a bit of code which bases that threshold off the semispace size. I believe they’re meant to be in proportion to each other or something?

    Sounds like that brings with it some collateral damage to performance, though it must not have been significant enough of a hit to prevent 250k concurrent. Maybe if I use a cleaner technique to prevent that external alloc triggered GC, performance would be good enough to handle 500k :]

    caustik

    15 Apr 12 at 4:11 am

  12. Well, 250 concurrent req/sec sounds very good, what is configuration of your server?

    Kirill

    1 May 12 at 1:35 am

  13. This was on a rackspace cloud server — I think it was on the 8GB size.

    caustik

    4 May 12 at 12:22 am

  14. Hello, thanks for you share.
    I insepect the v8 source code, find that (Heap.cc)Heap class,
    intptr_t max_old_generation_size_; –this is the v8 max_old_space_size
    so,the max size is 2GB because the type is int, and testify by like this ” node –max_old_space_size=1919 **.js”
    the max size is 1919 and larger then 1919 is failed.
    so you set –max-old-space-size=8192 is it success ???

    bigfish

    27 Jun 12 at 7:19 am

  15. The code you mention isn’t there in the version I used for this article (3.10.0).

    Also, the command line parameter for old space is measured in MB, so 8GB actually requires only 16 bits (0×2000, 8192). I haven’t checked the latest code, so this could be wrong if it is per-multiplied to bytes before being placed in that variable. Sorry that I don’t have the time to keep this blog up to date with the latest versions of V8.

    caustik

    28 Jun 12 at 2:06 am

  16. May be when you have time, could you test the same thing with Node.js 0.8?

    And possibly your 500K test :D

    Ed

    29 Jun 12 at 2:18 pm

  17. There was a interesting bit about IE10 latest JS Engine, being the world first Non Stop the World Concurrent GC, If V8 ever have that in near future it could boost the Node.Js to another level.

    Anyway i though of something you could try to beat :D , 2M concurrent connection.

    http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/

    Ed

    30 Jun 12 at 12:04 pm

  18. ulimit -n 999999

    I get the error bash: ulimit: open files: cannot modify limit: Operation not permitted, when setting a limit above 1024.

    Can I get away with using soft limits instead? ulimit -s 9999?

    Ash Cairo

    23 Jul 12 at 1:54 pm

  19. Ash, You’ll need root access

    caustik

    24 Jul 12 at 4:17 am

  20. [...] blog post : Escape the 1.4GB V8 heap limit in Node.js This entry was posted in Hacking, node.js, Programming and tagged Hacking, node.js, [...]

  21. [...] Upping the Heap Limit in the v8 JavaScript Engine for Node.JS Usage [...]

  22. […] sounds like you’re running into the default heap limit in V8. I wrote a blog post about removing this […]

  23. On a micro instance of EC2 I never get past say 1500 concurrent connections of node.js. I tried spawning 2 instances and having HAProxy in front of them but still it doesnt work beyond 1500. Till 1000 it looks almost fine but socket time out errors start to come on client side after that.
    Please suggest what can be the issue

    Shashank

    2 Feb 14 at 9:17 pm

Leave a Reply