Box fit algorithm uses huge amounts of memory for relatively simple scenarios

[marcnicholas]

Hi,

A relatively simple test of the box fitting algorithm seems to use huge amounts of memory and run forever.

45403 node 100.3 01:03.63 8/1 0 38 *1450M+* 0B 0B 45403 3364 running 5477428 0 501 1093442+ 240 63 22 50565+ 6035+ 47118+

That's nearly 1.5GByte of resident memory!

OSX Yosemite (10.10), node.js v0.10.21,

-marc

// Fit items in box.
postmaster.v1.box.fit({
    items: [{
        'width': 6,
        'length': 6,
        'height': 12,
        'count': 6,
        'sku': 'Grow',
        'weight': 75,
        'dimension_units': 'CM',
        'weight_units': 'G'
        } , 

        {
        'width': 6,
        'length': 6,
        'height': 12,
        'count': 6,
        'sku': 'Climate',
        'weight': 50,
        'dimension_units': 'CM',
        'weight_units': 'G'
        }


    ]
}, function(err, response) {
    if (err === null) err = 'No errors';
    console.log('box.fit', err, postmaster.stringify(response));
});

[CountZachula]

Hey @marcnicholas,

Thanks for trying out the Postmaster node.js client library.

I ran the the provided code snippet and it successfully completed with a 5-10 second response time. No memory issues. One thing to note is that the box fitting algorithm resides on Postmaster's servers, not within the client library itself. The client library merely consumes a RESTful API and registers callback events for responses.

That said, the amount of memory usage here is a bit alarming.

• By "run forever", do you mean the request isn't timing out? If not, this may be an issue with Node's native request library. I may need to force a quicker timeout.
• Yosemite (10.10) is still in public beta. Have you tried executing the same query on Mavericks? Maybe there's still optimization to be done on the Node or Google V8 side of things.
• Have you tried upgrading to the latest version of node.js (v0.10.30)?
• There may be issues server-side. Have you encountered prolonged requests for any other functions?

I will continue to test but the more information the better. My first order of business will be to require later dependency versions. I hope we can hone in on this issue together.

Thanks,
Z

...and for reference, I tested this on three separate environments:

• node.js v0.10.30
• Mavericks 10.9.4
• Fedora 20
• Arch 2014.08.01

[marcnicholas]

Hi @CountZachula!

Thanks for the response.

Yes, I ended up looking at your API code and did indeed notice virtually nothing is done client-side....so I'm also starting to wonder if there's a more systemic node.js issue at play.

I'm in the process of migrating the code to EC2 today for testing, so it'll be on Ubuntu Server....I'll report back if that cleans things up.

Unfortunately, I couldn't resist the Yosemite beta ;)

Cheers,

-m

[marcnicholas]

I get the same issue with node v0.10.28 on Ubuntu on Amazon EC2!

In fact, get the same issue with a different API call for shipping rates (see below code).

The only thing that might be different is I'm using the API in test mode?

I'll switch on some debugging to see if I can trace what's going on!

-m

// Initialize.
var postmaster = require('postmaster-shipping')({
    apiKey: 'XXX',
    password: 'XXX!'
}, false);


postmaster.v1.rate.list({
    from_zip: '14202',
    to_zip: '',
    from_country: 'US',
    to_country: 'DE',
    weight: 1.0,
    carrier: 'USPS'
}, function(err, response) {
    if (err === null) err = 'No errors';
    console.log('rate.list', err, postmaster.stringify(response));
});

[CountZachula]

Thanks for the updates, @marcnicholas.

Your rate.list snippet might raise an API validation error from the server (HTTP 400). Try entering a valid postal code in Germany (e.g. 14199 for Berlin).

Maybe it's a carrier-specific issue? I've experienced quite a few outages during development and testing of the Node.js and .NET libraries. I'll probably still lower the timeout value in case the carrier APIs take too long to respond.

[marcnicholas]

The postal code seems to fix it for the rates issues.....but that doesn't explain the box fitting issue?

I did try profiling to find the root cause, but the app bombs before the profiler can collect stats :(

[marcnicholas]

After digging and enabling some console logging....The box fit issue is a 504 response code....

"{"message": "Carrier API Timeout", "code": 504, "details": {"body": {}, "code": 101}}"

I can't seem to find any re-try code in the API, so I'm wondering if there's maybe an async + request memory leak somewhere?

[CountZachula]

@marcnicholas,

This info is helpful. Let me do some digging this evening and I'll get back to you. I'm gonna whiteboard out some smarter retry/fail logic to avoid the library from hanging. Hopefully it isn't a low-level problem.

Thanks,
Z

[marcnicholas]

Thanks, Zach.....if I get a minute I'll dig some more too. I know there're potential memory leak issues with request not used async....

I'd love this to work for us...even if it is non-realtime, batching packing requests would help. :)

[CountZachula]

I did some digging and apparently there are several reported memory leak issues for the request library. One commonality they all share is that it only seems to occur when making requests over HTTPS. That said, it may be an SSL or EventEmitter bug deep in the request library code. One user reported memory usage increasing 20%-30% every 60 seconds for long-running requests.

I'm going to update all my dependencies and set a hard timeout of 30 seconds as a workaround. Luckily most requests take 10-15 seconds max to complete with the Postmaster API unless the carrier endpoint doesn't respond.

The 'request' library is widely used so hopefully this issue is fixed soon. I apologize for any inconvenience this may have caused you. Expect to see an updated version tomorrow morning. I will be keeping this issue open until I either find a legitimate fix or the request crew applies one of their own.

Thanks,
Z

[marcnicholas]

Thanks, Zach.

I did a quick manual clone into my current node_modules, but don't seem to see the timeout working...will play some more as I might have done something stupid ;)

Cheers,

-m

[CountZachula]

@marcnicholas,

Really? Sheesh, no luck. I will run a few tests and get back to you. Sorry about that.

Thanks,
Z

[CountZachula]

@marcnicholas,

Could you paste a snippet that you're testing the timeout with? I'm actually able to force a timeout.