Load-test fails, especially with large number of clients

[ghost]

I'm seeing that the load-test is rather unstable. Especially when running a larger number of clients (say 20000 to make it fail faster). I think it's more an issue with the orders and user services, but figure I should open this issue against the load-test, since running the load-test is the only way to see this issue.

When running the load test with say 20000 clients. There are two sets of errors I see. The first is a reoccurring log in error. Kind of seems like there's a sequencing issue in the load-test script. The user doesn't get logged in before a checkout is performed? The logs show something like this:

...
Error: User not logged in.
500
Error with log in: true
...

These errors appear as soon as you start running the load-test. It's very hard to miss. The front-end service produces the errors.

The second error that happens is that the front-end service crashes and restarts. It looks to crash because it receives an empty response from the orders service. The issue happens within 5-10 minutes, sometimes faster. It also happens faster when the load-test is executed from a remote machine, not locally on the machine that is running the weavesocks demo.

When this crash happens you see a log that looks something like this:

Received response: "{"error":"hystrix: max concurrency","status_code":500,"status_text":"Internal Server Error"}\n"
/usr/src/app/api/orders/index.js:67
var customerlink = jsonBody._links.customer.href;
^

TypeError: Cannot read property 'customer' of undefined
at Request._callback (/usr/src/app/api/orders/index.js:67:47)
at Request.self.callback (/usr/src/app/node_modules/request/request.js:186:22)
at emitTwo (events.js:87:13)
at Request.emit (events.js:172:7)
at Request. (/usr/src/app/node_modules/request/request.js:1081:10)
at emitOne (events.js:77:13)
at Request.emit (events.js:169:7)
at IncomingMessage. (/usr/src/app/node_modules/request/request.js:1001:12)
at IncomingMessage.g (events.js:260:16)
at emitNone (events.js:72:20)
at IncomingMessage.emit (events.js:166:7)

npm ERR! Linux 4.4.36-8-default
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "start"
npm ERR! node v4.2.6
npm ERR! npm v3.5.2
npm ERR! code ELIFECYCLE
npm ERR! [email protected] start: node server.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script 'node server.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the weave-demo-frontend package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node server.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs weave-demo-frontend
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls weave-demo-frontend
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /usr/src/app/npm-debug.log

Note, I did try a newer version of nodeJS and npm, but the error still appears like above. This suggests it's probably an issue with the demo.

To run the services, I used "docker-compose up" in the following directory:
/microservices-demo/deploy/docker-compose

Example command to use with the load-test:
./runLocust.sh -h <host_name> -r 20000 -c 20000