artillery: Artillery runs forever and never exit (v2)

Hi, I am running artillery too for load testing.

Artillery: 1.6.1
Node.js: v10.15.3
OS: linux/x64

I am getting the same issue. Did anyone able to resolve this issue? If yes, please share. I am attaching the screens from my trial run. Here’s the config I used.

config:
    target: "someURL"
    ensure:
      maxErrorRate: 1
    phases:
      - duration: 10
        arrivalRate: 5
        rampTo: 10
        name: "Warming up"
      - duration: 20
        arrivalRate: 10
        rampTo: 50
        name: "Max load"
scenarios:
  - engine: "ws"
    flow:
      - send: "hello"
      - think: 3
      - send: "how are you?"

This is so annoying issue 😭

I have artillery for stress tests and I want to find a limits of server - and artillery found it, but I am not able to get some reasonable output from artillery due this issue.

My setup - I run this 2x parallel in docker containers to avoid problems with CPU overload.

config:
  target: "ws://raspberry:3000/ws?templateId=1&username=JakeTheDog"
  phases:
    - duration: 120
      arrivalRate: 30
      name: "Stress creating connection"
  processor: "./utils/elasticsearch.js"

scenarios:
  - engine: "ws"
    flow:
      - send: "ping pong"
      - think: 1

The problem begins when the server begins to have data processing problems and starts to have timeouts and eventually crashes. After that, artillery starts show

Report @ 14:05:46(+0000) 2020-01-17
Elapsed time: 11 minutes, 10 seconds
  Scenarios launched:  0
  Scenarios completed: 0
  Requests completed:  0
  RPS sent: NaN
  Request latency:
    min: NaN
    max: NaN
    median: NaN
    p95: NaN
    p99: NaN

Any solution for that? It is so annoying due I send results from artillery to elasticsearch and now, it doesn’t send anything.

Unsure if relevant but thought it worth posting.

Recently I came across this issue as well and it seemed to be issues with target and the scenarios url.

example:

  target: 'http://myurlexample.com/'
  http:
    pool: 10
  phases:
    - duration: 60
      arrivalRate: 10

scenarios:
  - flow:
    - get:
        url: "preview/1/"

resulted in: the NaN issue as described by @igorclark

Report @ 11:40:26(+0100) 2017-09-28
  Scenarios launched:  100
  Scenarios completed: 0
  Requests completed:  0
  Concurrent users:    289
  RPS sent: NaN
  Request latency:
    min: NaN
    max: NaN
    median: NaN
    p95: NaN
    p99: NaN
  Scenario duration:
    min: NaN
    max: NaN
    median: NaN
    p95: NaN
    p99: NaN

but changing the target to myurlexample.com and scenarios url to /preview/1/ removed the NaN’s from the results.

Summary report @ 11:42:41(+0100) 2017-09-28
  Scenarios launched:  600
  Scenarios completed: 600
  Requests completed:  600
  Concurrent users:    60
  RPS sent: 9.93
  Request latency:
    min: 95.5
    max: 228.7
    median: 108.5
    p95: 146.8
    p99: 182.9
  Scenario duration:
    min: 105
    max: 303
    median: 132
    p95: 191.5
    p99: 239.5
  Scenario counts:
    0: 600 (100%)
  Codes:
    200: 600

@hassy don’t know about every time, but certainly 9 times out of 10.

Would love to provide a full test case including back-end, but it’s not a public project. Here’s the majority of the test plan though - don’t know if it’ll be much use but hope so 😃

config:
  target: "ws://<host-name>/"
  phases:
    -
      duration: 60
      arrivalRate: 100
  ws:
    # Ignore SSL certificate errors
    # - useful in *development* with self-signed certs
    rejectUnauthorized: false
scenarios:
  -
    engine: "ws"
    flow:
      -
        loop:
          -
            send: '{<json-message-1>}'
          -
            send: '{<json-message-2>}'
          -
            send: '{<json-message-3>}'
          -
            send: '{<json-message-4>}'
        count: 100

Drop the latest link that will resolve the issue to help people to find it easier as this is the first link on search!

https://github.com/artilleryio/artillery/issues/1799

@hassy I am using errors to stop my scenario, its using http, version 1.6.1.

I seem to be seeing the same issue, but have a bit more info that might be useful. First, see this screenshot, showing the issue with the hung test:

The log output on the service has stopped, all the received requests have completed successfully. The test runner itself is in the hung state and sending no more requests (the test should have ended at 1 minute).

The interesting part is that, when I kill the test runner, it seems like one last small batch of requests is fired off just before artillery exits:

Notice the last two requests in the server log, that are received immediately upon killing artillery. You can also see that these requests are then immediately aborted (the finished=false implying that the connection was closed on these requests before a response was sent, likely because artillery exited).

My guess would be that artillery is hung because it’s waiting for these requests, but the requests were never actually sent. And then killing the process causes them to somehow get flushed out and finally sent.

Hey @hassy 👋

I’m experiencing the same symptoms.

Artillery: 1.6.0-28 Node.js: v12.2.0 OS: darwin/x64

My config looks something like:

config:
  target: "https://example.com"
  phases:
    - duration: 60
      arrivalRate: 20
  defaults:
    headers:
      x-api-key: 'XXXXXXX'
scenarios:
    flow:
      - get:
          url: "/example"
          qs:
            foo: "bar"

The issue is intermittent (occurring ~2/3rds of the time). Below are two runs, one run immediately after the other with no config changes.

Run no. 1 (error)

$ artillery run example.yml

Started phase 0, duration: 60s @ 16:57:05(+0100) 2019-05-20
Report @ 16:57:15(+0100) 2019-05-20
Elapsed time: 10 seconds
  Scenarios launched:  199
  Scenarios completed: 0
  Requests completed:  0
  RPS sent: 20.1
  Request latency:
    min: NaN
    max: NaN
    median: NaN
    p95: NaN
    p99: NaN

Run no. 2 (success)

$ artillery run example.yml

Started phase 0, duration: 60s @ 16:57:17(+0100) 2019-05-20
Report @ 16:57:27(+0100) 2019-05-20
Elapsed time: 10 seconds
  Scenarios launched:  199
  Scenarios completed: 86
  Requests completed:  86
  RPS sent: 20.12
  Request latency:
    min: 124.4
    max: 202.6
    median: 137.4
    p95: 182.5
    p99: 201.6
  Codes:
    200: 86

If I get to the bottom of it I’ll post here. If you have any ideas about what I might be doing wrong, or if I can provide any more information please let me know!

Thanks