cwh: InvalidSequenceTokenException

Hi,

I am using your cloud watch handler for symfony. I rolled out the cwh in multiple environments (dev, staging, demo, prod, … ) and are now experiencing something odd after a couple of successfull pushes:

  [Aws\CloudWatchLogs\Exception\CloudWatchLogsException]
  Error executing "PutLogEvents" on "https://logs.eu-central-1.amazonaws.com"; AWS HTTP error: Client error response [url] https://logs.eu-central-
  1.amazonaws.com [status code] 400 [reason phrase] Bad Request InvalidSequenceTokenException (client): The given sequenceToken is invalid. The nex
  t expected sequenceToken is: 49578481559872967104658335510065862421249324983057912274 - {"__type":"InvalidSequenceTokenException","expectedSequen
  ceToken":"49578481559872967104658335510065862421249324983057912274","message":"The given sequenceToken is invalid. The next expected sequenceToke
  n is: 49578481559872967104658335510065862421249324983057912274"}

This is my configuration:

        class: Maxbanton\Cwh\Handler\CloudWatch
        arguments:
            - "@cloudwatch_client"
            - "%elasticbeanstalk_app%"              # groupName
            - "%kernel.environment%" # streamName
            - 30                     # retentionDays
            - 5                  # logsInBatch
            - { mytag: "tag" }       # tags
            - NOTICE                # logLevel

Is it maybe an issue, that I chose batchsize of 5?

It is neccesary to use use different access key / secret for different environment?

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 1
  • Comments: 19 (8 by maintainers)

Commits related to this issue

Most upvoted comments

For anyone looking for a workaround for this I forked with some retry code.

Overview of changes:

  • Constructor accepts a $tokenRetries parameter to specify how many times you want to retry sending the buffer. Defaults to 0 (existing behavior)
  • On an invalidSequenceToken exception type send will try to get a new token making an api call using describeLogStreams and then retry itself if the number of attempts is less than $tokenRetries
    private function send(array $entries, $tokenAttempt = 0)
    {
        if (false === $this->initialized) {
            $this->initialize();
        }

        $data = [
            'logGroupName' => $this->group,
            'logStreamName' => $this->stream,
            'logEvents' => $entries
        ];

	    $this->checkThrottle();

        if (!empty($this->sequenceToken)) {
            $data['sequenceToken'] = $this->sequenceToken;
        } else {
        	// may have been cleared from previous exception
        	$this->setSequenceToken();
        }

	    try
	    {
		    $response            = $this->client->putLogEvents($data);
		    $this->sequenceToken = $response->get('nextSequenceToken');
	    }
	    catch (CloudWatchLogsException $e)
	    {
		    if (in_array($e->getAwsErrorCode(), ['InvalidSequenceTokenException', 'DataAlreadyAcceptedException'], true))
		    {
			    $tokenAttempt++;
			    // try to get token again if attempt count is less than allowed retries
			    if ($this->tokenRetries >= $tokenAttempt)
			    {
				    $this->setSequenceToken();
				    $this->send($entries, $tokenAttempt);
			    }
			    else
			    {
			    	// if attempts exceeds retries then clear the token so the incorrect token isn't used on the next send()
				    $this->sequenceToken = null;
				    throw $e;
			    }
		    }
		    else
		    {
			    throw $e;
		    }
	    }
    }

	/**
	 * Calls describeLogStreams to get the next uploadSequenceToken
	 */
	private function setSequenceToken() {
		$existingStreams =
			$this
				->client
				->describeLogStreams(
					[
						'logGroupName' => $this->group,
						'logStreamNamePrefix' => $this->stream,
					]
				)->get('logStreams');

		// extract existing streams names
		foreach($existingStreams as $stream) {
			if ($stream['logStreamName'] === $this->stream && isset($stream['uploadSequenceToken'])) {
				$this->sequenceToken = $stream['uploadSequenceToken'];
				break;
			}
		}
	}

@klausbreyer It seems that is not straight forward solution to this ? I could recommend alternative solution such as elastic stack.

@klausbreyer You can use this fork but please be aware that this is just a temporary solution and you will lose some logs https://github.com/banstola/cwh