sockeye: Bug with beam-size=1?

In trying to get tests to passing with scoring (#538), I have turned up some weird behavior with scores output by Sockeye. Here are two commands using a transformer model built in the system tests. Notice:

  • The invocations differ only in the beam size (1 or 2)
  • --skip-topk is not enabled
  • With beam size of 1, the scores output should be impossible, since Sockeye outputs negative logprobs.

Any ideas?

CC: @bricksdont

$ python3 -m sockeye.translate -i src --output-type translation_with_score --use-cpu -m model --beam-size 1 2> /dev/null | head
-10.556	7 5 2 7 3 6 5 4 7 7
-10.727	9 2 4 1 6 7 8 6 8
-12.788	8 6 8 7
-10.413	0 5 0 7 5 9 0 6 3 1
-10.731	7 9 2 6 8 5 0 6 5
-12.490	5 6 3 2
-inf	
-11.242	3 9 1 3 8 7
-15.759	2 1
-10.506	8 8 8 2 4 4 5 5 2 5
$ python3 -m sockeye.translate -i src --output-type translation_with_score --use-cpu -m model --beam-size 2 2> /dev/null | head
0.003	7 5 2 7 3 6 5 4 7 7
0.001	9 2 4 1 6 7 8 6 8
0.000	8 6 8 7
0.002	0 5 0 7 5 9 0 6 3 1
0.001	7 9 2 6 8 5 0 6 5
0.001	5 6 3 2
-inf	
0.001	3 9 1 3 8 7
0.001	2 1
0.002	8 8 8 2 4 4 5 5 2 5

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 18 (18 by maintainers)

Most upvoted comments

good point regarding beam size 1. What if we changed sockeye.translate to only enable softmax skipping if the user does not specify an output handler, which uses the score?

Yes - skip_softmax is enabled by default for beam size 1. Initially, this was a CLI option, but after some discussion we changed it to default:

PR Discussion: https://github.com/awslabs/sockeye/pull/519 Commit: https://github.com/awslabs/sockeye/pull/519/commits/5c05dc3de1686303787fb7b2d17ad9037ac4564e

I think we simply never considered scoring as a usecase so far. If there is consensus to change it back to optional for inference, I will open a PR.