godot: Audio Stream Playback Generator Broken

I have investigated this a little bit and found out that the commit that broke the audio stream generator is afd2bbaa5f, i.e., this PR https://github.com/godotengine/godot/pull/55846. The commit immediately before (3017530e26) still works fine.

After skimming the code and adding some debug statements, I have a rough understanding of what is happening: The reason why it sometimes works and sometimes doesn’t is a race condition between the audio server thread and the rendering thread. Typically user code calls playback.push_buffer() (if playback.get_frames_available() is non-zero) from the _process callback. The determining factor whether it works is whether the render thread (i.e., the _process function) gets to push something into buffer before AudioServer::_mix_step calls into AudioStreamPlaybackResampled::mix in the audio thread. If the _process function fills the buffer first, it works. However, if the audio server tries to “mix” from the playback while the buffer is still empty, it leads to an immediate erasure of the playback from that playback_list in the audio server. This happens because there is some logic that compares the requested number of frames with what actually has been produced by the playback. That’s the crucial part of AudioServer::_mix_step:

https://github.com/godotengine/godot/blob/1d0e7f0222e2f6da16bea959a4170ac754110d56/servers/audio_server.cpp#L351-L371

The buffer_size is always hard coded to 512 currently. Interestingly when calling stream_playback->mix in line 351 it seems to return 128 frames, even if the rendering thread has pushed nothing into the buffer so far (I still need to understand how that is supposed to work, and why it is always 128). Then in line 357 the comparison mixed_frames != buffer_size becomes 128 != 512, so we change the state of the playback to AWAITING_DELETION in line 369-370. Further down in AudioServer::_mix_step there is a cleanup functionality the erases playbacks from the playback_list that are in the AWAITING_DELETION/FADE_OUT_TO_DELETION.

https://github.com/godotengine/godot/blob/1d0e7f0222e2f6da16bea959a4170ac754110d56/servers/audio_server.cpp#L448-L458

So in summary, if the user thread doesn’t manage to push something into the generator playback, the audio server will immediately remove the generator playback the first time it renders an audio buffer.

---

Now the question is what’s the best option to resolve the issue. Since I’ve only read Godot’s audio code for a few minutes now and don’t understand the general design yet, I can only make some guesses. Would be great if one of the audio devs could chime in. Some thoughts:

• Perhaps the implementation of AudioStreamPlaybackResampled::mix should be adapted to always return the number of requested frames, no matter what. This would mean that it would have to generate silence if it doesn’t have the data available. Basically the typical buffer under-run effect.
• Alternatively the audio server could use a smarter logic the determine when a playback needs to be erased. I.e., instead of just checking mixed_frames != buffer_size it could rather ask the playback “are you still playing?” or so. Perhaps this could even be implemented based on some get_stream_length function, i.e., finite streams could tell their finite lengths, while infinite streams could indicate that they are infinite by returning e.g. -1. Or perhaps just a playback_stream.is_infinite() getter. Some of these solutions need to be considered carefully, because the call into the playback would then come from the audio server, i.e., a different thread, so any kind of querying function needs to be thread safe.

The PR that broke the behavior (https://github.com/godotengine/godot/pull/55846) actually modified AudioStreamPlaybackResampled::mix, which could indicate that this is the right place to fix it (i.e., keep the mixed_frames != buffer_size in the audio server as it is). I don’t really understand yet why this PR modified that function in the way it did, and why this change was necessary for fixing ogg edge cases. So simply reverting it would fix the generator playback, but presumably break something related to ogg which I don’t understand yet.

The simplest thing ended up working. @bluenote10 could you test #73162?

On the other hand I’m wondering if that isn’t redundant to the information provided by playback->stream_playback->is_playing(). I’m wondering if we could simple wrap that part that changes the state to AWAITING_DELETION under an if on that “is playing” as a very naive fix

This makes sense.

(the generator could always return true to stay alive, but not sure if it means it can never be deleted at all).

This would probably just be fine, because you can still just call stop() to begin deletion I think. I don’t have the code in front of me but I’m 90% sure that would work. If you play() a generator stream and then never fill it, it will stay in the playing state forever (until stop()) but I don’t really think that’s a problem.

I’ve been pretty out of the loop with what’s going on with releases and stuff so I don’t know the criteria for inclusion of a bugfix in the RC’s, but this may (?) have to wait for a 4.0.x patch release because it could be a medium-risk change depending on how it’s implemented. I’m up for trying to fix it today though, this piece of code has been bugging me for a while.

I decided to take a deep dive into the problems I was having this afternoon and made a minimal reproduction project with some of the code I was tinkering with. I’ve included it below. Apologies if the evil music generated makes your ears bleed.

Audio.zip

In the default configuration, the project is likely to play on startup. Of note–like many of you, I’ve noticed that, on startup, sound may “chirp” briefly as the initial audio buffer is consumed and overtaken by permanent silence. However, in my experience, it’s totally random whether generated audio chirps and dies or continues to play on in good health. I have found this is mostly affected by the sample rate.

For me

• at 48000hz, my sample project usually plays (my project’s default)
• at 96000hz, my sample project rarely plays
• at 24000hz, my sample project rarely breaks

On the other hand, I’ve had the sample rate as low as 4000hz and still had the audio randomly chirp and die. I haven’t found any way to recover from this error either, so generated audio is a non-starter until it’s fixed.

The length of the audio buffer, and how many samples it receives in ready() may also matter, but I might as well be talking about superstition and witchcraft. I might trying straining at the source code tomorrow to see if there are any obvious problems I can understandstand related to to the sample rate.

I’ve noted that the auto play workaround is spotty. If streaming a simple sine wave it’s about 50/50. When loading a file first, then starting to stream, it basically never works. I did get it to work at 50/50 by streaming silence until the file was loaded, then switching to that. It seems possible that the sooner you start streaming the more likely it is to work.

Also I wanted to add that this is great work on your part @bluenote10 tracking down the source of this issue. I scrolled up to the top and was surprised to see the way this issue ended up presenting itself. The cause is not at all obvious.

I have been playing with the audio generator as well. I also noticed that setting autoplay to true “fixed” the issue by allowing the audio to play for just a fraction of a second. However, I was able to get the demo to play a prolonged note by also commenting out the $Player.play() invocation, (in the demo of course).

Garuda Linux, Godot 4.0.beta3

I can confirm this also.

However, setting Autoplay to True does not really fix the problem for me. It gets rid of the error, but the sound will only play for a fraction of a second before cutting off. In Godot 3.5 the tone continues to play until you exit.

Godot v3.5.stable.official [991bb6ac7] Godot v4.0.beta2.official [f8745f2f7] Windows 10 Pro, Microsoft audioendpoint 10.0.19041.1

EDIT I did a little more playing around. Here is the code that I used for the tests. Code:

extends Node

var sample_hz = 44100/4.0 # Keep the number of samples to mix low, GDScript is not super fast.
var pulse_hz = 440.0
var phase = 0.0
var playback_is_not_null = false

var playback: AudioStreamPlayback = null # Actual playback stream, assigned in _ready().

func _ready():
	$Player.stream.mix_rate = sample_hz # Setting mix rate is only possible before play().
	playback = $Player.get_stream_playback()
	_try_fill()
	$Player.play()

func _process(_delta):
	_try_fill()

func _try_fill():
	if playback != null:
		if playback_is_not_null == false:
			print("Playback: ", playback)
			playback_is_not_null = true
		_fill_buffer()
	else:
		print("Playback: ", playback)

func _fill_buffer():
	var increment = pulse_hz / sample_hz

	var to_fill = playback.get_frames_available()
	print("To Fill: ", to_fill)
	while to_fill > 0:
		playback.push_frame(Vector2.ONE * sin(phase * TAU)) # Audio frames are stereo.
		phase = fmod(phase + increment, 1.0)
		to_fill -= 1

It prints out the playback object until it’s not null. Then it prints it one last time. Then it prints out how much of the buffer needs filling.

Godot 4 Output without AutoPlay:

--- Debugging process started ---
Godot Engine v4.0.beta2.official.f8745f2f7 - https://godotengine.org
Vulkan API 1.2.198 - Using Vulkan Device #0: NVIDIA - GeForce GTX 1050
 
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
Playback: <Object#null>
--- Debugging process stopped ---

Godot 4 Output with AutoPlay = true:

--- Debugging process started ---
Godot Engine v4.0.beta2.official.f8745f2f7 - https://godotengine.org
Vulkan API 1.2.198 - Using Vulkan Device #0: NVIDIA - GeForce GTX 1050
 
Playback: <AudioStreamGeneratorPlayback#-9223372011034639975>
To Fill: 32767
To Fill: 128
To Fill: 128
To Fill: 0
To Fill: 0
To Fill: 0
To Fill: 0
To Fill: 0
To Fill: 0
To Fill: 0
To Fill: 0
To Fill: 0
To Fill: 0
--- Debugging process stopped ---

Godot 3.5 Output:

--- Debugging process started ---
Godot Engine v3.5.stable.official.991bb6ac7 - https://godotengine.org
OpenGL ES 2.0 Renderer: GeForce GTX 1050/PCIe/SSE2
 
Playback: [AudioStreamGeneratorPlayback:1266]
To Fill: 8191
To Fill: 0
To Fill: 0
To Fill: 0
To Fill: 768
To Fill: 0
To Fill: 0
To Fill: 512
To Fill: 0
To Fill: 512
To Fill: 0
To Fill: 0
To Fill: 256
To Fill: 0
To Fill: 0
To Fill: 2304
--- Debugging process stopped ---

I think the problem in Godot 4 is two-fold.

1. You shouldn’t have to have AutoPlay on for it to work (Or there should be a way from code to instantiate it?).
2. The buffer fills, but then is never emptied, so there are never any new spots to fill… I think that’s what is going on anyway…

Hopefully this helps somehow.

I can confirm this issue as well in Godot 4. I was able to resolve by setting Autoplay to True on the AudioStreamPlayer object – rest of the code works fine after that change. I can also confirm when opening the tutorial project in Godot 3.4 they do NOT set Autoplay to True. Not sure if a bug or not, but hope this helps!