Writing an iPhone Game Engine (Part 5- Audio)

Introduction
Every games should have an audio system. In my little engine, it supports 2 types of sound: 3D effects sounds and BGM. Separate this 2 types because the BGM can be decoded on hardware. A sample project is provided for playing the BGM using the Audio Queue on iOS.

Effects sounds
Effect sounds are played using OpenAL, which is an API similar to the OpenGL. In the engine, there has an audio thread where all OpenAL calls are executed in this thread and this thread is communicate with main thread through a command buffer which is similar the one used in graphics programming. For example, during main thread update, the game logic may request to play an explosion sound, then an audio command is made and this command is pushed to the command buffer. When the audio thread find that there is a command inside the buffer, the command is executed and initiate an OpenAL call. I set up the audio buffer and thread because the OpenAL call may stall the calling thread according to this article.

BGM
On iPhone, there are hardware to decode audio which can be used through the Apple's Audio Queue API. By this API, you can play sound by creating an audio queue output using AudioQueueNewOutput() providing the audio file description such as sample rate. You can get this description by calling the AudioFileOpenURL(). Unfortunately, this is not suitable in my case as my audio file is already loaded in the memory when the game world tile is streamed, I don't want to called the AudioFileOpenURL() to open the audio file again to get the description of the audio file only. So I decided to get this data by myself and I picked the Apple CAF file format with the AAC compression method because it is an open format and Mac machine have a command line tools to convert file into this file format. (Note that on iPhone, the Audio Queue can only decompress 1 song using hardware decoding. If more audio need to be played, it will fall back to use software decoding.)

CAF file format
Just like the WAV file format, the CAF file format is divided into many different chunks, such as description chunk(which store the sample rate, channels per frame, ...) and the data chunk(which store the audio sample data). The specification of CAF can be found here. We need to get those data inside the CAF file to playback the BGM using AudioQueue. For details, you can take a look at the AudioCAFHelper.cpp in the sample project.

Screen Shot from Sample Project

Apple Audio Queue
To playback the audio using Audio Queue API, we need a couple of steps:

1. An audio output need to be created using AudioQueueNewOutput().
2. We need to set the property of the newly created queue by AudioQueueSetProperty() which supply the Magic cookie property which is required by the audio format.
3. A property listener should be set up using AudioQueueAddPropertyListener() to listen to the event occurs in the audio queue such as the playback is finished.
4. We need to allocate memory for audio queue to contain the packet description by AudioQueueAllocateBufferWithPacketDescriptions().
5. After setting up the description, the audio sample data need to be put into the audio queue by AudioQueueEnqueueBuffer().
6. After that, we need to tell the hardware to decode the audio samples by AudioQueuePrime().
7. Finally, the audio are ready to be playback using AudioQueueStart().

To stop an audio queue, 3 steps are needed:

1. AudioQueueStop() need to be called to stop the playback.
2. The property listener set up in step 3 above needed to be removed by AudioQueueRemovePropertyListener().
3. Finally, AudioQueueDispose() is called to release all the audio queue resources.

You may refer to the sample project to have a full understand on how to use the audio queue, especially on the part to enqueue the audio sample to audio queue buffer.

Conclusion
Playback 3D effects sound using OpenAL on iPhone is similar to the other platform, while playback the BGM takes some efforts because I need to get the audio file description by myself as the sample code from Apple only provide how to play back an audio by specifying a file path but not the audio file that is loaded in memory. Hope that my sample code can help someone faces the same problem with me.

Reference:
Sample code: http://code.google.com/p/audio-queue-caf-sample/downloads/list
CAF file format:  http://developer.apple.com/library/mac/#documentation/MusicAudio/Reference/CAFSpec/CAF_spec/CAF_spec.html%23//apple_ref/doc/uid/TP40001862-CH210-DontLinkElementID_64
Audio Queue Reference:  http://developer.apple.com/library/ios/#DOCUMENTATION/MusicAudio/Reference/AudioQueueReference/Reference/reference.html
Using Audio on iOS: http://developer.apple.com/library/IOS/#documentation/AudioVideo/Conceptual/MultimediaPG/UsingAudio/UsingAudio.html

Writing an iPhone Game Engine (Part 4- Streaming)

Introduction
My game is an open world game. The player can freely explore the game world. It is impossible to load all the game objects when the game start, so my engine should be able to stream in the game objects when the player is playing.

Loading
In order to stream the game objects, I have to partition the whole game world into many square tiles:

I will load the world tile(s) according to the player position. I divide each world tile into 9 regions as below:

There are 3 types of region in each tile (marked as A, B & C). When the player is in region A, only the tile that the player inside is loaded. When the player is inside region B, 1 of the adjacent tile will be loaded:

And within region C, 3 of the adjacent tiles will be loaded:

So, maximum number of tiles in memory will be 4.

Unloading
To maintain the maximum number of tiles in memory, unseen tiles need to be unloaded. I divide each tile into 4 region for unloading:

Say, when the player is in region 0, the nearby tiles X, Y, Z in the below figure will be unloaded:

The only thing I need to ensure is tiles X, Y, Z are completely unloaded before new tile need to be loaded according to the above rules. If this situation happens, I will block the game until they are unloaded.

Memory and Threading
Since we already know that there are only 4 maximum tiles. Beside the pool memory allocator used for physics/script in the main thread. I also have 4 linear allocator for streaming world tile and all tiles are within this limit. The memory allocation pattern is related to how the threading model works in the game. In the game there are 2 threads: main thread and streaming thread. The main thread is responsible for updating the game logic, physics and rendering, while the streaming thread is for loading resources, decompress textures, etc. When the player update the position of the ship in the main thread, it will signal the streaming thread to load the tile if needed. Several frames later, the streaming thread signal back the main thread finished loading. The communication between 2 threads are double buffered to achieve minimum locking and also ensure the linear allocator will only be used in the streaming thread which can avoid using any mutex. But things go complicated when some of the objects should be created in main thread such as graphics objects and Lua objects. For example, the streaming thread should notify back to the main thread to create an openGL texture handle after it finished decompressing a texture.

Advantages
Using linear allocator for streaming can avoid memory fragmentation. Partition the game world into tiles make managing memory easier as each tile should have roughly the same amount of memory used.

Disadvantages
To unload a world tile, those resources created on CPU side are simply freed by reseting the linear allocator. However, things are not that easy as I think originally... For example, when I want to unload the physics objects in a tile, I need to remove all of them from the collision world first before resetting the allocator. Also, for the graphics objects, I need to release all of the openGL objects in that tile before resetting the allocator, otherwise, leak will occur in the GPU side. I also need to release the script object in the tile so that those scripts can be garbage collected in Lua. Therefore, it is nearly the same as 'delete' all game objects in a tile one by one and cannot free all resources simply by resetting the linear allocator. Besides, creating objects in bullet physics using another custom allocator other than the currently hooked up allocator using btAlignedAllocSetCustom() is not easy too. Since bullet is not designed for allocating objects to another memory allocator(i.e. in my case, the linear allocator for streamed objects such as the rigid bodies and collision shapes). I need to modify bullet source code to make it works.

Conclusion
After making the streaming system, I have a more clear understanding between inter-thread communication as it should be planned carefully, and some of the objects needed to be created on specific thread. Also, I think that it is not a wise choice to divide a dedicated memory region using linear allocator for streaming objects as those objects will be deleted one by one when they need to be removed from the game world. And it costs too much work to modify bullet physics to cope with this memory model and this result in hard to maintain and update bullet physics library.

Reference:
[1] http://www.gamearchitect.net/Articles/StreamingBestiary.html
[2] Game Engine Architecture: http://www.gameenginebook.com/