Problem

Every batch, the amplitudes are min-max normalized to the batch.

def __batch_quantize(data, q_levels, q_type):
    """
    One of 'linear', 'a-law', 'mu-law' for q_type.
    """
    data = data.astype('float64')
    data = __normalize(data)

. . . 

def __normalize(data):
    """To range [0., 1.]"""
    data -= data.min(axis=1)[:, None]
    data /= data.max(axis=1)[:, None]
    return data

This is bad for three reasons:

First reason: DC offset. The normalization is calculated by subtracting the minimum and dividing by the maximum. If minimum peak and maximum peak are different, silence is no longer the middle value, so you introduce a DC offset into the audio.

Second reason: Each batch has different peaks, so each batch will have a different quantization value for silence.

Third reason: dynamics. let's say part of my dataset is soft, part is loud, and part is transitions between soft and loud. If randomly there's a batch of all soft sounds, they will be normalized to loud, interfering with SampleRNN's ability to learn transitions from loud and soft.

Solution

Dont batch normalize amplitudes.

Take this line out

I think the only acceptable amplitude normalization would be to the entire dataset and you could do so [with ffmpeg] when creating the dataset.