folk-rnn: Data formatting issue: missing unique IDs & stray HTML
I’ve been trying out ABC with GPT-2 along the lines of my poetry generation (max-likelihood training and then I’ll use OA’s RL preference-learning for finetuning), and I’ve found that some of the ABC files in data/ appear to have 2 issues:
-
there are ~2k stray
</html>lines which should be removed, as this is not ABC music (unsure if it’s syntactically invalid, but they definitely shouldn’t be there). This can be easily removed with any search-and-replace. -
lack of unique IDs: some ABC compilers (specifically,
abc2midi) require unique IDs likeX: 1for each ABC song. http://abcnotation.com/wiki/abc:standard:v2.1#xreference_number notes thatThe
X:field may be empty, although this is not recommended.(I worked around this by using an Emacs macro to prefix it and an incrementing integer.)
About this issue
- Original URL
- State: open
- Created 5 years ago
- Comments: 20
To update this: I figured out the problem which was causing divergence. It turned out to not be the prompt issue, but once I fixed that, the problem became more obvious. The OA code has a sort of heuristic where it looks for a particular character/BPE as a validity check, and ignores the score in favor of a fixed penalty if the check fails; the check they had was to search for ‘.’, which they interpret as indicating a valid English sentence. Unfortunately, it seems periods show up very infrequently in ABC music… So every sample was being given a heavy penalty and quality was being ignored. Once I removed that check entirely, the RL tuning code now runs just fine.
I’ve begun the iterative loop and so far have done ~3 loops and have rated ~2136 pairs of samples.
My first impressions are that the RL finetuning is improving the overall syntactic validity of the ABC samples, and has largely eliminated the repetition failure mode (like in the poetry samples). I think it has also made them more ‘musical’ but it’s hard for me to say for sure. (For example, there seem to be much more in the way of ‘endings’ than I usually see with char-RNN stuff, but I’m not sure how much the RL finetuning is helping there versus the Transformer seeing the entire ABC sample.)
Yesterday samples: https://www.dropbox.com/s/8sb11k185v7likq/rlsamples-irish-6.txt?dl=0
Today samples: https://www.dropbox.com/s/pgwuffx60s2ioof/rlsamples-irish-7.txt?dl=0
some cherrypicked samples from throughout:
Incidentally, I noticed some more problems with the original data. There are English comments dumped in at random in various places. For example, if you grep for ‘should’ you can find lines like
Which don’t look like ABC to me and make abc2midi unhappy when they pop up in GPT-2-generated outputs.
I haven’t put the models up so far because it’s not really done. For example, the 1k sample from above had a loss of 0.46, but it turns out it can go as low as 0.09 on just the original corpus (minus the other transformed versions) before my simple plagiarism checks using
grepbegan turning up hits of ABC in the original corpus indicating memorization of more than just titles. (Where more ‘musical’ level plagiarism begins, I couldn’t say.) I discovered I had simply given up too early in the training when I went back to modify the corpus more to make it more acceptable toabc2midiand trained it some more. After I got down to 0.09, I retrained on the concatenated corpus down to 0.29, so it roughly halved the loss. The result is that the syntax seems much more correct and now most samples passabc2midi, and the musical quality seems somewhat better to me. Anyway:current model: https://mega.nz/#!ffIW0Y4Q!s98tWDmdphJtmCD_j1nYoDiL12sPe4j40hJL_CtIFnk
samples thereof: https://www.gwern.net/docs/ai/2019-10-23-gwern-gpt2-folkrnn-irishmusic-samples.txt
while training that, I cherrypicked 3 samples:
So that’s where the max-likelihood model currently is.
I’m still struggling with the RL tuning. My hacks worked fine for poetry generation, but lead to really bad divergence in every training run for the currently-final 117M model I trained despite me rating 800+ pairs of samples. The average reward would get worse every iteration, and it would degenerate, until sometimes it discovered a highly repetitive sequence like ‘X. X. X.’ which somehow earned a relatively high reward from the reward model, and would continue with that…
My current theory as to what is going on is that the OpenAI RL tuning code is framed as a conditional generation task: a prompt->response. I’ve been using it for unconditional generation by hacking the config to change one of the tasks to poetry and expect the model to simply ignore the random English words being used to prompt it. This is OK for the poetry model because it just ignores the prompts or works off of them (poetry is robust like that), but I think what happens with the ABC 117M reward model is that it is so finely tuned to ABC notation that when it sees a string with the random English word in it, it knows it’s invalid ABC and penalizes it harshly, so every generated sample looks bad to it, and this destroys the training dynamics. What I need to do is somehow eliminate the prompt or make it innocuous, like zeroing it out with spaces, so all the generated samples can be valid ABC and the reward model can start producing a sane training signal for the generator… Haven’t quite done that because all of the rating of the poems and music samples have really exhausted me. I switched back to working on the poems while I meditated on what was going wrong with the music, since adding more ratings clearly was not fixing things.
(The poetry isn’t suffering any weird problems like that, but the improvements from RL training are thus far small, and I suspect poetry is intrinsically much more difficult than music and may simply require way more ratings to train a reward model good enough to make a dramatic difference.)
Ah. Now I see what you mean. Yes, that would explain it. I think that’s OK to leave in place? I thought it was some sort of error by the model, but since they are all settings or ‘variants’ in the original (right?), then it’d be interesting to see the model also generate multiple ‘variants’ on a theme.
Sure. You can always get decent performance with small models, because the log-loss decreases roughly logarithmically with parameter count. And NNs are always highly overparameterized, so you should be able to cut down the trained GPT-2 by a factor of 10 or 100 without degrading quality too badly. My belief is that even a 117M is more than big enough and expressive enough to generate great music, and the problems are architectural or training-related (optimizing for the wrong thing).
I did hope that starting with OA’s GPT-2-117M would make the English titles more interesting because the original model knows all sorts of names and objects, but it seems once you’ve trained it far enough to generate good ABC, it’s just largely memorized the titles. Too small a dataset to benefit from the pretraining. Oh well.
Yes.
Oh, my only real goal here was to see if switching to the RL setting could yield a qualitative improvement over likelihood training using the same data as the starting point, because RL seems to me philosophically the right way to approach the problem and likelihood fundamentally optimizing for the wrong thing. Aside from that, I am mildly curious how much a Transformer can improve over an older RNN.