With single tag predictions, we have achieved the following prediction accuracies with the baseline model (feature engineering and Random Forest) and the advanced model(Word2Vec+LSTM) as follows:
​
​
​
​
​
​
​
We have almost doubled accuracy with the more advanced methods to achieve a test accuracy of 38%. We can then look into the accuracy of each individual class, as reflected in the heat map.
​
​
​
​
​
​
​
​
​
​
​
​
​​
-
We observe relatively high true positive rate with most tags as we see the correlation on the diagonal is high.
-
“Love” appears to be the most confounding tag out of the entire set, and it is strongly correlated with emotional, memory, happy, relax and sad. That, is, our model has predicted many songs with the above tags to be love instead.
​
We might first be concerned with such misclassification, but if we think intuitively, it might not necessarily be an issue with our model. An emotional song, a happy or sad song, a memory song can always be about love. It could simply be an effect of the specific user tagging preference in how to describe the song. In this case, when it is possible that one song could match to multiple tags, we define a new prediction methodology. Instead of predicting only one single tag with the highest probability, we will choose the top 3 tags with the highest probabilities with our model, and if any one out of the three matches with the true tag, we will consider it to be a correct classification. In this case, we allow buffer for non-conflicting tags to be the top prediction - as long as the correct one also appears in top 3, we do not penalize the model prediction.
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
With the new prediction method, we can achieve 82% accuracy with the train set and 60% for test. Checking for the individual tag accuracy, we notice high accuracy for some themed tags like Christmas, love and religious, out of which we expect to have relatively special vocabulary within these categories. In contrast, the accuracies are relatively low for some mood tags including energetic and relax. It is highly possible that such emotions are less conveyed through lyrics, and thus our language based model is not as appropriate. Overall, we can conclude that lyrics do help in predicting themes and some moods with relatively high accuracy.
