A final transformation.

For my final project, I wanted to compare the performance of an LSTM, paired with an attention mechanism, with the performance of Google’s Transformer’s architecture on SemEval’s 2017 Semantic Textual Similarity (STS) task.

So far, I’m trying to use the learned embeddings of the RNN – encodings if you will – as inputs into the decoder of Tensor2Tensor’s (T2T). I’m currently running into issues combining both architectures. T2T’s transformer decode function states that the output of the decoder will be 3-D. However, in looking at the code, unless using XLA, the output is operated on by the ExpandDimensions op making them 4D! I wonder what other mismatches are in the code, causing the dimension mismatch issues I’ve been having!

However, I did figure out why my RNN was having near 100% accuracy! The associated labels were processed incorrectly making it easy for the model to learn them. Now that I’ve generalized my processing to multi-class classification on the STS data set, I’m getting roughly 57% accuracy when predicting.

I feel odd given that it’s 57% accuracy BUT 57% accuracy predicting among five classes is much better than 57% accuracy predicting among binary classes!

Interestingly my accuracy graph for train & validation are…odd to say the least

(train)

(validation)

Summer Vibes. Summer Thoughts.

This summer with OpenAI was truly amazing. In more ways than one. First, the scholars community has been truly supportive. Everyone was incredibly vested in our success – from Larissa to Greg to our mentors. As scholars, we shared jokes, cheered each other on, and were incredibly honest about our highs and lows. I applied to the program on a whim; I had an idea of what I wanted to pursue as I’d already been dabbling a bit and wanted time and support to see things through.

The main thing I learned was how important pictures are to a deeper understanding. While I have never been pictorially (?) gifted, my mentor encouraged me to draw out and explain the math and processes behind what I was doing. While I struggled with it, I found that the most useful resources, for me, weren’t online tutorials that went through the code. I was most helped by online blog posts drawing out what was going on. I also developed a much deeper appreciation for data generation and feature engineering!

Furthermore, I was able to play with different architectures and actually understand them and how they worked. I’ve always been the type to need to know how things are at their core – it’s part of why I’m an engineer. I’m so excited to continue exploring other models and other problems.

The most amazing thing I got out of this experience was the confidence to know that I could actually work in this space. For a while I struggled with a lack of understanding and someone I could go to and ask what to me seemed the most useless questions. But this program encouraged me to ask those questions and my mentor was such an amazing resource to not feel stupid. Going in, I knew that I wanted my final project to be a model with rather good performance on the SemEval STS Task, Track 4.

I’ve gone through different iterations of what my model will look like, from using LSTSMs to Google’s Transformers architecture and back again. Though I’m still working on the Transformer architecture – I wanted to do a comparison of performance – I’m amazed that I’ve really built and worked on an RNN. I’ve really gone through each part of the architecture, researched and read so many papers, and it’s empowering to feel as though I did a thing.

Link to my github (where you can find my paper)