Avi Singh's blog
https://avisingh599.github.io/
Deep Learning for Visual Question Answering<script type="text/javascript" src="//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<p><img src="/images/vqa/sample_results.jpg" alt="Teaser" /></p>
<p>In this blog post, I’ll talk about the <a href="http://www.visualqa.org">Visual Question Answering</a> problem, and I’ll also present neural network based approaches for same. The source code for this blog post is written in Python and <a href="http://keras.io">Keras</a>, and is available on <a href="http://github.com/avisingh599/visual-qa">Github</a>.</p>
<p>An year or so ago, a chatbot named <a href="https://en.wikipedia.org/wiki/Eugene_Goostman">Eugene Goostman</a> made it to the mainstream <a href="http://www.bbc.com/news/technology-27762088">news</a>, after having been reported as the first computer program to have passed the famed <a href="https://en.wikipedia.org/wiki/Turing_test">Turing Test</a> in an event organized at the University of Reading. While the organizers hailed it as a historical achievement, most of the scientific community wasn’t impressed. This leads us to the question: Is the Turing Test, in its original form, a suitable test for AI in the modern day?</p>
<p>In the last couple of years, a number of papers (like <a href="http://www.pnas.org/content/112/12/3618.abstract">this paper from JHU/Brown</a>, and <a href="http://arxiv.org/abs/1410.8027">this one from MPI</a>) have suggested that the task of Visual Question Answering (VQA, for short) can be used as an alternative Turing Test. The task involves answering an open-ended question (or a series of questions) about an image. An example is shown below:</p>
<h5 id="image-from-visualqaorg">Image from visualqa.org</h5>
<p><img src="/images/vqa/challenge.png" alt="Visual QA" /></p>
<p>The AI system needs to solve a number of sub-problems in Natural Language Processing and Computer Vision, in addition to being able to perform some kind of “common-sense” reasoning. It needs to localize the subject being referenced (the woman’s face, and more specifically the region around her lips), needs to detect objects (the banana), and should also have some common-sense knowledge that the word <em>mustache</em> is often used to refer to markings or objects on the face that are not actually mustaches (like milk mustaches). Since the problem cuts through two two very different modalities (vision and text), and requires high-level understanding of the scene, it appears to be an ideal candidate for a true Turing Test. The problem also has real world applications, like helping the <a href="https://itunes.apple.com/us/app/vizwiz/id439686043?mt=8">visually impaired</a>.</p>
<p>A few days ago, the <a href="http://visualqa.org/challenge.html">Visual QA Challenge</a> was launched, and along with it came a large dataset (~750K questions on ~250K images). After the <a href="http://mscoco.org/dataset/#captions-challenge2015">MS COCO Image Captioning Challenge</a> sparked a lot of interest in problem of <a href="https://pdollar.wordpress.com/2015/01/21/image-captioning/">image captioning</a> (or was it the interest that led to the challenge?), the time seems ripe to move onto a much harder problem at the intersection of NLP and Vision.</p>
<p>This post will present ways to model this problem using Neural Networks, exploring both Feedforward Neural Networks, and the much more exciting <strong>Recurrent Neural Networks</strong> (LSTMs, to be specific). If you do not know much about Neural Networks, then I encourage you to check these two awesome blogs: <a href="https://colah.github.io">Colah’s Blog</a> and <a href="https://karpathy.github.io">Karpathy’s Blog</a>. Specifically, check out the posts on Recurrent Neural Nets, Convolutional Neural Nets and LSTM Nets. The models in this post take inspiration from <a href="https://filebox.ece.vt.edu/~parikh/Publications/ICCV2015_VQA.pdf">this ICCV 2015 paper</a>, <a href="https://www.d2.mpi-inf.mpg.de/sites/default/files/iccv15-neural_qa.pdf">this ICCV 2015 paper</a>, and <a href="http://www.cs.toronto.edu/~mren/imageqa/">this NIPS 2015 paper</a>.</p>
<h2 id="generating-answers">Generating Answers</h2>
<p>An important aspect of solving this problem is to have a system that can generate new answers. While most of the answers in the VQA dataset are short (1-3 words), we would still like to a have a system that can generate arbitrarily long answers, keeping up with our spirit of the Turing test. We can perhaps take inspiration from papers on <a href="http://arxiv.org/abs/1409.3215">Sequence to Sequence Learning using RNNs</a>, that solve a similar problem when generating translations of arbitrary length. <a href="http://papers.nips.cc/paper/5411-a-multi-world-approach-to-question-answering-about-real-world-scenes-based-on-uncertain-input.pdf">Multi-word methods</a> have been presented for VQA too. However, for the purpose of this blog post, we will ignore this aspect of the problem. We will select the 1000 most frequent answers in the VQA training dataset, and solve the problem in a multi-class classification setting. These top 1000 answers cover over 80% of the answers in the VQA training set, so we can still expect to get reasonable results.</p>
<h2 id="the-feedforward-neural-model">The Feedforward Neural Model</h2>
<p><img src="/images/vqa/model_1.jpg" alt="The MLP Model" /></p>
<p>To get started, let’s first try to model the problem using a <a href="https://en.wikipedia.org/wiki/Multilayer_perceptron">MultiLayer Perceptron</a>. An MLP is a simple feedforward neural net that maps a feature vector (of fixed length) to an appropriate output. In our problem, this output will be a probability distribution over the set of possible answers. We will be using <a href="http://keras.io">Keras</a>, an awesome deep learning library based on <a href="http://deeplearning.net/software/theano/">Theano</a>, and written in Python. Setting up Keras is fairly easy, just have a look at their <a href="https://github.com/fchollet/keras#installation">readme</a> to get started.</p>
<p>In order to use the MLP model, we need to map all our input questions and images to a feature vector of fixed length. We perform the following operations to achieve this:</p>
<ol>
<li>For the question, we transform each word to its <a href="https://code.google.com/p/word2vec/">word vector</a>, and sum up all the vectors. The length of this feature vector will be same as the length of a single word vector, and the word vectors (also called embeddings) that we use have a length of <code class="highlighter-rouge">300</code>.</li>
<li>For the image, we pass it through a Deep Convolutional Neural Network (the well-known <a href="http://arxiv.org/abs/1409.1556">VGG Architecture</a>), and extract the activation from the second last layer (before the softmax layer, that is). Size of this feature vector is <code class="highlighter-rouge">4096</code>.</li>
</ol>
<p>Once we have generated the feature vectors, all we need to do now is to define a model in Keras, set up a cost function and an optimizer, and we’re good to go.
The following Keras code defines a multi-layer perceptron with two hidden layers, <code class="highlighter-rouge">1024</code> hidden units in each layer and dropout layers in the middle for regularization. The final layer is a softmax layer, and is responsible for generating the probability distribution over the set of possible answers. I have used the <code class="highlighter-rouge">categorical_crossentropy</code> loss function since it is a multi-class classification problem. The <code class="highlighter-rouge">rmsprop</code> method is used for optimzation. You can try experimenting with other optimizers, and see what kind of <a href="http://lossfunctions.tumblr.com/">learning curves</a> you get.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">keras.models</span> <span class="kn">import</span> <span class="n">Sequential</span>
<span class="kn">from</span> <span class="nn">keras.layers.core</span> <span class="kn">import</span> <span class="n">Dense</span><span class="p">,</span> <span class="n">Dropout</span><span class="p">,</span> <span class="n">Activation</span>
<span class="n">img_dim</span> <span class="o">=</span> <span class="mi">4096</span> <span class="c">#top layer of the VGG net</span>
<span class="n">word_vec_dim</span> <span class="o">=</span> <span class="mi">300</span> <span class="c">#dimension of pre-trained word vectors</span>
<span class="n">nb_hidden_units</span> <span class="o">=</span> <span class="mi">1024</span> <span class="c">#number of hidden units, a hyperparameter</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">()</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="n">nb_hidden_units</span><span class="p">,</span> <span class="n">input_dim</span><span class="o">=</span><span class="n">img_dim</span><span class="o">+</span><span class="n">word_vec_dim</span><span class="p">,</span>
<span class="n">init</span><span class="o">=</span><span class="s">'uniform'</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Activation</span><span class="p">(</span><span class="s">'tanh'</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dropout</span><span class="p">(</span><span class="mf">0.5</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="n">nb_hidden_units</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="s">'uniform'</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Activation</span><span class="p">(</span><span class="s">'tanh'</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dropout</span><span class="p">(</span><span class="mf">0.5</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="n">nb_classes</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="s">'uniform'</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Activation</span><span class="p">(</span><span class="s">'softmax'</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="nb">compile</span><span class="p">(</span><span class="n">loss</span><span class="o">=</span><span class="s">'categorical_crossentropy'</span><span class="p">,</span> <span class="n">optimizer</span><span class="o">=</span><span class="s">'rmsprop'</span><span class="p">)</span></code></pre></figure>
<p>Have a look at the <a href="https://github.com/avisingh599/visual-qa/blob/master/scripts/trainMLP.py">entire python script</a> to see the code for generating the features and training the network. It does not access the hard disk once the training begins, and uses about ~4GB of RAM. You can reduce memory usage by lowering the <code class="highlighter-rouge">batchSize</code> variable, but that would also lead to longer training times. It is able to process over 215K image-question pairs in less than <strong>160 seconds/epoch</strong> when working on a GTX 760 GPU with a batch size of 128. I ran my experiments for 100 epochs.</p>
<h2 id="the-recurrent-neural-model">The Recurrent Neural Model</h2>
<p><img src="/images/vqa/lstm_encoder.jpg" alt="The LSTM Model" align="middle" style="width: 500px;" /></p>
<p>A drawback of the previous approach is that we ignore the sequential nature of the questions. Regardless of what order the words appear in, we’ll get the same vector representing the question, à la <a href="https://en.wikipedia.org/wiki/Bag-of-words_model">bag-of-words (BOW)</a>. A way to tackle this limitation is by use of <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">Recurrent Neural Networks</a>, which are well-suited for sequential data. We’ll be using <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">LSTMs</a> here, since they avoid some common nuances of vanilla RNNs, and often give a slightly better performance. You can also experiment with other recurrent layers in Keras, such as <code class="highlighter-rouge">GRU</code>. The word vectors corresponding to the tokens in the question are passed to an LSTM in a sequential fashion, and the output of the LSTM (from its output gate) after all the tokens have been passed is chosen as the representation for the entire question. This fixed length vector is concatenated with the <code class="highlighter-rouge">4096</code> dimensional CNN vector for the image, and passed on to a multi-layer perceptron with fully connected layers. The last layer is once again softmax, and provides us with a probability distribution over the possible outputs.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">keras.models</span> <span class="kn">import</span> <span class="n">Sequential</span>
<span class="kn">from</span> <span class="nn">keras.layers.core</span> <span class="kn">import</span> <span class="n">Dense</span><span class="p">,</span> <span class="n">Activation</span><span class="p">,</span> <span class="n">Merge</span><span class="p">,</span> <span class="n">Dropout</span><span class="p">,</span> <span class="n">Reshape</span>
<span class="kn">from</span> <span class="nn">keras.layers.recurrent</span> <span class="kn">import</span> <span class="n">LSTM</span>
<span class="n">num_hidden_units_mlp</span> <span class="o">=</span> <span class="mi">1024</span>
<span class="n">num_hidden_units_lstm</span> <span class="o">=</span> <span class="mi">512</span>
<span class="n">img_dim</span> <span class="o">=</span> <span class="mi">4096</span>
<span class="n">word_vec_dim</span> <span class="o">=</span> <span class="mi">300</span>
<span class="n">image_model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">()</span>
<span class="n">image_model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Reshape</span><span class="p">(</span><span class="n">input_shape</span> <span class="o">=</span> <span class="p">(</span><span class="n">img_dim</span><span class="p">,),</span> <span class="n">dims</span><span class="o">=</span><span class="p">(</span><span class="n">img_dim</span><span class="p">,)))</span>
<span class="n">language_model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">()</span>
<span class="n">language_model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">LSTM</span><span class="p">(</span><span class="n">output_dim</span> <span class="o">=</span> <span class="n">num_hidden_units_lstm</span><span class="p">,</span>
<span class="n">return_sequences</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
<span class="n">input_shape</span><span class="o">=</span><span class="p">(</span><span class="n">max_len</span><span class="p">,</span> <span class="n">word_vec_dim</span><span class="p">)))</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">()</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Merge</span><span class="p">([</span><span class="n">language_model</span><span class="p">,</span> <span class="n">image_model</span><span class="p">],</span>
<span class="n">mode</span><span class="o">=</span><span class="s">'concat'</span><span class="p">,</span> <span class="n">concat_axis</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="n">num_hidden_units_mlp</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="s">'uniform'</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Activation</span><span class="p">(</span><span class="s">'tanh'</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dropout</span><span class="p">(</span><span class="mf">0.5</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="n">num_hidden_units_mlp</span><span class="p">,</span> <span class="n">init</span><span class="o">=</span><span class="s">'uniform'</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Activation</span><span class="p">(</span><span class="s">'tanh'</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dropout</span><span class="p">(</span><span class="mf">0.5</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="n">nb_classes</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Activation</span><span class="p">(</span><span class="s">'softmax'</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="nb">compile</span><span class="p">(</span><span class="n">loss</span><span class="o">=</span><span class="s">'categorical_crossentropy'</span><span class="p">,</span> <span class="n">optimizer</span><span class="o">=</span><span class="s">'rmsprop'</span><span class="p">)</span></code></pre></figure>
<p>A single <code class="highlighter-rouge">train_on_batch</code> method call in Keras expects the sequences to be of the same length (so that is can be represented as a Theano Tensor). There has been a lot of discussion regarding training LSTMs with variable length sequences, and I used the following technique: Sorted all the questions by their length, and then processed them in batches of <code class="highlighter-rouge">128</code> while training. Most batches had questions of the same length (say 9 or 10 words), and there was no need of zero-padding. For the few batched that did have questions of varying length, the shorter questions were zero-padded. I was able to achieve a training speed of <strong>200 seconds/epoch</strong> on a GTX 760 GPU.</p>
<h2 id="show-me-the-numbers">Show me the numbers</h2>
<p>I trained my system on the Training Set of the VQA dataset, and evaluated performance on the validation set, following the rules of the VQA challenge. The answer produced by the Neural Net is checked against every answer provided by humans (there are ten human answers for every question). If the answer produced by the neural net <em>exactly</em> matches <em>at least</em> three of the ten answers, then we classify it as a correct prediction. Here is the performance of the models that I trained:</p>
<table>
<thead>
<tr>
<th>Model</th>
<th style="text-align: center">Accuracy</th>
</tr>
</thead>
<tbody>
<tr>
<td>BOW+CNN</td>
<td style="text-align: center">48.46%</td>
</tr>
<tr>
<td>LSTM-Language only</td>
<td style="text-align: center">44.17%</td>
</tr>
<tr>
<td>LSTM+CNN</td>
<td style="text-align: center">51.63%</td>
</tr>
</tbody>
</table>
<p><strong>Update</strong>: The results that I reported earlier were based on a metric slightly different from the ones used on VQA. They have since been updated. Also, I was able to obtain a performance of <strong>53.34%</strong> on the test-dev set (LSTM+CNN), which is practically the same as those set by the VQA authors in their LSTM baseline.</p>
<p>It’s interesting to see that even a “blind” model is able to obtain an accuracy of 44.17%. This shows that the model is pretty good at guessing the answers once it has identified the type of question. The LSTM+CNN model shows an improvement of about 3% as compared to the Feedforward Model (BOW+CNN), which tells us that the temporal structure of the question is indeed helpful. These results are in line with what was obtained in the <a href="http://www.visualqa.org/VQA_ICCV2015.pdf">original VQA paper</a>. However, the results reported in the paper were on the <em>test</em> set (trained on train+val), while we have evaluated on the <em>validation</em> set (trained on only train). If we learn a model on both the training and the validation data, then we can expect a significant improvement in performance since the number of training examples will increase by 50%. Finally, there is a lot of scope for hyperparameter tuning (number of hidden units, number of MLP hidden layers, number of LSTM layers, dropout or no dropout etc.).</p>
<p>I carried out my experiments for 100 epochs<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>, and observed the following curve:</p>
<p><img src="/images/vqa/learning_curve.jpg" alt="Validation Accuracy with number of epochs" /></p>
<p>The LSTM+CNN model flattens out in performance after about 50 epochs. The BOW+CNN also showed similar behavior, but took a surprising dive at epoch 90, which was soon rectified by the 100th epoch. I’ll probably re-initialize and run the models for 500 epochs, and see if such behavior is seen again or not. <strong>Update</strong>: I did run it once more, and the dip was not observed!</p>
<h3 id="a-note-on-word-embeddings">A note on word embeddings</h3>
<p>We have a number of choices when using word embeddings, and I experimented with three of them:</p>
<ol>
<li>
<p><a href="http://nlp.stanford.edu/projects/glove/">GloVe Word Embeddings</a> trained on the common-crawl: These gave the best performance, and all results reported here are using these embeddings.</p>
</li>
<li>
<p><a href="https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/">Goldberg and Levy 2014</a>: These are the default embeddings that come with <a href="http://spacy.io/">spaCy</a>, and they gave significantly worse results.</p>
</li>
<li>
<p>Embeddings Trained on the VQA questions: I used <a href="https://radimrehurek.com/gensim/models/word2vec.html">Gensim’s word2vec</a> implementation to train my own embeddings on the questions in the training set of the VQA dataset. The performance was similar to, but slighly worse than the GloVe embeddings. This is primarily because the VQA training set alone is not sufficiently large (~2.5m words) to get reasonable word vectors, especially for less common words.</p>
</li>
</ol>
<h3 id="link-to-github-repo"><a href="https://github.com/avisingh599/visual-qa">Link to github repo</a></h3>
<hr />
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Validation was done once per 10 epochs for BOW+CNN, once every 5 epochs for LSTMs. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Mon, 02 Nov 2015 00:00:00 +0000
https://avisingh599.github.io/deeplearning/visual-qa/
https://avisingh599.github.io/deeplearning/visual-qa/Computer Vision for Predicting Facial Attractiveness<p><a href="http://www.learnopencv.com/computer-vision-for-predicting-facial-attractiveness/">Click here to go to the post.</a></p>
Mon, 27 Jul 2015 00:00:00 +0000
https://avisingh599.github.io/predicting-facial-attractiveness/
https://avisingh599.github.io/predicting-facial-attractiveness/Monocular Visual Odometry using OpenCV<script type="text/javascript" src="//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<p>Last month, I made a <a href="/vision/visual-odometry-full/">post</a> on Stereo Visual Odometry and its implementation in MATLAB.
This post would be focussing on <strong>Monocular Visual Odometry</strong>, and how we can implement it in <strong>OpenCV/C++</strong>.
The implementation that I describe in this post is once again freely available on <a href="https://github.com/avisingh599/mono-vo">github</a>.
It is also simpler to understand, and runs at 5fps, which is much faster than my older stereo implementation.</p>
<p>If you are new to Visual Odometry, I suggest having a look at the first few paragraphs (before all the math starts) of my
<a href="/vision/visual-odometry-full/">old post</a>. It talks about what Visual Odometry is, why we
need it, and also compares the monocular and stereo approaches.</p>
<p>Acquanted with all the basics of visual odometry? Cool. Let’s go ahead.</p>
<h2 id="demo">Demo</h2>
<p>Before I move onto describing the implementation, have a look at the algorithm in action!</p>
<style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style>
<div class="embed-container"><iframe src="https://www.youtube.com/embed/homos4vd_Zs" frameborder="0" allowfullscreen=""></iframe></div>
<p>Pretty cool, eh? Let’s dive into implementing it in OpenCV now.</p>
<h3 id="formulation-of-the-problem">Formulation of the problem</h3>
<h4 id="input">Input</h4>
<p>We have a stream of gray scale images coming from a camera. Let the frames, captured at time <script type="math/tex">t</script> and <script type="math/tex">t+1</script> be referred to as
<script type="math/tex">\mathit{I}^{t}</script>, <script type="math/tex">\mathit{I}^{t+1}</script>. We have prior knowledge of all the intrinsic parameters, obtained via calibration,
which can also be done in <a href="http://docs.opencv.org/3.0.0/d9/d0c/group__calib3d.html">OpenCV</a>.</p>
<h4 id="output">Output</h4>
<p>For every pair of images, we need to find the rotation matrix <script type="math/tex">R</script> and the translation vector <script type="math/tex">t</script>, which describes the motion of the vehicle between the two frames. The vector <script type="math/tex">t</script> can only be computed upto a scale factor in our monocular scheme.</p>
<h3 id="algorithm-outline">Algorithm Outline</h3>
<ol>
<li>Capture images: <script type="math/tex">\mathit{I}^t</script>, <script type="math/tex">\mathit{I}^{t+1}</script>,</li>
<li>Undistort the above images.</li>
<li>Use FAST algorithm to detect features in <script type="math/tex">\mathit{I}^t</script>, and track those features to <script type="math/tex">{I}^{t+1}</script>. A new detection is triggered if the number of features drop below a certain threshold.</li>
<li>Use Nister’s 5-point alogirthm with RANSAC to compute the essential matrix.</li>
<li>Estimate <script type="math/tex">R, t</script> from the essential matrix that was computed in the previous step.</li>
<li>Take scale information from some external source (like a speedometer), and concatenate the translation vectors, and rotation matrices.</li>
</ol>
<p>You may or may not understand all the steps that have been metioned above, but don’t worry. All the points
above will be explained in great detail in the text to follow.</p>
<h3 id="undistortion">Undistortion</h3>
<p>Distortion happens when lines that are straight in the real world become curved in the images. T
his step compensates for this lens distortion. It is performed with the help of the distortion parameters
that were obtained during calibration. Since the KITTI dataset that I’m using already comes with
undistorted images, I won’t write the code about it here. However, it is relatively straightforward to
<a href="http://docs.opencv.org/modules/imgproc/doc/geometric_transformations.html#undistort">undistort</a> with OpenCV.</p>
<h3 id="feature-detection">Feature Detection</h3>
<p>My approach uses the FAST corner detector, just like my stereo implementation. I’ll now explain in brief how the detector works, though you must have a look at the <a href="http://www.edwardrosten.com/work/fast.html">original paper and source code</a> if you want to really understand how it works. Suppose there is a point <script type="math/tex">\mathbf{P}</script> which we want to test if it is a corner or not. We draw a circle of 16px circumference around this point as shown in figure below. For every pixel which lies on the circumference of this circle, we see if there exits a continuous set of pixels whose intensity exceed the intensity of the original pixel by a certain factor <script type="math/tex">\mathbf{I}</script> and for another set of contiguous pixels if the intensity is less by at least the same factor <script type="math/tex">\mathbf{I}</script>. If yes, then we mark this point as a corner. A heuristic for rejecting the vast majority of non-corners is used, in which the pixel at 1,9,5,13 are examined first, and atleast three of them must have a higher intensity be amount at least <script type="math/tex">\mathbf{I}</script>, or must have an intensity lower by the same amount <script type="math/tex">\mathbf{I}</script> for the point to be a corner. This particular approach is selected due to its computational efficiency as compared to other popular interest point detectors such as SIFT.</p>
<figure>
<img src="/images/visodo/fast.png" />
<figcaption>Image from the original FAST feature detection paper</figcaption>
</figure>
<p>Using OpenCV, detecting features is trivial, and here is the code that does it.</p>
<figure class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="kt">void</span> <span class="nf">featureDetection</span><span class="p">(</span><span class="n">Mat</span> <span class="n">img_1</span><span class="p">,</span> <span class="n">vector</span><span class="o"><</span><span class="n">Point2f</span><span class="o">>&</span> <span class="n">points1</span><span class="p">)</span> <span class="p">{</span>
<span class="n">vector</span><span class="o"><</span><span class="n">KeyPoint</span><span class="o">></span> <span class="n">keypoints_1</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">fast_threshold</span> <span class="o">=</span> <span class="mi">20</span><span class="p">;</span>
<span class="kt">bool</span> <span class="n">nonmaxSuppression</span> <span class="o">=</span> <span class="nb">true</span><span class="p">;</span>
<span class="n">FAST</span><span class="p">(</span><span class="n">img_1</span><span class="p">,</span> <span class="n">keypoints_1</span><span class="p">,</span> <span class="n">fast_threshold</span><span class="p">,</span> <span class="n">nonmaxSuppression</span><span class="p">);</span>
<span class="n">KeyPoint</span><span class="o">::</span><span class="n">convert</span><span class="p">(</span><span class="n">keypoints_1</span><span class="p">,</span> <span class="n">points1</span><span class="p">,</span> <span class="n">vector</span><span class="o"><</span><span class="kt">int</span><span class="o">></span><span class="p">());</span>
<span class="p">}</span></code></pre></figure>
<p>The parameters in the code above are set such that it gives ~4000 features on one image from the KITTI dataset. You may want
tune these parameters so as to obtain the best performance on your own data.
Note that the code above also converts the datatype of the detected feature points from KeyPoints to a vector of Point2f, so
that we can directly pass it to the feature tracking step, described below:</p>
<h3 id="feature-tracking">Feature Tracking</h3>
<p>The fast corners detected in the previous step are fed to the next step, which uses a <a href="https://www.ces.clemson.edu/~stb/klt/">KLT tracker</a>. The KLT tracker basically looks around every corner to be tracked, and uses this local information to find the corner in the next image. You are welcome to look into the KLT link to know more. The corners detected in <script type="math/tex">\mathit{I}^{t}</script> are tracked in <script type="math/tex">\mathit{I}^{t+1}</script>. Let the set of features detected in <script type="math/tex">\mathit{I}^{t}</script> be <script type="math/tex">\mathcal{F}^{t}</script> , and the set of corresponding features in <script type="math/tex">\mathit{I}^{t+1}</script> be <script type="math/tex">\mathcal{F}^{t+1}</script>. Here is the function that does feature tracking in OpenCV using the KLT tracker:</p>
<figure class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="kt">void</span> <span class="nf">featureTracking</span><span class="p">(</span><span class="n">Mat</span> <span class="n">img_1</span><span class="p">,</span> <span class="n">Mat</span> <span class="n">img_2</span><span class="p">,</span> <span class="n">vector</span><span class="o"><</span><span class="n">Point2f</span><span class="o">>&</span> <span class="n">points1</span><span class="p">,</span> <span class="n">vector</span><span class="o"><</span><span class="n">Point2f</span><span class="o">>&</span> <span class="n">points2</span><span class="p">,</span> <span class="n">vector</span><span class="o"><</span><span class="n">uchar</span><span class="o">>&</span> <span class="n">status</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">//this function automatically gets rid of points for which tracking fails
</span>
<span class="n">vector</span><span class="o"><</span><span class="kt">float</span><span class="o">></span> <span class="n">err</span><span class="p">;</span>
<span class="n">Size</span> <span class="n">winSize</span><span class="o">=</span><span class="n">Size</span><span class="p">(</span><span class="mi">21</span><span class="p">,</span><span class="mi">21</span><span class="p">);</span>
<span class="n">TermCriteria</span> <span class="n">termcrit</span><span class="o">=</span><span class="n">TermCriteria</span><span class="p">(</span><span class="n">TermCriteria</span><span class="o">::</span><span class="n">COUNT</span><span class="o">+</span><span class="n">TermCriteria</span><span class="o">::</span><span class="n">EPS</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mf">0.01</span><span class="p">);</span>
<span class="n">calcOpticalFlowPyrLK</span><span class="p">(</span><span class="n">img_1</span><span class="p">,</span> <span class="n">img_2</span><span class="p">,</span> <span class="n">points1</span><span class="p">,</span> <span class="n">points2</span><span class="p">,</span> <span class="n">status</span><span class="p">,</span> <span class="n">err</span><span class="p">,</span> <span class="n">winSize</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="n">termcrit</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mf">0.001</span><span class="p">);</span>
<span class="c1">//getting rid of points for which the KLT tracking failed or those who have gone outside the frame
</span> <span class="kt">int</span> <span class="n">indexCorrection</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">for</span><span class="p">(</span> <span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o"><</span><span class="n">status</span><span class="p">.</span><span class="n">size</span><span class="p">();</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span>
<span class="p">{</span> <span class="n">Point2f</span> <span class="n">pt</span> <span class="o">=</span> <span class="n">points2</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="n">i</span><span class="o">-</span> <span class="n">indexCorrection</span><span class="p">);</span>
<span class="k">if</span> <span class="p">((</span><span class="n">status</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span><span class="o">||</span><span class="p">(</span><span class="n">pt</span><span class="p">.</span><span class="n">x</span><span class="o"><</span><span class="mi">0</span><span class="p">)</span><span class="o">||</span><span class="p">(</span><span class="n">pt</span><span class="p">.</span><span class="n">y</span><span class="o"><</span><span class="mi">0</span><span class="p">))</span> <span class="p">{</span>
<span class="k">if</span><span class="p">((</span><span class="n">pt</span><span class="p">.</span><span class="n">x</span><span class="o"><</span><span class="mi">0</span><span class="p">)</span><span class="o">||</span><span class="p">(</span><span class="n">pt</span><span class="p">.</span><span class="n">y</span><span class="o"><</span><span class="mi">0</span><span class="p">))</span> <span class="p">{</span>
<span class="n">status</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">points1</span><span class="p">.</span><span class="n">erase</span> <span class="p">(</span><span class="n">points1</span><span class="p">.</span><span class="n">begin</span><span class="p">()</span> <span class="o">+</span> <span class="n">i</span> <span class="o">-</span> <span class="n">indexCorrection</span><span class="p">);</span>
<span class="n">points2</span><span class="p">.</span><span class="n">erase</span> <span class="p">(</span><span class="n">points2</span><span class="p">.</span><span class="n">begin</span><span class="p">()</span> <span class="o">+</span> <span class="n">i</span> <span class="o">-</span> <span class="n">indexCorrection</span><span class="p">);</span>
<span class="n">indexCorrection</span><span class="o">++</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<h4 id="feature-re-detection">Feature Re-Detection</h4>
<p>Note that while doing KLT tracking, we will eventually lose some points (as they move out of the field of view of the car), and
we thus trigger a redetection whenver the total number of features go below a certain threshold (2000 in my implementation).</p>
<h3 id="essential-matrix-estimation">Essential Matrix Estimation</h3>
<p>Once we have point-correspondences, we have several techniques for the computation of an essential matrix. The essential matrix is defined as follows:
<script type="math/tex">\begin{equation}
y_{1}^{T}Ey_{2} = 0
\end{equation}</script>
Here, <script type="math/tex">y_{1}</script>, <script type="math/tex">y_{2}</script> are homogenous normalised image coordinates.
While a simple algorithm requiring eight point correspondences exists\cite{Higgins81}, a more recent approach that is shown to give better results is the five point algorithm<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>. It solves a number of non-linear equations, and requires the minimum number of points possible, since the Essential Matrix has only five degrees of freedom.</p>
<h4 id="ransac">RANSAC</h4>
<p>If all of our point correspondences were perfect, then we would have need only
five feature correspondences between two successive frames to estimate motion accurately.
However, the feature tracking algorithms are not perfect, and therefore we have several
erroneous correspondence. A standard technique of handling outliers when doing model estimation
is RANSAC. It is an iterative algorithm. At every iteration, it randomly samples five
points from out set of correspondences, estimates the Essential Matrix, and then checks
if the other points are inliers when using this essential matrix. The algorithm terminates
after a fixed number of iterations, and the Essential matrix with which the maximum number of points agree, is used.</p>
<p>Using the above in OpenCV is again pretty straightforward, and all you need is one line:</p>
<figure class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="n">E</span> <span class="o">=</span> <span class="n">findEssentialMat</span><span class="p">(</span><span class="n">points2</span><span class="p">,</span> <span class="n">points1</span><span class="p">,</span> <span class="n">focal</span><span class="p">,</span> <span class="n">pp</span><span class="p">,</span> <span class="n">RANSAC</span><span class="p">,</span> <span class="mf">0.999</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="n">mask</span><span class="p">);</span></code></pre></figure>
<h3 id="computing-r-t-from-the-essential-matrix">Computing R, t from the Essential Matrix</h3>
<p>Another definition of the Essential Matrix (consistent) with the definition mentioned earlier is as follows:
<script type="math/tex">\begin{equation}
E = R[t]_{x}
\end{equation}</script>
Here, <script type="math/tex">R</script> is the rotation matrix, while <script type="math/tex">[t]_{x}</script> is the matrix representation of a cross product with <script type="math/tex">t</script>. Taking the SVD of the essential matrix, and then exploiting the constraints on the rotation matrix, we get the following:</p>
<script type="math/tex; mode=display">E = U\Sigma V^{T}</script>
<script type="math/tex; mode=display">[t]_{x} = VW\Sigma V^{T}</script>
<script type="math/tex; mode=display">R = UW^{-1}V^{T}</script>
<p>Here’s the one-liner that implements it in OpenCV:</p>
<figure class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="n">recoverPose</span><span class="p">(</span><span class="n">E</span><span class="p">,</span> <span class="n">points2</span><span class="p">,</span> <span class="n">points1</span><span class="p">,</span> <span class="n">R</span><span class="p">,</span> <span class="n">t</span><span class="p">,</span> <span class="n">focal</span><span class="p">,</span> <span class="n">pp</span><span class="p">,</span> <span class="n">mask</span><span class="p">);</span></code></pre></figure>
<h3 id="constructing-trajectory">Constructing Trajectory</h3>
<p>Let the pose of the camera be denoted by <script type="math/tex">R_{pos}</script>, <script type="math/tex">t_{pos}</script>. We can then track the trajectory using the following equation:</p>
<script type="math/tex; mode=display">R_{pos} = R R_{pos}</script>
<script type="math/tex; mode=display">t_{pos} = t_{pos} + t R_{pos}</script>
<p>Note that the scale information of the translation vector <script type="math/tex">t</script> has to be obtained from some other source before concatenating.
In my implementation, I extract this information from the ground truth that is supplied by the KITTI dataset.</p>
<h3 id="heuristics">Heuristics</h3>
<p>Most Computer Vision algorithms are not complete without a few heuristics thrown in, and Visual Odometry is not an exception. The
heuristive that we use is explained below:</p>
<h4 id="dominant-motion-is-forward">Dominant Motion is Forward</h4>
<p>The entire visual odometry algorithm makes the assumption that most of the points in its environment are rigid. However, if we are in a scenario where the vehicle is at a stand still, and a buss passes by (on a road intersection, for example), it would lead the algorithm to believe that the car has moved sideways, which is physically impossible. As a result, if we ever find the translation is dominant in a direction other than forward, we simply ignore that motion.</p>
<h2 id="results">Results</h2>
<p>So, how good is the performance of the algorithm on the KITTI dataset? See for yourself.</p>
<figure>
<img src="/images/visodo/2K.png" />
<figcaption> Computed Trajectory vs Ground Truth for 2000 frames</figcaption>
</figure>
<h2 id="what-next">What next?</h2>
<p>A major limitation of my implementation is that it cannot evaluate relative scale. I did try implementing some methods, but I
encountered the problem which is known as “scale drift” i.e. small errors accumulate, leading to bad odometry estimates.
I hope I’ll soon implement a more robust relative scale computation pipeline, and write a post about it!</p>
<hr />
<div class="footnotes">
<ol>
<li id="fn:1">
<p>David Nister An efficient solution to the five-point relative pose problem (2004) <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Mon, 08 Jun 2015 00:00:00 +0000
https://avisingh599.github.io/vision/monocular-vo/
https://avisingh599.github.io/vision/monocular-vo/Recognizing Human Activities with Kinect - The implementation<p><em>Disclaimer: The work described in this post was done by me and my classmate at IIT-Kanpur, Ankit Goyal. <a href="/assets/activity-classification.pdf">Here</a> is a link to the presentation that we gave.</em></p>
<p>This is a follow up of my earlier <a href="/machinelearning/classifying-human-activities-kinect/">post</a>, in which I explored
temporal models, that can be applied to things like part-of-speech tagging, gesture recognition, and any sequential
or temporal sources of data in general. In this post, I will describe in more detail the implementation of our
project that classified RGBD videos according to the activity being performed in them.</p>
<h3 id="dataset">Dataset</h3>
<p>Quite a few <a href="http://research.microsoft.com/en-us/um/people/zliu/ActionRecoRsrc/">RGBD datasets</a>
are available for human activity detection/classification, and we chose to use the
MSR Daily Activity 3D dataset. Since we had limited computational resources (the mathserver of IITK),
and a limited time before the submission deadline, we chose to use a subset of the above dataset,
and worked with only 6 activities. So, our problem was now reduced to 6-class classification.</p>
<h3 id="features">Features</h3>
<p>In any machine learning problem, your model or learning algorithm is useless without a good set of
features. I read a <a href="http://www.sciencedirect.com/science/article/pii/S0167865514001299">recent paper</a> which
had a decent review of the various features used. They were:</p>
<ol>
<li>3D silhouettes - Finding the outline of the human body, and using the shape of this outline as features.</li>
<li>Skeletal joints or body part tracking - Kinect comes with an algorithm to determine the pose of the body
from the depth image alone. Pose here refers to the 3D coordinates of 15 body joints.</li>
<li>Local Spatio-temporal features - Just like some 2D/3D image feature detector, but with the added dimension of time.</li>
<li>Local 3D occupancy features - This one seemed the most interesting. What this does is to treat an RGBD video as
a function I(x, y, z, t). Now, this a very sparse function, and would be zero at most points in a 4D space. But,
whenever a certain activity is performed, certain regions of this 4D space will become filled. Inferring from
such data is now a matter sampling it efficiently, and this where all the innovation must lie, if this technique
is to work.</li>
<li>3D optical flow - The 3D counter part of the popular <a href="http://en.wikipedia.org/wiki/Optical_flow">optical flow</a>,
it is also known as [Scene Flow] in the academic literature. <a href="http://www.sciencedirect.com/science/article/pii/S1077314210001748">This</a>
is one paper that makes use of these features.</li>
</ol>
<p>The features that we ultimately went ahead were the skeletal joints. The MSR Daily Activity 3D dataset already provides
the skeletal joint coordinates to us, so all we had to was to take that data, and do some basic pre-processing on it.</p>
<h4 id="preprocessing-the-features">Preprocessing the features.</h4>
<p>The dataset provides us with the 3D coordinates of 15 human body joints. These cordinates are in the frame of reference of the Kinect.
The first operation that we perform on them is the following: to transform the points from the Kinect reference frame to the frame
of the person. By frame of the person, we refer to the joint corresponding to the torso.</p>
<p>Next thing that we do is what we call “body size normalization”. Basically all the body lengths, such as the distance between the elbo and
hand, are scaled up or down to a standard body size. This ensures that the variation in bosy sizes is captured at the feature level itself,
and our model does not have to worry about it anymore.</p>
<p><a href="https://gist.github.com/avisingh599/73ac41db59d87115c99e">Clicke here</a> to get the MATLAB code that does the feature extraction part from skeleton files that were obtained from the MSR dataset.</p>
<h3 id="model">Model</h3>
<p>Now, as I discussed in my <a href="/machinelearning/classifying-human-activities-kinect/">previous post</a>, Hidden Conditional
Random Fields (HCRFs) was the model that we finally selected. The original authors had released a well documented
<a href="http://sourceforge.net/projects/hcrf/">toolbox</a>, to which we directly fed the features that were computed above.</p>
<h3 id="results">Results</h3>
<p>Five-fold cross-validation without any hyper-parameter tuning yielded a precision of 71%. These results do not seem impressive
on first glance, but it must be noted that all our experiments were performed in the “new person” setting i.e. the person in the
test set did not appear in the training set, and we did not do any hyper parameter tuning. Our results can be summarised in the
ollowing heatmap:</p>
<figure>
<img src="/images/kinect_activity/heatmap.bmp" />
<figcaption>Where the algorithm succeeds and fails</figcaption>
</figure>
<p>The above figure made one thing clear: that accuracy is being seriously harmed by the algorithm’s inability
to correctly distinguish between drinking and talking on phone. The reason for this is relatively simple.
The features that we are using are skeletal features, and therefore we do not pay any attention to what
objects the human is interacting with. If you look at the skelat stream, talking on the phone, and drinking
water seem extrmemly similar! In both the cases, the human raises a hand, and brings it near his head.
Thus, in order to make a truly useful activity detection system, it is important to model these interactions
explicitly.</p>
<p>If we do get around to improving this model, I will post it here.</p>
Tue, 02 Jun 2015 00:00:00 +0000
https://avisingh599.github.io/machinelearning/classifying-human-activities-kinect-2/
https://avisingh599.github.io/machinelearning/classifying-human-activities-kinect-2/Recognizing Human Activities with Kinect - Choosing a temporal model<p><em>Update: I have posted the sequel to this post <a href="/machinelearning/classifying-human-activities-kinect-2/">here</a></em></p>
<p>In this blog post, I will very briefly talk about some popular models used for <strong>temporal/sequence classification</strong>,
their advantages/disadvantages, which one I used for my human activity recognition project, and why.
This post is intended for people who would like to delve into sequence classification, but don’t know where to start.
I plan to follow up on this post with another post that explains in detail our implementation of recognizing human
activities from RGBD data. However, if you want to have a look at it now,
<a href="/assets/activity-classification.pdf">here</a> are the slides.</p>
<p>In one my graduate-level course <strong>Machine Learning for Computer Vision</strong>, we were asked to select
a research paper to review and present. We selected the paper
<a href="http://www.cs.cornell.edu/~jysung/paper/unstructured_human_activity_learning.pdf">Unstructured Human Activity Detection from RGBD Images</a>.
Our reasons for this selection were several: it was fairly recent (2012), had a large number of citations (according to google scholar, at least), and it dealt with sequential data (RGBD videos). Temporal models, or sequence classification, was
something that was not covered in our course, and so we were eager to explore this area of Machine Learning.
We read the paper, made a <a href="/assets/activity-poster.pdf">poster</a> out of it, and presented it to our peers, TAs and the professor.</p>
<p>The next part of the course was more interesting, and it involved us picking up a Machine Learning problem, and
we then had the option of either implementing an existing approach to the problem, or we could come with our
own approach to solve it. We could have implemented the paper that we reviewed, but it seemed to more interesting
to have a look at the models available for sequence classification, and then use one
such model for our problem.</p>
<p>So we started looking around, and found that that following three models (and their variations)
seem to be the most popular:</p>
<ol>
<li>Hidden Markov Models (HMMs)</li>
<li>Maximum Entropy Markov Models (MEMMs)</li>
<li>Conditional Random Fields (CRFs)</li>
</ol>
<p>Here’s the very basic intuition about temporal models: Suppose you are reading some text character by character. The first
character that you observe is an “i”. Now, what do you think are the chances of you observing another “i”. Pretty slim, right?
This is because consecutive “i” are pretty rare while reading english text. Modeling such probabilistic relationships
in a mathematical form is precisely why we use temporal models, instead of just using some regular classifier (such as
logistic regression). There’s two more popular models for sequential classification (or structured prediction, as some people
like to call it), and they are: 1) <strong>Structural SVM</strong>, 2) <strong>Recurrent Neural Nets</strong>. I won’t talk about for either of them,
as I have not used them, but you are welcome to check them out.</p>
<p>Hidden Markov Models are the oldest, and have been used in things like speech-to-text since the 1960s. MEMMs came
around in 2000, only to be followed (and overshadowed) by Conditional Random Fields an year later. Both MEMMs and CRF
came from the <a href="http://people.cs.umass.edu/~mccallum/">Andrew McCallum’s research group</a>, and were focused on <a href="http://en.wikipedia.org/wiki/Natural_language_processing">Natural Language Processing</a> tasks.
However, once you have extracted features from sequential data, you can use these models as long as your features
satisfy the assumptions made by these models. Note that all of these models are special cases of
<a href="http://en.wikipedia.org/wiki/Graphical_model">probabilistic graphical models</a>, so all the inference and learning algorithms from
there can directly be applied here.</p>
<h3 id="hidden-markov-models">Hidden Markov Models</h3>
<figure>
<img img="" height="155" width="410" src="/images/kinect_activity/hmm.png" />
<figcaption>Graphical Model Representation of a stack of HMMs</figcaption>
</figure>
<p>As I mentioned earlier, Hidden Markov Models have been around for a long time, and were heavily used by the speech processing community.
I won’t much into the details/code of HMMs, as there are a large number of resources that describe the topic, targeted both at
<a href="http://www.comp.leeds.ac.uk/roger/HiddenMarkovModels/html_dev/main.html">beginners</a> and those who want to go into all the
<a href="http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/tutorial%20on%20hmm%20and%20applications.pdf">details</a>.
HMMs are <a href="http://en.wikipedia.org/wiki/Generative_model"><strong>generative models</strong></a>, and efficient dynamic programming algorithms
are available for both training and inference. The models uses <strong>hidden states</strong>, and assumes that the <strong>observed states</strong> are independent of each other, given their hidden states. A common way to go about doing classification with HMMS is the following: Train an HMM
for every class, and then for every new example, find the probability of that example being generated by each HMM, the HMM that gives the
maximum probability is your final class.</p>
<p>However, with HMMs come a number of disadvantages, with the major ones being:</p>
<ol>
<li>Requires enumeration of all possible observation sequences.</li>
<li>Requires the observations to be independent of each other (given the hidden state).</li>
<li>Generative approach for solving a conditional problem leading to unnecessary computations.</li>
</ol>
<h3 id="maximum-entropy-markov-models">Maximum Entropy Markov Models</h3>
<p>So, let’s move onto a new model, which, in theory, solves all of the above problems: MEMMs.
MEMMs were introduced in 2000, and were at that time used in NLP tasks, and showed
improvements in tasks where assumption [2] mentioned above was not true. MEMMs are discriminative models, so
they also do away with problems [1] and [3]. There’s also a hierarchical version of the same model,
and a Hierarchical MEMM is what was used in the <a href="http://www.cs.cornell.edu/~jysung/paper/unstructured_human_activity_learning.pdf">paper</a>
that we reviewed. The paper contains an interesting way of selecting graph structure, and I recommend checking it out.</p>
<figure>
<img src="/images/kinect_activity/memm.png" />
<figcaption>Graphical Representation of an MEMM. Note how the direction of arrow from observation to hidden state has been reversed.
</figcaption>
</figure>
<p>But along with MEMMs comes it’s own problem, commonly called as the label-bias problem.</p>
<h4 id="label-bias-problem">Label bias problem</h4>
<ol>
<li>States with low-entropy transition distributions ”effectively ignore” their observations. States with lower transitions have ”unfair advantage”.</li>
<li>Since training is always done with respect to known previous tags, so the model struggles at test time when there is uncertainty in the previous tag.</li>
</ol>
<p>It is impossible to understand the above without some background on what MEMMs are, so it is advisable
to first look at <a href="http://courses.ischool.berkeley.edu/i290-dm/s11/SECURE/gidofalvi.pdf">how MEMMs work</a>
, and then at the original <a href="http://www.cs.columbia.edu/~jebara/6772/papers/crf.pdf">CRF paper</a>
which talks about the label bias problem.</p>
<h3 id="conditional-random-fields---star-of-the-show">Conditional Random Fields -> Star of the show</h3>
<figure>
<img img="" height="155" width="410" src="/images/kinect_activity/crf.png" />
<figcaption>Graphical Representation of a CRF. Note that this an undirected graphical model, as opposed to HMM/MEMM</figcaption>
</figure>
<p>To overcome the label-bias problem of MEMMs, CRFs were introduced an year later, and demonstrated superior or
equivalent performance in almost every NLP task that the authors tested it on. CRFs (and its variants) are considered as
state-of-the-art in a number of machine learning problems, specially in Computer Vision. They are used not only
for temporal modeling, but can also model more complicated relationships in high-dimensional data, and some applications include
image segmentation and depth estimation from monocular images. Understanding CRFs is a little more challenging than
HMMs or MEMMs, so I will list a few resources for you to get started with. For beginners, the best resource is this
<a href="http://videolectures.net/cikm08_elkan_llmacrf/">short course</a> by <a href="http://cseweb.ucsd.edu/~elkan/">Charles Elkan</a>.
It also has accompanying course notes,
and if you go to this guy’s academic website, you can also find some programming assignments to implement CRFs.
<a href="https://onionesquereality.wordpress.com/2011/08/20/conditional-random-fields-a-beginners-survey/">Here</a> is a
more comprehensive list of resources related to CRFs, and it’s pretty thorough.</p>
<p>Now, in 2006, there was an extension to CRF by the MIT CSAIL lab, called hidden CRFs. Here is the original paper<a href="http://people.csail.mit.edu/sybor/cvpr06_wang.pdf">original paper</a>. What this does, in essence, is to introduce another layer of hidden states, and is designed to
assign a single label to every sequence. This is different from MEMMs and CRFs, which assigned a label to every observation in
a sequence, and different from HMMs too (wherein a stack of HMMs was trained for classification).</p>
<figure>
<img img="" height="155" width="410" src="/images/kinect_activity/hcrf.png" />
<figcaption>Graphical Representation of an hCRF. Note the extra hidden layer.</figcaption>
</figure>
<p>The original hCRF paper applied it to gesture recognition from RGB videos, and demonstrated superior
performance to CRF in classifying gestures, so we zeroed down on this model, to be used for our
Human Activity Classification task (note that activities are not exactly the same as gestures).</p>
<p>The real icing on the cake was this-> MIT CSAIL had released a well documented <a href="http://sourceforge.net/projects/hcrf/">toolbox</a>,
making it ridiculously easy for us to use this model on whichever dataset that we wanted, and
the only major programming part that was left to us now was was the feature extraction stage.</p>
<p>In a future blog post, I will describe in detail the implementation of our project: the dataset, the features we used,
and what results we got.</p>
Wed, 27 May 2015 00:00:00 +0000
https://avisingh599.github.io/machinelearning/classifying-human-activities-kinect/
https://avisingh599.github.io/machinelearning/classifying-human-activities-kinect/Visual Odmetry from scratch - A tutorial for beginners<script type="text/javascript" src="//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<p>I made a post regarding Visual Odometry several months ago, but
never followed it up with a post on the actual work that I did.
I am hoping that this blog post will serve as a starting point for
beginners looking to implement a Visual Odometry system for their robots.
I will basically present the algorithm described in the paper
<a href="https://www-robotics.jpl.nasa.gov/publications/Andrew_Howard/howard_iros08_visodom.pdf">Real-Time Stereo Visual Odometry for Autonomous Ground Vehicles(Howard2008)</a>, with some of my own changes. It’s a somewhat old paper,
but very easy to understand, which is why I used it for my very first implementation. The MATLAB
source code for the same is available on <a href="https://github.com/avisingh599/vo-howard08">github</a>.</p>
<h3 id="what-is-odometry">What is odometry?</h3>
<p>Have you seen that little gadget on a car’s dashboard that tells you how much
distance the car has travelled? It’s called an <a href="http://en.wikipedia.org/wiki/Odometer">odometer</a>.
It (probably) measures the number of rotations that the wheel is undergoing, and multiplies that
by the circumference to get an estimate of the distance travlled by the car. <a href="http://simreal.com/content/Odometry">Odometry</a>
in Robotics is a more general term, and often refers to estimating not only the distance traveled,
but the entire trajectory of a moving robot. So for every time instance <script type="math/tex">t</script>, there is a vector
<script type="math/tex">[ x^{t} y^{t} z^{t} \alpha^{t} \beta^{t} \gamma^{t}]</script> which describes the complete <a href="http://en.wikipedia.org/wiki/Pose_(computer_vision)">pose</a> of the robot at that instance.
Note that <script type="math/tex">\alpha^{t}, \beta^{t}, \gamma^{t}</script> here are the <a href="http://mathworld.wolfram.com/EulerAngles.html">euler angles</a>,
while <script type="math/tex">x^{t}, y^{t} ,z^{t}</script> are <a href="http://en.wikipedia.org/wiki/Cartesian_coordinate_system"> caetesian coordinates</a> of the robot.</p>
<h3 id="whats-visual-odometry">What’s visual odometry?</h3>
<p>There are more than one ways to determine the trajectory of a moving robot, but the one that we
will focus on in this blog post is called Visual Odometry. In this approach we have a camera (or an
array of cameras) rigidly attached to a moving object (such as a car or a robot), and our job is
to construct a <a href="http://en.wikipedia.org/wiki/Six_degrees_of_freedom">6-DOF</a> trajectory using the
video stream coming from this camera(s). When we are using just one camera, it’s called
<strong><em>Monocular Visual Odometry</em></strong>. When we’re using two (or more) cameras, it’s refered to as
<strong><em>Stereo Visual Odometry</em></strong>.</p>
<h3 id="why-stereo-or-why-monocular">Why stereo, or why monocular?</h3>
<p>There are certain advantages and disadvantages associated with both the stereo and the monocular
scheme of things, and I’ll briefly describe some of the main ones here. (Note that this blog post will
only concentrate on stereo as of now, but I might document and post my monocular implementation also).
The advantage of stereo is that you can estimate the exact trajectory, while in monocular you can
only estimate the trajectory, <a href="http://stackoverflow.com/questions/17114880/up-to-a-scale-factor">unique only up to a scale factor</a>.
So, in monocular VO, you can only say that you moved one unit in x, two units in y, and so on, while in stereo,
you can say that you moved one meter in x, two meters in y, and so on. Also, stereo VO is usually much more robust
(due to more data being available). But, in cases where the distance of the objects from the camera are too high (
as compared to the distance between to the two cameras of the stereo system), the stereo case degenerates to the monocular case.
So, let’s say you have a very small robot (like the <a href="http://robobees.seas.harvard.edu/publications">robobees</a>), then
it’s useless to have a stereo system, and you would be much better off with a monocular VO algorithm like <a href="https://github.com/uzh-rpg/rpg_svo">SVO</a>. Alos, there’s a general trend of drones becoming smaller and smaller, so groups like those of <a href="http://rpg.ifi.uzh.ch/people_scaramuzza.html">Davide Scaramuzza</a> are now focusing more on monocular VO approaches (or so he said in a talk that I happened to attend).</p>
<h3 id="enough-english-lets-talk-math-now">Enough english, let’s talk math now</h3>
<h4 id="formulation-of-the-problem">Formulation of the problem</h4>
<h5 id="input"><strong>Input</strong></h5>
<p>We have a stream of (grayscale/color) images coming from a pair of cameras. Let the left and right frames, captured at time t and t+1 be referred to as <script type="math/tex">\mathit{I}_l^t</script>, <script type="math/tex">\mathit{I}_r^t</script>, <script type="math/tex">\mathit{I}_l^{t+1}</script>, <script type="math/tex">\mathit{I}_r^{t+1}</script>. We have prior knowledge of all the intrinsic as well as extrinsic calibration parameters of the stereo rig, obtained via any one of the numerous stereo calibration algorithms available.</p>
<h5 id="output"><strong>Output</strong></h5>
<p>For every pair of stereo images, we need to find the rotation matrix <script type="math/tex">R</script> and the translation vector <script type="math/tex">t</script>, which describes the motion of the vehicle between the two frames.</p>
<h3 id="the-algorithm">The algorithm</h3>
<p>An outline:</p>
<ol>
<li>
<p>Capture images: <script type="math/tex">\mathit{I}_l^t</script>, <script type="math/tex">\mathit{I}_r^t</script>, <script type="math/tex">\mathit{I}_l^{t+1}</script>, <script type="math/tex">\mathit{I}_r^{t+1}</script></p>
</li>
<li>
<p>Undistort, Rectify the above images.</p>
</li>
<li>
<p>Compute the disparity map <script type="math/tex">\mathit{D}^t</script> from <script type="math/tex">\mathit{I}_l^t</script>, <script type="math/tex">\mathit{I}_r^t</script> and the map <script type="math/tex">\mathit{D}^{t+1}</script> from <script type="math/tex">\mathit{I}_l^{t+1}</script>, <script type="math/tex">\mathit{I}_r^{t+1}</script>.</p>
</li>
<li>
<p>Use FAST algorithm to detect features in <script type="math/tex">\mathit{I}_l^t</script>, <script type="math/tex">\mathit{I}_l^{t+1}</script> and match them.</p>
</li>
<li>
<p>Use the disparity maps <script type="math/tex">\mathit{D}^t</script>, <script type="math/tex">\mathit{D}^{t+1}</script> to calculate the 3D posistions of the features detected in the previous steps. Two point Clouds <script type="math/tex">\mathcal{W}^{t}</script>, <script type="math/tex">\mathcal{W}^{t+1}</script> will be obtained</p>
</li>
<li>
<p>Select a subset of points from the above point cloud such that all the matches are mutually compatible.</p>
</li>
<li>
<p>Estimate <script type="math/tex">R, t</script> from the inliers that were detected in the previous step.</p>
</li>
</ol>
<p>Do not worry if you do not understand some of the terminologies like disparity maps or FAST features that you see above.
Most of them will be explained in greater detail in the text to follow, along with the code to use them in MATLAB.</p>
<h4 id="undistortion-rectification">Undistortion, Rectification</h4>
<p>Before computing the disparity maps, we must perform a number of preprocessing steps.</p>
<p>Undistrortion: This step compensates for lens distortion. It is performed with the help of the distortion parameters that were obtained during calibration.</p>
<p>Rectification: This step is performed so as to ease up the problem of disparity map computation. After this step, all the epipolar lines become parallel to the horizontal, and the disparity computation step needs to perform its search for matching blocks only in one direction.</p>
<figure>
<img src="/images/visodo/epi.jpg" />
<figcaption>Stereo images overlayed from KITTI dataset, notice the feature matches are along parallel (horizontal) lines</figcaption>
</figure>
<p>Both of these operations are implemented in MATLAB, and since the KITTI Visual Odometry dataset that I used in my implmentation
already has these operations implemented, you won’t find the code for them in my implmenation. You can see how to use these functions <a href="http://www.mathworks.com/help/vision/ref/rectifystereoimages.html?searchHighlight=rectifyStereoImages">here</a> and <a href="http://www.mathworks.com/help/vision/ref/undistortimage.html">here</a>. Note that you need the Computer Vision Toolbox, and MATLAB R2014a or newer for these functions.</p>
<h4 id="disparity-map-computation">Disparity Map Computation</h4>
<p>Given a pair of images from a stereo camera, we can compute a disparity map. Suppose a particular 3D in the physical world <script type="math/tex">F</script> is located at the position <script type="math/tex">(x,y)</script> in the left image, and the same feature is located on <script type="math/tex">(x+d,y)</script> in the second image, then the location <script type="math/tex">(x,y)</script> on the disparity map holds the value <script type="math/tex">d</script>. Note that the y-cordinates are the same since the images have been rectified. Thus, we can define disparity at each point in the image plane as:
<script type="math/tex">\begin{equation}
d = x_{l} - x_{r}
\end{equation}</script></p>
<figure>
<img src="/images/visodo/disp.jpg" />
<figcaption>A disparity map computed on frames from KITTI VO dataset</figcaption>
</figure>
<h5 id="block-matching-algorithm">Block-Matching Algorithm</h5>
<p>Disparity at each point is computed using a sliding window.
For every pixel in the left image a 15x15 pixels wide window is generated around it,
and the value of all the pixels in the windows is stored. This window is then constructed
at the same coordinate in the right image, and is slid horizontally, until the Sum-of-Absolute-Differences (SAD) is minimized.
The algorithm used in our implementation is an advanced version of this block-matching technique, called the <a href="http://zone.ni.com/reference/en-XX/help/372916M-01/nivisionconceptsdita/guid-53310181-e4af-4093-bba1-f80b8c5da2f4/">Semi-Global Block Matching algorithm</a>. A function directly implements this algorithm in MATLAB:</p>
<figure class="highlight"><pre><code class="language-matlab" data-lang="matlab"><span class="n">disparityMap1</span> <span class="o">=</span> <span class="n">disparity</span><span class="p">(</span><span class="n">I1_l</span><span class="p">,</span><span class="n">I1_r</span><span class="p">,</span> <span class="s1">'DistanceThreshold'</span><span class="p">,</span> <span class="mi">5</span><span class="p">);</span></code></pre></figure>
<h4 id="feature-detection">Feature Detection</h4>
<p>My approach uses the FAST corner detector. I’ll now explain in brief how the detector works, though you must have a look at the <a href="http://www.edwardrosten.com/work/fast.html">original paper and source code</a> if you want to really understand how it works. Suppose there is a point <script type="math/tex">\mathbf{P}</script> which we want to test if it is a corner or not. We draw a circle of 16px circumference around this point as shown in figure below. For every pixel which lies on the circumference of this circle, we see if there exits a continuous set of pixels whose intensity exceed the intensity of the original pixel by a certain factor <script type="math/tex">\mathbf{I}</script> and for another set of contiguous pixels if the intensity is less by at least the same factor <script type="math/tex">\mathbf{I}</script>. If yes, then we mark this point as a corner. A heuristic for rejecting the vast majority of non-corners is used, in which the pixel at 1,9,5,13 are examined first, and atleast three of them must have a higher intensity be amount at least <script type="math/tex">\mathbf{I}</script>, or must have an intensity lower by the same amount <script type="math/tex">\mathbf{I}</script> for the point to be a corner. This particular approach is selected due to its computational efficiency as compared to other popular interest point detectors such as SIFT.</p>
<figure>
<img src="/images/visodo/fast.png" />
<figcaption>Image from the original FAST feature detection paper</figcaption>
</figure>
<p>Another thing that we do in this approach is something that is called “bucketing”.
If we just run a feature detector over an entire image, there is a very good chance
that most of the features would be concentrated in certain rich regions of the image,
while certain other regions would not have any representation. This is not good for
our algorithm, since it relies on the assumption of a static scene, and to find the
“true” static scene, we must look at all of the image, instead of just certain regions
of it. In order to tackle this issue, we divide the images into grids (of roughly 100x100px),
and extract at most 20 features from each of this grid, thus maintaing a more uniform distribution
of fetures.</p>
<p>In the code, you will find the following line:</p>
<figure class="highlight"><pre><code class="language-matlab" data-lang="matlab"><span class="n">points1_l</span> <span class="o">=</span> <span class="n">bucketFeatures</span><span class="p">(</span><span class="n">I1_l</span><span class="p">,</span> <span class="n">h</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">h_break</span><span class="p">,</span> <span class="n">b_break</span><span class="p">,</span> <span class="n">numCorners</span><span class="p">);</span></code></pre></figure>
<p>This line calls the following function:</p>
<figure class="highlight"><pre><code class="language-matlab" data-lang="matlab"><span class="k">function</span> <span class="n">points</span> <span class="o">=</span> <span class="n">bucketFeatures</span><span class="p">(</span><span class="n">I</span><span class="p">,</span> <span class="n">h</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">h_break</span><span class="p">,</span> <span class="n">b_break</span><span class="p">,</span> <span class="n">numCorners</span><span class="p">)</span>
<span class="c1">% input image I should be grayscale</span>
<span class="n">y</span> <span class="o">=</span> <span class="nb">floor</span><span class="p">(</span><span class="nb">linspace</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">h</span> <span class="o">-</span> <span class="n">h</span><span class="p">/</span><span class="n">h_break</span><span class="p">,</span> <span class="n">h_break</span><span class="p">));</span>
<span class="n">x</span> <span class="o">=</span> <span class="nb">floor</span><span class="p">(</span><span class="nb">linspace</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">b</span> <span class="o">-</span> <span class="n">b</span><span class="p">/</span><span class="n">b_break</span><span class="p">,</span> <span class="n">b_break</span><span class="p">));</span>
<span class="n">final_points</span> <span class="o">=</span> <span class="p">[];</span>
<span class="k">for</span> <span class="nb">i</span><span class="o">=</span><span class="mi">1</span><span class="p">:</span><span class="nb">length</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="k">for</span> <span class="nb">j</span><span class="o">=</span><span class="mi">1</span><span class="p">:</span><span class="nb">length</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">roi</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="p">(</span><span class="nb">j</span><span class="p">),</span><span class="n">y</span><span class="p">(</span><span class="nb">i</span><span class="p">),</span><span class="nb">floor</span><span class="p">(</span><span class="n">b</span><span class="p">/</span><span class="n">b_break</span><span class="p">),</span><span class="nb">floor</span><span class="p">(</span><span class="n">h</span><span class="p">/</span><span class="n">h_break</span><span class="p">)];</span>
<span class="n">corners</span> <span class="o">=</span> <span class="n">detectFASTFeatures</span><span class="p">(</span><span class="n">I</span><span class="p">,</span> <span class="s1">'MinQuality'</span><span class="p">,</span> <span class="mf">0.00</span><span class="p">,</span> <span class="s1">'MinContrast'</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="s1">'ROI'</span><span class="p">,</span><span class="n">roi</span> <span class="p">);</span>
<span class="n">corners</span> <span class="o">=</span> <span class="n">corners</span><span class="o">.</span><span class="n">selectStrongest</span><span class="p">(</span><span class="n">numCorners</span><span class="p">);</span>
<span class="n">final_points</span> <span class="o">=</span> <span class="nb">vertcat</span><span class="p">(</span><span class="n">final_points</span><span class="p">,</span> <span class="n">corners</span><span class="o">.</span><span class="n">Location</span><span class="p">);</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">points</span> <span class="o">=</span> <span class="n">cornerPoints</span><span class="p">(</span><span class="n">final_points</span><span class="p">);</span></code></pre></figure>
<p>As you can see, the image is divided into grids, and the strongest corners from each grid are
selected for the subsequent steps.</p>
<h4 id="feature-description-and-matching">Feature Description and Matching</h4>
<p>The fast corners detected in the previous step are fed to the next step, which uses a <a href="https://www.ces.clemson.edu/~stb/klt/">KLT tracker</a>. The KLT tracker basically looks around every corner to be tracked, and uses this local information to find the corner in the next image. You are welcome to look into the KLT link to know more. The corners detected in <script type="math/tex">\mathit{I}_{l}^{t}</script> are tracked in <script type="math/tex">\mathit{I}_{l}^{t+1}</script> Let the set of features detected in <script type="math/tex">\mathit{I}_{l}^{t}</script> be <script type="math/tex">\mathcal{F}^{t}</script> , and the set of corresponding features in <script type="math/tex">\mathit{I}_{l}^{t+1}</script> be <script type="math/tex">\mathcal{F}^{t+1}</script>.</p>
<p>In MATLAB, this is again super-easy to do, and the following three lines intialize the tracker, and run it once.</p>
<figure class="highlight"><pre><code class="language-matlab" data-lang="matlab"><span class="n">tracker</span> <span class="o">=</span> <span class="n">vision</span><span class="o">.</span><span class="n">PointTracker</span><span class="p">(</span><span class="s1">'MaxBidirectionalError'</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span>
<span class="n">initialize</span><span class="p">(</span><span class="n">tracker</span><span class="p">,</span> <span class="n">points1_l</span><span class="o">.</span><span class="n">Location</span><span class="p">,</span> <span class="n">I1_l</span><span class="p">);</span>
<span class="p">[</span><span class="n">points2_l</span><span class="p">,</span> <span class="n">validity</span><span class="p">]</span> <span class="o">=</span> <span class="nb">step</span><span class="p">(</span><span class="n">tracker</span><span class="p">,</span> <span class="n">I2_l</span><span class="p">);</span></code></pre></figure>
<p>Note that in my current implementation, I am just tracking the point from one frame to the next, and then again doing the detection part,
but in a better implmentation, one would track these points as long as the number of points do not drop below a particular threshold.</p>
<h4 id="triangulation-of-3d-pointcloud">Triangulation of 3D PointCloud</h4>
<p>The real world 3D coordinates of all the point in <script type="math/tex">\mathcal{F}^{t}</script> and <script type="math/tex">\mathcal{F}^{t+1}</script> are computed with respect to the left camera using the disparity value corresponding to these features from the disparity map, and the known projection matrices of the two cameras <script type="math/tex">\mathbf{P}_{1}</script> and <script type="math/tex">\mathbf{P}_{2}</script>.
We first form the reprojection matrix <script type="math/tex">\mathbf{Q}</script>, using data from <script type="math/tex">\mathbf{P1}</script> and <script type="math/tex">\mathbf{P2}</script>:</p>
<script type="math/tex; mode=display">% <![CDATA[
Q=
\left[ {\begin{array}{cccc}
1 & 0 & 0 & -c_{x} \\
0 & 1 & 0 & -c_{y} \\
0 & 0 & 0 & -f \\
0 & 0 & -1/T_{x} & 0 \\
\end{array} } \right] %]]></script>
<p><script type="math/tex">c_{x}</script> = x-coordinate of the optical center of the left camera (in pixels)<br />
<script type="math/tex">c_{y}</script> = y-coordinate of the optical center of the left camera (in pixels)<br />
<script type="math/tex">f</script> = focal length of the first camera<br />
<script type="math/tex">T_{x}</script> = The x-coordinate of the right camera with respect to the first camera (in meters)</p>
<p>We use the following relation to obtain the 3D coordinates of every feature in <script type="math/tex">\mathcal{F}_{l}^{t}</script> and <script type="math/tex">\mathcal{F}_{l}^{t+1}</script></p>
<script type="math/tex; mode=display">\begin{equation}
\left[ \begin{array}{c} X \\ Y \\ Z \\ 1\end{array} \right] = \mathbf{Q} \times \left[ \begin{array}{c} x \\ y \\ d \\ 1\end{array} \right]
\end{equation}</script>
<p>Let the set of point clouds obtained from be referred to as <script type="math/tex">\mathcal{W}^{t}</script> and <script type="math/tex">\mathcal{W}^{t+1}</script>. To have a better understanding of
the geometry that goes on in the above equations, you can have a look at the Bible of visual geometry i.e. Hartley and Zisserman’s <a href="http://www.robots.ox.ac.uk/~vgg/hzbook/">Multiple View Geometry</a>.</p>
<h4 id="the-inlier-detection-step">The Inlier Detection Step</h4>
<p>This algorithm defers from most other visual odometry algorithms in the sense that it does not have an outlier detection step, but it has an inlier detection step. We assume that the scene is rigid, and hence it must not change between the time instance <script type="math/tex">t</script> and <script type="math/tex">t+1</script>. As a result, the distance between any two features in the point cloud <script type="math/tex">\mathcal{W}^{t}</script> must be same as the distance between the corresponding points in <script type="math/tex">\mathcal{W}^{t+1}</script>. If any such distance is not same, then either there is an error in 3D triangulation of at least one of the two features, or we have triangulated a moving, which we cannot use in the next step. In order to have the maximum set of consistent matches, we form the consistency matrix <script type="math/tex">\mathbf{M}</script> such that:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{equation}
\mathbf{M}_{i,j} = \begin{cases} 1, & \mbox{if the distance between i and j points is same in both the point clouds} \\ 0, & \mbox{otherwise} \end{cases}
\end{equation} %]]></script>
<p>From the original point clouds, we now wish to select the largest subset such that they are all the points in this subset are consistent with each other (every element in the reduced consistency matrix is 1). This problem is equivalent to the <a href="http://en.wikipedia.org/wiki/Clique_problem">Maximum Clique Problem</a>, with <script type="math/tex">\mathbf{M}</script> as an adjacency matrix. A cliques is basically a subset of a graph, that only contains nodes that are all connected to each other. An easy way to visualise this is to think of a graph as a social network, and then trying to find the largest group of people who all know each other.</p>
<figure>
<img src="/images/visodo/clique.png" />
<figcaption>This is how clique looks like.</figcaption>
</figure>
<p>This problem is known to be NP-complete, and thus an optimal solution cannot be found for any practical situation. We therefore employ a greedy heuristic that gives us a clique which is close to the optimal solution:</p>
<ol>
<li>Select the node with the maximum degree, and initialize the clique to contain this node.</li>
<li>From the existing clique, determine the subset of nodes <script type="math/tex">\mathit{v}</script> which are connected to all the nodes present in the clique.</li>
<li>From the set <script type="math/tex">\mathit{v}</script>, select a node which is connected to the maximum number of other nodes in <script type="math/tex">\mathit{v}</script>. Repeat from step 2 till no more nodes can be added to the clique.</li>
</ol>
<p>The above algorithm is implemented in the following two functions in my code:</p>
<figure class="highlight"><pre><code class="language-matlab" data-lang="matlab"><span class="k">function</span> <span class="n">cl</span> <span class="o">=</span> <span class="n">updateClique</span><span class="p">(</span><span class="n">potentialNodes</span><span class="p">,</span> <span class="n">clique</span><span class="p">,</span> <span class="n">M</span><span class="p">)</span>
<span class="n">maxNumMatches</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">curr_max</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">for</span> <span class="nb">i</span> <span class="o">=</span> <span class="mi">1</span><span class="p">:</span><span class="nb">length</span><span class="p">(</span><span class="n">potentialNodes</span><span class="p">)</span>
<span class="k">if</span><span class="p">(</span><span class="n">potentialNodes</span><span class="p">(</span><span class="nb">i</span><span class="p">)</span><span class="o">==</span><span class="mi">1</span><span class="p">)</span>
<span class="n">numMatches</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">for</span> <span class="nb">j</span> <span class="o">=</span> <span class="mi">1</span><span class="p">:</span><span class="nb">length</span><span class="p">(</span><span class="n">potentialNodes</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(</span><span class="n">potentialNodes</span><span class="p">(</span><span class="nb">j</span><span class="p">)</span> <span class="o">&</span> <span class="n">M</span><span class="p">(</span><span class="nb">i</span><span class="p">,</span><span class="nb">j</span><span class="p">))</span>
<span class="n">numMatches</span> <span class="o">=</span> <span class="n">numMatches</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">if</span> <span class="p">(</span><span class="n">numMatches</span><span class="o">>=</span><span class="n">maxNumMatches</span><span class="p">)</span>
<span class="n">curr_max</span> <span class="o">=</span> <span class="nb">i</span><span class="p">;</span>
<span class="n">maxNumMatches</span> <span class="o">=</span> <span class="n">numMatches</span><span class="p">;</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">if</span> <span class="p">(</span><span class="n">maxNumMatches</span><span class="o">~=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">clique</span><span class="p">(</span><span class="nb">length</span><span class="p">(</span><span class="n">clique</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">=</span> <span class="n">curr_max</span><span class="p">;</span>
<span class="k">end</span>
<span class="n">cl</span> <span class="o">=</span> <span class="n">clique</span><span class="p">;</span>
<span class="k">function</span> <span class="n">newSet</span> <span class="o">=</span> <span class="n">findPotentialNodes</span><span class="p">(</span><span class="n">clique</span><span class="p">,</span> <span class="n">M</span><span class="p">)</span>
<span class="n">newSet</span> <span class="o">=</span> <span class="n">M</span><span class="p">(:,</span><span class="n">clique</span><span class="p">(</span><span class="mi">1</span><span class="p">));</span>
<span class="k">if</span> <span class="p">(</span><span class="nb">size</span><span class="p">(</span><span class="n">clique</span><span class="p">)</span><span class="o">></span><span class="mi">1</span><span class="p">)</span>
<span class="k">for</span> <span class="nb">i</span><span class="o">=</span><span class="mi">2</span><span class="p">:</span><span class="nb">length</span><span class="p">(</span><span class="n">clique</span><span class="p">)</span>
<span class="n">newSet</span> <span class="o">=</span> <span class="n">newSet</span> <span class="o">&</span> <span class="n">M</span><span class="p">(:,</span><span class="n">clique</span><span class="p">(</span><span class="nb">i</span><span class="p">));</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">for</span> <span class="nb">i</span><span class="o">=</span><span class="mi">1</span><span class="p">:</span><span class="nb">length</span><span class="p">(</span><span class="n">clique</span><span class="p">)</span>
<span class="n">newSet</span><span class="p">(</span><span class="n">clique</span><span class="p">(</span><span class="nb">i</span><span class="p">))</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">end</span></code></pre></figure>
<h4 id="computation-of-mathbfr-and-mathbft">Computation of <script type="math/tex">\mathbf{R}</script> and <script type="math/tex">\mathbf{t}</script></h4>
<p>In order to determine the rotation matrix <script type="math/tex">\mathbf{R}</script> and translation vector <script type="math/tex">\mathbf{t}</script>, we use Levenberg-Marquardt non-linear least squares minimization to minimize the following sum:</p>
<script type="math/tex; mode=display">\begin{equation}
\epsilon = \sum_{\mathcal{F}^{t}, \mathcal{F}^{t+1}} (\mathbf{j_{t}} - \mathbf{P}\mathbf{T}\mathbf{w_{t+1}})^{2} + (\mathbf{j_{t+1}} - \mathbf{P}\mathbf{T^{-1}}\mathbf{w_{t}})^{2}
\end{equation}</script>
<p><script type="math/tex">\mathcal{F}^{t}, \mathcal{F}^{t+1}</script>: Features in the left image at time <script type="math/tex">t</script> and <script type="math/tex">t+1</script>
<script type="math/tex">\mathbf{j_{t}}, \mathbf{j_{t+1}}</script>: 2D Homogeneous coordinates of the features <script type="math/tex">\mathcal{F}^{t}, \mathcal{F}^{t+1}</script><br />
<script type="math/tex">\mathbf{w_{t}}, \mathbf{w_{t+1}}</script>: 3D Homogeneous coordinates of the features <script type="math/tex">\mathcal{F}^{t}, \mathcal{F}^{t+1}</script><br />
<script type="math/tex">\mathbf{P}</script>: <script type="math/tex">3\times4</script> Projection matrix of left camera<br />
<script type="math/tex">\mathbf{T}</script>: <script type="math/tex">4\times4</script> Homogeneous Transformation matrix\</p>
<p>The Optimization Toolbox in MATLAB directly implements the Levenberg-Marquardt algorithm in the function lsqnonlin, which needs to be supplied with a vector objective function that needs to be minimized, and a set of parameters that can be varied.</p>
<p>This is how the function to be minimized is represented in MATLAB. This part of the algorithm,
is the most computationally expensive one.</p>
<figure class="highlight"><pre><code class="language-matlab" data-lang="matlab"><span class="k">function</span> <span class="n">F</span> <span class="o">=</span> <span class="n">minimize</span><span class="p">(</span><span class="n">PAR</span><span class="p">,</span> <span class="n">F1</span><span class="p">,</span> <span class="n">F2</span><span class="p">,</span> <span class="n">W1</span><span class="p">,</span> <span class="n">W2</span><span class="p">,</span> <span class="n">P1</span><span class="p">)</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">PAR</span><span class="p">(</span><span class="mi">1</span><span class="p">:</span><span class="mi">3</span><span class="p">);</span>
<span class="n">t</span> <span class="o">=</span> <span class="n">PAR</span><span class="p">(</span><span class="mi">4</span><span class="p">:</span><span class="mi">6</span><span class="p">);</span>
<span class="c1">%F1, F2 -> 2d coordinates of features in I1_l, I2_l</span>
<span class="c1">%W1, W2 -> 3d coordinates of the features that have been triangulated</span>
<span class="c1">%P1, P2 -> Projection matrices for the two cameras</span>
<span class="c1">%r, t -> 3x1 vectors, need to be varied for the minimization</span>
<span class="n">F</span> <span class="o">=</span> <span class="nb">zeros</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="nb">size</span><span class="p">(</span><span class="n">F1</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span> <span class="mi">3</span><span class="p">);</span>
<span class="n">reproj1</span> <span class="o">=</span> <span class="nb">zeros</span><span class="p">(</span><span class="nb">size</span><span class="p">(</span><span class="n">F1</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span> <span class="mi">3</span><span class="p">);</span>
<span class="n">reproj2</span> <span class="o">=</span> <span class="nb">zeros</span><span class="p">(</span><span class="nb">size</span><span class="p">(</span><span class="n">F1</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span> <span class="mi">3</span><span class="p">);</span>
<span class="n">dcm</span> <span class="o">=</span> <span class="n">angle2dcm</span><span class="p">(</span> <span class="n">r</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="n">r</span><span class="p">(</span><span class="mi">2</span><span class="p">),</span> <span class="n">r</span><span class="p">(</span><span class="mi">3</span><span class="p">),</span> <span class="s1">'ZXZ'</span> <span class="p">);</span>
<span class="n">tran</span> <span class="o">=</span> <span class="p">[</span> <span class="nb">horzcat</span><span class="p">(</span><span class="n">dcm</span><span class="p">,</span> <span class="n">t</span><span class="p">);</span> <span class="p">[</span><span class="mi">0</span> <span class="mi">0</span> <span class="mi">0</span> <span class="mi">1</span><span class="p">]];</span>
<span class="k">for</span> <span class="n">k</span> <span class="o">=</span> <span class="mi">1</span><span class="p">:</span><span class="nb">size</span><span class="p">(</span><span class="n">F1</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">f1</span> <span class="o">=</span> <span class="n">F1</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="p">:)</span><span class="o">'</span><span class="p">;</span>
<span class="n">f1</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">w2</span> <span class="o">=</span> <span class="n">W2</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="p">:)</span><span class="o">'</span><span class="p">;</span>
<span class="n">w2</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">f2</span> <span class="o">=</span> <span class="n">F2</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="p">:)</span><span class="o">'</span><span class="p">;</span>
<span class="n">f2</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">w1</span> <span class="o">=</span> <span class="n">W1</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="p">:)</span><span class="o">'</span><span class="p">;</span>
<span class="n">w1</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">f1_repr</span> <span class="o">=</span> <span class="n">P1</span><span class="o">*</span><span class="p">(</span><span class="n">tran</span><span class="p">)</span><span class="o">*</span><span class="n">w2</span><span class="p">;</span>
<span class="n">f1_repr</span> <span class="o">=</span> <span class="n">f1_repr</span><span class="p">/</span><span class="n">f1_repr</span><span class="p">(</span><span class="mi">3</span><span class="p">);</span>
<span class="n">f2_repr</span> <span class="o">=</span> <span class="n">P1</span><span class="o">*</span><span class="nb">pinv</span><span class="p">(</span><span class="n">tran</span><span class="p">)</span><span class="o">*</span><span class="n">w1</span><span class="p">;</span>
<span class="n">f2_repr</span> <span class="o">=</span> <span class="n">f2_repr</span><span class="p">/</span><span class="n">f2_repr</span><span class="p">(</span><span class="mi">3</span><span class="p">);</span>
<span class="n">reproj1</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="p">:)</span> <span class="o">=</span> <span class="p">(</span><span class="n">f1</span> <span class="o">-</span> <span class="n">f1_repr</span><span class="p">);</span>
<span class="n">reproj2</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="p">:)</span> <span class="o">=</span> <span class="p">(</span><span class="n">f2</span> <span class="o">-</span> <span class="n">f2_repr</span><span class="p">);</span>
<span class="k">end</span>
<span class="n">F</span> <span class="o">=</span> <span class="p">[</span><span class="n">reproj1</span><span class="p">;</span> <span class="n">reproj2</span><span class="p">];</span></code></pre></figure>
<h4 id="validation-of-results">Validation of results</h4>
<p>A particular set of <script type="math/tex">\mathbf{R}</script> and <script type="math/tex">\mathbf{t}</script> is said to be valid if it satisfies the following conditions:</p>
<ol>
<li>If the number of features in the clique is at least 8.</li>
<li>The reprojection error <script type="math/tex">\epsilon</script> is less than a certain threshold.</li>
</ol>
<p>The above constraints help in dealing with noisy data.</p>
<h4 id="an-important-hack">An important “hack”</h4>
<p>If you run the above algorithm on real-world sequences, you will encounter a
rather big problem. The assumption of scene rigidity stops holding when a large vehicle
such as a truck or a van occupies a majority of the field of view of the camera. In order
to deal with such data, we introduce a simple hack: accept a tranlsation/rotation matrix
only if the dominant motion is in the forward direction. This is known to improve results
significantly on the KITTI dataset, though you won’t find in this hack explicitly written
in most of the papers that are published on the same!</p>
Mon, 25 May 2015 00:00:00 +0000
https://avisingh599.github.io/vision/visual-odometry-full/
https://avisingh599.github.io/vision/visual-odometry-full/Stitching Intra-Oral Images<p><em>Note: This is a repost of my <a href="https://mitredxcampjan2015.wordpress.com/2015/01/28/dental-imaging-project-the-stitching-story/">January post</a> on MIT Media Lab’s Wordpress blog of their RedX 2015 Camp held at IIT-Bombay. There are a few minor modifications though.</em></p>
<p>Most intraoral cameras have a relative narrow field of view, and the entire jaw is never visible in a single image. We are trying to stitch several images into one, so that the user has complete view of the jaw, and we can then segment the tooth from it, and keep a track for every individual tooth.</p>
<p>A basic image stitching pipeline has the following steps:</p>
<ol>
<li>Matching features between two images</li>
<li>Computing the homography with RANSAC (minimal set is four matches)</li>
<li>Transforming , concatenating and blending the images.</li>
</ol>
<p>Most of the existing panaroma building algorithms are well-suited for applications in which the object being photographed is quite far away from the camera, such as in the image shown below (<a href="http://www.cs.bath.ac.uk/brown/autostitch/autostitch.html">obtained from the Autostitch page</a>):</p>
<figure>
<img src="/images/dental/panaroma.png" />
<figcaption>Panorama construction</figcaption>
</figure>
<p>However, we are photographing the teeth at a really close range, and minor changes in perspective are fatal for these algorithms. In order to overcome the problems imposed by changes in perspective, we are using ASIFT, a feature detection/description/matching algorithm which is robust to perspective changes when compared to SIFT. The next steps (homography computation, blending) are pretty standard, and here are some results:</p>
<figure>
<img src="/images/dental/stitched.png" />
<figcaption>A stitch of three images taken from an intraoral camera</figcaption>
</figure>
Sat, 23 May 2015 00:00:00 +0000
https://avisingh599.github.io/vision/stichting-story/
https://avisingh599.github.io/vision/stichting-story/Every Tooth Tracked<p><em>Note: This is a repost of my <a href="https://mitredxcampjan2015.wordpress.com/2015/01/30/dental-imaging-project-every-tooth-tracked/">January post</a> on MIT Media Lab’s Wordpress blog of their RedX 2015 Camp held at IIT-Bombay. There are a few minor modifications though.</em></p>
<p>We want to track the health of every tooth over time, and therefore wanted an algorithm that could extract the image of every single tooth from the stitch that we obtained in our previous step. Our first attempt was at a completely automated approach, and we soon found a <a href="http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6482414&tag=1">paper</a> which attempted to solve a problem that was a subset of ours. They wanted to separate the teeth part from the rest of the image, while we wanted to segment every teeth from the rest of the image. The algorithm that these guys had used was pretty basic (<a href="http://cdanup.com/10.1.1.2.1828.pdf">Active Contours Without Edges</a>), and I got it working within half an hour on MATLAB, with the following results:</p>
<figure>
<img src="/images/dental/3k_with_removal.png" />
<figcaption>Obtained using Active Contours Without Edges (Chan-Vese)</figcaption>
</figure>
<p>But this approach had a few problems. It was computationally expensive (~ 2min to run on my Intel Core i7 machine), and could not be used to segment an individual tooth out.</p>
<p>So, I started looking at other algorithms, and soon stumbled across the <a href="http://www.cs.rug.nl/~roe/publications/parwshed.pdf">Watershed transform</a>. In order to generate good results, watershed needs certain markers, and these markers can be generated using both automated or manual methods. One popular automated method for generating these markers is ‘opening-by-reconstruction’ and ‘closing-by-reconstruction’. The following results were obtained using MATLAB’s watershed example:</p>
<figure>
<img src="/images/dental/49_seg_man.png" />
<figcaption>Vanilla Watershed with automatic marker generation</figcaption>
</figure>
<p>As you can see, the above is a complete mess. A lot of unwanted segments are obtained, and some superpixels (clusters of pixels) flow into each other.
So, I then tried a manual-marker approach, and the results were much better:</p>
<figure>
<img src="/images/dental/49_final.png" />
<figcaption>Watershed with manually-annotated markers</figcaption>
</figure>
<p>A <a href="http://www.mathworks.com/matlabcentral/fileexchange/44469-gui-image-mask-sample">matlab-based GUI</a> is used to generate the masks as follows:</p>
<figure>
<img src="/images/dental/gui_marker.png" />
</figure>
<p>The mask file looks something like this:</p>
<figure>
<img src="/images/dental/49_msk.png" />
<figcaption>The mask used to generate the above results</figcaption>
</figure>
<p>In the final product, we can assume to have a touchscreen based user interface, wherein the user slashes with his finger across every tooth once, and then gets the segmented image as an output. One several such images have been mannually annotated, we could use a learning algorithm that can automatically generate these masks.</p>
Sat, 23 May 2015 00:00:00 +0000
https://avisingh599.github.io/vision/segmenting-teeth/
https://avisingh599.github.io/vision/segmenting-teeth/Visual Odometry - The Reading List<p>I am thinking of taking up a project on ‘Visual Odometry’ as UGP-1 (Undergraduate Project) here in my fifth semester at IIT-Kanpur.
This post is primarily a list of some useful links which will get one acquainted with the basics of Visual Odometry.</p>
<p>The first thing that anyone should read is this wonderful two-part review by Davide Scaramuzza and Friedrich Fraundorfer:</p>
<ul>
<li><a href="http://www.roboticsschool.ethz.ch/airobots/programme/presentations/VO_part_I.pdf">Visual Odometry Tutorial Part 1</a></li>
<li><a href="http://rpg.ifi.uzh.ch/docs/VO_Part_II_Scaramuzza.pdf">Visual Odometry Tutorial Part 2</a></li>
</ul>
<p>One thing that I did not understand from the above tutorials was the ‘5-point algorithm’ by Nister in 2003. The original paper is <a href="http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1288525">here</a>. But, this paper also seemed quite complicated for me to implement without any background, so I moved onto a simpler algorithm, called the ‘8-point algorithm’, which was published a long time ago by Longuet-Higgins. You can find it <a href="http://www2.ece.ohio-state.edu/~aleix/Longuet-Higgins.pdf">here</a>. There are some lecture slides which explain this in a simple manner, and you can find them <a href="http://www.cse.psu.edu/~rcollins/CSE486/lecture20_6pp.pdf">here</a>.</p>
<p>Note, there are more papers that one should read regarding this, most notably:</p>
<ul>
<li><a href="http://www.cse.unr.edu/~bebis/CS485/Handouts/hartley.pdf">In Defense of the 8-point Algorithmm</a></li>
<li><a href="http://users.cecs.anu.edu.au/~hongdong/new5pt_cameraREady_ver_1.pdf">5-point Motion Estimation Made Easy</a></li>
</ul>
<p>In my next post, I will hopefully start working on my implementation.</p>
Tue, 29 Jul 2014 00:00:00 +0000
https://avisingh599.github.io/vision/visual-odometry-read/
https://avisingh599.github.io/vision/visual-odometry-read/RANSAC<p>This post is about the popular outlier rejection algorithm RANSAC. It stands for RANdom SAmple Consensus. It is widely used in computer vision, with one of the application being in rejection of false feature matches in a pair of images from a stereo camera set.</p>
<p>Suppose you have been given a dataset and you want to fit a mathematical model on it. We now assume that this data has certain <em>inliers</em> and some <em>outliers</em>. Inliers refer to the data points whose presence can be explained with the help of a mathematical model, while outliers are data points whose presence can never be explained via any reasonable mathematical model. Usually their presence in the dataset deteriorates the quality of the mathematical model that we can fit to the data. For best results, we should ignore these outliers while estimating the parameters of our mathematical model. RANSAC helps us in identifying these points so that we can obtain a better fir for the inliers.</p>
<p>Note that even the inliers do not <em>exactly</em> fit the mathematical model as they might have some noise, but the outliers either have an extremely large amount of noise or they are obtained due to faults in measurement, or because of problems in the sensor from which we are obtaining the data.</p>
<h2 id="the-algorithm">The Algorithm</h2>
<h3 id="the-input">The Input</h3>
<ul>
<li>Data points</li>
<li>Some parametrized model (we need to estimate the parameters for this model)</li>
<li>Some confidence parameters</li>
</ul>
<h3 id="algo">Algo</h3>
<ul>
<li>A set points from the original dataset are randomly selected, and are assumed to be the inliers.</li>
<li>Parameters are estimated to fit to this hypothetical inlier set.</li>
<li>Every point that was not a part of this hypothetical inlier set is tested against the mathematical model that we just fit.</li>
<li>The points that fit the model become a part of the <em>consensus</em> set. The model is good if a particular number of points have been classified as part of the consensus set.</li>
<li>This model is then re-estimated using all the members of a consensus set.</li>
<li>The above process is repeated a fixed number of times, and the model with the largest consensus set is kept.</li>
</ul>
<h3 id="how-many-times-do-we-repeat">How many times do we repeat?</h3>
<p>It is possible to theoretically determine the fixed number of iterations ‘k’ which are needed, if we have an estimate of the percentage of outliers present in the data.</p>
Mon, 21 Jul 2014 00:00:00 +0000
https://avisingh599.github.io/stats/ransac/
https://avisingh599.github.io/stats/ransac/