Most of the recent work in sentiment analysis is carried out on textual data. The text based sentiment analysis mainly relies on construction of word dictionaries, using machine learning techniques that learn and extract opinion from large text corpora. Text based sentiment analysis has numerous applications such as customer satisfaction analysis about a brand or product perception, to gauge voting intentions etc. With the rapid growth of social media, users post humongous volume of data in various modalities such as text, image, audio, and video. These multimodal data streams bring new opportunities for going beyond text based sentiment analysis and improving possible results. Since sentiment can be extracted from facial and vocal expressions, prosody and body posture, multimodal sentiment analysis offers new avenues in sentiment analysis. In multimodal sentiment analysis, sentiment is extracted from transcribed content, visual and vocal features. This survey defines sentiment, sentiment analysis, states problems and challenges in multimodal sentiment analysis and finally reviews some of the recent computational approaches used multimodal sentiment analysis.

Sentiment Analysis, Computational Approaches, Multimodal Sentiment Analysis, Challenges, Machine Learning

[1] L. Z. Bing Liu, "A survey of opinion mining and sentiment analysis," in Mining Text Data, Springer US, 2013, pp. 415-463.
[2] M. Prem, G. Wojciech and R. D. Lawrence, "Sentiment analysis of blogs by combining lexical knowledge with text classification," in ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France, 2009.
[3] B. J. Jansen, M. Zhang, K. Sobel and A. Chowdury, "Twitter power: Tweets as electronic word of mouth," Journal of the Association for Information Science and Technology, vol. 60, no. 11, p. 2169–2188, 2009.
[4] E. Cambria, B. Schuller and Y. Xia, "New Avenues in Opinion Mining and Sentiment Analysis," IEEE Intelligent Systems, vol. 28, no. 2, pp. 15 - 21, 2013.
[5] M. Munezero, C. S. Montero, E. Sutinen and J. Pajunen, "Are They Different? Affect, Feeling, Emotion, Sentiment, and Opinion Detection in Text," IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, vol. 5, no. 2, pp. 101-111, 2014.
[6] J. Deonna, C. Tappolet and F. Teroni, "Emotion, philosophical issues about," WIREs Cogn Sci , pp. 1-15, 2015.
[7] S.-M. Kim and E. Hovy, "Determining the sentiment of opinions," in COLING `04 Proceedings of the 20th international conference on Computational Linguistics, PA, USA, 2004.
[8] J. M. Wiebet, R. F. Bruce and T. P. O`Harat, "Development and Use of a Gold-Standard Data Set for Subjectivity Classifications," in Annual Meeting of the Association for Computational Linguistics on Computational Linguistics, 1999.
[9] J. Bollen, H. Mao and X.-J. Zeng, "Twitter mood predicts the stock market," Journal of Computer Science, vol. 2, no. 1, pp. 1-8, 2011.
[10] A. Tumasjan, T. O. Sprenger, P. G. Sandner and I. M. Welpe, "Predicting elections with Twitter: what 140 characters reveal about political sentiment," in Proceedings of the Fourth International AAAI Conference on Weblogs and Social Media, George Washington University, 2010.
[11] S. Asur and B. A. Huberman, "Predicting the Future With Social Media," Web Intelligence and. Intelligent Agent Technology (WI-IAT), vol. 1, no. 6, pp. 492-299, 2010.
[12] B. Pang, L. Lee and S. Vaithyanathan, "Thumbs up? Sentiment Classification using Machine Learning Techniques," in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2002.
[13] B. Pang and L. Lee, "Opinion Mining and Sentiment Analysis," Foundations and Trends in Information Retrieval, vol. 2, no. 1-2, pp. 1-135, 2008.
[14] P. Nakov, T. Zesch, D. Cer and D. Jurgens, "International Workshop on Semantic Evaluation (SemEval 2013),," in Association for Computational Linguistics., Atlanta, Georgia, 2013.
[15] S. M. Mohammad, S. Kiritchenko and X. Zhu, "NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets," in In Proceedings of the seventh international workshop on Semantic Evaluation Exercises (SemEval-2013), Atlanta, USA, 2013.
[16] Y. Miura, S. Sakaki, K. Hattori and T. Ohkuma, "TeamX: A Sentiment Analyzer with Enhanced Lexicon Mapping and Weighting Scheme for Unbalanced Data," in Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), Dublin, Ireland, 2014.
[17] M. Hagen, M. Potthast and B. Stein, "Webis: An Ensemble for Twitter Sentiment Detection," in Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), Denver, Colorado, 2015.
[18] J. Deriu, M. Gonzenbach, F. Uzdilli, A. Lucchi, V. D. Luca and M. Jaggi, "SwissCheese at SemEval-2016 Task 4: Sentiment Classification Using an Ensemble of Convolutional Neural Networks with Distant Supervision," in Proceedings of the 10th International Workshop on Semantic Evaluation, San Diego, CA, USA, 2016.
[19] S. Rosenthal, N. Farra and P. Nakov, "SemEval-2017: Sentiment Analysis in Twitter," in Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, 2017.
[20] A. Abbasi, A. Hassan and M. Dhar, "Benchmarking Twitter Sentiment Analysis Tools," LREC, p. 823–829, 2014.
[21] R. Socher, A. Perelygin, J. Wu, J. Chuang, M. C. D, A. Ng and C. Potts, "Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank," in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013.
[22] T. Wilson, J. Wiebe and P. Hoffmann, "Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis," in In: HLT/EMNLP, Vancouver, BC, Canada, 2005.
[23] J. W. Pennebaker, R. L. Boyd, K. Jordan and K. Blackburn, "The Development and Psychometric Properties of LIWC2015," Development Manual, University of Texas at Austin, 2016.
[24] Q. T. Ain, M. Ali, A. Riaz, A. Noureen, M. Kamran, B. Hayat and A. Rehman, "Sentiment Analysis Using Deep Learning Techniques: A Review," in (IJACSA) International Journal of Advanced Computer Science and Applications, 2017.
[25] D. Tang, F. Wei, N. Yang, M. Zhou, T. Liu and B. Qin, "Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification," in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, Maryland, 2014.
[26] O. Irsoy and C. Cardie, "Opinion mining with deep recurrent neural networks," EMNLP, p. 720–728, 2014.
[27] S. G. Koolagudi and K. S. Rao, "Emotion recognition from speech: a review," International Journal of Speech Technology, vol. 15, no. 2, pp. 99-117, 2012.
[28] R. M. Ver´onica P´erez-Rosas, "Sentiment Analysis of Online Spoken Reviews," in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2012.
[29] T. Johnstone, "Emotional speech elicited using computer games," in Proceedings., Fourth International Conference on Spoken Language, Philadelphia, PA, USA, 2002.
[30] C. L. Shao-Hsien, "Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction," Doctoral Dissertation Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction, USA, 2000.
[31] E. Navas, I. Hernaez and I. Luengo, "An objective and subjective study of the role of semantics and prosodic features in building corpora for emotional TTS," IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, no. 4, pp. 1117-1127, 2006.
[32] P. Pudil, F. Ferri and J. Novovicova, "Floating search methods for feature selection with nonmonotonic criterion functions," in 12th IAPR International Conference on Pattern Recognition, 1994.
[33] G.N. Peerzade, R.R. Deshmukh, S.D. Waghmare, "A Review: Speech Emotion Recognition", International Journal of Computer Sciences and Engineering, Vol.6, Issue.3, pp.400-402, 2018.
[34] H. Atassi and A. Esposito, "A Speaker Independent Approach to the Classification of Emotional Vocal Expressions," in 20th IEEE International Conference on Tools with Artificial Intelligence, 2008.
[35] G. Caridakis, G. Castellano, L. Kessous, A. Raouzaiou, L. Malatesta, S. Asteriadis and K. Karpouzis, "Multimodal emotion recognition from expressive faces, body gestures and speech," in Artificial Intelligence and Innovations, 2007.
[36] C. Colombo, A. D. Bimbo and P. Pala, "Semantics in visual information retrieval," IEEE MultiMedia, vol. 8, no. 6, pp. 38 - 53, 1999.
[37] S. Jindal and S. Singh, "Image Sentiment Analysis using Deep Convolutional Neural Networks with Domain Specific Fine Tuning," in International Conference onInformation Processing (ICIP), Pune, India, 2015.
[38] J. Islam and Y. Zhang, "Visual Sentiment Analysis for Social Images Using Transfer Learning Approach," in 2016 IEEE International Conferences on Social Computing and Networking, Atlanta, GA, USA, 2016.
[39] Y. Wang, Y. Hu, S. Kambhampati and B. Li, "Inferring Sentiment from Web Images with Joint Inference on Visual and Social Cues: A Regulated Matrix Factorization Approach," in Proceedings of the Ninth International AAAI Conference on Web and Social, 2015.
[40] Y. Yu, H. Lin, J. Meng and Z. Zhao, "Visual and Textual Sentiment Analysis of a Microblog Using Deep Convolutional Neural Networks," MDPI journals on algorithms, vol. 9, no. 41, pp. 1-11, 2016.
[41] Y. Wang and B. Li, "Sentiment Analysis for Social Media Images," in IEEE International Conference onData Mining Workshop (ICDMW), Atlantic City, NJ, USA, 2015.
[42] Q. You, J. Luo, H. Jin and J. Yang, "Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks by Quanzeng," Association for the Advancement of Artificial, pp. 1-9, 2015.
[43] Y. Yang, J. Jia, S. Zhang, B. Wu, Q. Chen, J. Li, C. Xing and J. Tang, "How do your friends on social media disclose your emotions?," in Twenty-Eighth AAAI Conference on Artificial Intelligence, Canada, 2014.
[44] A. Frome, G. Corrado, J. Shlens, S. Bengio, J. Dean and T. Mikolov, "Devise: A deep visual-semantic embedding model," in Advances in neural information processing systems, Lake Tahoe, 2013.
[45] L.-P. Morency, R. Mihalcea and P. Doshi, "Towards multimodal sentiment analysis: harvesting opinions from the web," in ACM International Conference on Multimodal Interfaces (ICMI), Alicante, Spain, 2011.
[46] M. Wöllmer, F. Weninger, T. Knaup and B. Schuller, "YouTube movie reviews: ssentiment analysis in an audio-visual context," IEEE Intelligent Systems, vol. 28, no. 3, pp. 46-52, 2013.
[47] Ketan Sarvakar, Urvashi K Kuchara, "Sentiment Analysis of movie reviews: A new feature-based sentiment classification", International Journal of Scientific Research in Computer Science and Engineering, Vol.6, Issue.3, pp.8-12, 2018
[48] V. P´erez-Rosas, R. Mihalcea and L.-P. Morency, "Utterance-Level Multimodal Sentiment Analysis," in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, 2013.
[49] J. G. Ellis, B. Jou and S.-F. Chang, "Why We Watch the News: A Dataset for Exploring Sentiment in Broadcast Video News," in ICMI’14, Istanbul, Turkey, 2014.
[50] F. Eyben, M. Wöllmer and B. Schuller, "OpenEAR — Introducing the munich open-source emotion and affect recognition toolkit," in 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, 2009.
[51] J. Wiebe, T. Wilson and C. Cardie, "Annotating Expressions of Opinions and Emotions in Language," Language Resources and Evaluation, vol. 39, no. 2, p. 165–210, 2015.
[52] S. Poria, E. Cambria, N. Howard, G. B. Huang and A. Hussain, "Fusing audio, visual and textual clues for sentiment analysis from multimodal content," Neurocomputing, vol. vol. 174, pp. pp. 50-59.
[53] F. Eyben, M. Wöllmer and B. Schuller, "openSMILE: The Munich versatile and fast open-source audio feature extractor," in ACM International Conference on Multimedia (MM), 2010.
[54] T. Mikolov, I. Sutskever, K. Chen, G. Corrado and J. Dean, "Distributed representations of words and phrases and their compositionality," in Advances in Neural Information Processing Systems, 2013.
[55] S. Poria, E. Cambria, N. Howard, G.-B. Huang and A. Hussain, "Fusing audio, visual and textual clues for sentiment analysis from multimodal content," Neurocomputing, vol. 174, pp. 50-59, 20176.
[56] Gagandeep Kaur, Kamaldeep Kaur, "Sentiment Detection from Punjabi Text using Support Vector Machine", International Journal of Scientific Research in Computer Science and Engineering, Vol.5, Issue.6, pp.39-46, 2017
[57] M. Wöllmer, M. Kaiser, F. Eyben, B. Schuller and G. Rigoll, "LSTM-modeling of continuous emotions in an audiovisual affect recognition framework," Image and Vision Computing, vol. 31, no. 2, p. 53–163, 2013.
[58] V. P. Rosas, R. Mihalcea and L.-P. Morency, "Multimodal sentiment analysis of Spanish online videos," IEEE Intelligent Systems, vol. 28, no. 3, pp. 38-45, 2013.
[59] D. McDuff, R. Kaliouby, E. Kodra and R. Picard, "Measuring voter’s candidate preference based on affective responses to election debates," in Conference on Affective Computing and Intelligent Interaction (ACII), US, 2013.
[60] S. Poria, E. Cambria, R. Bajpai and A. Hussain, "A review of affective computing: From unimodal analysis to multimodal fusion," Information Fusion, vol. 37, pp. 98-125, 2017.
[61] B. Siddiquie, D. Chisholm and A. Divakaran, "Exploiting multimodal affect and semantics to identify politically persuasive web videos," in ACM on International Conference on Multimodal Interaction, Seattle, Washington, 2015.
[62] A. Zadehy, M. Chen, S. Poria, E. Cambria and L.-P. Morency, "Tensor Fusion Network for Multimodal Sentiment Analysis," arXiv, 2017.
[63] D. Borth, R. Ji, T. Chen, T. Breuel and S.-F. Chang, "Large-scale Visual Sentiment Ontology and detectors using adjective noun pairs," in ACM International Conference on Multimedia, Barcelona, Spain, 2013.
[64] A. Khosla, A. D. Sarma and R. Hamid, "What makes an image popular?," in International Conference on World Wide Web (WWW), 2014.
[65] X.-L. Zheng, C.-C. Chen, J.-L. Hung, W. He, F.-X. Hong and Z. Lin, "A Hybrid Trust-Based Recommender System for Online Communities of Practice," IEEE Transactions on Learning Technologies, vol. 8, no. 4, pp. 345 - 356, 2015.
[66] A. HedayatiMoghaddam, M. HedayatiMoghaddam and MortezaEsfandyari, "Stock market index prediction using artificial neural network," Journal of Economics, Finance and Administrative Science, vol. 21, no. 41, pp. 89-93, Dec 2016.
[67] Shrija Madhu, "An approach to analyze suicidal tendency in blogs and tweets using Sentiment Analysis", International Journal of Scientific Research in Computer Science and Engineering, Vol.6, Issue.4, pp.34-36, 2018
[68] J. Ellis, B. Jou and S. Chang, "Why we watch the news: a dataset for exploring," in ACM International Conference on Multimodal, 2014.
[69] P. Nagamma, H. R. Pruthvi and K. K. Nisha, "An improved sentiment analysis of online movie reviews based on clustering for box-office prediction," in International Conference on Computing, Communication & Automation (ICCCA), Noida, India, 2015.
[70] K. R. Apala, M. Jose and S. Motnam, "Prediction of movies box office performance using social media," in IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), Niagara Falls, ON, Canada, 2013.
[71] L. Curini, A. Ceron and S. M. Iacus, "To what extent sentiment analysis of Twitter is able to forecast electoral results? Evidence from France, Italy and the United States," in 7th ECPR General Conference, Bordeaux, 2013.