{"id":1172540,"date":"2021-10-16T08:01:10","date_gmt":"2021-10-16T12:01:10","guid":{"rendered":"https:\/\/www.analyticsvidhya.com\/?p=85362"},"modified":"2021-10-16T08:01:10","modified_gmt":"2021-10-16T12:01:10","slug":"cleaning-and-pre-processing-textual-data-with-neattext-library","status":"publish","type":"station","link":"https:\/\/platodata.io\/plato-data\/cleaning-and-pre-processing-textual-data-with-neattext-library\/","title":{"rendered":"Cleaning and Pre-processing textual data with NeatText library"},"content":{"rendered":"<br \/>\n<h3><\/h3>\n<p class>Unstructured text data can be a problem while solving NLP problems. There is a need to pre-process any unstructured text data in order for us to build an effective NLP model. Hence pre-processing textual data is an important step while building any NLP model. Converting text into numbers is important as the machine learning models take only numbers as inputs. Therefore converting string objects(text) into \u2018int\u2019 objects is necessary. There are many ways to pre-process text. One way is to hard code every step and processes the text data through that code. Another way is to use any Natural Language Processing package that does the work for us using simple commands. One such package is <a href=\"https:\/\/jcharis.github.io\/neattext\/\" target=\"_blank\" rel=\"nofollow noopener\">NeatText<\/a>.<\/p>\n<h3>Table of Contents<\/h3>\n<ol>\n<li>What is NeatText<\/li>\n<li>Components of NeatText<\/li>\n<li>Installation<\/li>\n<li>Using TextFrame<\/li>\n<li>Using TextExtractor<\/li>\n<li>Using Functions<\/li>\n<li>Using Explainer<\/li>\n<li>Conclusion<\/li>\n<\/ol>\n<h2>What is NeatText<\/h2>\n<p>NeatText<span> is a simple Natural Language Processing package for cleaning text data and pre-processing text data. It can be used to clean sentences, extract emails, phone numbers, weblinks, and emojis from sentences. It can also be used to set up text pre-processing pipelines.<\/span><\/p>\n<p class>This library is intended to solve the following problems :<\/p>\n<ul>\n<li>Cleaning unstructured text data.<\/li>\n<li><span>Reducing noise in text.<\/span><\/li>\n<li><span>Avoid repetition of the same code for pre-processing.<\/span><\/li>\n<\/ul>\n<p class>In this article, we shall explore the different components and functionalities of this package using examples. First, let us see the different components in this package.<\/p>\n<h2>Components of NeatText<\/h2>\n<p class>This library offers four components also called objects. They are:<\/p>\n<ol>\n<li>TextFrame: It\u2019s a frame-like object for cleaning and analyzing text.<\/li>\n<li>TextExtractor: It\u2019s also a frame-like object that is used to extract different entities in a given sentence.<\/li>\n<li>Functions: These facilitate us to do different tasks at the same time saving lines of code.<\/li>\n<li>Explainer: This object is used to deal with emojis in sentences.<\/li>\n<\/ol>\n<div class=\"medium-insert-images medium-insert-images-wide\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library.jpg\" alt=\"NeatText\" width=\"761\" height=\"290\"><figcaption class>Image 1<\/figcaption><\/figure>\n<\/div>\n<p class>We shall go through each object in the rest of the article from installing the library to pre-processing text data.<\/p>\n<h2>Installation of NeatText<\/h2>\n<pre>pip install neattext<\/pre>\n<p class>Now that we&nbsp;have installed the NeatText library let\u2019s import and use it.<\/p>\n<pre>import neattext as nt<\/pre>\n<p class>We shall see the functionality of different objects of this library in a sentence.<\/p>\n<p class>Let\u2019s create a sentence.<\/p>\n<pre>my_text = 'His name is Donald trump \ud83d\ude06. He is former president of USA !!!! \ud83d\ude0e . He abused women. His contact number is 0123456789. His email is <a href=\"https:\/\/www.analyticsvidhya.com\/cdn-cgi\/l\/email-protection\" class=\"__cf_email__\" data-cfemail=\"1f6b6d6a726f7b70717e737b5f78727e7673317c707231\">[email&nbsp;protected]<\/a> His website is https:\/\/www.donaldjtrump.com\/'<\/pre>\n<p class>The above sentence contains so much noise like emojis, emails, weblinks, and some special characters. So we will clean that sentence.<\/p>\n<h2>Using TextFrame in NeatText<\/h2>\n<p class>Generally, sentences scrapped from the internet contain a lot of noise like punctuations, emojis, and other special characters. These can be a problem if not removed from the sentence. TextFrame object enables us to remove all this noise and help us get a cleaned sentence.<\/p>\n<p class>This object keeps the text as a TextFrame object which will allow us to do more with text. To use the TextFrame object we have to create an instance of TextFrame on the given sentence. Let\u2019s create an instance called&nbsp;sentx.<\/p>\n<pre>sentx = nt.TextFrame(text=my_text)<\/pre>\n<p>Let\u2019s print the text from the instance.<\/p>\n<pre>sentx.text<\/pre>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-1.jpg\" alt=\"Using TextFrame in NeatText\" width=\"980\" height=\"46\"><\/figure>\n<\/div>\n<p>Now we have an instance sentx, let\u2019s use describe() command.<\/p>\n<pre>sentx.describe()<\/pre>\n<h4>output:<\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-2.jpg\" alt=\"Using TextFrame in NeatText - output\" width=\"169\" height=\"172\"><\/figure>\n<\/div>\n<p class>We can see that this command outputs a detailed description of sentences like the length of sentence, number of stopwords present in the sentence, etc.,<\/p>\n<p class>To count the number of vowels in the sentence \u2013<\/p>\n<pre>sentx.count_vowels()<\/pre>\n<h4>Output:<\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-3.jpg\" alt=\"Using TextFrame in NeatText - vowel count\" width=\"202\" height=\"30\"><\/figure>\n<\/div>\n<p class>To count the number of stopwords in the sentence \u2013<\/p>\n<pre>sentx.count_stopwords()<\/pre>\n<h4>Output:<\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-4.jpg\" alt=\"stopwords - Using TextFrame in NeatText\" width=\"321\" height=\"31\"><\/figure>\n<\/div>\n<p class>Now that we have seen some basic text metrics that this library has to offer, let\u2019s see some text cleaning methods.<\/p>\n<p class>To remove punctuations in the sentence \u2013<\/p>\n<pre>print(sentx.remove_puncts())<\/pre>\n<h4>Output:<\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-5.jpg\" alt=\"remove puncts Using TextFrame in NeatText\" width=\"980\" height=\"42\"><\/figure>\n<\/div>\n<p class>we can see that all punctuations like exclamatory marks and full stops are removed from the sentence.<\/p>\n<p class>To remove stopwords in the sentence \u2013<\/p>\n<pre>print(sentx.remove_stopwords())<\/pre>\n<h4>Output:<\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-6.jpg\" alt=\"remove stopwords | Using TextFrame in NeatText\" width=\"947\" height=\"27\"><\/figure>\n<\/div>\n<p class>We can see that the sentence is free of punctuations and stopwords.<\/p>\n<p class>Next, we remove emojis. To do that use this simple command \u2013<\/p>\n<pre>print(sentx.remove_emojis())<\/pre>\n<h4>Output:<\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-7.jpg\" alt=\"output\" width=\"908\" height=\"28\"><\/figure>\n<\/div>\n<p class>In the same way, we can remove emails and numbers<\/p>\n<h2>Using TextExtractor in NeatText<\/h2>\n<p class>So far we have seen some simple commands to remove punctuations, stopwords, and emojis in a given sentence using the TextFrame object.<\/p>\n<p class>Sometimes sentences contain valuable data like emails, phone numbers, emojis. For example in a given tweet emoji can give us valuable information like the sentiment of the tweet. So Let us use TextExtractor to extract some valuable information from the sentence.<\/p>\n<p class>First, we shall extract the email from the sentence. To do that we have to import TextExtractor and create an instance of it.<\/p>\n<pre>from neattext import TextExtractor\nsentx1 = TextExtractor(text=my_text)<\/pre>\n<p>Now that we have created an instance of it let\u2019s use it to extract email \u2013<\/p>\n<pre>sentx1.extract_emails()<\/pre>\n<h4>Output:<\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-8.jpg\" alt=\"Using TextExtractor in NeatText\" width=\"182\" height=\"26\"><\/figure>\n<\/div>\n<p class>We have extracted the email in the sentence with just one single command. Now let\u2019s extract the emojis from the sentence \u2013<\/p>\n<pre>sentx1.extract_emojis()<\/pre>\n<h4>Output:<\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-9.jpg\" alt=\"Using TextExtractor in NeatText - output\" width=\"86\" height=\"33\"><\/figure>\n<\/div>\n<h2>Using Functions in Neattext<\/h2>\n<p class>Functions offer a unique way to pre-process the sentence. We can do all activities that we did above in one line using functions that will save some lines of code.<\/p>\n<p class>Now let\u2019s use functions to process the sentence. To do that we have to import necessary modules \u2013<\/p>\n<pre>from neattext.functions import clean_text<\/pre>\n<p>Now we will use this to remove URLs, stopwords from the original sentence my_text as follows \u2013<\/p>\n<pre>clean_text(my_text, urls=True, stopwords=True)<\/pre>\n<h4>Output:<\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-10.jpg\" alt=\"Using TextExtractor in NeatText 2\" width=\"770\" height=\"32\"><\/figure>\n<\/div>\n<p class>We can see that the given sentence doesn\u2019t contain any URL or stopwords.<\/p>\n<h2>Using Explainer<\/h2>\n<p class>So far we have seen TextFrame, TextExtractor, and Functions. Now let us see another object of the NeatText library i.e., Explainer.<\/p>\n<p class>Sometimes we might get confused regarding what an emoji could mean and also sometimes we want to create an emoji based on a given expression and in some applications we might want to use Unicode instead of emojis. So there is a need to convert these Unicodes into emojis in the output. Explainer object does all these for us.<\/p>\n<p>The explainer can do three kinds of activities. They are<\/p>\n<ul>\n<li>emojify() \u2013&nbsp; It creates an emoji based on a given emotional expression.<\/li>\n<li>emoji_explainer() \u2013 It explains the meaning behind the given emoji.<\/li>\n<li>unicode_2_emoji() \u2013 It converts Unicode to emoji.<\/li>\n<\/ul>\n<p class>Let\u2019s see how each one works. Firstly we will import these functions from the explainer in the following way \u2013<\/p>\n<pre>from neattext.explainer import emojify, emoji_explainer, unicode_2_emoji<\/pre>\n<p>Now that we have imported them let\u2019s use them. First, we will see emojify()<\/p>\n<pre>emojify('cry')<\/pre>\n<p>Output:<\/p>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-11.jpg\" alt=\"explainer\" width=\"35\" height=\"27\"><\/figure>\n<\/div>\n<p class>We can see that it has produced an emoji based on the emotion we gave to it.<\/p>\n<p class>Now let\u2019s see emoji_explainer() \u2013<\/p>\n<pre>emoji_explainer('<span>\ud83d\ude0d')<\/span><\/pre>\n<h4><span>Output:<\/span><\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-12.jpg\" alt=\"explainer - output\" width=\"303\" height=\"29\"><\/figure>\n<\/div>\n<p class>The output shows the description of the given emoji.<\/p>\n<p class>Let\u2019s see unicode_2_emoji()<\/p>\n<pre>unicode_2_emoji('0x1f49b')<\/pre>\n<h4>Output:<\/h4>\n<div class=\"medium-insert-images\">\n<figure><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" src=\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library-13.jpg\" alt=\"explainer 2\" width=\"33\" height=\"30\"><\/figure>\n<\/div>\n<h2>Conclusion<\/h2>\n<p>We have seen all the components of the NeatText library. We have removed several unwanted things from the sentence and made it ready for further processing. We also saved a lot of time writing functions for pre-processing. This library comes in handy to do any kind of pre-processing on text data.<\/p>\n<h3>Image Source-<\/h3>\n<p>Image 1: https:\/\/jcharis.github.io\/neattext\/<\/p>\n<div class=\"medium-insert-buttons\" data-active-addon=\"images\">\n<div>\n<div class=\"medium-insert-buttons\">\n<div class=\"medium-insert-buttons\" data-active-addon=\"images\">\n<div class=\"medium-insert-buttons\">\n<div readability=\"6.5\">\n<div class=\"medium-insert-buttons\" readability=\"8\">\n<p><strong>The media shown in this article are not owned by Analytics Vidhya and are used at the Author\u2019s discretion.<\/strong><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<p> <a href=\"https:\/\/www.analyticsvidhya.com\/blog\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library\/\">Source: https:\/\/www.analyticsvidhya.com\/blog\/2021\/10\/cleaning-and-pre-processing-textual-data-with-neattext-library\/<\/a><\/p>\n","protected":false},"author":1,"featured_media":1172541,"template":"","meta":{"_eb_attr":"","type":"","auto_type":false,"post":"","stream":"","stream_url":"","waveform_data":[],"duration":0,"start":0,"end":0,"bpm":0,"downloadable":false,"download_url":"","purchase_title":"","purchase_url":"","post-count-all":0,"like_count":0,"download_count":0,"editor_note":"","copyright":"","captions":[],"sources":[]},"genre":[42022],"station_tag":[4520,3761,4046,3681,4135,4526,4244,7101,3720,3642,4155,3834,3949,4167,3899,44292,12620,3772,4742,4178,5646,40320,3694,3650,3908,4191,3653,4572,4010,5388,457,4089,4353,475,4094,43581,16265,16762,7017,4748,3855,4096,3662,3663,4323,3969,4430,4260,4624,3812,3976,4461,3779,4392,3671,3980,3675,44293,3984,5306,44004,4396,6277,3935,4240,467],"artist":[42023],"mood":[],"activity":[],"_links":{"self":[{"href":"https:\/\/platodata.io\/wp-json\/wp\/v2\/station\/1172540"}],"collection":[{"href":"https:\/\/platodata.io\/wp-json\/wp\/v2\/station"}],"about":[{"href":"https:\/\/platodata.io\/wp-json\/wp\/v2\/types\/station"}],"author":[{"embeddable":true,"href":"https:\/\/platodata.io\/wp-json\/wp\/v2\/users\/1"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/platodata.io\/wp-json\/"}],"wp:attachment":[{"href":"https:\/\/platodata.io\/wp-json\/wp\/v2\/media?parent=1172540"}],"wp:term":[{"taxonomy":"genre","embeddable":true,"href":"https:\/\/platodata.io\/wp-json\/wp\/v2\/genre?post=1172540"},{"taxonomy":"station_tag","embeddable":true,"href":"https:\/\/platodata.io\/wp-json\/wp\/v2\/station_tag?post=1172540"},{"taxonomy":"artist","embeddable":true,"href":"https:\/\/platodata.io\/wp-json\/wp\/v2\/artist?post=1172540"},{"taxonomy":"mood","embeddable":true,"href":"https:\/\/platodata.io\/wp-json\/wp\/v2\/mood?post=1172540"},{"taxonomy":"activity","embeddable":true,"href":"https:\/\/platodata.io\/wp-json\/wp\/v2\/activity?post=1172540"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}