nltk movie reviews corpus download

Sentiment Analysis: First Steps With Python's NLTK Library

Table of Contents

Installing and Importing

Compiling data, creating frequency distributions, extracting concordance and collocations, using nltk’s pre-trained sentiment analyzer, selecting useful features, training and using a classifier, installing and importing scikit-learn, using scikit-learn classifiers with nltk.

Once you understand the basics of Python , familiarizing yourself with its most popular packages will not only boost your mastery over the language but also rapidly increase your versatility. In this tutorial, you’ll learn the amazing capabilities of the Natural Language Toolkit (NLTK) for processing and analyzing text, from basic functions to sentiment analysis powered by machine learning !

Sentiment analysis can help you determine the ratio of positive to negative engagements about a specific topic. You can analyze bodies of text, such as comments, tweets, and product reviews, to obtain insights from your audience. In this tutorial, you’ll learn the important features of NLTK for processing text data and the different approaches you can use to perform sentiment analysis on your data.

By the end of this tutorial, you’ll be ready to:

• Split and filter text data in preparation for analysis
• Analyze word frequency
• Find concordance and collocations using different methods
• Perform quick sentiment analysis with NLTK’s built-in classifier
• Define features for custom classification
• Use and compare classifiers for sentiment analysis with NLTK

Free Bonus: Click here to get our free Python Cheat Sheet that shows you the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Getting Started With NLTK

The NLTK library contains various utilities that allow you to effectively manipulate and analyze linguistic data. Among its advanced features are text classifiers that you can use for many kinds of classification, including sentiment analysis.

Sentiment analysis is the practice of using algorithms to classify various samples of related text into overall positive and negative categories. With NLTK, you can employ these algorithms through powerful built-in machine learning operations to obtain insights from linguistic data.

You’ll begin by installing some prerequisites, including NLTK itself as well as specific resources you’ll need throughout this tutorial.

First, use pip to install NLTK:

While this will install the NLTK module, you’ll still need to obtain a few additional resources. Some of them are text samples, and others are data models that certain NLTK functions require.

To get the resources you’ll need, use nltk.download() :

NLTK will display a download manager showing all available and installed resources. Here are the ones you’ll need to download for this tutorial:

• names : A list of common English names compiled by Mark Kantrowitz
• stopwords : A list of really common words, like articles, pronouns, prepositions, and conjunctions
• state_union : A sample of transcribed State of the Union addresses by different US presidents, compiled by Kathleen Ahrens
• twitter_samples : A list of social media phrases posted to Twitter
• movie_reviews : Two thousand movie reviews categorized by Bo Pang and Lillian Lee
• averaged_perceptron_tagger : A data model that NLTK uses to categorize words into their part of speech
• vader_lexicon : A scored list of words and jargon that NLTK references when performing sentiment analysis, created by C.J. Hutto and Eric Gilbert
• punkt : A data model created by Jan Strunk that NLTK uses to split full texts into word lists

Note: Throughout this tutorial, you’ll find many references to the word corpus and its plural form, corpora . A corpus is a large collection of related text samples. In the context of NLTK, corpora are compiled with features for natural language processing (NLP) , such as categories and numerical scores for particular features.

A quick way to download specific resources directly from the console is to pass a list to nltk.download() :

This will tell NLTK to find and download each resource based on its identifier.

Should NLTK require additional resources that you haven’t installed, you’ll see a helpful LookupError with details and instructions to download the resource:

The LookupError specifies which resource is necessary for the requested operation along with instructions to download it using its identifier.

NLTK provides a number of functions that you can call with few or no arguments that will help you meaningfully analyze text before you even touch its machine learning capabilities. Many of NLTK’s utilities are helpful in preparing your data for more advanced analysis.

Soon, you’ll learn about frequency distributions, concordance, and collocations. But first, you need some data.

Start by loading the State of the Union corpus you downloaded earlier:

Note that you build a list of individual words with the corpus’s .words() method, but you use str.isalpha() to include only the words that are made up of letters. Otherwise, your word list may end up with “words” that are only punctuation marks.

Have a look at your list. You’ll notice lots of little words like “of,” “a,” “the,” and similar. These common words are called stop words , and they can have a negative effect on your analysis because they occur so often in the text. Thankfully, there’s a convenient way to filter them out.

NLTK provides a small corpus of stop words that you can load into a list:

Make sure to specify english as the desired language since this corpus contains stop words in various languages.

Now you can remove stop words from your original word list:

Since all words in the stopwords list are lowercase, and those in the original list may not be, you use str.lower() to account for any discrepancies. Otherwise, you may end up with mixedCase or capitalized stop words still in your list.

While you’ll use corpora provided by NLTK for this tutorial, it’s possible to build your own text corpora from any source. Building a corpus can be as simple as loading some plain text or as complex as labeling and categorizing each sentence. Refer to NLTK’s documentation for more information on how to work with corpus readers .

For some quick analysis, creating a corpus could be overkill. If all you need is a word list, there are simpler ways to achieve that goal. Beyond Python’s own string manipulation methods, NLTK provides nltk.word_tokenize() , a function that splits raw text into individual words. While tokenization is itself a bigger topic (and likely one of the steps you’ll take when creating a custom corpus), this tokenizer delivers simple word lists really well.

To use it, call word_tokenize() with the raw text you want to split:

Now you have a workable word list! Remember that punctuation will be counted as individual words, so use str.isalpha() to filter them out later.

Now you’re ready for frequency distributions . A frequency distribution is essentially a table that tells you how many times each word appears within a given text. In NLTK, frequency distributions are a specific object type implemented as a distinct class called FreqDist . This class provides useful operations for word frequency analysis.

To build a frequency distribution with NLTK, construct the nltk.FreqDist class with a word list:

This will create a frequency distribution object similar to a Python dictionary but with added features.

Note: Type hints with generics as you saw above in words: list[str] = ... is a new feature in Python 3.9 !

After building the object, you can use methods like .most_common() and .tabulate() to start visualizing information:

These methods allow you to quickly determine frequently used words in a sample. With .most_common() , you get a list of tuples containing each word and how many times it appears in your text. You can get the same information in a more readable format with .tabulate() .

In addition to these two methods, you can use frequency distributions to query particular words. You can also use them as iterators to perform some custom analysis on word properties.

For example, to discover differences in case, you can query for different variations of the same word:

These return values indicate the number of times each word occurs exactly as given.

Since frequency distribution objects are iterable , you can use them within list comprehensions to create subsets of the initial distribution. You can focus these subsets on properties that are useful for your own analysis.

Try creating a new frequency distribution that’s based on the initial one but normalizes all words to lowercase:

Now you have a more accurate representation of word usage regardless of case.

Think of the possibilities: You could create frequency distributions of words starting with a particular letter, or of a particular length, or containing certain letters. Your imagination is the limit!

In the context of NLP, a concordance is a collection of word locations along with their context. You can use concordances to find:

• How many times a word appears
• Where each occurrence appears
• What words surround each occurrence

In NLTK, you can do this by calling .concordance() . To use it, you need an instance of the nltk.Text class, which can also be constructed with a word list.

Before invoking .concordance() , build a new word list from the original corpus text so that all the context, even stop words, will be there:

Note that .concordance() already ignores case, allowing you to see the context of all case variants of a word in order of appearance. Note also that this function doesn’t show you the location of each word in the text.

Additionally, since .concordance() only prints information to the console, it’s not ideal for data manipulation. To obtain a usable list that will also give you information about the location of each occurrence, use .concordance_list() :

.concordance_list() gives you a list of ConcordanceLine objects, which contain information about where each word occurs as well as a few more properties worth exploring. The list is also sorted in order of appearance.

The nltk.Text class itself has a few other interesting features. One of them is .vocab() , which is worth mentioning because it creates a frequency distribution for a given text.

Revisiting nltk.word_tokenize() , check out how quickly you can create a custom nltk.Text instance and an accompanying frequency distribution:

.vocab() is essentially a shortcut to create a frequency distribution from an instance of nltk.Text . That way, you don’t have to make a separate call to instantiate a new nltk.FreqDist object.

Another powerful feature of NLTK is its ability to quickly find collocations with simple function calls. Collocations are series of words that frequently appear together in a given text. In the State of the Union corpus, for example, you’d expect to find the words United and States appearing next to each other very often. Those two words appearing together is a collocation.

Collocations can be made up of two or more words. NLTK provides classes to handle several types of collocations:

• Bigrams: Frequent two-word combinations
• Trigrams: Frequent three-word combinations
• Quadgrams: Frequent four-word combinations

NLTK provides specific classes for you to find collocations in your text. Following the pattern you’ve seen so far, these classes are also built from lists of words:

The TrigramCollocationFinder instance will search specifically for trigrams. As you may have guessed, NLTK also has the BigramCollocationFinder and QuadgramCollocationFinder classes for bigrams and quadgrams, respectively. All these classes have a number of utilities to give you information about all identified collocations.

One of their most useful tools is the ngram_fd property. This property holds a frequency distribution that is built for each collocation rather than for individual words.

Using ngram_fd , you can find the most common collocations in the supplied text:

You don’t even have to create the frequency distribution, as it’s already a property of the collocation finder instance.

Now that you’ve learned about some of NLTK’s most useful tools, it’s time to jump into sentiment analysis!

NLTK already has a built-in, pretrained sentiment analyzer called VADER ( V alence A ware D ictionary and s E ntiment R easoner).

Since VADER is pretrained, you can get results more quickly than with many other analyzers. However, VADER is best suited for language used in social media, like short sentences with some slang and abbreviations. It’s less accurate when rating longer, structured sentences, but it’s often a good launching point.

To use VADER, first create an instance of nltk.sentiment.SentimentIntensityAnalyzer , then use .polarity_scores() on a raw string :

You’ll get back a dictionary of different scores. The negative, neutral, and positive scores are related: They all add up to 1 and can’t be negative. The compound score is calculated differently. It’s not just an average, and it can range from -1 to 1.

Now you’ll put it to the test against real data using two different corpora. First, load the twitter_samples corpus into a list of strings, making a replacement to render URLs inactive to avoid accidental clicks:

Notice that you use a different corpus method, .strings() , instead of .words() . This gives you a list of raw tweets as strings.

Different corpora have different features, so you may need to use Python’s help() , as in help(nltk.corpus.tweet_samples) , or consult NLTK’s documentation to learn how to use a given corpus.

Now use the .polarity_scores() function of your SentimentIntensityAnalyzer instance to classify tweets:

In this case, is_positive() uses only the positivity of the compound score to make the call. You can choose any combination of VADER scores to tweak the classification to your needs.

Now take a look at the second corpus, movie_reviews . As the name implies, this is a collection of movie reviews. The special thing about this corpus is that it’s already been classified. Therefore, you can use it to judge the accuracy of the algorithms you choose when rating similar texts.

Keep in mind that VADER is likely better at rating tweets than it is at rating long movie reviews. To get better results, you’ll set up VADER to rate individual sentences within the review rather than the entire text.

Since VADER needs raw strings for its rating, you can’t use .words() like you did earlier. Instead, make a list of the file IDs that the corpus uses, which you can use later to reference individual reviews:

.fileids() exists in most, if not all, corpora. In the case of movie_reviews , each file corresponds to a single review. Note also that you’re able to filter the list of file IDs by specifying categories. This categorization is a feature specific to this corpus and others of the same type.

Next, redefine is_positive() to work on an entire review. You’ll need to obtain that specific review using its file ID and then split it into sentences before rating:

.raw() is another method that exists in most corpora. By specifying a file ID or a list of file IDs, you can obtain specific data from the corpus. Here, you get a single review, then use nltk.sent_tokenize() to obtain a list of sentences from the review. Finally, is_positive() calculates the average compound score for all sentences and associates a positive result with a positive review.

You can take the opportunity to rate all the reviews and see how accurate VADER is with this setup:

After rating all reviews, you can see that only 64 percent were correctly classified by VADER using the logic defined in is_positive() .

A 64 percent accuracy rating isn’t great, but it’s a start. Have a little fun tweaking is_positive() to see if you can increase the accuracy.

In the next section, you’ll build a custom classifier that allows you to use additional features for classification and eventually increase its accuracy to an acceptable level.

Customizing NLTK’s Sentiment Analysis

NLTK offers a few built-in classifiers that are suitable for various types of analyses, including sentiment analysis. The trick is to figure out which properties of your dataset are useful in classifying each piece of data into your desired categories.

In the world of machine learning, these data properties are known as features , which you must reveal and select as you work with your data. While this tutorial won’t dive too deeply into feature selection and feature engineering , you’ll be able to see their effects on the accuracy of classifiers.

Since you’ve learned how to use frequency distributions, why not use them as a launching point for an additional feature?

By using the predefined categories in the movie_reviews corpus, you can create sets of positive and negative words, then determine which ones occur most frequently across each set. Begin by excluding unwanted words and building the initial category groups:

This time, you also add words from the names corpus to the unwanted list on line 2 since movie reviews are likely to have lots of actor names, which shouldn’t be part of your feature sets. Notice pos_tag() on lines 14 and 18, which tags words by their part of speech.

It’s important to call pos_tag() before filtering your word lists so that NLTK can more accurately tag all words. skip_unwanted() , defined on line 4, then uses those tags to exclude nouns, according to NLTK’s default tag set .

Now you’re ready to create the frequency distributions for your custom feature. Since many words are present in both positive and negative sets, begin by finding the common set so you can remove it from the distribution objects:

Once you’re left with unique positive and negative words in each frequency distribution object, you can finally build sets from the most common words in each distribution. The amount of words in each set is something you could tweak in order to determine its effect on sentiment analysis.

This is one example of a feature you can extract from your data, and it’s far from perfect. Looking closely at these sets, you’ll notice some uncommon names and words that aren’t necessarily positive or negative. Additionally, the other NLTK tools you’ve learned so far can be useful for building more features. One possibility is to leverage collocations that carry positive meaning, like the bigram “thumbs up!”

Here’s how you can set up the positive and negative bigram finders:

The rest is up to you! Try different combinations of features, think of ways to use the negative VADER scores, create ratios, polish the frequency distributions. The possibilities are endless!

With your new feature set ready to use, the first prerequisite for training a classifier is to define a function that will extract features from a given piece of data.

Since you’re looking for positive movie reviews, focus on the features that indicate positivity, including VADER scores:

extract_features() should return a dictionary, and it will create three features for each piece of text:

• The average compound score
• The average positive score
• The amount of words in the text that are also part of the top 100 words in all positive reviews

In order to train and evaluate a classifier, you’ll need to build a list of features for each text you’ll analyze:

Each item in this list of features needs to be a tuple whose first item is the dictionary returned by extract_features and whose second item is the predefined category for the text. After initially training the classifier with some data that has already been categorized (such as the movie_reviews corpus), you’ll be able to classify new data.

Training the classifier involves splitting the feature set so that one portion can be used for training and the other for evaluation, then calling .train() :

Since you’re shuffling the feature list, each run will give you different results. In fact, it’s important to shuffle the list to avoid accidentally grouping similarly classified reviews in the first quarter of the list.

Adding a single feature has marginally improved VADER’s initial accuracy, from 64 percent to 67 percent. More features could help, as long as they truly indicate how positive a review is. You can use classifier.show_most_informative_features() to determine which features are most indicative of a specific property.

To classify new data, find a movie review somewhere and pass it to classifier.classify() . You can also use extract_features() to tell you exactly how it was scored:

Was it correct? Based on the scoring output from extract_features() , what can you improve?

Feature engineering is a big part of improving the accuracy of a given algorithm, but it’s not the whole story. Another strategy is to use and compare different classifiers.

Comparing Additional Classifiers

NLTK provides a class that can use most classifiers from the popular machine learning framework scikit-learn .

Many of the classifiers that scikit-learn provides can be instantiated quickly since they have defaults that often work well. In this section, you’ll learn how to integrate them within NLTK to classify linguistic data.

Like NLTK, scikit-learn is a third-party Python library, so you’ll have to install it with pip :

After you’ve installed scikit-learn, you’ll be able to use its classifiers directly within NLTK.

The following classifiers are a subset of all classifiers available to you. These will work within NLTK for sentiment analysis:

With these classifiers imported, you’ll first have to instantiate each one. Thankfully, all of these have pretty good defaults and don’t require much tweaking.

To aid in accuracy evaluation, it’s helpful to have a mapping of classifier names and their instances:

Now you can use these instances for training and accuracy evaluation.

Since NLTK allows you to integrate scikit-learn classifiers directly into its own classifier class, the training and classification processes will use the same methods you’ve already seen, .train() and .classify() .

You’ll also be able to leverage the same features list you built earlier by means of extract_features() . To refresh your memory, here’s how you built the features list:

The features list contains tuples whose first item is a set of features given by extract_features() , and whose second item is the classification label from preclassified data in the movie_reviews corpus.

Since the first half of the list contains only positive reviews, begin by shuffling it, then iterate over all classifiers to train and evaluate each one:

For each scikit-learn classifier, call nltk.classify.SklearnClassifier to create a usable NLTK classifier that can be trained and evaluated exactly like you’ve seen before with nltk.NaiveBayesClassifier and its other built-in classifiers. The .train() and .accuracy() methods should receive different portions of the same list of features.

Now you’ve reached over 73 percent accuracy before even adding a second feature! While this doesn’t mean that the MLPClassifier will continue to be the best one as you engineer new features, having additional classification algorithms at your disposal is clearly advantageous.

You’re now familiar with the features of NTLK that allow you to process text into objects that you can filter and manipulate, which allows you to analyze text data to gain information about its properties. You can also use different classifiers to perform sentiment analysis on your data and gain insights about how your audience is responding to content.

In this tutorial, you learned how to:

• Perform quick sentiment analysis with NLTK’s built-in VADER
• Use and compare classifiers from scikit-learn for sentiment analysis within NLTK

With these tools, you can start using NLTK in your own projects. For some inspiration, have a look at a sentiment analysis visualizer , or try augmenting the text processing in a Python web application while learning about additional popular packages!

🐍 Python Tricks 💌

Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.

About Marius Mogyorosi

Marius is a tinkerer who loves using Python for creative projects within and beyond the software security field.

Each tutorial at Real Python is created by a team of developers so that it meets our high quality standards. The team members who worked on this tutorial are:

Master Real-World Python Skills With Unlimited Access to Real Python

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

What Do You Think?

What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.

Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students. Get tips for asking good questions and get answers to common questions in our support portal . Looking for a real-time conversation? Visit the Real Python Community Chat or join the next “Office Hours” Live Q&A Session . Happy Pythoning!

Keep Learning

Related Topics: intermediate data-science machine-learning

Keep reading Real Python by creating a free account or signing in:

Already have an account? Sign-In

Almost there! Complete this form and click the button below to gain instant access:

Get the Python Cheat Sheet (Free PDF)

🔒 No spam. We take your privacy seriously.

Build a Sentiment Analysis app with Movie Reviews

Shantnu Tiwari

0. Introduction to NLP and Sentiment Analysis

1. Natural Language Processing with NTLK

2. Intro to NTLK, Part 2

3. Build a sentiment analysis program

4. Sentiment Analysis with Twitter

5. Analysing the Enron Email Corpus

6. Build a Spam Filter using the Enron Corpus

So now we use everything we have learnt to build a Sentiment Analysis app.

Sentiment Analysis means finding the mood of the public about things like movies, politicians, stocks, or even current events. We will analyse the sentiment of the movie reviews corpus we saw earlier.

Only interested in videos? Go here for the next part: Sentiment Analysis with Twitter

Let’s import our libraries:

We will be using the Naive Bayes classifier for this example. The Naive Bayes is a fairly simple machine learning algorithm, that works mainly with probabilities. Stack Overflow has a great (if slightly long) explanation of how it works. The top 2 answers are worth reading.

Before we start, there is something that had me stumped for a long time. I saw it in all the examples, but it didn’t make sense. But the Naive Bayes classifier, especially in the Nltk library, expects the input to be in this format: Every word must be followed by true. So for example, if you have these words:

you need to pass it in as:

NOTE: It’s just a quirk of the nltk library. What bothers me is none of the dozens of tutorials/videos I looked at make this clear. They just write this weird code to do so, and expect you to figure it out for yourselves. (Hurray for us).

I’ll show you the function I wrote, and hopefully, you will understand why we need to do it this way. Here is the function:

Let’s go over it line by line.

The first thing we do is remove all stopwords. This is what we did in the last lesson. This step is optional.

For each word, we create a dictionary with all the words and True . Why a dictionary? So that words are not repeated. If a word already exists, it won’t be added to the dictionary.

Let’s see how this works:

We call our function with the string “the quick brown quick a fox”.

You can see that a) The stop words are removed b) Repeat words are removed c) There is a True with each word.

Again, this is just the format the Naive Bayes classifier in nltk expects.

Okay, let’s start with the code. Remember, the sentiment analysis code is just a machine learning algorithm that has been trained to identify positive/negative reviews.

We create an empty list called neg_reviews . Next, we loop over all the files in the neg folder.

We get all the words in that file.

Then we use the function we wrote earlier to create word features in the format nltk expects. Here is a sample of the output:

So there are a 1000 negative reviews.

Let’s do the same for the positive reviews. The code is exactly the same:

So we have a 1000 negative and 1000 positive reviews, for a total of 2000. We will now create our test and train samples, this time manually:

We end up with 1500 training samples and 500 test.

Let’s create our Naive Bayes Classifier, and train it with our training set.

And let’s use our test set to find the accuracy:

Ac accuracy of 72%. Could you improve it? How?

For now, I want to show you how to classify a review as negative or positive. But before that, a warning.

The problem with sentiment analysis, as with any machine learning approach, is that your algorithm is only as good as your data. If your data is crap, your algorithm will be crap.

Not only that, the algorithm depends on the type of input you train it with. So if you train your data with long movie reviews, it will not work with Twitter data, which is much shorter.

This particular dataset is, imo, a bit short. Also, the reviews are very informal, using a lot of swear words etc. Which is why I found it not very accurate when comparing it to Imdb reviews, where swearing is discouraged and reviews are (slightly) more formal.

Anyway, I was looking for negative and positive reviews. Our algorithm is more accurate when the review contains stronger words ( horrible instead of bad ). For the bad reviews, I found this gem of a movie. A real masterpiece:

We need to word_tokenize the text, call our function it, and then use the classify() function to let our algorithm decide if this is a positive or negative review.

That was correct, but only because the review was really scathing.

For the positive review, I chose one of my favourite movies, Spirited Away , a very beautiful movie:

Repeat the steps:

Correct again, but I’d like to repeat, the classifier isn’t very accurate overall, I suspect because the original sample is very small and not very representative for Imdb reviews. But it’s good enough for learning.

Okay, the next video is not just a practice session, but also contains some learning exercises, so I strongly recommend you do it. We will build a Sentiment analysis engine with Twitter data.

Sign up to know when the next post is out, and get exclusive content

Master Sentiment Analysis in Python with NLTK using Python 3: A Comprehensive Guide for Aspiring Python Developers

Introduction

Whether you’re a budding developer or a seasoned coder, sentiment analysis is a powerful skill to add to your toolkit. In this comprehensive guide, we’ll explore the ins and outs of sentiment analysis using Python and NLTK. So, buckle up, fire up PyCharm, and let’s dive in!

Understanding Sentiment Analysis

Before we jump into coding, let’s take a moment to understand what sentiment analysis is all about. In a nutshell, sentiment analysis involves determining the emotional tone behind a piece of text. Is it positive, negative, or neutral? This skill is particularly useful in various applications, from social media monitoring to customer feedback analysis.

Setting Up Your Environment

First things first, let’s ensure our development environment is ready to roll. Open up PyCharm and create a new Python project. Make sure you have NLTK installed by running:

Now, let’s get our hands dirty with some real code!

Loading NLTK and Preparing Data

We’ll start by importing the NLTK library and loading a sample dataset. For this guide, we’ll use the classic IMDb movie reviews dataset.

Preprocessing Text Data

Raw text data needs a bit of cleaning before we can extract meaningful insights. Let’s tokenize the words and apply some stemming to reduce words to their root form.

Feature Extraction

Now, let’s prepare our feature set. We’ll use the most common words as features.

Creating a Feature Set

Our next step is to create a feature set for each review, indicating which of the top words are present.

Training and Testing the Model

Now comes the exciting part – training our sentiment analysis model using the Naive Bayes classifier.

Congratulations! You’ve just trained your first sentiment analysis model using NLTK. But what’s the fun without visualization?

Visualizing Results

Let’s add some visual flair to our analysis by plotting the most informative features.

And there you have it – a detailed guide on sentiment analysis using NLTK and Python. We’ve covered everything from setting up your environment to training a model and visualizing the results. The journey doesn’t end here – sentiment analysis is a vast field with plenty of room for exploration.

As you celebrate another year of coding and learning, take a moment to appreciate how far you’ve come. Sentiment analysis is just one stepping stone in your Python journey. Here’s to another year of growth, learning, and mastering the art of Pythonic magic!

Now, fire up your PyCharm, experiment with different datasets, and let the world know how you’re using sentiment analysis in your projects.

Also, check out our other playlist  Rasa Chatbot ,  Internet of things ,  Docker ,  Python Programming ,  Machine Learning , Natural Language Processing ,  MQTT ,  Tech News ,  ESP-IDF  etc. Become a member of our social family on  youtube  here. Stay tuned and Happy Learning. ✌🏻😃 Happy coding, and may your NLP endeavors be both enlightening and rewarding! ❤️🔥🚀

Share this:

About Ashish saini

You may like these posts.

Know the Secrets of Topic Identification in NLTK with Python 3: A Step-by-Step Guide for Aspiring Python Pros

Build your Chatbots and Virtual Assistants with NLTK in Python 3: A Comprehensive Guide

Unleashing the Power of Market Intelligence: A Comprehensive Guide to NLTK with Python 3

Question Answering in NLTK in Python 3: A Comprehensive Guide to Question Answering

Master Information Retrieval with NLTK in Python 3: A Comprehensive Guide for Python Enthusiasts

Leave a reply cancel reply.

You must be logged in to post a comment.

Privacy Policy - Cancellation/Refund Policy - Terms and Conditions

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications You must be signed in to change notification settings

The movie_reviews corpus contains 2K movie reviews with sentiment polarity classification.Here, we have two categories for classification. They are: positive and negative. The movie_reviews corpus already has the reviews categorized as positive and negative.Using NLTK , we can predict whether a review is positive and negative.

tanishq9/Movie-Reviews-NLP

Folders and files.

Repository files navigation

Movie-reviews-nlp.

The Movie Review Data is a collection of movie reviews retrieved from the imdb.com website in the early 2000s by Bo Pang and Lillian Lee. The reviews were collected and made available as part of their research on natural language processing.The dataset is comprised of 1,000 positive and 1,000 negative movie reviews drawn from an archive of the rec.arts.movies.reviews newsgroup hosted at IMDB. The authors refer to this dataset as the “polarity dataset“.

They are: positive and negative. The movie_reviews corpus already has the reviews categorized as positive and negative.We predict whether a review is positive and negative.

• Jupyter Notebook 100.0%

HyperionDev Blog

Software Development and Coding

Python NLP tutorial: Using NLTK for natural language processing

In the broad field of artificial intelligence, the ability to parse and understand natural language is an important goal with many applications. Online retailers and service providers may wish to analyse the feedback of their reviews; governments may need to understand the content of large-scale surveys and responses; or researchers may attempt to determine the sentiments expressed towards certain topics or people on social media. There are many areas where the processing of natural language is required, and this NLP tutorial will step through some of the key elements involved in Natural Language Processing.

The difficulty of understanding natural language is tied to the fact that text data is unstructured . This means it doesn’t come in an easy format to analyse and interpret. When performing data analysis, we want to be able to evaluate the information quantitatively, but text is inherently qualitative. Natural language processing is the process of transforming a piece of text into a structured format that a computer can process and begin to understand.

Requirements and Setup

For this tutorial we will be using Python 3, so check that this is installed by opening up your terminal and running the following command.

If you get stuck at any stage during this tutorial, get your own personal mentor to help you learn coding and switch careers.

Now we can set up a new directory for our project and navigate into it.

You can also optionally use a virtual environment for this project, with the following two lines of code which create our virtual environment and then activate it.

We’re now ready to install the library that we will be using, called Natural Language Toolkit (NLTK). This library provides us with many language processing tools to help format our data. We can use pip , a tool for installing Python packages, to install NLTK on our machine.

We also need to download the necessary data within NLTK. As it’s a large library, we only need to download the parts of it which we intend to use. To do this, first open an interactive Python shell or new file, import the NLTK library, and then open the download window:

A window should open which looks similar to the one below. Here we want to select the book collection, and click download.

The download screen from NLTK

Our setup is now complete and we can begin the tutorial.

To perform natural language processing, we need some data containing natural language to work with. Often you can collect your own data for projects by scraping the web or downloading existing files. The NLTK library helpfully comes with a few large datasets built in and these are easy to import directly. These include classic novels, film scripts, tweets, speeches, and even real-life conversations overheard in New York. We will be using a set of movie reviews for our analysis.

After importing a selected dataset, you can call a ‘readme’ function to learn more about the structure and purpose of the collected data.

We can start trying to understand the data by simply printing words and frequencies to the console, to see what we are dealing with. To get the entire collection of movie reviews as one chunk of data, we use the raw text function:

As you will see, this outputs each of the reviews with no formatting at all. If we print just the first element, the output is a single character.

We can use the ‘words’ function to split our movie reviews into individual words, and store them all in one corpus. Now we can start to analyse things like individual words and their frequencies within the corpus.

We can calculate the frequency distribution (counting the number of times each word is used in the corpus), and then plot or print out the top k words. This should show us which words are used most often. Let’s take a look at the top 50.

The first line of our output is telling us how many total words are in our corpus (1,583,820 outcomes), and how many unique words this contains (39,768 samples). The next line then begins the list of the top 50 most frequently appearing words, and we can save our plotted graph as a .png file.

We can start to see that attempting analysis on the text in its natural format is not yielding useful results – the inclusion of punctuation and common words such as ‘the’ does not help us understand the content or meaning of the text. The next few sections will explain some of the most common and useful steps involved in NLP, which begin to transform the given text documents into something that can be analysed and utilised more effectively.

Tokenization

When we previously split our raw text document into individual words, this was a process very similar to the concept of tokenisation. Tokenisation aims to take a document and break it down into individual ‘tokens’ (often words), and store these in a new data structure. Other forms of minor formatting can be applied here too. For example, all punctuation could be removed.

To test this out on an individual review, first we will need to split our raw text document up, this time by review instead of by word.

The movie reviews are stored according to a file ID such as ‘neg/cv000_29416.txt’, so this code loops  through each of the file IDs, and assigning the raw text associated with that ID to an empty ‘reviews’ list. We will now be able to call, for example, review[0] to look at the first review by itself.

Now taking the first review in its natural form, we can break it down even further using tokenizers. The sent_tokenize function will split the review into tokens of sentences, and word_tokenize will split the review into tokens of words.

There are lots of options for tokenizing in NLTK which you can read about in the API documentation [here]( http://www.nltk.org/api/nltk.tokenize.html ). We are going to combine a regular expression tokenizer to remove punctuation, along with the Python function for transforming strings to lowercase, to build our final tokenizer.

We’re starting to head towards a more concise representation of the reviews, which only holds important and useful parts of the text. One further key step in NLP is the removal of stop words, for example ‘the’, ‘and’, ‘to’, which add no value in terms of content or meaning and are used very frequently in almost all forms of text. To do this we can run our document against a predefined list of stop words and remove matching instances.

This has reduced the number of tokens in the first review from 726 to 343 – so we can see that nearly half the words in this instance were essentially redundant.

Lemmatization and Stemming

Often different inflections of a word have the same general meaning (at least in terms of data analysis), and it may be useful to group them together as one. For example, instead of handling the words ‘walk’, ‘walks’, ‘walked’, ‘walking’ individually, we may want to treat them all as the same word. There are two common approaches to this – lemmatization and stemming.

Lemmatization takes any inflected form of a word and returns its base form – the lemma . To achieve this, we need some context to the word use, such as whether it is a noun or adjective. Stemming is a somewhat cruder attempt at generating a root form, often returning a word which is simply the first few characters that are consistent in any form of the word (but not always a real word itself). To understand this better, let’s test an example.

First we’ll import and define a stemmer and lemmatizer – there are different versions available but these are two of the most popularly used. Next we’ll test the resulting word generated by each approach.

As you can see, the stemmer in this case has outputted the word ‘worri’ which is not a real word, but a stemmed version of ‘worry’. For the lemmatizer to correctly lemmatize the word ‘worrying’ it needs to know whether this word has been used as a verb or adjective. The process for assigning these contextual tags is called part-of-speech tagging and is explained in the following section. Although stemming is a less thorough approach compared to lemmatization, in practise it oftens perform equally or only negibly worse. You can test other words such as ‘walking’ and see that in this case the stemmer and lemmatizer would give the same result.  

Part-of-Speech Tagging

As briefly mentioned, part-of-speech tagging refers to tagging a word with a grammatical category. To do this requires context on the sentence in which the word appears – for example, which words it is adjacent to, and how its definition could change depending on this. The following function can be called to view a list of all possible part-of-speech tags.

The extensive list includes PoS tags such as VB (verb in base form), VBD (verb in past tense), VBG (verb as present participle) and so on.

To generate the tags for our tokens, we simply import the library and call the pos_tag function.

So far we’ve mostly been testing these NLP tools on the first individual review in our corpus, but in a real task we would be using all 2000 reviews. If this were the case, our corpus would generate a total of 1,336,782 word tokens after the tokenization step. But the resulting list is a collection of each use of a word, even if the same word is repeated multiple times. For instance the sentence “He walked and walked” generates the tokens [‘he’, ‘walked’, ‘and’, ‘walked’]. This is useful for counting frequencies and other such analysis, but we may want a data structure containing every unique word used in the corpus, for example [‘he’, ‘walked’, ‘and’]. For this, we can use the Python set function.

This gives us the vocabulary of a corpus, and we can use it to compare sizes of vocabulary in different texts, or percentages of the vocabulary which refer to a certain topic or are a certain PoS type. For example, whilst the total size of the corpus of reviews contains 1,336,782 words (after tokenization), the size of the vocabulary is 39,696.

In this NLP tutorial, we have been able to transform movie reviews from long raw text documents into something that a computer can begin to understand. These structured forms can be used for data analysis or as input into machine learning algorithms to determine topics discussed, analyse sentiment expressed, or infer meaning.

You can now go ahead and use some of the NLP concepts discussed to start working with text-based data in your own projects. The NLTK movie reviews data set is already categorised into positive and negative reviews, so see if you can use the tools from this tutorial to aid you in creating a machine learning classifier to predict the label of a given review.

A Jupyter notebook with the complete running code can be found here .

Find out more about the courses offered at HyperionDev , including our popular online bootcamps in Full Stack Web Development , Mobile Development and Software Engineering . For more articles like this once, take a look at the HyperionDev blog .

This post was contributed by Ellie Birbeck .

Home » Machine Learning » Natural Language Processing (NLP) » Python NLTK: Sentiment Analysis on Movie Reviews [Natural Language Processing (NLP)]

Python NLTK: Sentiment Analysis on Movie Reviews [Natural Language Processing (NLP)]

This article shows how you can perform sentiment analysis on movie reviews using Python and Natural Language Toolkit (NLTK).

Sentiment Analysis means analyzing the sentiment of a given text or document and categorizing the text/document into a specific class or category (like positive and negative). In other words, we can say that sentiment analysis classifies any particular text or document as positive or negative. Basically, the classification is done for two classes: positive and negative. However, we can add more classes like neutral, highly positive, highly negative, etc.

Sentiment Analysis is also referred as Opinion Mining. It’s mostly used in social media and customer reviews data.

In this article, we will learn about labeling data, extracting features, training classifier, and testing the accuracy of the classifier.

Table of Contents

Supervised Classification

Here, we will be doing supervised text classification. In supervised classification, the classifier is trained with labeled training data .

In this article, we will use the NLTK’s movie_reviews corpus as our labeled training data. The movie_reviews corpus contains 2K movie reviews with sentiment polarity classification. It’s compiled by Pang, Lee.

Here, we have two categories for classification. They are: positive and negative. The movie_reviews corpus already has the reviews categorized as positive and negative.

Create list of movie review document

This list contains array containing tuples of all movie review words and their respective category (pos or neg).

Feature Extraction

To classify the text into any category, we need to define some criteria. On the basis of those criteria, our classifier will learn that a particular kind of text falls in a particular category. This kind of criteria is known as feature . We can define one or more feature to train our classifier.

In this example, we will use the top-N words feature .

Fetch all words from the movie reviews corpus

We first fetch all the words from all the movie reviews and create a list.

Create Frequency Distribution of all words

Frequency Distribution will calculate the number of occurence of each word in the entire list of words.

Removing Punctuation and Stopwords

From the above frequency distribution of words, we can see the most frequently occurring words are either punctuation marks or stopwords.

Stop words are those frequently words which do not carry any significant meaning in text analysis. For example, I, me, my, the, a, and, is, are, he, she, we, etc.

Punctuation marks like comma, fullstop. inverted comma, etc. occur highly in any text data.

We will do data cleaning by removing stop words and punctuations.

Remove Stop Words

You can see that after removing stopwords, the words to and a has been removed from the first 10 words result.

Remove Punctuation

You can see that on the list that all punctuations like semi-colon : , comma , are removed.

Remove both Stopwords & Punctuation

In the above examples, at first, we only removed stopwords and then in the next code, we only removed punctuation.

Below, we will remove both stopwords and punctuation from the all_words list.

Frequency Distribution of cleaned words list

Below is the frequency distribution of the new list after removing stopwords and punctuation.

Previously, before removing stopwords and punctuation, the frequency distribution was:

FreqDist with 39768 samples and 1583820 outcomes

Now, the frequency distribution is:

FreqDist with 39586 samples and 710578 outcomes

This shows that after removing around 200 stop words and punctuation, the outcomes/words number has reduced to around half of the original size.

The most common words or highly occurring words list has also got meaningful words in the list. Before, the first 10 frequently occurring words were only stop-words and punctuations.

Create Word Feature using 2000 most frequently occurring words

We take 2000 most frequently occurring words as our feature.

Create Feature Set

Now, we write a function that will be used to create feature set. The feature set is used to train the classifier.

We define a feature extractor function that checks if the words in a given document are present in the word_features list or not.

In the beginning of this article, we have created the documents list which contains data of all the movie reviews. Its elements are tuples with word list as first item and review category as the second item of the tuple.

We now loop through the documents list and create a feature set list using the document_features function defined above.

• Each item of the feature_set list is a tuple.
• The first item of the tuple is the dictionary returned from document_features function
• The second item of the tuple is the category (pos or neg) of the movie review

Training Classifier

From the feature set we created above, we now create a separate training set and a separate testing/validation set. The train set is used to train the classifier and the test set is used to test the classifier to check how accurately it classifies the given text.

Creating Train and Test Dataset

In this example, we use the first 400 elements of the feature set array as a test set and the rest of the data as a train set. Generally, 80/20 percent is a fair split between training and testing set, i.e. 80 percent training set and 20 percent testing set.

Training a Classifier

Now, we train a classifier using the training dataset. There are different kind of classifiers namely Naive Bayes Classifier, Maximum Entropy Classifier, Decision Tree Classifier, Support Vector Machine Classifier, etc.

In this example, we use the Naive Bayes Classifier . It’s a simple, fast, and easy classifier which performs well for small datasets. It’s a simple probabilistic classifier based on applying Bayes’ theorem. Bayes’ theorem describes the probability of an event, based on prior knowledge of conditions that might be related to the event.

Testing the trained Classifier

Let’s see the accuracy percentage of the trained classifier. The accuracy value changes each time you run the program because of the names array being shuffled above.

Let’s see the output of the classifier by providing some custom reviews.

Let’s see the most informative features among the entire features in the feature set.

The result shows that the word outstanding is used in positive reviews 14.7 times more often than it is used in negative reviews the word poorly is used in negative reviews 7.7 times more often than it is used in positive reviews. Similarly, for other letters. These ratios are also called likelihood ratios .

Therefore, a review has a high chance to be classified as positive if it contains words like outstanding and wonderfully . Similarly, a review has a high chance of being classified as negative if it contains words like poorly , awful , waste , etc.

Note: You can modify the document_features function to generate the feature set which can improve the accuracy of the trained classifier. Feature extractors are built through a process of trail-and-error & guided by intuitions.

Bag of Words Feature

In the above example, we used top-N words feature. We used 2000 most frequently occurring words as our top-N words feature. The classifier identified negative review as negative. However, the classifier was not able to classify positive review correctly. It classified a positive review as negative.

Top-N words feature – The top-N words feature is also a bag-of-words feature. – But in the top-N feature, we only used the top 2000 words in the feature set. – We combined the positive and negative reviews into a single list, randomized the list, and then separated the train and test set. – This approach can result in the un-even distribution of positive and negative reviews across the train and test set. Bag-of-words feature shown below In the bag-of-words feature as shown below: – We will use all the useful words of each review while creating the feature set. – We take a fixed number of positive and negative reviews for train and test set. – This result in equal distribution of positive and negative reviews across train and test set.

In the approach shown below, we will modify the feature extractor function.

• We form a list of unique words of each review.
• The category (pos or neg) is assigned to each bag of words.
• Then the category of any given text is calculated by matching the different bag-of-words & their respective category.

We use the bag-of-words feature. Here, we clean the word list (i.e. remove stop words and punctuation). Then, we create a dictionary of cleaned words.

We use the bag-of-words feature and tag each review with its respective category as positive or negative.

Create Train and Test Set

There are 1000 positive reviews set and 1000 negative reviews set. We take 20% (i.e. 200) of positive reviews and 20% (i.e. 200) of negative reviews as a test set. The remaining negative and positive reviews will be taken as a training set.

Note: – There is difference between pos_reviews & pos_reviews_set array which are defined above. – pos_reviews array contains words list only – pos_reviews_set array contains words feature list – pos_reviews_set & neg_reviews_set arrays are used to create train and test set as shown below

Training Classifier and Calculating Accuracy

We train Naive Bayes Classifier using the training set and calculate the classification accuracy of the trained classifier using the test set.

Testing Classifier with Custom Review

We provide custom review text and check the classification output of the trained classifier. The classifier correctly predicts both negative and positive reviews provided.

Bi-gram Features

N-grams are common terms in text processing and analysis. N-grams are related with words of a text. There are different n-grams like unigram, bigram, trigram, etc.

Unigram = Item having a single word, i.e. the n-gram of size 1. For example, good. Bigram = Item having two words, i.e. the n-gram of size 2. For example, very good. Trigram = Item having three words, i.e. the n-gram of size 3. For example, not so good.

In the above bag-of-words model, we only used the unigram feature. In the example below, we will use both unigram and bigram feature, i.e. we will deal with both single words and double words.

In this case, both unigrams and bigrams are used as features.

We define two functions:

• bag_of_words : that extracts only unigram features from the movie review words
• bag_of_ngrams : that extracts only bigram features from the movie review words

We then define another function:

• bag_of_all_words : that combines both unigram and bigram features

Working with NLTK’s movie reviews corpus

There are 1000 positive reviews set and 1000 negative reviews set. We take 20% (i.e. 200) of positive reviews and 20% (i.e. 200) of negative reviews as the test set. The remaining negative and positive reviews will be taken as the training set.

Note: – The accuracy of the classifier has significantly increased when trained with combined feature set (unigram + bigram). – Accuracy was 73% while using only Unigram features. – Accuracy has increased to 80% while using combined (unigram + bigram) features.

References: 1. Learning to Classify Text 2. From Text Classification to Sentiment Analysis

Hope this helps. Thanks.

Share this:

Related posts:.

• Python NLTK: Twitter Sentiment Analysis [Natural Language Processing (NLP)]
• Python NLTK: Stop Words [Natural Language Processing (NLP)]
• Python: Twitter Sentiment Analysis using TextBlob
• Python NLTK: Text Classification [Natural Language Processing (NLP)]
• Machine Learning & Sentiment Analysis: Text Classification using Python & NLTK

TensorFlow.js Livestream: Deep Learning in the browser

• Text Classification with NLTK

Now that we're comfortable with NLTK, let's try to tackle text classification. The goal with text classification can be pretty broad. Maybe we're trying to classify text as about politics or the military. Maybe we're trying to classify it by the gender of the author who wrote it. A fairly popular text classification task is to identify a body of text as either spam or not spam, for things like email filters. In our case, we're going to try to create a sentiment analysis algorithm.

To do this, we're going to start by trying to use the movie reviews database that is part of the NLTK corpus. From there we'll try to use words as "features" which are a part of either a positive or negative movie review. The NLTK corpus movie_reviews data set has the reviews, and they are labeled already as positive or negative. This means we can train and test with this data. First, let's wrangle our data.

It may take a moment to run this script, as the movie reviews dataset is somewhat large. Let's cover what is happening here.

After importing the data set we want, you see:

Basically, in plain English, the above code is translated to: In each category (we have pos or neg), take all of the file IDs (each review has its own ID), then store the word_tokenized version (a list of words) for the file ID, followed by the positive or negative label in one big list.

Next, we use random to shuffle our documents. This is because we're going to be training and testing. If we left them in order, chances are we'd train on all of the negatives, some positives, and then test only against positives. We don't want that, so we shuffle the data.

Then, just so you can see the data you are working with, we print out documents[1], which is a big list, where the first element is a list the words, and the 2nd element is the "pos" or "neg" label.

Next, we want to collect all words that we find, so we can have a massive list of typical words. From here, we can perform a frequency distribution, to then find out the most common words. As you will see, the most popular "words" are actually things like punctuation, "the," "a" and so on, but quickly we get to legitimate words. We intend to store a few thousand of the most popular words, so this shouldn't be a problem.

The above gives you the 15 most common words. You can also find out how many occurences a word has by doing:

Next up, we'll begin storing our words as features of either positive or negative movie reviews.

The next tutorial: Converting words to Features with NLTK

Documentation

NLTK Documentation

• API Reference
• Example Usage
• Module Index
• Open Issues
• NLTK on GitHub

Installation

• Installing NLTK
• Installing NLTK Data
• Release Notes
• Contributing to NLTK

nltk.corpus.reader.reviews module ¶

CorpusReader for reviews corpora (syntax based on Customer Review Corpus).

Customer Review Corpus information ¶

Department of Computer Science University of Illinois at Chicago

https://www.cs.uic.edu/~liub

Distributed with permission.

The “product_reviews_1” and “product_reviews_2” datasets respectively contain annotated customer reviews of 5 and 9 products from amazon.com.

Related papers:

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD-04), 2004.

Proceedings of Nineteeth National Conference on Artificial Intelligence (AAAI-2004), 2004.

Opinion Mining.” Proceedings of First ACM International Conference on Web Search and Data Mining (WSDM-2008), Feb 11-12, 2008, Stanford University, Stanford, California, USA.

Symbols used in the annotated reviews:

[t] : the title of the review: Each [t] tag starts a review. xxxx[+|-n] : xxxx is a product feature. [+n] : Positive opinion, n is the opinion strength: 3 strongest, and 1 weakest. Note that the strength is quite subjective. You may want ignore it, but only considering + and - [-n] : Negative opinion ## : start of each sentence. Each line is a sentence. [u] : feature not appeared in the sentence. [p] : feature not appeared in the sentence. Pronoun resolution is needed. [s] : suggestion or recommendation. [cc] : comparison with a competing product from a different brand. [cs] : comparison with a competing product from the same brand.

provide separation between different reviews. This is due to the fact that the dataset was specifically designed for aspect/feature-based sentiment analysis, for which sentence-level annotation is sufficient. For document- level classification and analysis, this peculiarity should be taken into consideration.

Bases: object

A Review is the main block of a ReviewsCorpusReader.

title – the title of the review.

review_lines – the list of the ReviewLines that belong to the Review.

Add a line (ReviewLine) to the review.

review_line – a ReviewLine instance that belongs to the Review.

Return a list of features in the review. Each feature is a tuple made of the specific item feature and the opinion strength about that feature.

all features of the review as a list of tuples (feat, score).

list(tuple)

Return all tokenized sentences in the review.

all sentences of the review as lists of tokens.

list(list(str))

A ReviewLine represents a sentence of the review, together with (optional) annotations of its features and notes about the reviewed item.

Bases: CorpusReader

Reader for the Customer Review Data dataset by Hu, Liu (2004). Note: we are not applying any sentence tokenization at the moment, just word tokenization.

We can also reach the same information directly from the stream:

We can compute stats for specific product features:

alias of StreamBackedCorpusView

root – The root directory for the corpus.

fileids – a list or regexp specifying the fileids in the corpus.

word_tokenizer – a tokenizer for breaking sentences or paragraphs into words. Default: WordPunctTokenizer

encoding – the encoding that should be used to read the corpus.

Return a list of features. Each feature is a tuple made of the specific item feature and the opinion strength about that feature.

fileids – a list or regexp specifying the ids of the files whose features have to be returned.

all features for the item(s) in the given file(s).

Return all the reviews as a list of Review objects. If fileids is specified, return all the reviews from each of the specified files.

fileids – a list or regexp specifying the ids of the files whose reviews have to be returned.

the given file(s) as a list of reviews.

Return all sentences in the corpus or in the specified files.

fileids – a list or regexp specifying the ids of the files whose sentences have to be returned.

the given file(s) as a list of sentences, each encoded as a list of word strings.

Return all words and punctuation symbols in the corpus or in the specified files.

fileids – a list or regexp specifying the ids of the files whose words have to be returned.

the given file(s) as a list of words and punctuation symbols.

• Stack Overflow for Teams Where developers & technologists share private knowledge with coworkers
• Advertising & Talent Reach devs & technologists worldwide about your product, service or employer brand
• OverflowAI GenAI features for Teams
• OverflowAPI Train & fine-tune LLMs
• Labs The future of collective knowledge sharing
• About the company Visit the blog

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Get early access and see previews of new features.

How do I download NLTK data?

Updated answer:NLTK works for 2.7 well. I had 3.2. I uninstalled 3.2 and installed 2.7. Now it works!!

I have installed NLTK and tried to download NLTK Data. What I did was to follow the instrution on this site: http://www.nltk.org/data.html

I downloaded NLTK, installed it, and then tried to run the following code:

It gave me the error message like below:

Tried both nltk.download() and nltk.downloader() , both gave me error messages.

Then I used help(nltk) to pull out the package, it shows the following info:

I do see Downloader there, not sure why it does not work. Python 3.2.2, system Windows vista.

• 1 Short note: I do not know what the problem is, but what you are doing is correct and should give you a GUI to choose what to download (i.e. you're not doing it wrong, but something is wrong) –  Miquel Commented Mar 5, 2014 at 23:24
• From where did you install NLTK? I highly suggest you install it through a package manager like pip to handle all the dependencies for you. –  Michael Aquilina Commented Mar 5, 2014 at 23:30
• I am not sure how to do it. Do you mean I should install pip first and then use it to install NLTK? –  Q-ximi Commented Mar 5, 2014 at 23:35
• Correct that is what @MichaelAquilina means. –  Jason Commented Mar 5, 2014 at 23:37
• 2 installation from terminal python3 -m nltk.downloader all –  avr-girl Commented Feb 25, 2021 at 18:18

15 Answers 15

To download a particular dataset/models, use the nltk.download() function, e.g. if you are looking to download the punkt sentence tokenizer, use:

If you're unsure of which data/model you need, you can start out with the basic list of data + models with:

It will download a list of "popular" resources, these includes:

In case anyone is avoiding errors from downloading larger datasets from nltk , from https://stackoverflow.com/a/38135306/610569

From v3.2.5, NLTK has a more informative error message when nltk_data resource is not found, e.g.:

To find nltk_data directory (auto-magically), see https://stackoverflow.com/a/36383314/610569

To download nltk_data to a different path , see https://stackoverflow.com/a/48634212/610569

To config nltk_data path (i.e. set a different path for NLTK to find nltk_data ), see https://stackoverflow.com/a/22987374/610569

• 1 To install nltk_data in a conda environment, see stackoverflow.com/a/53464117/501086 –  Shatu Commented Nov 25, 2018 at 2:18

nltk.download('all')

this will download all the data and no need to download individually.

• in my case it was not loading UI.. do not know why... but this helped me. thanks. –  desaiankitb Commented Mar 27, 2018 at 17:36
• 6 FYI - as of 2019/12/15, that whole folder is about 3.2 GB, including the zip files. –  Florin Andrei Commented Dec 15, 2019 at 23:21
• Thanks a lot @FlorinAndrei for that info. –  Deepam Gupta Commented Dec 18, 2020 at 14:43

Install Pip: run in terminal : sudo easy_install pip

Install Numpy (optional): run : sudo pip install -U numpy

Install NLTK: run : sudo pip install -U nltk

Test installation: run: python

then type : import nltk

To download the corpus

run : python -m nltk.downloader all

Do not name your file nltk.py I used the same code and name it nltk, and got the same error as you have, I changed the file name and it went well.

This worked for me:

After running this you get something like this

Then, Press d

Do As Follows:

You will get following message on completion, and Prompt then Press q Done downloading collection all

you can't have a saved python file called nltk.py because the interpreter is reading from that and not from the actual file.

Change the name of your file that the python shell is reading from and try what you were doing originally:

import nltk and then nltk.download()

It's very simple....

• Open pyScripter or any editor
• Create a python file eg: install.py
• write the below code in it.
• A pop-up window will apper and click on download .

• The pop up window is not opening . I have tried many times . The version of the nltk is also new one which is 3.4.1 . Now tell what should be issue ? –  Hamza Tahir Commented May 31, 2019 at 11:24
• @HamzaTahir Same happened with me, I restarted my kernel –  Lakhani Aliraza Commented Apr 3, 2020 at 4:56

I had the similar issue. Probably check if you are using proxy.

If yes, set up the proxy before doing download:

If you are running a really old version of nltk, then there is indeed no download module available ( reference )

As per the reference, anything after 0.9.5 should be fine

• 2 This is actually what I suspected, which is why I suggested the OP uses pip to install NLTK instead. –  Michael Aquilina Commented Mar 5, 2014 at 23:41
• It won't even print out the version info. The version is 2.0.4. That is the link I followed from the book Natural Language Processing with Python . Here is the [link] nltk.org/install.html . –  Q-ximi Commented Mar 5, 2014 at 23:42
• Ok, that's the latest. I guess this isn't it then. Let's leave it here for future reference –  Miquel Commented Mar 5, 2014 at 23:45
• Does any of nltk work for you? try running: nltk.word_tokenize("hello world") and see if gives you any output –  Michael Aquilina Commented Mar 6, 2014 at 9:25

you should add python to your PATH during installation of python...after installation.. open cmd prompt type command- pip install nltk then go to IDLE and open a new file..save it as file.py..then open file.py type the following: import nltk

Try download the zip files from http://www.nltk.org/nltk_data/ and then unzip, save in your Python folder, such as C:\ProgramData\Anaconda3\nltk_data

• This is the method I had to use (in modified form) for a Linux server that is not connected to the internet. I unzipped it under a directory I created, /usr/share/nltk_data/tokenizers/ –  Mike Maxwell Commented Aug 5, 2021 at 21:13

if you have already saved a file name nltk.py and again rename as my_nltk_script.py. check whether you have still the file nltk.py existing. If yes, then delete them and run the file my_nltk.scripts.py it should work!

just do like

then you will be show a popup asking what to download , select 'all'. it will take some time because of its size, but eventually we will get it.

and if you are using Google Colab, you can use

after running that you will be asked to select from a list

here you have to enter d as you want to download. after that you will be asked to enter the identifier that you want to download . You can see the list of available indentifier with l command or if you want all of them just enter 'all' in the input box. then you will see something like -

at last you can enter q to quit.

You may try:

happy nlp'ing.

Not the answer you're looking for? Browse other questions tagged python nltk or ask your own question .

• The Overflow Blog
• At scale, anything that could fail definitely will
• Featured on Meta
• Announcing a change to the data-dump process
• Bringing clarity to status tag usage on meta sites
• What does a new user need in a homepage experience on Stack Overflow?
• Feedback requested: How do you use tag hover descriptions for curating and do...
• Staging Ground Reviewer Motivation

Hot Network Questions

• Does it make sense for the governments of my world to genetically engineer soldiers?
• In Lord Rosse's 1845 drawing of M51, was the galaxy depicted in white or black?
• Why is a USB memory stick getting hotter when connected to USB-3 (compared to USB-2)?
• Unable to upgrade from Ubuntu Server 22.04 to 24.04.1
• MANIFEST_UNKNOWN error: OCI index found, but Accept header does not support OCI indexes
• What does "if you ever get up this way" mean?
• Not a cross, not a word (number crossword)
• How best to cut (slightly) varying size notches in long piece of trim
• Is it possible to recover from a graveyard spiral?
• Does an unseen creature with a burrow speed have advantage when attacking from underground?
• If a Palestinian converts to Judaism, can they get Israeli citizenship?
• Using rule-based symbology for overlapping layers in QGIS
• Why is there so much salt in cheese?
• Why is the wiper fluid hose on the Mk7 Golf covered in cloth tape?
• How to reconcile the effect of my time magic spell with my timeline
• What is Zion's depth in the Matrix?
• Marie-Sklodowska-Curie actions: publish a part of the proposal just after the deadline?
• How to prevent my frozen dessert from going solid?
• quantulum abest, quo minus . .
• How would you slow the speed of a rogue solar system?
• Should I install our toilet on top of the subfloor or on top of the PVT?
• Convert 8 Bit brainfuck to 1 bit Brainfuck / Boolfuck
• Whats the safest way to store a password in database?
• Help writing block matrix