酒店评论的情感分析

酒店评论的情感分析

现在你已经详细地探索了数据集，是时候过滤掉一些列，然后使用NLP技术来获得关于酒店的新见解。

课前测验

过滤和情感分析操作

你可能已经注意到，数据集有一些问题。有些列充满了无用的信息，其他列似乎不正确。如果它们是正确的，那么不清楚它们是如何计算出来的，而且答案无法通过自己的计算进行独立验证。

练习：更多的数据处理

再清理一下数据。添加一些以后可能会有用的列，更改其他列的值，并完全删除某些列。

1. 初始列处理

   1. 删除 lat 和 lng

   2. 使用以下值替换 Hotel_Address 值（如果地址包含城市名和国家名，则将其更改为仅城市名和国家名）。

      数据集中只包含以下城市和国家：

      荷兰阿姆斯特丹

      西班牙巴塞罗那

      英国伦敦

      意大利米兰

      法国巴黎

      奥地利维也纳

      def replace_address(row):
          if "Netherlands" in row["Hotel_Address"]:
              return "Amsterdam, Netherlands"
          elif "Barcelona" in row["Hotel_Address"]:
              return "Barcelona, Spain"
          elif "United Kingdom" in row["Hotel_Address"]:
              return "London, United Kingdom"
          elif "Milan" in row["Hotel_Address"]:        
              return "Milan, Italy"
          elif "France" in row["Hotel_Address"]:
              return "Paris, France"
          elif "Vienna" in row["Hotel_Address"]:
              return "Vienna, Austria" 
      
      # Replace all the addresses with a shortened, more useful form
      df["Hotel_Address"] = df.apply(replace_address, axis = 1)
      # The sum of the value_counts() should add up to the total number of reviews
      print(df["Hotel_Address"].value_counts())
      ​

      现在你可以查询国家层面的数据：

      display(df.groupby("Hotel_Address").agg({"Hotel_Name": "nunique"}))
      ​

      酒店地址	酒店名称
      荷兰阿姆斯特丹	105
      西班牙巴塞罗那	211
      英国伦敦	400
      意大利米兰	162
      法国巴黎	458
      奥地利维也纳	158

2. 处理酒店元评论列

   1. 删除 Additional_Number_of_Scoring

3. Replace Total_Number_of_Reviews with the total number of reviews for that hotel that are actually in the dataset

4. Replace Average_Score 并用我们自己计算的分数替换

# Drop `Additional_Number_of_Scoring`
df.drop(["Additional_Number_of_Scoring"], axis = 1, inplace=True)
# Replace `Total_Number_of_Reviews` and `Average_Score` with our own calculated values
df.Total_Number_of_Reviews = df.groupby('Hotel_Name').transform('count')
df.Average_Score = round(df.groupby('Hotel_Name').Reviewer_Score.transform('mean'), 1)
​

3. 处理评论列

   1. 删除 Review_Total_Negative_Word_Counts, Review_Total_Positive_Word_Counts, Review_Date and days_since_review

   2. Keep Reviewer_Score, Negative_Review, and Positive_Review as they are,

   3. Keep Tags for now

   • We'll be doing some additional filtering operations on the tags in the next section and then tags will be dropped

4. Process reviewer columns

5. Drop Total_Number_of_Reviews_Reviewer_Has_Given

6. Keep Reviewer_Nationality

Tag columns

The Tag column is problematic as it is a list (in text form) stored in the column. Unfortunately the order and number of sub sections in this column are not always the same. It's hard for a human to identify the correct phrases to be interested in, because there are 515,000 rows, and 1427 hotels, and each has slightly different options a reviewer could choose. This is where NLP shines. You can scan the text and find the most common phrases, and count them.

Unfortunately, we are not interested in single words, but multi-word phrases (e.g. Business trip). Running a multi-word frequency distribution algorithm on that much data (6762646 words) could take an extraordinary amount of time, but without looking at the data, it would seem that is a necessary expense. This is where exploratory data analysis comes in useful, because you've seen a sample of the tags such as [' 商务旅行', ' 单独旅行者', ' 单人间', ' 住了5晚', ' 从移动设备提交'] ，你可以开始询问是否可以大大减少需要处理的工作量。幸运的是，这是可行的——但首先你需要遵循几个步骤来确定感兴趣的标签。

过滤标签

记住，数据集的目标是添加情感和有助于选择最佳酒店（无论是为自己还是为客户任务创建酒店推荐机器人）。你需要问自己这些标签在最终数据集中是否有用。这里有一种解释（如果你需要这个数据集用于其他原因，不同的标签可能会保留在或移出选择中）：

1. 旅行类型是相关的，应该保留
2. 客人组类型很重要，应该保留
3. 客人入住的房间、套间或工作室类型无关紧要（所有酒店基本上都有相同的房间）
4. 评论提交的设备无关紧要
5. 如果你认为较长的停留时间与客人对酒店的喜爱程度相关，那么住的天数可能是相关的，但这有点牵强，可能无关紧要

总之，保留两种类型的标签并移除其他标签。

首先，你不希望在标签格式更好之前就计数，这意味着要去掉方括号和引号。你可以用几种方法来做这件事，但你想要最快的方法，因为它可能会花费很长时间来处理大量数据。幸运的是，pandas 提供了一个简单的方法来完成每一步。

# Remove opening and closing brackets
df.Tags = df.Tags.str.strip("[']")
# remove all quotes too
df.Tags = df.Tags.str.replace(" ', '", ",", regex = False)
​

每个标签会变成类似这样的内容：商务旅行, 单独旅行者, 单人间, 住了5晚, 从移动设备提交.

Next we find a problem. Some reviews, or rows, have 5 columns, some 3, some 6. This is a result of how the dataset was created, and hard to fix. You want to get a frequency count of each phrase, but they are in different order in each review, so the count might be off, and a hotel might not get a tag assigned to it that it deserved.

Instead you will use the different order to our advantage, because each tag is multi-word but also separated by a comma! The simplest way to do this is to create 6 temporary columns with each tag inserted in to the column corresponding to its order in the tag. You can then merge the 6 columns into one big column and run the value_counts() method on the resulting column. Printing that out, you'll see there was 2428 unique tags. Here is a small sample:

Some of the common tags like 从移动设备提交 are of no use to us, so it might be a smart thing to remove them before counting phrase occurrence, but it is such a fast operation you can leave them in and ignore them.

Removing the length of stay tags

Removing these tags is step 1, it reduces the total number of tags to be considered slightly. Note you do not remove them from the dataset, just choose to remove them from consideration as values to count/keep in the reviews dataset.

There are a huge variety of rooms, suites, studios, apartments and so on. They all mean roughly the same thing and not relevant to you, so remove them from consideration.

Finally, and this is delightful (because it didn't take much processing at all), you will be left with the following useful tags:

You could argue that 与朋友一起旅行 is the same as 团体 more or less, and that would be fair to combine the two as above. The code for identifying the correct tags is the Tags notebook.

The final step is to create new columns for each of these tags. Then, for every review row, if the 标签 列匹配新列中的一个，则添加1，如果不匹配，则添加0。最终结果将显示有多少评论者（总计）选择了这个酒店，例如，是为了商务还是休闲，或者是否带宠物，这在推荐酒店时是非常有用的信息。

# Process the Tags into new columns
# The file Hotel_Reviews_Tags.py, identifies the most important tags
# Leisure trip, Couple, Solo traveler, Business trip, Group combined with Travelers with friends, 
# Family with young children, Family with older children, With a pet
df["Leisure_trip"] = df.Tags.apply(lambda tag: 1 if "Leisure trip" in tag else 0)
df["Couple"] = df.Tags.apply(lambda tag: 1 if "Couple" in tag else 0)
df["Solo_traveler"] = df.Tags.apply(lambda tag: 1 if "Solo traveler" in tag else 0)
df["Business_trip"] = df.Tags.apply(lambda tag: 1 if "Business trip" in tag else 0)
df["Group"] = df.Tags.apply(lambda tag: 1 if "Group" in tag or "Travelers with friends" in tag else 0)
df["Family_with_young_children"] = df.Tags.apply(lambda tag: 1 if "Family with young children" in tag else 0)
df["Family_with_older_children"] = df.Tags.apply(lambda tag: 1 if "Family with older children" in tag else 0)
df["With_a_pet"] = df.Tags.apply(lambda tag: 1 if "With a pet" in tag else 0)

​

保存文件

最后，以新的名称保存当前的数据集。

df.drop(["Review_Total_Negative_Word_Counts", "Review_Total_Positive_Word_Counts", "days_since_review", "Total_Number_of_Reviews_Reviewer_Has_Given"], axis = 1, inplace=True)

# Saving new data file with calculated columns
print("Saving results to Hotel_Reviews_Filtered.csv")
df.to_csv(r'../data/Hotel_Reviews_Filtered.csv', index = False)
​

情感分析操作

在本节中，你将对评论列应用情感分析并将结果保存到数据集中。

练习：加载并保存过滤后的数据

请注意，你现在加载的是在上一节中保存的过滤后的数据集，而不是原始数据集。

import time
import pandas as pd
import nltk as nltk
from nltk.corpus import stopwords
from nltk.sentiment.vader import SentimentIntensityAnalyzer
nltk.download('vader_lexicon')

# Load the filtered hotel reviews from CSV
df = pd.read_csv('../../data/Hotel_Reviews_Filtered.csv')

# You code will be added here


# Finally remember to save the hotel reviews with new NLP data added
print("Saving results to Hotel_Reviews_NLP.csv")
df.to_csv(r'../data/Hotel_Reviews_NLP.csv', index = False)
​

移除停用词

如果你直接对负面和正面评论列运行情感分析，可能需要很长时间。在一台强大的测试笔记本电脑上，使用快速CPU，耗时大约12到14分钟，这相对较长，因此值得研究是否可以加快速度。

移除停用词（即不影响句子情感的常见英语单词）是第一步。通过移除这些词，情感分析应该更快，但不会降低准确性（因为停用词不影响情感，但确实减慢了分析速度）。

最长的负面评论有395个单词，但在移除停用词后，它只有195个单词。

移除停用词也是一个快速的操作，在测试设备上，从两个评论列中移除515,000行的停用词耗时3.3秒。根据你的设备CPU速度、RAM、是否拥有SSD等因素，耗时可能会稍多或稍少。由于该操作相对短暂，如果它能提高情感分析的速度，那就值得一试。

from nltk.corpus import stopwords

# Load the hotel reviews from CSV
df = pd.read_csv("../../data/Hotel_Reviews_Filtered.csv")

# Remove stop words - can be slow for a lot of text!
# Ryan Han (ryanxjhan on Kaggle) has a great post measuring performance of different stop words removal approaches
# https://www.kaggle.com/ryanxjhan/fast-stop-words-removal # using the approach that Ryan recommends
start = time.time()
cache = set(stopwords.words("english"))
def remove_stopwords(review):
    text = " ".join([word for word in review.split() if word not in cache])
    return text

# Remove the stop words from both columns
df.Negative_Review = df.Negative_Review.apply(remove_stopwords)   
df.Positive_Review = df.Positive_Review.apply(remove_stopwords)
​

执行情感分析

现在你应该计算负面和正面评论列的情感分析，并将结果存储在两个新列中。情感测试的依据是将其与同一评论的评分进行比较。例如，如果情感分析认为负面评论的情感评分为1（非常积极的情感），而正面评论的情感评分为1，但评论者给酒店的评分是最低分，那么要么评论文本与评分不符，要么情感分析器未能正确识别情感。你应该期望有些情感评分完全错误，而且通常这些错误是可以解释的，例如评论可能是极度讽刺的“当然我爱上了没有暖气的房间”，而情感分析器认为这是积极的情感，尽管人类阅读时会知道这是讽刺。

NLTK提供了不同的情感分析器来学习，并且你可以替换它们来看看情感是否更准确或不准确。这里使用VADER情感分析。

Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. 第八届博客与社交媒体国际会议（ICWSM-14）。2014年6月，美国密歇根州安阿伯市。

from nltk.sentiment.vader import SentimentIntensityAnalyzer

# Create the vader sentiment analyser (there are others in NLTK you can try too)
vader_sentiment = SentimentIntensityAnalyzer()
# Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.

# There are 3 possibilities of input for a review:
# It could be "No Negative", in which case, return 0
# It could be "No Positive", in which case, return 0
# It could be a review, in which case calculate the sentiment
def calc_sentiment(review):    
    if review == "No Negative" or review == "No Positive":
        return 0
    return vader_sentiment.polarity_scores(review)["compound"]    
​

在你的程序后期，当你准备好计算情感时，可以按如下方式应用于每个评论：

# Add a negative sentiment and positive sentiment column
print("Calculating sentiment columns for both positive and negative reviews")
start = time.time()
df["Negative_Sentiment"] = df.Negative_Review.apply(calc_sentiment)
df["Positive_Sentiment"] = df.Positive_Review.apply(calc_sentiment)
end = time.time()
print("Calculating sentiment took " + str(round(end - start, 2)) + " seconds")
​

在我的电脑上，这大约需要120秒，但每个电脑的时间可能会有所不同。如果你想打印结果并查看情感是否与评论相符：

df = df.sort_values(by=["Negative_Sentiment"], ascending=True)
print(df[["Negative_Review", "Negative_Sentiment"]])
df = df.sort_values(by=["Positive_Sentiment"], ascending=True)
print(df[["Positive_Review", "Positive_Sentiment"]])
​

在使用文件参加挑战之前，需要做的最后一件事就是保存！你也应该考虑重新排列所有新列，使其易于使用（对于人类来说，这只是外观上的变化）。

# Reorder the columns (This is cosmetic, but to make it easier to explore the data later)
df = df.reindex(["Hotel_Name", "Hotel_Address", "Total_Number_of_Reviews", "Average_Score", "Reviewer_Score", "Negative_Sentiment", "Positive_Sentiment", "Reviewer_Nationality", "Leisure_trip", "Couple", "Solo_traveler", "Business_trip", "Group", "Family_with_young_children", "Family_with_older_children", "With_a_pet", "Negative_Review", "Positive_Review"], axis=1)

print("Saving results to Hotel_Reviews_NLP.csv")
df.to_csv(r"../data/Hotel_Reviews_NLP.csv", index = False)
​

你应该运行整个代码进行分析笔记本（在你运行过滤笔记本生成Hotel_Reviews_Filtered.csv文件之后）。

回顾一下，步骤如下：

1. 在之前的课程中使用探索笔记本探索原始数据集文件 Hotel_Reviews.csv
2. Hotel_Reviews.csv 由 过滤笔记本 进行过滤，生成 Hotel_Reviews_Filtered.csv
3. Hotel_Reviews_Filtered.csv 由 情感分析笔记本 进行处理，生成 Hotel_Reviews_NLP.csv
4. 在下面的NLP挑战中使用 Hotel_Reviews_NLP.csv

结论

当你开始时，你有一个带有列和数据的数据集，但并不是所有的数据都能被验证或使用。你已经探索了数据，过滤掉了不需要的内容，将标签转换成有用的东西，计算了自己的平均值，添加了一些情感列，并希望学到了关于自然语言处理的一些有趣的事情。

课后测验

挑战

现在你已经对数据集进行了情感分析，看看是否可以使用在此课程中学到的策略（聚类等）来确定情感方面的模式。

复习与自学

参加此学习模块以了解更多内容，并使用不同的工具来探索文本中的情感。

作业

尝试不同的数据集

声明:
本文件灏天文库团队进行了翻译。尽管我们力求准确，但请注意，翻译可能包含错误或不准确之处。原文档以其原始语言为准。我们不对因使用此翻译而产生的任何误解或误译负责。