Data Exploration

The codes described in this section can be found in preprocessor.py. Thus, all the functions/methods stated in this section can be found on the linked page.

• Handling missing values. (see the method handle_missing_values)

  For the columns with string data type, all the empty values are set to the empty string. On the other hand, for the columns with numerical data type, only the 'views' columns have empty values. The empty values of the said column are set to 0.

• Handling outliers. (see the method handle_outliers)

  We decided to not remove any of the numerical outliers.

• Ensuring formatting consistency. (see the method ensure_formatting_consistency)

  We ensured formatting consistency for all the manually-labeled columns, which are: leni_sentiment, marcos_sentiment, incident, account_type, tweet_type, content_type, keywords, and alt-text. After lower-casing all the above columns, we used regex pattern matching to verify that all the following criteria are met:

  • leni_sentiment - values should be one of the following only: "negative", "neutral", "positive".
  • marcos_sentiment - values should be one of the following only: "negative", "neutral", "positive".
  • account_type - values should be one of the following only: "identified", "anonymous", "media".
  • tweet_type - values should be a combination of the following values only: "text", "image", "video", "url", "reply".
  • country - values should be one of the following values only: "unspecified", "" (empty string), any alphabetic string.
  • content_type - values should be a combination of the following values only: "rational", "emotional", "transactional".
  • keywords - any alphanumeric string that is not empty.
  • alt-text - values should be one of the following values only: any string enclosed by curly braces { }, "" (empty string)
  Fortunately, all our labels passed all the above criteria.
  Note that there is no need to ensure formatting consistency for the date columns. This is because the scraper already provided us with date values so it is guaranteed that the date values are correct and properly formatted.

• Normalization and standardization. (see the method norm_std_ize)

  For context, here are all the numerical features/columns:

  following	followers	likes	replies	retweets
  quote_tweets	views	has_leni_ref	joined_unix*	date_posted_unix*

  *Note: we converted the date columns (the joined and date_posted columns) to numerical values by converting them to unix time, so we can perform normalization and standardization to their values as well.
  We normalized each numerical column (all the features in the table above) by translating and scaling their values to the range [0, 1]. We also standardized each numerical column by getting their z-scores. This whole process resulted to two new columns for each numerical column.
  

• Categorical data encoding. (see the method encode_cat_feats)

  For context, here are all the categorical data columns.

  leni_sentiment	marcos_sentiment	incident	account_type
  country	tweet_type	content_type

  There are two types of categorical data columns. First are those columns that can only have a single category (these are leni_sentiment, marcos_sentiment, incident, account_type, and country). Second are those columns that can have multiple categories (tweet_type and content_type). You may refer again to the Ensuring formatting consistency bullet above for the valid values of these columns.
  We'll discuss first the single category columns. We converted each single category columns to numerical data by following the mapping below.
  • leni_sentiment: (negative, neutral, positive) → (-1, 0, 1)
  • marcos_sentiment: (negative, neutral, positive) → (-1, 0, 1)
  • incident: (baguio, ladder, scandal, quarantine, others) → (0, 1, 2, 3, 4)
  • account_type: (anonymous, identified, media) → (0, 1, 2)
  • country: ("" or unspecified, any alphabetic string) → (0, 1)
  For the multiple category columns, it's a little bit more complicated. First, we converted multiple-categorical-values to single category only. You will understand what we mean by this through the illustration below.

  tweet_type	tweet_type_code	content_type	content_type_code
  text	T	emotional	E
  text, image, url	ITU	rational, emotional	ER
  text, url, image, reply	IRTU	emotional, transactional	ET
  reply, video, text, url, image	IRTUV	transactional, rational, emotional	ERT
  text, url, image, reply, video	IRTUV	rational, emotional, transactional	ERT

  As you may have observed, the tweet_type_code is obtained by getting the first letter of each category, sorting all of the first-letter-characters, and concatenating them to form a string (the tweet_type_code). The same process is done for the content_type_code.
  The possible values of tweet_type_code is essentially all the possible combinations of the set {'I', 'R', 'T', 'U', 'V'} sorted lexicographically, which is 32 combinations in total (thus, 32 possible tweet_type_codes). What this means is that we can map each 32 codes/combinations to integers from 0 to 31. This mapping is then used to convert a tweet_type_code to its corresponding numerical value.
  The same process goes for the content_type_code (where the content_type_codes are essentially all the possible combinations of the set {'E', 'R', 'T'}).

The codes described in this section can be found in nlp.py. Thus, all the functions/methods stated in this section can be found on the linked page.

• Tokenization and lower casing.
• Stop words removal.
• Stemming and lemmatization.

The team wrote a code inspired by the natural language processing notebook provided by the professor. First, emojis and punctuation were replaced with their word counterparts. It can be noted that a number of emojis and punctuations have persisted. To address this, all the unaffected punctuation and emojis were manually translated. The resulting text was casted into lowercase.

Next, the team installed the googletrans api version 3.1.0a0. This was the method used in translating the Tagalog tweets into English. The team does not have a focus on NLP anyway. However, there were empty tweets because such tweets were only images. The error was finally caught and empty tweets translated into empty strings.

Lastly, the tweets were tokenized using the nltk. The rest of the professor's code has worked nicely. In the end, the disinformation dataframe was appended the stemmed and lemmatized versions of the tweets. It should be noted that the tweets were translated poorly by the automatic translator.

But, actually, the group has performed some kind of natural language processing manually. The team has identified the aliases being mentioned in the disinformation tweets. More about this will be discuss at the latter part of this article below.

The codes described in this section can be found in time_series.py. Thus, all the functions/methods stated in this section can be found on the linked page.

• Interpolation

  All our samples have no missing datetime values. Thus, there is no need to interpolate any datetime values.

• Binning (see the method bin)

  The two datetime columns (joined and date_posted) are binned daily, weekly, monthly, and yearly. Thus, this results in eight new columns, which are:

  joined_day	joined_week	joined_month	joined_year
  date_posted_day	date_posted_week	date_posted_month	date_posted_year

  Below are some of the values of each bin:
  • For yearly bins: 2016, 2017, ... (yyyy)
  • For monthly bins: 2016-1, 2016-2, ... (yyyy-mm)
  • For weekly bins: 2016-1, 2016-2, ... (yyyy-ww)
  • For daily bins: 2016-1-1, 2016-1-2, ... (yyyy-mm-dd)