# Pre-processing text

## Remove punctuation

```python
import string
string.punctuation
def remove_punct(text):
  text_nopunct = "".join[char for char in text if char not in string.punctuation]
  return text_nopunct
```

## Tokenize

Splitting sentence into words

```python
def tokenize(t):
  tokens = re.split('\W+', text)
  return tokens
```

## Lowercase

.lower()

```python
data['body_text_token'] = data['body_text_clean'].apply(lambda x: tokenize(x.lower()))
```

## Remove StopWords

```python
def remove_stopwords(tokenized_list):
  [word for word in tokenized_list if word not in nltk.corpus.stopwords.words('english')]
```

## Stemming

Leave root word, chopping off suffix

* Problems: Meanness, Meaning -> Mean
*   Explicitly correlates words with similar meanings

    **PorterStemmer**

    ```python
    ps = nltk.PorterStemmer()
    ps.stem('goose') #goos
    ps.stem('geese') #gees
    ```

    **Lemmatizing**
* Grouping together inflected form of words, basically has goal of stemming
* Lemmatizing is more accurate as uses more informed analysis, but takes more time
*   If not in wordnet, just leave word

    **Wordnet lemmatizer**

    ```python
    wn = nltk.WordNetLemmatizer()
    wn.lemmatize('goose') #goose
    wn.lemmatize('geese') #goose
    ```
