Text processing

Real text is messy. It has extra spaces and punctuation stuck to words, and to a computer Python and python are two different words. So before you count anything, the text goes through three steps: normalize the case, clean it, then split it.

You normalize with .lower(), so Python, PYTHON and python all become the same word. Then you clean with re.sub(r"[^\w\s]", " ", text), which turns everything that isn't a letter or digit (\w) or whitespace (\s) into a space. Finally you split with .split() and empty parentheses: it splits on any run of whitespace and ignores whitespace at both ends, just like .strip(). .split(" "), on the other hand, gives you an empty string between any two spaces in a row.

There's a trap here: string.punctuation only holds ASCII punctuation. Curly quotes “ ”, the ellipsis … and the em dash — aren't in it, and text pasted from a phone or a word processor often has them. Strip punctuation with it and “excellent” and excellent stay two different words. \w, on the other hand, knows every Unicode letter, so when you remove everything that isn't a letter or digit, every mark goes and letters like é stay.