What method do you use to normalize punctuation? For example, “ vs ". I attempted to use unicode normalization with NFKC, but it didn't normalize these two quotation marks to be equal (==). In addition to quotation marks, there are many other punctuation marks that are nearly equivalent but are not normalized using NFKC. Any recommendations or thoughts about normalizing them?
That's great bro, clean and simple explanation loved it a lot !
Thank you very much, you were a great help.
Nice explanations, thank you!
What method do you use to normalize punctuation? For example, “ vs ". I attempted to use unicode normalization with NFKC, but it didn't normalize these two quotation marks to be equal (==). In addition to quotation marks, there are many other punctuation marks that are nearly equivalent but are not normalized using NFKC. Any recommendations or thoughts about normalizing them?
yes there are some stubborn characters unfortunately, I usually convert them manually with regex/.replace
@@jamesbriggs Do you generally use use punctuation in NLP-based models/applications or is it strictly use case dependent?
nice one bro
happy it helps!