PySBD: Pragmatic Sentence Boundary Disambiguation

Nipun Sadvilkar, Mark Neumann


Abstract
We present a rule-based sentence boundary disambiguation Python package that works out-of-the-box for 22 languages. We aim to provide a realistic segmenter which can provide logical sentences even when the format and domain of the input text is unknown. In our work, we adapt the Golden Rules Set (a language specific set of sentence boundary exemplars) originally implemented as a ruby gem pragmatic segmenter which we ported to Python with additional improvements and functionality. PySBD passes 97.92% of the Golden Rule Set examplars for English, an improvement of 25% over the next best open source Python tool.
Anthology ID:
2020.nlposs-1.15
Volume:
Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)
Month:
November
Year:
2020
Address:
Online
Editors:
Eunjeong L. Park, Masato Hagiwara, Dmitrijs Milajevs, Nelson F. Liu, Geeticka Chauhan, Liling Tan
Venue:
NLPOSS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
110–114
Language:
URL:
https://aclanthology.org/2020.nlposs-1.15
DOI:
10.18653/v1/2020.nlposs-1.15
Bibkey:
Cite (ACL):
Nipun Sadvilkar and Mark Neumann. 2020. PySBD: Pragmatic Sentence Boundary Disambiguation. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 110–114, Online. Association for Computational Linguistics.
Cite (Informal):
PySBD: Pragmatic Sentence Boundary Disambiguation (Sadvilkar & Neumann, NLPOSS 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.nlposs-1.15.pdf
Optional supplementary material:
 2020.nlposs-1.15.OptionalSupplementaryMaterial.zip
Video:
 https://slideslive.com/38939754
Code
 nipunsadvilkar/pySBD
Data
GENIA