We released a minor updated version of HanLP, bringing the following features to you.
- Tokenizers can be configured to output the span of each token now. So integration with ElasticSeach becomes possible for v2.1.
- Note that the span
["HanLPv2.1", 6, 15]now aligns with the input text even when there exist spaces.
- Note that the span
HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)
HanLP['tok/fine'].config.output_spans = True
doc = HanLP(['2021年 HanLPv2.1 为生产环境带来次世代最先进的多语种NLP技术。', '阿婆主来到北京立方庭参观自然语义科技公司。'], tasks='tok')
print(doc)
{
"tok/fine": [
[["2021年", 0, 5], ["HanLPv2.1", 6, 15], ["为", 16, 17], ["生产", 17, 19], ["环境", 19, 21], ["带来", 21, 23], ["次", 23, 24], ["世代", 24, 26], ["最", 26, 27], ["先进", 27, 29], ["的", 29, 30], ["多", 30, 31], ["语种", 31, 33], ["NLP", 33, 36], ["技术", 36, 38], ["。", 38, 39]],
[["阿婆主", 0, 3], ["来到", 3, 5], ["北京", 5, 7], ["立方庭", 7, 10], ["参观", 10, 12], ["自然", 12, 14], ["语义", 14, 16], ["科技", 16, 18], ["公司", 18, 20], ["。", 20, 21]]
]
}
-
Improvement on apostrophe tokenization for English.
-
An updated Semantic Textual Similarity model for Chinese.
-
Mirrors of xlm-roberta-base and bert-base-japanese-char for faster loading.
An upgrade is highly recommended.