What is the best way to make the API return token - pos pairs?
HanLP.parse(‘哎呀！北京冬天超级冷’, tasks=‘pos/pku’).pretty_print()
returns
哎呀/e ！/w
北京/ns 冬天/t 超级/d 冷/a
which is not a format that can be easily used.
How do I get something like
[(‘哎呀’, ‘e’), (’！’, ‘w’), …]
Lack of Token Offsets/Indices: The most critical missing piece is character or byte offsets (start and end positions) for each token in both the coarse and fine segmentation lists. If these were present, you could simply check for overlapping spans. For example:
Coarse: {“word”: “西方文化”, “start”: 0, “end”: 4}
Fine: {“word”: “西方”, “start”: 0, “end”: 2}, {“word”: “文化”, “start”: 2, “end”: 4} This would make linking trivial.
This would also help linking a POS tag to a word. Is something like this possible or am I missing something?