Hanlp2.1无法加载2.0训练的模型吗

Gary · September 6, 2022, 7:10am

何博士，你好，我之前使用hanlp2.0版本训练了一个自己的模型，因为发现tf存在内存泄漏，后来看到你说升级tf2.7可以解决，所以我升级了hanlp2.1.0版本，但是在加载我原来的模型的时候，提示hanlp版本不对，需要pip install hanlp[full]。可是我进行pip install hanlp[full]安装之后，仍然报了同样的错误。请问有没有哪个版本可以解决这个问题，因为我的模型已经上线了，希望只需要升级hanlp就可以了。

hankcs · September 9, 2022, 8:39pm

分情况讨论，对于没用到BERT的模型，升级到2.1只需将模型路径下的meta.json中的类路径替换掉即可，参考：

github.com

hankcs/HanLP/blob/68e6527dfd55c913593dc598596299cb2455c554/hanlp/utils/component_util.py#L75

      
        
                               'embed': {'classpath': 'hanlp.layers.embeddings.fast_text.FastTextEmbedding',
                                         'filepath': identifier, 'src': 'token'},
                               'hanlp_version': version.__version__}, metapath)
                else:
                    raise FileNotFoundError(f'The identifier {save_dir} resolves to a nonexistent meta file {metapath}. {tips}')
            meta: dict = load_json(metapath)
            cls = meta.get('classpath', cls)
            if not cls:
                cls = meta.get('class_path', None)  # For older version
            if tf_model:
                # tf models are trained with version < 2.1. To migrate them to 2.1, map their classpath to new locations
                upgrade = {
                    'hanlp.components.tok_tf.TransformerTokenizerTF': 'hanlp.components.tokenizers.tok_tf.TransformerTokenizerTF',
                    'hanlp.components.pos.RNNPartOfSpeechTagger': 'hanlp.components.taggers.pos_tf.RNNPartOfSpeechTaggerTF',
                    'hanlp.components.pos_tf.RNNPartOfSpeechTaggerTF': 'hanlp.components.taggers.pos_tf.RNNPartOfSpeechTaggerTF',
                    'hanlp.components.pos_tf.CNNPartOfSpeechTaggerTF': 'hanlp.components.taggers.pos_tf.CNNPartOfSpeechTaggerTF',
                    'hanlp.components.ner_tf.TransformerNamedEntityRecognizerTF': 'hanlp.components.ner.ner_tf.TransformerNamedEntityRecognizerTF',
                    'hanlp.components.parsers.biaffine_parser.BiaffineDependencyParser': 'hanlp.components.parsers.biaffine_parser_tf.BiaffineDependencyParserTF',
                    'hanlp.components.parsers.biaffine_parser.BiaffineSemanticDependencyParser': 'hanlp.components.parsers.biaffine_parser_tf.BiaffineSemanticDependencyParserTF',
                    'hanlp.components.tok_tf.NgramConvTokenizerTF': 'hanlp.components.tokenizers.tok_tf.NgramConvTokenizerTF',
                    'hanlp.components.classifiers.transformer_classifier.TransformerClassifier': 'hanlp.components.classifiers.transformer_classifier_tf.TransformerClassifierTF',

对于用到BERT的模型，由于bert-for-tf2被替换为transformers，这类模型只能重新训练。