加载腾讯AI LAB模型报错文件不存在

ericding1314 · April 7, 2020, 11:52am

Hanlp 2.0直接load(‘TENCENT_AI_LAB_EMBEDDING’)，压缩包下载好了，但是却报下面的错误。
FileNotFoundError: The identifier $HANLP_HOME/thirdparty/ai.tencent.com/ailab/nlp/data/Tencent_AILab_ChineseEmbedding.tar/Tencent_AILab_ChineseEmbedding.txt resolves to a non-exist meta file $HANLP_HOME/thirdparty/ai.tencent.com/ailab/nlp/data/Tencent_AILab_ChineseEmbedding.tar/Tencent_AILab_ChineseEmbedding.txt/meta.json.

hankcs · April 10, 2020, 6:46pm

大部分第三方模型不符合hanlp规范，不能直接hanlp.load。你可以通过设计好的类型load：

import hanlp
from hanlp.layers.embeddings.word2vec import Word2VecEmbedding

print(Word2VecEmbedding(filepath=hanlp.pretrained.word2vec.TENCENT_AI_LAB_EMBEDDING).input_dim)

ericding1314 · April 12, 2020, 6:46am

感谢解答，可以加载了。

lyling · January 18, 2021, 10:27am

要加载的文件TENCENT_AI_LAB_EMBEDDING应该放到哪?

turkeymz · May 31, 2021, 8:47am

我在使用Word2Vec预训练模型时无法加载，缺少配置文件。但我尝试用网盘分享的下载zip解压，删掉全部重新运行代码。依然报这个错误。请问是什么问题？
另外有没有2.x的比较全的文档包，因为我的python执行环境无法使用在线api，
import hanlp

model = hanlp.load(hanlp.pretrained.word2vec.CONVSEG_W2V_NEWS_TENSITE_WORD_PKU)
model([
‘看图猜一电影名’,
‘无线路由器怎么无线上网’,
‘北京到上海的动车票’,
])

Traceback (most recent call last):
File “E:/Project/python/jxnlp-sdk/test/Word2VecTest.py”, line 9, in
model = hanlp.load(hanlp.pretrained.word2vec.CONVSEG_W2V_NEWS_TENSITE_WORD_PKU)
File “E:\Project\python\jxnlp-sdk\hanlp_init_.py”, line 43, in load
return load_from_meta_file(save_dir, ‘meta.json’, verbose=verbose, **kwargs)
File “E:\Project\python\jxnlp-sdk\hanlp\utils\component_util.py”, line 53, in load_from_meta_file
raise FileNotFoundError(f’The identifier {save_dir} resolves to a non-exist meta file {metapath}. {tips}’)
FileNotFoundError: The identifier C:\Users\Administrator\AppData\Roaming\hanlp\hanlp\embeddings\convseg_embeddings\news_tensite.pku.words.w2v50 resolves to a non-exist meta file C:\Users\Administrator\AppData\Roaming\hanlp\hanlp\embeddings\convseg_embeddings\news_tensite.pku.words.w2v50\config.json.

turkeymz · June 1, 2021, 8:58am

你好，我是从这个帖子转过来的，但遇到了下面两个问题（https://bbs.hankcs.com/t/topic/3804）

我使用的是源码预训练的模型（hanlp.pretrained.word2vec.SEMEVAL16_EMBEDDINGS_300_TEXT_CN）
我屏蔽然后执行上述这段代码，发现还是报错？（Traceback (most recent call last):
File “E:/Project/python/jxnlp-sdk/test/Word2VecTest.py”, line 24, in
print(Word2VecEmbedding(filepath=hanlp.pretrained.word2vec.TENCENT_AI_LAB_EMBEDDING).input_dim)
TypeError: init() got an unexpected keyword argument ‘filepath’）

我是从git 克隆的，应该属于2.X。然后本地随便建一个python文件测试

hankcs · June 1, 2021, 1:13pm

在2.1中更名为：

github.com

hankcs/HanLP/blob/bce4a4e2dabcc9e0a07bfa3f8eed3ae351657041/hanlp/layers/embeddings/word2vec_tf.py#L111


            return vectors
        elif isinstance(word, tf.Tensor):
            if word.dtype == tf.string:
                word_ids = self.vocab.token_to_idx_table.lookup(word)
                return tf.nn.embedding_lookup(self.array_tf, word_ids)
            elif word.dtype == tf.int32 or word.dtype == tf.int64:
                return tf.nn.embedding_lookup(self.array_tf, word)
@hanlp_register
class Word2VecEmbeddingTF(tf.keras.layers.Embedding):
    def __init__(self, filepath: str = None, vocab: VocabTF = None, expand_vocab=True, lowercase=True,
                 input_dim=None, output_dim=None, unk=None, normalize=False,
                 embeddings_initializer='VarianceScaling',
                 embeddings_regularizer=None,
                 activity_regularizer=None, embeddings_constraint=None, mask_zero=True, input_length=None,
                 name=None, cpu=True, **kwargs):
        filepath = get_resource(filepath)
        word2vec, _output_dim = load_word2vec(filepath)
        if output_dim:

word2vec、FastText之类的模块被设计为其他网络的embedding层，不提供面向终端用户进行句向量计算这种弱功能。

JLUGQQ · December 11, 2021, 1:49am

import hanlp
Hanlp = hanlp.load(hanlp.pretrained.word2vec.RADICAL_CHAR_EMBEDDING_100)

FileNotFoundError: The identifier /home/gavin/.hanlp/embeddings/radical_char_vec_20191229_013849/character.vec.txt resolves to a non-exist meta file /home/gavin/.hanlp/embeddings/radical_char_vec_20191229_013849/character.vec.txt/config.json.

看了一下文件已经下载好了，没有出现网络问题。