Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained...

Linguistics Semantic Textual Similarity
#1

Disentangling semantics and syntax is an effective approach in text style transfer as well as unsupervised sentence embedding learning. This paper uses separate encoders for semantics and syntax to learn paraphrasing on ParaNMT dataset, resulting in a sentence semantics encoder disentangled from syntax.

The syntax encoder operates on linearized and de-lexicalized constituency trees. They further applies adversarial learning to encourage disentanglement by adding a classifier to predict BoW of level constituent tags from the semantic embedding, with the following objective:

\min\limits_{E_{sem}, E_{syn}, D_{dec}} \left(\max\limits_{D_{dis}} \left(\mathcal{L}_{para}-\lambda_{adv}\mathcal{L}_{adv}\right)\right)

where \lambda_{adv} is a hyperparameter to balance loss terms. In each iteration, the D_{dis} is updated by considering the inner optimization so \mathcal{L}_{para} is constant to D_{dis} (that’s why it can be placed inside the \max), and then update E_{sem}, E_{syn} and D_{dec} by considering the outer optimization.

Comments

  • Overall quality is good, especially the adversarial part.
  • The syntax encoder is quite crude. There exists many tree RNNs for constituent trees.
Rating
  • 5: Transformative: This paper is likely to change our field. It should be considered for a best paper award.
  • 4.5: Exciting: It changed my thinking on this topic. I would fight for it to be accepted.
  • 4: Strong: I learned a lot from it. I would like to see it accepted.
  • 3.5: Leaning positive: It can be accepted more or less in its current form. However, the work it describes is not particularly exciting and/or inspiring, so it will not be a big loss if people don’t see it in this conference.
  • 3: Ambivalent: It has merits (e.g., it reports state-of-the-art results, the idea is nice), but there are key weaknesses (e.g., I didn’t learn much from it, evaluation is not convincing, it describes incremental work). I believe it can significantly benefit from another round of revision, but I won’t object to accepting it if my co-reviewers are willing to champion it.
  • 2.5: Leaning negative: I am leaning towards rejection, but I can be persuaded if my co-reviewers think otherwise.
  • 2: Mediocre: I would rather not see it in the conference.
  • 1.5: Weak: I am pretty confident that it should be rejected.
  • 1: Poor: I would fight to have it rejected.

0 投票者