Harnessing Pre-Trained Neural Networks with Rules for Formality Style Transfer

Text Generation Style Transfer
#1

Rule-based systems often generate cheap but noisy silver data which serves as a useful source of augmentation. This paper explores 3 ways to harness the seq2seq model with rules for formality style transfer.

1

These 3 methods are:

  1. Concatenate original input and rule-based output as input using a separator.
  2. Feed original input and rule-based output to 2 separate encoder/decoder pairs then average the outputs using a voting mechanism.
  3. Like 2 but compute a hierarchical attention to aggregate decoder outputs.

Interestingly, the 1st method (concatenation) performs the best, which is probably because it introduces no extra parameters.

Overall, this is a good engineering paper. Its novelty is that it is the first to explore how to integrate rule-based systems into neural style transfer models. Its weaknesses are mostly methodologies and experiments. It could have tried to simply mix the original input and rule-based outputs and see what happens. I would bet naïve mixing data would not yield much worse results. The baseline model they are using is too weak. A copy mechanism is usually essential to handle named entities for these tasks.

Overall Recommendation

  • 5: Transformative: This paper is likely to change our field. It should be considered for a best paper award.
  • 4.5: Exciting: It changed my thinking on this topic. I would fight for it to be accepted.
  • 4: Strong: I learned a lot from it. I would like to see it accepted.
  • 3.5: Leaning positive: It can be accepted more or less in its current form. However, the work it describes is not particularly exciting and/or inspiring, so it will not be a big loss if people don’t see it in this conference.
  • 3: Ambivalent: It has merits (e.g., it reports state-of-the-art results, the idea is nice), but there are key weaknesses (e.g., I didn’t learn much from it, evaluation is not convincing, it describes incremental work). I believe it can significantly benefit from another round of revision, but I won’t object to accepting it if my co-reviewers are willing to champion it.
  • 2.5: Leaning negative: I am leaning towards rejection, but I can be persuaded if my co-reviewers think otherwise.
  • 2: Mediocre: I would rather not see it in the conference.
  • 1.5: Weak: I am pretty confident that it should be rejected.
  • 1: Poor: I would fight to have it rejected.