Convolutions and Self-Attention: Re-interpreting Relative Positions in...

This paper reveals the similarity of convolutions and self-attention from the perspective of relative position embeddings in the following formula.

\begin{equation}\label{eq:convq-convfixed} \begin{aligned} % Just using this to add a space below. \underbrace{\textstyle \frac{(\textbf{x}_i W^Q) (\textbf{x}_j W^K)^{T}}{\sqrt{d_h}}}_{\textrm{Self-attention}} + \underbrace{\textstyle \frac{(\textbf{x}_i W^Q)(W^C_{j-i})^T}{\sqrt{d_h}}}_{\substack{\textrm{Dynamic convolution} \\ \textrm{(relative embeddings)}}} + \underbrace{{\beta_{j-i}}}_{\substack{\textrm{Fixed} \\ \textrm{convolution}}} \\ \\ \end{aligned} % \addstackgap[6pt] \end{equation}

By combining these 3 terms together, their pre-trained BERT is able to obtain a very marginal improvement.

Comments

  • Not so novel, better to write a short paper.
Rating
  • 5: Transformative: This paper is likely to change our field. It should be considered for a best paper award.
  • 4.5: Exciting: It changed my thinking on this topic. I would fight for it to be accepted.
  • 4: Strong: I learned a lot from it. I would like to see it accepted.
  • 3.5: Leaning positive: It can be accepted more or less in its current form. However, the work it describes is not particularly exciting and/or inspiring, so it will not be a big loss if people don’t see it in this conference.
  • 3: Ambivalent: It has merits (e.g., it reports state-of-the-art results, the idea is nice), but there are key weaknesses (e.g., I didn’t learn much from it, evaluation is not convincing, it describes incremental work). I believe it can significantly benefit from another round of revision, but I won’t object to accepting it if my co-reviewers are willing to champion it.
  • 2.5: Leaning negative: I am leaning towards rejection, but I can be persuaded if my co-reviewers think otherwise.
  • 2: Mediocre: I would rather not see it in the conference.
  • 1.5: Weak: I am pretty confident that it should be rejected.
  • 1: Poor: I would fight to have it rejected.

0 投票者