Simple and Effective Paraphrastic Similarity from Parallel Translations

Wieting, Gimpel, Neubig and Berg-Kirkpatrick, ACL 2019 (short paper), arXiv:1909.13872. T1 (peer-reviewed venue) — a training-methodology paper for paraphrastic-sentence-embeddings, and the corpus's first source on representation learning as its own subject rather than a downstream use of a model someone else trained.

The method

Sentence encoders are usually trained on paraphrase pairs, and paraphrase corpora are expensive: john-wieting's own earlier work reports 10,000 GPU hours to build one by backtranslating one side of a bitext corpus. This paper skips that step and trains directly on the bitext itself — ordinary parallel sentence pairs from translation data, which occur far more naturally at scale than paraphrase data ever will.

The objective is a margin loss over cosine similarity: push a source sentence and its true translation closer than the source and a hard negative — the highest-scoring wrong translation in the current mega-batch, following the mega-batching procedure from john-wieting's prior work. The encoder itself is deliberately simple: averaged SentencePiece (SP) token embeddings, not an LSTM or transformer. This repeats a finding already established for monolingual paraphrase embeddings — pooling beats deep architectures — and the paper shows the same pattern holds in the bilingual setting too.

Results

• It beats both deep baselines on 4 of 6 SemEval-2017 cross-lingual STS datasets (ar-ar, ar-en, es-es, es-en, en-en, tr-en) — Schwenk (2018, 3-layer BiLSTM on 9 Europarl languages) and Chidambaram et al. (2018, 3-layer transformer, multi-task dual-encoder) — despite both being structurally more complex and trained on more data.
• It's faster by ~330×: 855,571 sentences/sec (SP, 300-dim) versus Schwenk's 2,601/sec, measured on the same GPU and batch size. The paper's "orders of magnitude faster" claim is this number, not a rounded estimate.
• It wins bitext mining too — F1 on BUCC en-de/en-fr, though by a narrower margin than on STS, which the authors attribute to a domain mismatch between the training data (Europarl, legislative text) and the STS evaluation data (image captions).

The finding that undercuts its own field

Section 3.3's second result is the one worth recording on its own: random, untrained encoders — especially averaged random trigram embeddings — are competitive with or beat every literature baseline on monolingual STS. They're doing it by measuring lexical overlap between the two sentences, not semantics. The effect disappears cross-lingually, where there's no lexical overlap to exploit between different alphabets — which is exactly why the paper's cross-lingual results are the ones that isolate a genuine semantic signal. It's a direct, measured instance of the same caution tabular-deep-learning-benchmark and pervasive-label-errors make from different angles: a benchmark result is only as informative as the baseline it beat, and an untrained baseline nobody checked can be winning for the wrong reason.

Does the choice of language matter?

Training English embeddings against 46 different bitext languages, STS performance correlates with SentencePiece vocabulary overlap with English first, and with syntactic distance (URIEL features) once overlap is controlled for. Related languages give a cleaner training signal — plausibly because their machine-translation data is itself less noisy. This is a training-data selection result, not just a curiosity: it says which non-English language to pair with English if the deployment target is English semantic similarity.

Related

paraphrastic-sentence-embeddings · john-wieting · machine-learning · tabular-deep-learning-benchmark · pervasive-label-errors · synthesis