Paraphrastic sentence embeddings

A sentence encoder trained so that sentences with the same meaning land close together in vector space, measured by cosine similarity. "Paraphrastic" names the training signal, not the architecture: the model learns from pairs that mean the same thing — paraphrases, or translations of each other — rather than from a labelled similarity score.

Two ways to get the training signal

• Build a paraphrase corpus first, then train on it — via bilingual pivoting or backtranslating one side of a bitext corpus. Effective, but the corpus-building step is its own expensive pipeline (10,000 GPU hours, in one reported case).
• Train directly on bitext — ordinary parallel-translation data, skipping paraphrase-corpus construction entirely. paraphrastic-similarity-parallel-translations is the source for this route: a margin loss with hard-negative mining over cosine similarity, on plain translation pairs.

The second route has a byproduct the first doesn't: because training pairs cross a language boundary, the resulting embeddings are also cross-lingual — usable for mining or filtering parallel corpora, and for cross-lingual semantic similarity, without ever training on that task directly.

What the corpus learned from it

Simple pooled encoders (averaged token embeddings) have repeatedly outperformed deeper architectures on this task, monolingually and now bilingually — the corpus that trains best isn't the one with the most parameters. And the standard evaluation, STS (semantic textual similarity), rewards lexical overlap unless the comparison is cross-lingual: an untrained random encoder can win a monolingual STS leaderboard for the wrong reason, which is why paraphrastic-similarity-parallel-translations treats its cross-lingual results, not its monolingual ones, as the real evidence of a semantic signal.

Related

paraphrastic-similarity-parallel-translations · john-wieting · machine-learning