Similarity Measure–Phonetic

Phonetic Matching is based on the principal that pronunciation depends on the language. So the first step is to determine the language from the spelling of the name. Then the name is converted into a sequence of phonetic tokens using pronunciation rules specific to that particular language. And, finally, names are compared based on their phonetic-token sequence.

  • Soundex

Soundex is a phonetic algorithm for indexing names by sound, as pronounced in English. The goal is for homophones to be encoded to the same representation so that they can be matched despite minor differences in spelling.[1] The algorithm mainly encodes consonants; a vowel will not be encoded unless it is the first letter. Soundex is the most widely known of all phonetic algorithms (in part because it is a standard feature of popular database software such as DB2, PostgreSQL,[2] MySQL,[3] Ingres,MS SQL Server[4] and Oracle[5]) and is often used (incorrectly) as a synonym for “phonetic algorithm”.

American Soundex

The Soundex code for a name consists of a letter followed by three numerical digits: the letter is the first letter of the name, and the digits encode the remaining consonants. Consonants at a similar place of articulation share the same digit so, for example, the labial consonants B, F, P, and V are each encoded as the number 1.

The correct value can be found as follows:

  1. Retain the first letter of the name and drop all other occurrences of a, e, i, o, u, y, h, w.
  2. Replace consonants with digits as follows (after the first letter):
    • b, f, p, v → 1
    • c, g, j, k, q, s, x, z → 2
    • d, t → 3
    • l → 4
    • m, n → 5
    • r → 6
  3. If two or more letters with the same number are adjacent in the original name (before step 1), only retain the first letter; also two letters with the same number separated by ‘h’ or ‘w’ are coded as a single number, whereas such letters separated by a vowel are coded twice. This rule also applies to the first letter.
  4. Iterate the previous step until you have one letter and three numbers. If you have too few letters in your word that you can’t assign three numbers, append with zeros until there are three numbers. If you have more than 3 letters, just retain the first 3 numbers.

Using this algorithm, both “Robert” and “Rupert” return the same string “R163” while “Rubin” yields “R150”. “Ashcraft” and “Ashcroft” both yield “A261” and not “A226” (the chars ‘s’ and ‘c’ in the name would receive a single number of 2 and not 22 since an ‘h’ lies in between them). “Tymczak” yields “T522” not “T520” (the chars ‘z’ and ‘k’ in the name are coded as 2 twice since a vowel lies in between them). “Pfister” yields “P236” not “P123” (the first two letters have the same number and are coded once as ‘P’).

 

  • NYIIS

The New York State Identification and Intelligence System Phonetic Code, commonly known as NYSIIS, is a phonetic algorithm devised in 1970 as part of the New York State Identification and Intelligence System (now a part of the New York State Division of Criminal Justice Services). It features an accuracy increase of 2.7% over the traditional Soundex algorithm.[1]

The algorithm, as described in Name Search Techniques,[2] is:

  1. Translate first characters of name: MAC → MCC, KN → N, K → C, PH, PF → FF, SCH → SSS
  2. Translate last characters of name: EE → Y, IE → Y, DT, RT, RD, NT, ND → D
  3. First character of key = first character of name.
  4. Translate remaining characters by following rules, incrementing by one character each time:
    1. EV → AF else A, E, I, O, U → A
    2. Q → G, Z → S, M → N
    3. KN → N else K → C
    4. SCH → SSS, PH → FF
    5. H → If previous or next is non-vowel, previous.
    6. W → If previous is vowel, A.
    7. Add current to key if current is not same as the last key character.
  5. If last character is S, remove it.
  6. If last characters are AY, replace with Y.
  7. If last character is A, remove it.
  8. Append translated key to value from step 3 (removed first character)
  9. If longer than 6 characters, truncate to first 6 characters. (only needed for true NYSIIS, some versions use the full key)

 

  • Metaphone

Metaphone is a phonetic algorithm, published by Lawrence Philips in 1990, for indexing words by their English pronunciation.[1] It fundamentally improves on the Soundex algorithm by using information about variations and inconsistencies in English spelling and pronunciation to produce a more accurate encoding, which does a better job of matching words and names which sound similar. As with Soundex, similar sounding words should share the same keys.

Original Metaphone codes use the 16 consonant symbols 0BFHJKLMNPRSTWXY.[2] The ‘0’ represents “th” (as an ASCII approximation of Θ), ‘X’ represents “sh” or “ch“, and the others represent their usual English pronunciations. The vowels AEIOU are also used, but only at the beginning of the code.[3] This table summarizes most of the rules in the original implementation:

  1. Drop duplicate adjacent letters, except for C.
  2. If the word begins with ‘KN’, ‘GN’, ‘PN’, ‘AE’, ‘WR’, drop the first letter.
  3. Drop ‘B’ if after ‘M’ at the end of the word.
  4. ‘C’ transforms to ‘X’ if followed by ‘IA’ or ‘H’ (unless in latter case, it is part of ‘-SCH-‘, in which case it transforms to ‘K’). ‘C’ transforms to ‘S’ if followed by ‘I’, ‘E’, or ‘Y’. Otherwise, ‘C’ transforms to ‘K’.
  5. ‘D’ transforms to ‘J’ if followed by ‘GE’, ‘GY’, or ‘GI’. Otherwise, ‘D’ transforms to ‘T’.
  6. Drop ‘G’ if followed by ‘H’ and ‘H’ is not at the end or before a vowel. Drop ‘G’ if followed by ‘N’ or ‘NED’ and is at the end.
  7. ‘G’ transforms to ‘J’ if before ‘I’, ‘E’, or ‘Y’, and it is not in ‘GG’. Otherwise, ‘G’ transforms to ‘K’.
  8. Drop ‘H’ if after vowel and not before a vowel.
  9. ‘CK’ transforms to ‘K’.
  10. ‘PH’ transforms to ‘F’.
  11. ‘Q’ transforms to ‘K’.
  12. ‘S’ transforms to ‘X’ if followed by ‘H’, ‘IO’, or ‘IA’.
  13. ‘T’ transforms to ‘X’ if followed by ‘IA’ or ‘IO’. ‘TH’ transforms to ‘0’. Drop ‘T’ if followed by ‘CH’.
  14. ‘V’ transforms to ‘F’.
  15. ‘WH’ transforms to ‘W’ if at the beginning. Drop ‘W’ if not followed by a vowel.
  16. ‘X’ transforms to ‘S’ if at the beginning. Otherwise, ‘X’ transforms to ‘KS’.
  17. Drop ‘Y’ if not followed by a vowel.
  18. ‘Z’ transforms to ‘S’.
  19. Drop all vowels unless it is the beginning.

It should be noted, however, that this table does not constitute a complete description of the original Metaphone algorithm, and the algorithm cannot be coded correctly from it. Original Metaphone contained many errors and was superseded by Double Metaphone, and in turn Double Metaphone and original Metaphone were superseded by Metaphone 3, which corrects thousands of miscodings that will be produced by the first two versions.

 

Double Metaphone

The Double Metaphone phonetic encoding algorithm is the second generation of this algorithm. Its implementation was described in the June 2000 issue of C/C++ Users Journal. It makes a number of fundamental design improvements over the original Metaphone algorithm.

It is called “Double” because it can return both a primary and a secondary code for a string; this accounts for some ambiguous cases as well as for multiple variants of surnames with common ancestry. For example, encoding the name “Smith” yields a primary code of SM0 and a secondary code of XMT, while the name “Schmidt” yields a primary code of XMT and a secondary code of SMT—both have XMT in common.

Double Metaphone tries to account for myriad irregularities in English of Slavic, Germanic, Celtic, Greek, French, Italian, Spanish, Chinese, and other origin. Thus it uses a much more complex ruleset for coding than its predecessor; for example, it tests for approximately 100 different contexts of the use of the letter C alone.

To implement Metaphone without purchasing a (source code) copy of Metaphone 3, the best guide would be the reference implementation of Double Metaphone, which may be found here.

 

  • Metaphone 3

Metaphone 3 is the third generation of the Metaphone algorithm. It increases the accuracy of phonetic encoding from the 89% of Double Metaphone to 98%, as tested against a database of the most common English words, and names and non-English words familiar in North America. This produces an extremely reliable phonetic encoding for American pronunciations.
Metaphone 3 was designed and developed by Lawrence Philips, who designed and developed the original Metaphone and Double Metaphone algorithms.

A professional version was released in October 2009, developed by the same author, Lawrence Philips. It is a commercial product but is sold as source code. Metaphone 3 further improves phonetic encoding of words in the English language, non-English words familiar to Americans, and first names and family names commonly found in the United States.[4] It improves encoding for proper names in particular to a considerable extent.[5] The author claims that in general it improves accuracy for all words from the approximately 89% of Double Metaphone to 98%. Developers can also now set switches in to code to cause the algorithm to encode Metaphone keys 1) taking non-initial vowels into account, as well as 2) encoding voiced and unvoiced consonants differently. This allows the result set to be more closely focused if the developer finds that the search results include too many words that don’t resemble the search term closely enough.[6] Metaphone 3 is sold as C++, Java, C#, PHP, Perl, and PL/SQL source, as well as Metaphone 3 for Spanish and German available as Java source.[7] The latest revision of the Metaphone 3 algorithm is v2.5.2, released February 2015.

 

 

References:

http://stevemorse.org/phonetics/bmpm2.htm

http://en.wikipedia.org/wiki/Soundex

http://en.wikipedia.org/wiki/New_York_State_Identification_and_Intelligence_System

http://en.wikipedia.org/wiki/Metaphone

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s