OpenQuizz
Une application de gestion des contenus pédagogiques
HebrewProber Class Reference

General ideas of the Hebrew charset recognition ###. More...

Inheritance diagram for HebrewProber:
Collaboration diagram for HebrewProber:

Public Member Functions

def __init__ (self)
 
def reset (self)
 
def set_model_probers (self, logicalProber, visualProber)
 
def is_final (self, c)
 
def is_non_final (self, c)
 
def feed (self, byte_str)
 
def charset_name (self)
 
def language (self)
 
def state (self)
 
- Public Member Functions inherited from CharSetProber
def __init__ (self, lang_filter=None)
 
def get_confidence (self)
 

Data Fields

 state
 
- Data Fields inherited from CharSetProber
 lang_filter
 
 logger
 

Static Public Attributes

 FINAL_KAF
 
 NORMAL_KAF
 
 FINAL_MEM
 
 NORMAL_MEM
 
 FINAL_NUN
 
 NORMAL_NUN
 
 FINAL_PE
 
 NORMAL_PE
 
 FINAL_TSADI
 
 NORMAL_TSADI
 
 MIN_FINAL_CHAR_DISTANCE
 
 MIN_MODEL_DISTANCE
 
 VISUAL_HEBREW_NAME
 
 LOGICAL_HEBREW_NAME
 
- Static Public Attributes inherited from CharSetProber
 SHORTCUT_THRESHOLD
 

Additional Inherited Members

- Static Public Member Functions inherited from CharSetProber
def filter_high_byte_only (buf)
 
def filter_international_words (buf)
 
def filter_with_english_letters (buf)
 

Detailed Description

General ideas of the Hebrew charset recognition ###.

Four main charsets exist in Hebrew: "ISO-8859-8" - Visual Hebrew "windows-1255" - Logical Hebrew "ISO-8859-8-I" - Logical Hebrew "x-mac-hebrew" - ?? Logical Hebrew ??

Both "ISO" charsets use a completely identical set of code points, whereas "windows-1255" and "x-mac-hebrew" are two different proper supersets of these code points. windows-1255 defines additional characters in the range 0x80-0x9F as some misc punctuation marks as well as some Hebrew-specific diacritics and additional 'Yiddish' ligature letters in the range 0xc0-0xd6. x-mac-hebrew defines similar additional code points but with a different mapping.

As far as an average Hebrew text with no diacritics is concerned, all four charsets are identical with respect to code points. Meaning that for the main Hebrew alphabet, all four map the same values to all 27 Hebrew letters (including final letters).

The dominant difference between these charsets is their directionality. "Visual" directionality means that the text is ordered as if the renderer is not aware of a BIDI rendering algorithm. The renderer sees the text and draws it from left to right. The text itself when ordered naturally is read backwards. A buffer of Visual Hebrew generally looks like so: "[last word of first line spelled backwards] [whole line ordered backwards and spelled backwards] [first word of first line spelled backwards] [end of line] [last word of second line] ... etc' " adding punctuation marks, numbers and English text to visual text is naturally also "visual" and from left to right.

"Logical" directionality means the text is ordered "naturally" according to the order it is read. It is the responsibility of the renderer to display the text from right to left. A BIDI algorithm is used to place general punctuation marks, numbers and English text in the text.

Texts in x-mac-hebrew are almost impossible to find on the Internet. From what little evidence I could find, it seems that its general directionality is Logical.

To sum up all of the above, the Hebrew probing mechanism knows about two charsets: Visual Hebrew - "ISO-8859-8" - backwards text - Words and sentences are backwards while line order is natural. For charset recognition purposes the line order is unimportant (In fact, for this implementation, even word order is unimportant). Logical Hebrew - "windows-1255" - normal, naturally ordered text.

"ISO-8859-8-I" is a subset of windows-1255 and doesn't need to be specifically identified. "x-mac-hebrew" is also identified as windows-1255. A text in x-mac-hebrew that contain special punctuation marks or diacritics is displayed with some unconverted characters showing as question marks. This problem might be corrected using another model prober for x-mac-hebrew. Due to the fact that x-mac-hebrew texts are so rare, writing another model prober isn't worth the effort and performance hit.

The Prober ####

The prober is divided between two SBCharSetProbers and a HebrewProber, all of which are managed, created, fed data, inquired and deleted by the SBCSGroupProber. The two SBCharSetProbers identify that the text is in fact some kind of Hebrew, Logical or Visual. The final decision about which one is it is made by the HebrewProber by combining final-letter scores with the scores of the two SBCharSetProbers to produce a final answer.

The SBCSGroupProber is responsible for stripping the original text of HTML tags, English characters, numbers, low-ASCII punctuation characters, spaces and new lines. It reduces any sequence of such characters to a single space. The buffer fed to each prober in the SBCS group prober is pure text in high-ASCII. The two SBCharSetProbers (model probers) share the same language model: Win1255Model. The first SBCharSetProber uses the model normally as any other SBCharSetProber does, to recognize windows-1255, upon which this model was built. The second SBCharSetProber is told to make the pair-of-letter lookup in the language model backwards. This in practice exactly simulates a visual Hebrew model using the windows-1255 logical Hebrew model.

The HebrewProber is not using any language model. All it does is look for final-letter evidence suggesting the text is either logical Hebrew or visual Hebrew. Disjointed from the model probers, the results of the HebrewProber alone are meaningless. HebrewProber always returns 0.00 as confidence since it never identifies a charset by itself. Instead, the pointer to the HebrewProber is passed to the model probers as a helper "Name Prober". When the Group prober receives a positive identification from any prober, it asks for the name of the charset identified. If the prober queried is a Hebrew model prober, the model prober forwards the call to the HebrewProber to make the final decision. In the HebrewProber, the decision is made according to the final-letters scores maintained and Both model probers scores. The answer is returned in the form of the name of the charset identified, either "windows-1255" or "ISO-8859-8".

Constructor & Destructor Documentation

◆ __init__()

def __init__ (   self)

Member Function Documentation

◆ charset_name()

def charset_name (   self)

Reimplemented from CharSetProber.

◆ feed()

def feed (   self,
  byte_str 
)

Reimplemented from CharSetProber.

◆ is_final()

def is_final (   self,
  c 
)

◆ is_non_final()

def is_non_final (   self,
  c 
)

◆ language()

def language (   self)

◆ reset()

def reset (   self)

Reimplemented from CharSetProber.

◆ set_model_probers()

def set_model_probers (   self,
  logicalProber,
  visualProber 
)

◆ state()

def state (   self)

Reimplemented from CharSetProber.

Field Documentation

◆ FINAL_KAF

FINAL_KAF
static

◆ FINAL_MEM

FINAL_MEM
static

◆ FINAL_NUN

FINAL_NUN
static

◆ FINAL_PE

FINAL_PE
static

◆ FINAL_TSADI

FINAL_TSADI
static

◆ LOGICAL_HEBREW_NAME

LOGICAL_HEBREW_NAME
static

◆ MIN_FINAL_CHAR_DISTANCE

MIN_FINAL_CHAR_DISTANCE
static

◆ MIN_MODEL_DISTANCE

MIN_MODEL_DISTANCE
static

◆ NORMAL_KAF

NORMAL_KAF
static

◆ NORMAL_MEM

NORMAL_MEM
static

◆ NORMAL_NUN

NORMAL_NUN
static

◆ NORMAL_PE

NORMAL_PE
static

◆ NORMAL_TSADI

NORMAL_TSADI
static

◆ state

state

◆ VISUAL_HEBREW_NAME

VISUAL_HEBREW_NAME
static

The documentation for this class was generated from the following file: