DETECTION OF UNKNOWN CODE PAGE INDEXING TOKENS

A method for determining an encoding used for a sequence of bytes may be provided. The method comprises providing a set of candidate code pages and transforming them into different groups of sequences of bytes, wherein each group of sequences of bytes corresponds to one of the candidate code pages. Thereby each code point is transformed by applying a transformation from one of the candidate code pages to a reference code point value relating to a reference encoding for each code point. The method comprises further separating each of the transformed sequences of bytes into groups of tokens, wherein each group of tokens relates to one candidate code page, and providing an index relating to a text corpus. Furthermore, the method comprises selecting a code page from the set of candidate code pages at least partially based on how many tokens are found in the index.