BString and UTF8

Don't use chars. Use string:

BString s = "Bärenstraße";
int p = s.FindFirst("ä");
printf("%d", p); // Out: 1

This is a C++ Linux equivalent... and it works no tricks ì is "a char" and is found in position 4... I had not to use a string (that is a char * = "ì" trick!).

#include <iostream>
#include <string>
using namespace std;
 
int
main(void)
{
   string testUTF8 = "Birìbò";
   char toFind = 'ì';
   size_t pos;
 
   pos = testUTF8.find(toFind);
   if (!pos) {
      cout << "The char is NOT found";
      return -1;
   }
 
   cout << "The char is found at position " << (int) pos << endl; // return 4 and it is right!
   return 0;
}

So Haiku's FindFirst(char c) should work in an analogous way...

Do you realise the difference between 'ä' and "ä", in memory? The first item occupies sizeof(char) bytes, which is usually one byte. The second item is a zero terminated byte array. Also, the reason your code worked under Linux was a coincidence which depends on the local users code page. On a typical US system, it would give different results to a west european code page. When you realise the difference, you've taken your first step towards understanding Unicode.

BString was created at a time when std::string was under-featured. Today it's a different story. But then again, the entire codepage/unicode text handling facility in C based languages is also a mess. This is the price of legacy.

Maybe you missed what I edited to my last post: "Don't mistake these forum chats with the official development process. That is really done on the dev mailing list.". If really want to work on the issue, you should consider taking it to the official list.

Regards,
Humdinger

That's odd. You just have to subscribe with the email address you're going to use when posting to the list and confirm the email sent to you by freelists.org once. After that all mails will be delivered to the account you specified and posting should a simple mail to haiku-development@freelists.org.

Regards,
Humdinger

I believe that the BString class can be extended in a compatible manner to handle UTF-8. I believe it can be done without changing the parameter types in the existing methods. It can be done so that all of the existing behavior can be done without changing the binary compatibility or the behavior of the existing methods. The one thing that my implementation won't do is locale sensitive linguistic comparisons. For that, a Collator class needs to be added to the Locale Kit.

I'm working on an implementation and am planning on writing some articles that demonstrate the new methods that will be added.

I've been working on adding full support in the BString class for UTF-8, and wanted to give an update on this issue.

First the bad news, then an explanation, then some good news.

Using national characters with single quotes (apostrophe) will not work with the BString. The problem is not the design or implementation of the BString. The problem is caused by a 'feature' of the C/C++ language.

The C/C++ language does not know about UTF-8. So, you can't natively handle UTF-8 in C/C++. From a purely technical point of view, using "char*" with UTF-8 strings is incorrect. In C/C++, a "char" is a "single byte character". A UTF-8 string is an array of "unsigned bytes", where a single character can be between one and four bytes in length. A more accurate type for a pointer to a UTF-8 string is "uint8*".

The '€' character constant is two bytes in length. The C/C++ language does not consider it a "char". So, what type is a character constant that uses single quotes and is two to four bytes? I tested this issue with several C++ compilers to see what type they considered a character constant that was between two and four bytes. One treated it as an "int". Another treated it as a "long". The C++ compilers used for Haiku treat it as an "unsigned long". Another compiler treated it as an "unsigned long", but also allowed characters constants up to 8 bytes. That compiler treated a character constant that was between 5 and 8 bytes as an "unsigned long long".

So where does that leave us?

If you want to use "operator<<" with characters in Haiku, you will have to use double quotes (if the character is not a 7-bit ASCII char).

When Be, Inc. designed BeOS, they used this 'feature' of C/C++ to their advantage. The "message constant" parameter to a BMessage is a "character constant" that can be between one and four bytes in length. One of the implications of this is that you can use national characters in a "message constant" for the BMessage system. You, of course, need to make sure that the value does not exceed four bytes in length. (The compiler will flag a compile error if you do that.)

Now for the good news.

My update of the BString class to enable full UTF-8 support is in good shape. I'm focusing on several aspects:
1) Ensure that the BString can only contain valid UTF-8 strings.
2) Add full support for both "code point" and "UTF-8 character" functions.
3) Enhance the Locale kit to complete the ability to fully handle UTF-8 characters for any language.

Beyond the code changes, I am also going to provide:
1) Updated API documentation for the "Haiku Book".
2) An article that is an introductory description of UTF-8, internationalization (i18n), collation, code points vs characters, normalization, etc... This document will also give code examples showing how to use these classes to provide full i18n support. (I would like this to be a chapter in the "overview" section of the Haiku Book.)

I'm targeting the first part of this for the end of November, 2011, with the remainder rolling out in 2012.

Notes:
A) There is often a misconception that in Unicode a "code point" and a "character" are the same thing. They are not. There are times when a character consists of a single code point, but in other cases, a character encompasses two or more code points. The BString class needs to understand (and support) both code points and characters. (New functions in the BString class will provide an easy way for applications to get the correct behavior.)
B) One of the main reasons for only allowing valid UTF-8 strings in the BString class is that not doing so would expose a security vulnerability. "String" exploits are a common target for computer viruses. Closing this security vulnerability is important.

I was going to try and explain this, but, mibrid beat me to it, so what he said. Using '€' just won't work, you have to use "€" because using '' indicates a single byte char and € isn't. This isn't a problem with BString() it is a fact set by the compiler.

Pleas in English :)

Hate to miss if a solution was given ;)

Selbstverständlich ist "€" etwas anderes als '€' - "€" ist im C++ Standard klar definiert als ein String, '€' ist laut Standard undefiniert und compilerabhängig. Dass '€' in anderen Betriebssystem funktioniert ist wohl Glücksache.

Sich in BString auf Eigenheiten des Compilers zu verlassen wäre gefährlich, BString könnte dann also mit einem Wechsel oder gar Update des Compilers plötzlich nicht mehr funktionieren.

Die Eigenschaft die du beschreibst, ist unglücklich, aber leider eine Eigenschaft von C/C++ selbst und BString ist dafür nicht verantwortlich. Die einzig vernünftige Methode um '€' zu benutzen, wäre zu einer Programmiersprache zu wechseln die UTF8 direkt unterstützt, innerhalb von C/C++ muss man sich mit "€" wohl oder übel abfinden.

Hi Octopus.

Hast Du einen Link zu dem Mailinglist Thread, in dem Du Deine Ideen/Routinen vorgestellt hast? Hier ist das Archiv.

Gruß,
Humdinger