Banjo API 1.1.0-alpha
Low-level C99 game development API
Loading...
Searching...
No Matches
Unicode
Collaboration diagram for Unicode:

Data Structures

struct  bj_utf8_report

Typedefs

typedef bj_bool(* bj_utf8_fn) (const struct bj_utf8_report *report, void *user_data)

Enumerations

enum  bj_utf8_state {
  BJ_UTF8_STATE_VALID = 0 , BJ_UTF8_STATE_BAD_LEAD , BJ_UTF8_STATE_LONE_CONTINUATION , BJ_UTF8_STATE_BAD_SLOT ,
  BJ_UTF8_STATE_TRUNCATED
}

Functions

bj_bool bj_utf8_next (struct bj_str view, size_t *cursor, struct bj_utf8_report *report)
size_t bj_utf8_scan (struct bj_str view, bj_utf8_fn callback, void *user_data)
bj_bool bj_utf8_prev (struct bj_str view, size_t *cursor, uint32_t *value)
size_t bj_utf8_encode (uint32_t value, char bytes[4])
bj_bool bj_utf8_validate (struct bj_str view, size_t *first_error)

Detailed Description

Decode and encode UTF-8 text.

UTF-8 represents each Unicode character with one to four bytes. This module provides functions to read those characters, write them as UTF-8, and check whether a byte sequence is valid.

Use bj_utf8_next to read one character at a time, or bj_utf8_scan to process an entire view with a callback. Each read produces a bj_utf8_report describing the bytes and the result.

Invalid UTF-8

When a byte sequence is malformed, the decoder reports the valid prefix of the attempted character and leaves the byte that caused the error for the next read. This lets decoding continue after malformed input without losing subsequent valid characters. For example, E0 80 AF produces three failed reports.

Use bj_utf8_validate to check input before storing or processing it. The string constructor bj_create_string instead replaces malformed sequences with U+FFFD.

Warning
The functions that read forward can report malformed bytes. bj_utf8_prev reads backwards and therefore requires valid UTF-8.

Data Structure Documentation

◆ bj_utf8_report

struct bj_utf8_report

Details about one UTF-8 decoding attempt.

Each report describes the bytes consumed by one call to bj_utf8_next. For a valid character, value contains its Unicode code point. For an error, value contains the bits decoded before the error was detected.

A failed report covers the part of the sequence that could still have been valid. The byte that caused the error is left for the next decoding attempt. As a result, every input byte belongs to exactly one report.

Examples
utf8_decode.c.

Definition at line 64 of file unicode.h.

Data Fields
size_t byte_offset Offset of the first byte in the view.
const char * bytes The bytes covered by the report, borrowed from the view.
size_t n_bytes Number of bytes covered by the report.
enum bj_utf8_state state Result of the decoding attempt.
uint32_t value Decoded code point, or the partial value for an error.

Typedef Documentation

◆ bj_utf8_fn

typedef bj_bool(* bj_utf8_fn) (const struct bj_utf8_report *report, void *user_data)

Callback used by bj_utf8_scan.

Parameters
reportDetails about the current decoding result. The report is only valid for the duration of the callback.
user_dataThe pointer supplied to bj_utf8_scan.
Returns
BJ_TRUE to continue scanning, or BJ_FALSE to stop.

Definition at line 82 of file unicode.h.

Enumeration Type Documentation

◆ bj_utf8_state

Result of decoding the bytes in a UTF-8 report.

Enumerator
BJ_UTF8_STATE_VALID 

A valid UTF-8 character was decoded.

BJ_UTF8_STATE_BAD_LEAD 

The first byte cannot start a UTF-8 character (C0, C1, F5..FF).

BJ_UTF8_STATE_LONE_CONTINUATION 

A continuation byte appeared where a character should start.

BJ_UTF8_STATE_BAD_SLOT 

A byte is not allowed at this position in the sequence.

BJ_UTF8_STATE_TRUNCATED 

The view ended before the sequence was complete.

Definition at line 43 of file unicode.h.

Function Documentation

◆ bj_utf8_encode()

size_t bj_utf8_encode ( uint32_t value,
char bytes[4] )

Encode a Unicode code point as UTF-8.

The function writes the shortest valid UTF-8 sequence for value. Surrogate values (U+D800 to U+DFFF) and values above U+10FFFF cannot be encoded.

Parameters
[in]valueCode point to encode.
[out]bytesBuffer that can hold four bytes.
Returns
Number of bytes written (one to four), or zero if value cannot be encoded or bytes is NULL. The buffer is unchanged when zero is returned.
Examples
utf8_decode.c.

Referenced by main().

◆ bj_utf8_next()

bj_bool bj_utf8_next ( struct bj_str view,
size_t * cursor,
struct bj_utf8_report * report )

Read one UTF-8 character or decoding error from a view.

The function reads from the byte offset in cursor and advances the cursor past the bytes described by the report. The cursor may point to any byte, including the middle of a multi-byte character.

Parameters
[in]viewBytes to read.
[in,out]cursorByte offset to read from. It is advanced when a report is produced. If it is at or past the end of the view, it is left unchanged.
[out]reportReceives the decoding result. May be NULL if you only need to advance the cursor.
Returns
BJ_TRUE when a report was produced, or BJ_FALSE when the cursor is at or past the end of the view.
Examples
utf8_decode.c.

Referenced by main().

◆ bj_utf8_prev()

bj_bool bj_utf8_prev ( struct bj_str view,
size_t * cursor,
uint32_t * value )

Read the UTF-8 character immediately before a byte offset.

The function moves cursor back to the first byte of the character ending at that offset. The input must contain valid UTF-8.

Parameters
[in]viewUTF-8 bytes to read.
[in,out]cursorOffset immediately after the character. On success, it is moved to the character's first byte.
[out]valueReceives the decoded Unicode code point.
Returns
BJ_TRUE when a character ends at the given offset. Returns BJ_FALSE without changing cursor or value when the offset is zero, past the end of the view, or not at the end of a character.

◆ bj_utf8_scan()

size_t bj_utf8_scan ( struct bj_str view,
bj_utf8_fn callback,
void * user_data )

Process every UTF-8 character or decoding error in a view.

The function calls callback once for each report produced by bj_utf8_next. Return BJ_FALSE from the callback to stop the scan.

Parameters
[in]viewBytes to read.
[in]callbackFunction called for each report, or NULL to count the reports without processing them.
[in]user_dataPointer passed to each callback invocation.
Returns
Number of reports produced, including the report that caused the callback to stop scanning.
Examples
utf8_decode.c.

Referenced by main().

◆ bj_utf8_validate()

bj_bool bj_utf8_validate ( struct bj_str view,
size_t * first_error )

Check whether a view contains only valid UTF-8.

The function reads the entire view and returns the byte offset of the first decoding error when one is found.

Parameters
[in]viewBytes to validate.
[out]first_errorReceives the offset of the first invalid byte, or may be NULL if the offset is not needed.
Returns
BJ_TRUE if the view contains only valid UTF-8, or BJ_FALSE if an invalid sequence is found.
Examples
utf8_decode.c.

Referenced by main().