Banjo API 1.1.0-alpha
Low-level C99 game development API
Loading...
Searching...
No Matches
utf8_decode.c

The UTF-8 codec: one call per character, reports included.

The UTF-8 codec: one call per character, reports included.The decoder is one call, bj_utf8_next: a decode remembers nothing between characters, so every other way in, the scan and the validator included, is a loop over it. This example reads a view both ways, then meets the failures the codec reports instead of hiding: a byte that cannot start a character, a sequence cut off by the end of the view, and the maximal-subpart policy that keeps a valid character after garbage.

#include <banjo/log.h>
#include <banjo/main.h>
#include <banjo/string.h>
#include <banjo/unicode.h>
// A callback for bj_utf8_scan: prints one line per character attempt,
// then keeps scanning. The report says what the bytes are; this decides
// what they become on screen.
static bj_bool print_report(const struct bj_utf8_report* report, void* user_data) {
(void)user_data;
const int valid = (report->state == BJ_UTF8_STATE_VALID);
bj_info(" %3zu..%3zu %s U+%04X", report->byte_offset,
report->byte_offset + report->n_bytes,
valid ? "valid " : "replaced ",
valid ? report->value : 0xFFFDu);
return BJ_TRUE;
}
int main(int argc, char* argv[]) {
(void)argc; (void)argv;
// The pull door: one character per call, the cursor walking the view.
const bj_str text = bj_str_from_cstr("Hello, UTF-8!");
struct bj_utf8_report report;
size_t cursor = 0;
size_t count = 0;
while (bj_utf8_next(text, &cursor, &report)) {
++count;
}
bj_info("The pull door read %zu characters", count);
// The push door: the same reports, handed to a callback over the
// whole view. Protocols and diagnostics read this way.
bj_info("The push door, same view:");
const size_t report_count = bj_utf8_scan(text, print_report, NULL);
bj_info(" %zu reports", report_count);
// A view with a defect: E9 decodes, then E0 80 cannot sit where it
// is. Each failed report covers the plausible prefix without the
// byte that broke it, and the A after the garbage comes out whole.
static const char bytes[] = "\xC3\xA9\xE0\x80\x41";
const bj_str damaged = { bytes, 5 };
bj_info("A damaged view, E9 then E0 80 41:");
bj_utf8_scan(damaged, print_report, NULL);
// Validation answers before anything is built: the door protocols
// and editors use. first_error names the byte where the failure
// starts.
size_t first_error = 0;
if (!bj_utf8_validate(damaged, &first_error)) {
bj_info("The view is not valid UTF-8: first error at byte %zu",
first_error);
}
// Writing a character back: the shortest form, and nothing at all
// for a value with no spelling, as a surrogate.
char encoded[4];
const size_t written_e9 = bj_utf8_encode(0xE9, encoded);
const size_t written_surrogate = bj_utf8_encode(0xD800, encoded);
bj_info("U+00E9 writes %zu bytes, a surrogate writes %zu",
written_e9, written_surrogate);
return 0;
}
uint32_t bj_bool
Boolean type used throughout the Banjo API.
Definition api.h:264
#define BJ_TRUE
Boolean true value (1).
Definition api.h:282
#define bj_info(...)
Log a message using the BJ_LOG_INFO level.
Definition log.h:141
struct bj_str bj_str_from_cstr(const char *c_string)
Create a view from a NUL-terminated C string.
A borrowed view of a sequence of bytes.
Definition string.h:57
uint32_t value
Decoded code point, or the partial value for an error.
Definition unicode.h:68
enum bj_utf8_state state
Result of the decoding attempt.
Definition unicode.h:69
const char * bytes
The bytes covered by the report, borrowed from the view.
Definition unicode.h:67
size_t n_bytes
Number of bytes covered by the report.
Definition unicode.h:66
size_t byte_offset
Offset of the first byte in the view.
Definition unicode.h:65
size_t bj_utf8_scan(struct bj_str view, bj_utf8_fn callback, void *user_data)
Process every UTF-8 character or decoding error in a view.
bj_bool bj_utf8_validate(struct bj_str view, size_t *first_error)
Check whether a view contains only valid UTF-8.
size_t bj_utf8_encode(uint32_t value, char bytes[4])
Encode a Unicode code point as UTF-8.
bj_bool bj_utf8_next(struct bj_str view, size_t *cursor, struct bj_utf8_report *report)
Read one UTF-8 character or decoding error from a view.
@ BJ_UTF8_STATE_VALID
A valid UTF-8 character was decoded.
Definition unicode.h:44
Details about one UTF-8 decoding attempt.
Definition unicode.h:64
Logging utility functions.
Portable main() replacement with platform-aware entry shim.
UTF-8 string views and owning strings.
int main(void)
Read, write and validate UTF-8 characters.
static bj_bool print_report(const struct bj_utf8_report *report, void *user_data)
Definition utf8_decode.c:23