tecs.data.utf8#

UTF-8 codepoint decoding, encoding, validation, and byte-safe truncation.

Offsets are one-based byte cursor positions, matching Lua string indices. decodeAt reads the codepoint beginning at an offset and returns the cursor after it. decodeBefore reads the codepoint before a cursor and returns its starting offset:

local utf8 <const> = tecs.data.utf8
local text <const> = "A€"

local codepoint, nextOffset = utf8.decodeAt(text, 2)
assert(codepoint == 0x20ac)
assert(nextOffset == #text + 1)

codepoint, nextOffset = utf8.decodeBefore(text, nextOffset)
assert(codepoint == 0x20ac)
assert(nextOffset == 2)

The end cursor is the byte length plus one. decodeAt returns nil there, and decodeBefore returns nil at cursor 1. An offset inside a multibyte sequence is not moved to a boundary automatically. SDL reports the malformed fragment at that exact position as U+FFFD.

Every inspection function accepts a Lua string or an open ByteView. A view is read through its retained pointer without first becoming a Lua string. validPrefixLength answers the byte count a caller can use to retain a zero-copy view at a UTF-8 boundary.

Malformed UTF-8 is recoverable rather than exceptional. Forward decoding returns U+FFFD and consumes one malformed byte, so callers can keep scanning. A valid encoded U+FFFD remains indistinguishable by value but consumes its three bytes. Embedded NUL is the valid U+0000 codepoint and does not terminate a Lua string.

These functions operate on Unicode codepoints, not user-perceived characters. A combining mark, variation selector, or member of an emoji sequence is a separate result. Text editing that needs cursor movement by grapheme cluster requires a Unicode segmentation implementation above this module.

Module contents#

Functions#

Functions#

tecs.data.utf8.decodeAt Static#

Decodes the codepoint beginning at a byte offset.

The function returns U+FFFD and advances one byte when the offset begins malformed or truncated UTF-8. It does not search for a nearby codepoint boundary when the caller supplies an offset inside a multibyte sequence.

function tecs.data.utf8.decodeAt(
    bytes: ByteInput, byteOffset: integer
): integer | nil, integer

Arguments#

Returns#

Examples#

Reads forward from a one-based byte offset.

local utf8 <const> = require("tecs.data.utf8")
local codepoint, nextOffset = utf8.decodeAt("A€", 2)
assert(codepoint == 0x20ac)
assert(nextOffset == 5)

tecs.data.utf8.decodeBefore Static#

Decodes the codepoint before a byte cursor.

The decoder searches backward for a possible sequence leader. When that candidate does not form one scalar ending at the cursor, it reports the final byte as U+FFFD instead. Forward and reverse scans therefore recover from malformed input one byte at a time.

function tecs.data.utf8.decodeBefore(
    bytes: ByteInput, byteOffset: integer
): integer | nil, integer

Arguments#

Returns#

Examples#

Reads backward from the end cursor.

local utf8 <const> = require("tecs.data.utf8")
local codepoint, startOffset = utf8.decodeBefore("A€", 5)
assert(codepoint == 0x20ac)
assert(startOffset == 2)

tecs.data.utf8.encode Static#

Encodes one Unicode scalar value as UTF-8.

function tecs.data.utf8.encode(codepoint: integer): string

Arguments#

Returns#

Examples#

Encodes a Unicode scalar value.

local utf8 <const> = require("tecs.data.utf8")
assert(utf8.encode(0x20ac) == "€")

tecs.data.utf8.isValid Static#

Reports whether every byte forms canonical UTF-8.

RFC 3629 rules reject overlong encodings, UTF-16 surrogate values, codepoints above U+10FFFF, stray continuation bytes, and truncated sequences. Embedded NUL and an encoded U+FFFD are valid.

function tecs.data.utf8.isValid(bytes: ByteInput): boolean

Arguments#

Returns#

Examples#

Rejects malformed UTF-8 bytes.

local utf8 <const> = require("tecs.data.utf8")
assert(utf8.isValid("café"))
assert(not utf8.isValid("\xff"))

tecs.data.utf8.length Static#

Counts Unicode codepoints in bytes.

This is not a byte count or a grapheme count. Each malformed byte contributes one U+FFFD result, while an embedded NUL contributes the valid U+0000 codepoint and does not end the input.

function tecs.data.utf8.length(bytes: ByteInput): integer

Arguments#

Returns#

Examples#

Counts Unicode codepoints rather than bytes.

local utf8 <const> = require("tecs.data.utf8")
assert(utf8.length("A€") == 2)

tecs.data.utf8.truncate Static#

Truncates a string without splitting a valid UTF-8 sequence.

The result is a byte prefix, not a normalization. Malformed bytes are one-byte decoding units and remain unchanged when they fit. The function does not combine marks or emoji into grapheme clusters.

function tecs.data.utf8.truncate(
    text: string, maxBytes: integer
): string

Arguments#

Returns#

Examples#

Leaves a multibyte codepoint intact at the byte limit.

local utf8 <const> = require("tecs.data.utf8")
assert(utf8.truncate("A€B", 4) == "A€")

tecs.data.utf8.validPrefixLength Static#

Returns the longest prefix ending at a complete UTF-8 decoding unit.

The function returns a byte count and never copies the input. A caller can pass the result to ByteView:view to retain the prefix. Malformed bytes remain one-byte units and therefore fit whenever their byte does.

function tecs.data.utf8.validPrefixLength(
    bytes: ByteInput, maxBytes: integer
): integer

Arguments#

Returns#

Examples#

Retains a zero-copy prefix at a complete codepoint boundary.

local tecs <const> = require("tecs")
local bytes <const> = tecs.io.newBuffer("A€B")
local all <const> = bytes:view()
local count <const> = tecs.data.utf8.validPrefixLength(all, 3)
local prefix <const> = all:view(0, count)
assert(prefix:getString() == "A")
prefix:close()
all:close()
bytes:close()