# tecs.data.utf8


UTF-8 codepoint decoding, encoding, validation, and byte-safe truncation.

Offsets are one-based byte cursor positions, matching Lua string indices.
`decodeAt` reads the codepoint beginning at an offset and returns the cursor
after it. `decodeBefore` reads the codepoint before a cursor and returns its
starting offset:

```teal
local utf8 <const> = tecs.data.utf8
local text <const> = "A€"

local codepoint, nextOffset = utf8.decodeAt(text, 2)
assert(codepoint == 0x20ac)
assert(nextOffset == #text + 1)

codepoint, nextOffset = utf8.decodeBefore(text, nextOffset)
assert(codepoint == 0x20ac)
assert(nextOffset == 2)
```

The end cursor is the byte length plus one. `decodeAt` returns nil there, and
`decodeBefore` returns nil at cursor 1. An offset inside a multibyte sequence
is not moved to a boundary automatically. SDL reports the malformed fragment
at that exact position as U+FFFD.

Every inspection function accepts a Lua string or an open
[`ByteView`](/modules/io/#tecs.io.ByteView). A view is read through its retained
pointer without first becoming a Lua string. `validPrefixLength` answers the
byte count a caller can use to retain a zero-copy view at a UTF-8 boundary.

Malformed UTF-8 is recoverable rather than exceptional. Forward decoding
returns U+FFFD and consumes one malformed byte, so callers can keep scanning.
A valid encoded U+FFFD remains indistinguishable by value but consumes its
three bytes. Embedded NUL is the valid U+0000 codepoint and does not terminate
a Lua string.

These functions operate on Unicode codepoints, not user-perceived characters.
A combining mark, variation selector, or member of an emoji sequence is a
separate result. Text editing that needs cursor movement by grapheme cluster
requires a Unicode segmentation implementation above this module.


## Module contents

### Functions

| Function | Kind | Description |
| --- | --- | --- |
| [`decodeAt`](/modules/data/utf8/#tecs.data.utf8.decodeAt) | <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span> | Decodes the codepoint beginning at a byte offset. |
| [`decodeBefore`](/modules/data/utf8/#tecs.data.utf8.decodeBefore) | <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span> | Decodes the codepoint before a byte cursor. |
| [`encode`](/modules/data/utf8/#tecs.data.utf8.encode) | <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span> | Encodes one Unicode scalar value as UTF-8. |
| [`isValid`](/modules/data/utf8/#tecs.data.utf8.isValid) | <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span> | Reports whether every byte forms canonical UTF-8. |
| [`length`](/modules/data/utf8/#tecs.data.utf8.length) | <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span> | Counts Unicode codepoints in bytes. |
| [`truncate`](/modules/data/utf8/#tecs.data.utf8.truncate) | <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span> | Truncates a string without splitting a valid UTF-8 sequence. |
| [`validPrefixLength`](/modules/data/utf8/#tecs.data.utf8.validPrefixLength) | <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span> | Returns the longest prefix ending at a complete UTF-8 decoding unit. |

## Functions

<a id="tecs.data.utf8.decodeAt"></a>
### tecs.data.utf8.decodeAt <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span>

Decodes the codepoint beginning at a byte offset.

The function returns U+FFFD and advances one byte when the offset
begins malformed or truncated UTF-8. It does not search for a nearby
codepoint boundary when the caller supplies an offset inside a
multibyte sequence.



```teal
function tecs.data.utf8.decodeAt(
    bytes: ByteInput, byteOffset: integer
): integer | nil, integer
```

#### Arguments

| Name | Type | Description |
| --- | --- | --- |
| `bytes` | `ByteInput` | The caller supplies a complete byte string or open [`ByteView`](/modules/io/#tecs.io.ByteView). Embedded NUL is U+0000. |
| `byteOffset` | `integer` | A one-based byte cursor from 1 through the byte length plus one. The final value is the end cursor. Other values raise. |

#### Returns

| Type | Description |
| --- | --- |
| <code>integer &#124; nil</code> | The Unicode codepoint, or nil at the end cursor. |
| `integer` | The one-based cursor immediately after the result, or the unchanged end cursor when no codepoint remains. |

#### Examples


Reads forward from a one-based byte offset.


```teal
local utf8 <const> = require("tecs.data.utf8")
local codepoint, nextOffset = utf8.decodeAt("A€", 2)
assert(codepoint == 0x20ac)
assert(nextOffset == 5)
```

<a id="tecs.data.utf8.decodeBefore"></a>
### tecs.data.utf8.decodeBefore <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span>

Decodes the codepoint before a byte cursor.

The decoder searches backward for a possible sequence leader. When
that candidate does not form one scalar ending at the cursor, it
reports the final byte as U+FFFD instead. Forward and reverse scans
therefore recover from malformed input one byte at a time.



```teal
function tecs.data.utf8.decodeBefore(
    bytes: ByteInput, byteOffset: integer
): integer | nil, integer
```

#### Arguments

| Name | Type | Description |
| --- | --- | --- |
| `bytes` | `ByteInput` | The caller supplies a complete byte string or open [`ByteView`](/modules/io/#tecs.io.ByteView). Embedded NUL is U+0000. |
| `byteOffset` | `integer` | A one-based byte cursor from 1 through the byte length plus one. It points after the codepoint to read. Other values raise. |

#### Returns

| Type | Description |
| --- | --- |
| <code>integer &#124; nil</code> | The preceding Unicode codepoint, or nil at cursor 1. |
| `integer` | The one-based starting offset of the result, or 1 when no codepoint precedes the cursor. |

#### Examples


Reads backward from the end cursor.


```teal
local utf8 <const> = require("tecs.data.utf8")
local codepoint, startOffset = utf8.decodeBefore("A€", 5)
assert(codepoint == 0x20ac)
assert(startOffset == 2)
```

<a id="tecs.data.utf8.encode"></a>
### tecs.data.utf8.encode <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span>

Encodes one Unicode scalar value as UTF-8.



```teal
function tecs.data.utf8.encode(codepoint: integer): string
```

#### Arguments

| Name | Type | Description |
| --- | --- | --- |
| `codepoint` | `integer` | An integer from U+0000 through U+10FFFF, excluding the UTF-16 surrogate range U+D800 through U+DFFF. Other values raise. |

#### Returns

| Type | Description |
| --- | --- |
| `string` | One through four bytes. U+0000 returns a one-byte Lua string containing NUL. |

#### Examples


Encodes a Unicode scalar value.


```teal
local utf8 <const> = require("tecs.data.utf8")
assert(utf8.encode(0x20ac) == "€")
```

<a id="tecs.data.utf8.isValid"></a>
### tecs.data.utf8.isValid <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span>

Reports whether every byte forms canonical UTF-8.

RFC 3629 rules reject overlong encodings, UTF-16 surrogate values,
codepoints above U+10FFFF, stray continuation bytes, and truncated
sequences. Embedded NUL and an encoded U+FFFD are valid.



```teal
function tecs.data.utf8.isValid(bytes: ByteInput): boolean
```

#### Arguments

| Name | Type | Description |
| --- | --- | --- |
| `bytes` | `ByteInput` | The caller supplies a complete byte string or open [`ByteView`](/modules/io/#tecs.io.ByteView), including embedded NUL. |

#### Returns

| Type | Description |
| --- | --- |
| `boolean` | True only when forward decoding never needs a replacement for a malformed byte. |

#### Examples


Rejects malformed UTF-8 bytes.


```teal
local utf8 <const> = require("tecs.data.utf8")
assert(utf8.isValid("café"))
assert(not utf8.isValid("\xff"))
```

<a id="tecs.data.utf8.length"></a>
### tecs.data.utf8.length <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span>

Counts Unicode codepoints in bytes.

This is not a byte count or a grapheme count. Each malformed byte
contributes one U+FFFD result, while an embedded NUL contributes the
valid U+0000 codepoint and does not end the input.



```teal
function tecs.data.utf8.length(bytes: ByteInput): integer
```

#### Arguments

| Name | Type | Description |
| --- | --- | --- |
| `bytes` | `ByteInput` | The caller supplies a complete byte string or open [`ByteView`](/modules/io/#tecs.io.ByteView), including embedded NUL. |

#### Returns

| Type | Description |
| --- | --- |
| `integer` | The number of forward decoding steps. |

#### Examples


Counts Unicode codepoints rather than bytes.


```teal
local utf8 <const> = require("tecs.data.utf8")
assert(utf8.length("A€") == 2)
```

<a id="tecs.data.utf8.truncate"></a>
### tecs.data.utf8.truncate <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span>

Truncates a string without splitting a valid UTF-8 sequence.

The result is a byte prefix, not a normalization. Malformed bytes are
one-byte decoding units and remain unchanged when they fit. The
function does not combine marks or emoji into grapheme clusters.



```teal
function tecs.data.utf8.truncate(
    text: string, maxBytes: integer
): string
```

#### Arguments

| Name | Type | Description |
| --- | --- | --- |
| `text` | `string` | The complete byte string. |
| `maxBytes` | `integer` | A non-negative integer byte ceiling. Other values raise. |

#### Returns

| Type | Description |
| --- | --- |
| `string` | The longest prefix no larger than `maxBytes` whose final decoding unit is complete. Returns `text` itself when it fits. |

#### Examples


Leaves a multibyte codepoint intact at the byte limit.


```teal
local utf8 <const> = require("tecs.data.utf8")
assert(utf8.truncate("A€B", 4) == "A€")
```

<a id="tecs.data.utf8.validPrefixLength"></a>
### tecs.data.utf8.validPrefixLength <span class="tealdoc-kind-badge tealdoc-kind-static">Static</span>

Returns the longest prefix ending at a complete UTF-8 decoding unit.

The function returns a byte count and never copies the input. A caller
can pass the result to `ByteView:view` to retain the prefix. Malformed
bytes remain one-byte units and therefore fit whenever their byte does.



```teal
function tecs.data.utf8.validPrefixLength(
    bytes: ByteInput, maxBytes: integer
): integer
```

#### Arguments

| Name | Type | Description |
| --- | --- | --- |
| `bytes` | `ByteInput` | The caller supplies a complete byte string or open [`ByteView`](/modules/io/#tecs.io.ByteView). |
| `maxBytes` | `integer` | A non-negative integer byte ceiling. Other values raise. |

#### Returns

| Type | Description |
| --- | --- |
| `integer` | Returns the largest safe prefix length no greater than `maxBytes`. |


#### Examples


Retains a zero-copy prefix at a complete codepoint boundary.


```teal
local tecs <const> = require("tecs")
local bytes <const> = tecs.io.newBuffer("A€B")
local all <const> = bytes:view()
local count <const> = tecs.data.utf8.validPrefixLength(all, 3)
local prefix <const> = all:view(0, count)
assert(prefix:getString() == "A")
prefix:close()
all:close()
bytes:close()
```