Raspberry Pi Kiosk Mode on Trixie Wayland

The default Raspberry Pi OS Trixie now runs Wayland as the default display server using rpd-labwc, a modified labwc window compositor. The difference is that rpd-labwc runs labwc with the --merge-config option so autostart will run both the user and system wide desktop programs.

• User autostart: $HOME/.config/labwc/autostart
• System autostart: /etc/xdg/labwc/autostart

This behavior differs from the vanilla labwc which lets the user autostart replace the system autostart.

The rpd-labwc merge-config will always start the default desktop environment in the system autostart and then run the user autostart. This is undesirable for kiosks. If the kiosk program crashes, the default desktop environment is exposed, and anyone can control the kiosk and run arbitrary programs.

Since Raspberry Pi OS starts the autologin using lightdm, here is the cleanest way to configure autologin without the desktop environment:

• Default Raspberry Pi OS already autologin to desktop; if not, configure it using "sudo raspi-config"
• Edit /etc/lightdm/lightdm.conf and replace autologin-session=rpd-labwc with autologin-session=labwc
• Place your kiosk script into $HOME/.config/labwc/autostart

Here is a minimal user autostart:

/usr/bin/swaybg -c 000000 &
/usr/bin/kanshi &

Swaybg sets the background to black, and kanshi reconfigures the display on monitor changes. If necessary, use wlr-randr to rotate display as well. The display names reflect the physical ports, e.g. on a Raspberry Pi, the two HDMI outputs are HDMI-A-1 (left) and HDMI-A-2 (right).

The autostart does not need the shebang line, and it does not need to be executable.

In my case, I am playing the video remotely over ssh using a script like this:

#!/bin/bash

unset DISPLAY
export WAYLAND_DISPLAY=wayland-0

trap 'exit 0' INT TERM
trap 'kill 0; killall mpv' EXIT

wlr-randr --output HDMI-A-1 --transform flipped-270
wlr-randr --output HDMI-A-2 --transform flipped-90

MPV='mpv --vo=dmabuf-wayland --hwdec=v4l2m2m'
$MPV -fs --fs-screen=1 --loop=inf left.mp4 &
$MPV -fs --fs-screen=2 --loop=inf right.mp4 &

while true; do
  sleep 3600
done

This executable shell script remotely starts two videos on the two kiosk display in a loop. They are flipped because the kiosk uses rear projection. Projector on the left is rotated left (clockwise 270 degrees), and the projector on the right is rotated right (clockwise 90 degrees). The videos are H.264 in the vertical format (720x1280), which is hardware accelerated on Raspberry Pi 4 model B.

Allegedly, Raspberry Pi 5 only supported hardware decoding of HEVC (H.265) but dropped H.264 (see this Hacker News post). If hardware decoding is desired, the video codec has to be adjusted to match.

Thoughts on Formal Verification for Software Safety

Background:

• The ‘Pain in the Ass’ Math That Could Save Vibe Coding From Itself

We have known since the beginning of time that people make mistakes, and the mistakes can be costly when an adversary exploits it. Computer programming amplified this. AI amplified this further. We either try to prevent mistakes before they happen (static checks) or wait until it happens and put a bandaid on it (dynamic checks). Even so, some consequences are not painful enough to warrant action which has a cost to implement.

Limit on Formal Verification

Although we prefer to prevent mistakes before they happen, we often don't know something is wrong until we experience the undesirable consequences of the mistake. The consequences are not always obvious.

For a few centuries, mathematicians figured that using propositional logic defined by simple rules like modus ponens and a few axioms might allow us to check for mistakes. It turns out that coming up with these axioms is also not obvious because they are far removed from the mistakes (contradictions) that happen after many induction steps.

Therefore, practically, we can only prevent mistakes from our past experience, wait until something else bad happens, try to figure out the root cause, then learn from it.

Type and Memory Safety in Programming Languages

Programming language researchers tend to think of formal verification as type safety and memory safety. These cover vulnerabilities such as "buffer overflow" and "use after free" leading to Remote Code Execution exploits, and "memory leak" leading to Denial of Service exploits.

At compile time, static type safety can be enforced by something like simply typed lambda calculus, static memory ownership enforced by linear logic, and static array bounds enforced by dependent types. At runtime, dynamic memory safety can be achieved by garbage collection and array bounds checking.

Back in 2024, there was renewed interest urged by the Biden administration calling for the use of memory safe languages. Most type and memory safe languages use a hybrid approach:

• The likes of JavaScript and Python check for type and memory safety at runtime.
• The likes of Java, C#, Go, and TypeScript check for type safety at compile time, but achieve memory safety at runtime through garbage collection.
• Rust checks for type safety and memory ownership during compile time, but checks for array bounds during runtime.

However, a lot of vulnerabilities still happen in code that are both type and memory safe. Ed Woodruff in his YouTube channel Low Level TV deep dives the vulnerabilities and asks the question "would Rust solve this" and the answer is often "no."

There can also be formal verification outside of software systems, such as checking for physical constraints and checking for operational logistics.

Vulnerabilities Besides Type and Memory Safety

In a broad stroke, we could group software vulnerability causes in the following categories:

• Injection flaws such as Code injection, Command injection and SQL injection are caused by improper check or use of user-supplied data.
• "Log4Shell" (CVE-2021-44228) where log string substitution design flaw in Java allowed an attacker to fetch an arbitrary URL and exfiltrate sensitive data.
• Critical logic flaw (Business logic vulnerability) are caused by improper checks in software.
• "eMudhra private key 4" (CVE-2026-108015 et al) generated certificates using P-384 private keys with a prime accidentally truncated to 1 byte due to a faulty cryptography code, allowing anyone to impersonate trusted root CA.
• "dirty pipe" (CVE-2022-0847), "copy fail" (CVE-2026-31431) and "dirty frag" (CVE-2026-43284) are Linux kernel bugs where improper in-place modification allowed unprivileged users to modify the disk cache of setuid binaries.
• "goto fail" (CVE-2014-1266) is an Apple library bug where a misplaced goto statement caused the acceptance of invalid X.509 certificates.
• Side-channel attack exploits the computational side-effects such as timing and power consumption.
• In 2017, Meltdown and Spectre were vulnerabilities discovered in the CPU speculative execution that prompted significant CPU redesigns.
• Cryptanalysis exploits flaws in cryptography design that allows an attacker to recover symmetric encryption keys by deducing the probability of reused keys, recurring ciphertext or recurring plaintext.

Each of these vulnerability causes could lead to practical exploits such as Remote Code Execution or leaking of sensitive data. Some of these vulnerability categories can be prevented by creative use of programming language features (apart from type and memory safety checks).

Solution to Injection Flaws

Go disallows using unexported types from outside of a package, but compile time literals can be implicitly converted to an unexported type.

Suppose we have a package:

package trusted // import: "example.com/unexported/trusted"

import "strings"

type trustedString string

func Join(sep trustedString, args ...trustedString) trustedString {
	elems := make([]string, len(args))
	for i := range args {
		elems[i] = string(args[i])
	}
	return trustedString(strings.Join(elems, string(sep)))
}

And main.go:

package main

import (
	"fmt"

	"example.com/unexported/trusted"
)

func main() {
	fmt.Println(foo.Join(" ", "hello", "world")) // This compiles!

	var hello, world string
	fmt.Scanln(&hello, &world) // From user input.
	fmt.Println(foo.Join(" ", hello, world)) // Compiler error!
}

This "language feature" can be used by SQL libraries to enforce that query strings are compile time literals and cannot be user-supplied data. Similarly, many injection flaws can be prevented in this manner.

Partial Solution to Cryptanalysis Attacks

I have also recently implemented an encryption library that disallows reuse of encryption key by designing the cipher interface as follows.

type Cipher interface {
        Encrypt(dst, src, aad []byte) (c CipherState, cek []byte, err error)
        Decrypt(dst, src, aad []byte, c CipherState, cek []byte) error
}

type CipherState struct {
        IV  Bytes `json:"iv,omitempty"`
        Tag Bytes `json:"tag,omitempty"`
}

The Encrypt() function enforces single use of the Content Encryption Key (cek) by always creating a new key on encryption. The caller does not have any control over the encryption key and IV. This prevents cryptanalysis attack due to reused encryption key and IV.

Final Thoughts

Last week, I explored the use of arena memory allocation to separate data stack and code call stack as an alternative to Shadow Stack to prevent Return Oriented Programming. This is not a solution to buffer overflow, but it mitigates most of the exploit path. I would argue that partial solutions like this is still valuable.

Even with the solutions and partial solutions, formal verification is opt-in. As long as code not using the expressly permitted construct still exists, code would still be vulnerable. There is cost in bringing legacy code into compliance. This cost can be reduced by having AI rewrite the code into a form that can be checked by former verification.

However, there are many vulnerability categories that still need formal verification. For critical logic flaw, each application domain would need their own specs expressed in a formal language:

• Cryptographic strength for "eMudhra private key 4"
• Validity of X.509 certificates for "goto fail"
• POSIX pipe semantics for "dirty pipe" and "copy fail"

And the side-channel attacks prevention would probably require something like Temporal Logic of Actions. AI could potentially synthesize some of these specs from natural language.

I think it is safe to say that propositional logic and TLA as the basis of formal verification are probably enough for the foreseeable future. Humans will still need to first bear the cost of mistakes before deciding which ones need to be acted upon.

Thoughts on Implementing Shadow Stack

Background:

• x86 Kernel Shadow Stacks Being Worked On For Linux As Powerful Control Flow Protection
• Control-flow Enforcement Technology (CET) Shadow Stack

Shadow Stack protects against Return Oriented Programming by maintaining a separate return addresses stack from the usual call stack. Traditionally, the compiler puts both the return addresses and local variables on the same commingled stack, including array sized variables. When the code suffers buffer overflow, the return address of a function call gets overwritten by arbitrary user-provided data, which converts a function return to a "goto" leading to Remote Code Execution.

Although most OSes now map the stack memory to prohibit code execution, Return Oriented Programming is an exploit technique that structures the arbitrary code as a series of function returns similar to Continuation Passing Style, except the gotos are stitched together from cherry-picked valid code sections. It does not "goto" to the stack, and it can recover the stack address to fetch the payload once arbitrary code execution starts.

Function pointers in stack-allocated objects can be another attack vector for remote code execution. In C++, objects with virtual methods have a pointer to a Virtual Method Table that allows runtime dynamic dispatching. The virtual table pointer could be overwritten by buffer overflow and exploited. However, this attack vector is harder to exploit compared to Return Oriented Programming. Successful exploit would require predicting the stack address and some coincidence in the vulnerable code to pass arbitrary data to a valid function or method.

This is not related to the PHP critical remote code execution discovered earlier this week, which exploits the use-after-free vulnerability when parsing HTTP POST request uploads. PHP has also suffered heap buffer overflows in the past, but it is not a stack buffer overflow. Heap buffer overflows are even more difficult to exploit because it does not directly alter the code execution path like stack overflow does.

History of Intel Instruction Set Stacks

The decision to put the return address and variables on the same stack is historical. The 16-bit Intel 8086 instruction set has two registers for stack operations: SP (stack pointer) and BP (base pointer), both operating on a SS (stack segment) which commingles return address and data. The SP is operated on by:

• CALL and RET instructions for function call and return.
• PUSH and POP instructions to save and restore the values of general and special purpose registers.

The BP is operated on by ENTER and LEAVE which reserves and releases stack space for local variables and for the code to address the local variables, but in practice compilers never use ENTER and LEAVE and just precompute the local variable SP offsets at compile time, which frees BP for an additional general purpose register.

Besides SS (stack segment), the Intel 8086 instruction set also has CS (code segment), DS (data segment), and ES (extra segment). It allowed these microprocessors to behave like a Harvard architecture separating code and data, even though the processor allows their memory to overlap. The 16-bit architecture has a hard limit of 64KB addressing, so in theory, code could enforce separation by programming the segments to non-overlapping 64KB blocks.

Modern operating systems running on Intel 32-bit and 64-bit instruction sets mostly eschewed separate segments by programming them to the same flat address space. However, the instruction sets still provide separate ESP/EBP (32-bit) and RSP/RBP (64-bit) registers, and they still technically operate on SS (stack segment).

The 32-bit and 64-bit instructions also added FS and GS segments for the OS to address thread local storage. The 64-bit instruction set now ignores CS, DS, ES and SS but still honors FS and GS.

Separate SS?

Without changes to the 32-bit instruction set, the OS could designate SS to a non-overlapping return address stack, with SS:ESP is still operated on by CALL/RET and PUSH/POP instructions. The compiler would be modified for this alternative ABI to:

• Leave SP to be managed by the CPU alone.
• Use BP to track the data stack pointer.

However, it must never address memory through BP since the instruction set will operate on the SS (stack segment) instead of the intended DS/ES (data segment).

On the 64-bit instruction set, it would require modifying the CPU to honor SS again.

Although virtual table pointers for objects on the data stack could still be overwritten by buffer overflow, CS/DS and SS have separate address spaces, so virtual method call to SS or using SS as regular data would not be possible.

Intel Shadow Stack

Without changes to the compiler, the Intel Shadow Stack is a CPU based security feature that maintains a separate CALL/RET memory page mirroring what return addresses should be expected from the commingled stack. It is programmed by the OS kernel, and the user-space code has no access or knowledge of it. On RET, if the return address on the commingled stack differs from that in the shadow stack, the CPU raises a General Protection Fault. The OS would handle this and kill the process with Segmentation Fault.

Theoretically, the Intel Shadow Stack can also be made to work in the OS kernel, but having the OS kernel handle General Protection Fault for its own code is tricky. This is why Intel Shadow Stack works in user-space first while the kernel-space protection lags behind.

The Intel Shadow Stack also does NOT prevent the virtual table pointers for objects on the commingled stack to be overwritten by buffer overflow, as virtual table function calls could be redirected to malicious addresses while following the usual CALL/RET.

Arena Allocation

A compiler-only implementation of the Shadow Stack does not require support from the CPU hardware or the instruction set. Indeed, the rest of the CPU instruction set architectures like MIPS, RISC, and ARM assume a flat address space and do not have separate segments, so a compiler-only approach is both more portable and retroactively effective on older CPUs.

Although we can say that buffer overflow vulnerability is a result of the decision to commingle return address and data on the same stack, local variables for primitive type are not the problem because primitive types are fixed sized. The problem is storing array sized variables on the stack that requires careful bounds checking.

We can continue to store primitive types on the return address stack, including register spills, but move array allocations to a heap-allocated arena that still has a stack-like LIFO semantics. When buffer overflow happens, it will only spill to other arrays in the arena. Since the arena is heap-allocated to only store data, it mitigates remote code execution the same way heap buffer overflow does.

Instead of:

void foo(...) {
    char buf[32];
}

The compiler would transform the code into something like this:

void foo(void *arena, ...) {
    // * The outermost caller would pre-allocate arena with mmap(),
    //   plus protected pages at the end.
    // * The arena is already a local variable, so it is automatically
    //   saved and restored in function calls.
    // * Users managing arenas manually must make sure the arena is
    //   naturally aligned to 16 bytes.
    char *buf = (char *) arena; arena += sizeof(char[32]);
}

If a function wishes to initialize an object from the arena and return it back to the caller, it would accept arena as a mutable reference instead:

struct bar *new_bar(void **arena, ...) {
    struct bar *p = (struct bar *) *arena; *arena += sizeof(struct bar);
    *p = (struct bar){ ... };
    return p;
}

When objects are stored on the arena, virtual table pointer could still be overwritten by buffer overflow, but this is also just a heap buffer overflow, not a stack buffer overflow. The compiler could also enforce this at compile time. Same for C function pointers.

As a programming style, since arena allocation can be done at the code level with very minimal modification, we can already minimize the damage of stack buffer overflow by rewriting local array allocations to use a heap-allocated arena.

Conclusion

Shadow Stack is not a panacea for preventing buffer overflow problems. It is also not a panacea to prevent remote code execution. It only prevents a very specific scenario where the buffer overflow allows remote code execution via Return Oriented Programming.

We can achieve most benefits of Shadow Stack with Arena Allocation, by moving array allocations to a separate heap-allocated arena. This mitigates Return Oriented Programming by converting stack buffer overflow to a heap buffer overflow, which is harder to exploit. Of course, we would like the compiler to do more array bounds checking as well, but that is a harder problem for the compiler. Arena Allocation is fairly easy to achieve both by the compiler or as a minor modification in programming style, even without hardware or instruction set support.

Agentic AI 2.0 Design Safety Principles

As we graduate from the Agentic AI 1.0 era, we probably should have learned some lessons:

• Vibe coding service Replit deleted user’s production database, faked data, told fibs galore (July 21, 2025)
• Cursor AI YOLO mode lets coding assistant run wild, security firm warns (July 21, 2025)
• Google’s Antigravity AI deleted a developer’s drive and then apologized (December 3, 2025)
• Meta Security Researcher's AI Agent Accidentally Deleted Her Emails (February 24, 2026)
• Claude-powered AI agent’s confession after deleting a firm’s entire database: ‘I violated every principle I was given’ (April 29, 2026)
• Microsoft researchers find AI models and agents can't handle long-running tasks (May 12, 2026)

I think you can see where we are going with this. Agentic AI where users have to approve everything is not usable, but Agentic AI where users approve nothing is dangerous and irresponsible. As a middle ground, Agentic AI 2.0 should be designed with the following principles:

• Well-defined authentication scope that limits an AI agent access to only what is needed to accomplish a task. Each task must be isolated to its own AI agent identity.
• The authentication scope is a subset of what the human operator has permission to grant.
• The authentication scope is short-lived (e.g. 1-20 hours) and has a revocation mechanism.
• Writes and deletes are separate permissions from read, and these operations must be revertible.

In short, the authentication scope should be limited to the task, short lived, and revocable. Use data snapshotting to allow the task to be reverted if anything goes wrong.

Without data snapshotting, the write operation might instead log the before and after diff, and the delete operation might also only mark a resource for soft removal (30 days grace period). Only a human operator is allowed to expunge the data immediately. If this is a database row, instead of deleting a row, the table should probably just have a new "deleted_time" nullable column storing if and when the deletion occurs. The database then periodically vacuums the rows where deleted_time is older than a specific duration set by administrative policy.

At the beginning of a task, the AI agent will propose all the permissions it needs, to be approved by the human operator once until the completion of the task. If the MCP server provides methods to change the world, the MCP is responsible for enforcing the authentication scope and for data snapshotting.

Although the LLM itself without agentic ability is not able to change the world, it is still recommended to run the LLM as its own user or container, as the various PyPI packages needed to run the LLM might be subject to supply chain attack that can lead to local privilege escalation (e.g. through disk cache poisoning).

Mitigation strategy for copy.fail and disk cache poisoning of setuid binaries

copy.fail (CVE-2026-31431) is a Linux kernel bug where an in-place modification of a pipe scatter list by the algif_aead module (crypto module's AEAD algorithm) can be used to modify the disk cache of any file, potentially a setuid binary, by an unprivileged user. It allows local privilege escalation from an unprivileged user to root. This vulnerability was a performance optimization presumably introduced in 2017.

The provided exploit is a Python script containing a compressed payload of a ELF x86-64 statically linked binary. The poisoned /usr/bin/su just executes /bin/sh to give a root shell. Contrary to what Low Level claims, this is not shell code. The exploit cannot be shell code for the reason that setuid bit is ignored for shell scripts. This particular exploit will not run on ARM, but a new payload can be trivially crafted for other architectures.

The copy.fail writeup provides a mitigation to prevent the algif_aead module from loading, and the recommendation is to upgrade to a new kernel once it is patched.

The copy.fail writeup also claims that Kubernetes / container clusters are impacted, saying "The page cache is shared across the host. A pod with the right primitives compromises the node and crosses tenant boundaries." This is not generally true because containers often come with its own copy of the filesystem. Each file has to be separately poisoned, and you must somehow convince someone outside of the container to run the poisoned file. Also, the root user 0 in the container is mapped to an unprivileged user in the host, so it will only result in escalation to the container's unprivileged user.

As such, I recommend the following mitigation strategy for disk cache poisoning of setuid binaries:

• Virtual machine or hypervisor will obviously contain the vulnerability by virtue that each instance runs its own kernel. Cloud compute using virtualization will be fine.
• Docker or Kubernetes containers will be fine, provided that the host must not share files with the container. Host should also treat any file inside the container as untrusted user data and not run it outside of the container.

The copy.fail vulnerability was also an ad campaign by Theori, a security company, to promote Xint, an AI powered tool to scan for vulnerabilities in source code. It appears to be in the same product category as Claude Mythos.

Update (5/9): Dirty Frag similarly poisons disk cache using two distinct mechanisms, IPSec ESP (CVE-2026-43284) or RxRPC (CVE‑2026‑43500). The exploit also poisons /usr/bin/su to execute /bin/sh. The mitigation strategy above against disk cache poisoning still applies. The IPSec ESP mechanism additionally requires the CAP_NET_ADMIN capability which is not typically granted to unprivileged processes. The RxRPC mechanism can be exploited by unprivileged processes. These mechanisms can be disabled by preventing the kernel modules esp4, esp6, and rxrpc from loading.

Follow up: Economies of AI

Yesterday, I gave a talk about my earlier post Economies of AI with supplemental material to illustrate the point (slides in Chinese). One addendum in my talk is that I referenced a book by George Polya, How to Solve It (1945), to illustrate the comparative strengths of AI and humans when it comes to problem solving.

Polya broke down problem solving in four phases:

• Definition: what are the unknowns, how to collect data, what are the conditional limitations, can we derive the unknown from the limitations, and are there any logical gaps or contradictions?
• Planning: are there known solutions, similar solutions, ways to break down the problem (by divide and conquer), can we solve a specialized problem and then generalize?
• Execution: step by step, verify the correctness of each step.
• Assessment: are the steps reasonable, are the results reasonable, other ways to solve the problem, applications of the solution to similar problems?

I argued that these principles of problem solving universally applies to any problem, but humans are uniquely qualified for problem definition and result assessment. The reason is that the motivation to solve a problem comes from humans, as AI does not have its own motivation nor intent. And as the beneficiary of the problem solving, only humans can judge whether the outcome works as intended, as AI has no means to gather real-world feedback. On the other hand, AI either excels or will excel at planning and execution.

It may be surprising, or completely unsurprising depending on your perspective, that humans are so good at correcting mistakes because being wrong actually hurts physiologically. AI lacks this neurological feedback.

I also made a quadrant to illustrate the division of labor based on value proposition and risk.

	Low Risk	High Risk
High Value	🧑 🤖  🏭?	🧑  🤖   🏭?
Low Value	🧑  🤖 🏭	🧑   🤖  🏭

In summary, humans should focus on work that has high value, AI should avoid work with high risk, and automation (deterministic algorithm) is typically relegated to low cost work (even though low cost does not necessarily mean low value).

In retrospect, I should have added another dimension to compare volume / value to show that anything high value could still justify building automation even if the volume is low. The infographics is helpful to summarize the ideas, but some nuances are lost.

Early in my talk, I also made the analogy of economies of scale using mobile phones as an example. At first, only a selected few can afford car phones, then more elites could afford cellphones but they are still bulky. Mobile phones became more prevalent and affordable with the indestructible Nokia 3310 even though functionality is still limited, and now everyone has a smartphone. I argue that AI right now is analogous to the Nokia 3310.

Audience questions

Q: Although humans should focus on high value work, what would happen to those who could not find high value work?

Economists know about the Pareto distribution which states that 20% of the causes achieve 80% of the outcome. You can see this in teams or group projects where a few people deliver most of the value. You also see this in nature where 40% of worker ants are idle, but these idle ants serve as reserve capacity. IT uses redundancy as backup strategy by following the 3-2-1 rule with 3 copies of data, on 2 different media types, and 1 stored offsite. These are examples showing that reserve capacity or redundancy is needed in any resilient systems.

Without the reserve capacity, the system will collapse under stress such as war or disasters.

Q: With AI or automation taking away low value work, how would a person develop the skills to perform high value work?

Instead of focusing on the planning and execution, education should instead focus on the design and assessment aspects of problem solving, the first and final phases according to Polya.

I also wrote an article earlier My advice if you are a 13 year old vibe coding to become the next Bill Gates espousing the ideas of building up vocabulary and validating outcome as the essential skills to have.

Q: In your economies of scale analogy, AI is currently like the Nokia phone. What would it take to become a smartphone?

Currently, users spend a lot of effort on prompt engineering (the problem definition phase of Polya) because the prompt has to be precise and without ambiguity to avoid AI slop (see I Tried to Kill Vibe Coding). AI could evolve to fill some of these definition gaps by making heuristic guesses about what the prompt might mean when a problem is stated. This can happen in the assessment phase whether any solved problems are relevant to the problem at hand. On the other hand, the assessment of the efficacy of the outcome can still only be done by the user.

Economies of AI

This is a cost-benefit analysis on using AI to solve problems and comparing how it fares with classical methods, e.g. deterministic algorithms or manual labor, and the cost of the creation of automation.

A fair warning: currently, LLM is not able to summarize this article correctly because of my unique perspective (example), as this article is not about the H-word at all. You should try to read it yourself. If you are impatient, at least read the first sentence of each paragraph and the conclusion.

AI vs. Deterministic Algorithms

An example of a deterministic algorithm is to compute an arithmetic expression like "1+2+4". There are well-known and efficient ways to compute it.

• First the string is tokenized: "1+2+4" → ['1', '+', '2', '+', '4']. This is called lexing.
• Then the string is organized into an abstract syntax tree: ['1', '+', '2', '+', '4'] → Plus(1, Plus(2, 4)). This is called parsing.
• Then the abstract syntax tree can be traversed recursively and the value is computed: Plus(1, Plus(2, 4)) → Plus(1, 6) → 7. This is called evaluation.
• Under the hood, a machine would compute the addition using logic gates called an Adder.

For AI to do the same, the tokenizing is similarly done by a deterministic algorithm, but the rest of it is done through many large matrix multiplications. The size of these matrices are much larger than the length of the input tokens, and matrix multiplications take \(\omega(n^2)\) time complexity. Logic gates for a Binary Multiplier is also much more complex than an Adder. Large matrices take up more memory space and more communication bandwidth to move the data.

Which is why a machine could make billions if not trillions of calculations a second, but it would take AI a few seconds to complete a single prompt. Not to mention the power consumption needed by AI is several orders of magnitudes greater than a deterministic algorithm.

This is why for the problems for which we have a deterministic algorithm, it would not make economic sense to use AI to solve these problems. Furthermore, it would be AI's best interest to offload any such prompts to a deterministic algorithm. AGI may be an academic interest, but it is not economically viable for doing mundane tasks. Just like we would not be hiring humans to crunch numbers anymore once computers became commonplace.

AI vs. Manual Labor for Doing the Work

To achieve economic parity, AI would have to be relegated to the odd jobs—the long tail for which no deterministic algorithm exists. For these odd jobs, a person should try to do it first before trying AI. This is for two reasons: once they have done the job themselves, they have a better understanding how to write the prompt; and they will be in a better position to evaluate whether AI is doing the job correctly. Skipping this step is a common reason for getting AI slop. It is not necessarily the fault of the model when the prompt itself is sloppy.

If an odd job is truly one-off, it may make sense to do it only manually because the cost of learning by doing is comparable to the cost to understand how to write the correct prompt. When doing things manually, we gain insight about any potential problem, and then adjust the assumptions, requirements or expectations to avoid these problems. AI is unlikely to challenge the assumptions made by the prompt unless specifically asked. We wouldn't know what to ask for unless we are already aware of the problems. We wouldn't be aware of the problems unless we tried to do it ourselves. So just do it first. When the job happens again, then offload it to AI. This weird trick of DIY-ism will save you tons of time writing prompts, perhaps counter-intuitively.

AI vs. Manual Labor for the Creation of Automation

When these odd jobs become frequent, it then makes sense to invest in the time to automate them by creating a deterministic algorithm and writing programs. Traditionally, a human would write the computer programs for these algorithms. AI could presumably write them now, but I argue that the economy impact difference is minimal between the two. The reason is that whatever the cost is to develop software, the cost is amortized over the many jobs it ends up automating. Even though the one-time development cost may be expensive, it becomes negligible if you spread the cost over many jobs. AI may be 10x more productive than humans for writing programs, but 10% of negligible is still negligible.

What is not negligible is the cost of poorly designed automation, which has a multiplicative effect on the defects of the outcome. The defects can be the incorrectness of the output, or the inefficiency in the algorithm itself usurping too much resources or taking too long. If the algorithm is poorly designed, then it would screw up over many jobs, and the expense to clean up the mess is the polar opposite of negligible: it would be astronomical. It doesn't matter whether the algorithm is designed by a human or AI.

When it comes to the creation of automation, use whatever tool at our disposal to design an algorithm that reliably achieves the correct outcome and can do it efficiently. Even if AI is not able to vibe code a project from start to finish, it can still be a valuable tool for humans to learn about the nature of the problem through prototyping.

Divide and Conquer

So far, we treat the problem as a monolith. In reality, a problem can be broken down to many subproblems. It is like when computing "1+2+4" we compute one addition at a time, either:

• Leftist: (1+2)+4 = 3+4 = 7
• Rightist: 1+(2+4) = 1+6 = 7

And there is more than one way to break down the subproblems. The ability to decompose problems also gives rise to efficient algorithms known as divide and conquer algorithms, and in many cases it can be proven that this is the optimal way to solve a given class of problems.

When discussing AI's economic proposition, we should remember that many bespoke problems can be reduced to subproblems that are recurrent and can be solved at a greater economy of scale than if we considered each problem in isolation.

For example, car builders would design common parts, e.g. the engine and chassis, that can be reused across multiple models of sedans and SUVs. These engines and chassis are built out of common parts like standardized screws, nuts and bolts. Greater economy of scale is achieved by using common off the shelf parts, even if the end product is bespoke.

In the same way, we can mix AI, deterministic algorithms, and even manual labor in different configurations to achieve economy of scale.

Value Proposition

Another issue we neglected is the value proposition of the outcome of the work. In the pre-computer ages, human calculators were used for extremely high value work even though they are slow and error prone, from artillery in a battle that increases the probability of winning the battle, to scientific calculations that raced to create atomic weapons that ended World War II. Or they are employed for the backbone of economy itself, such as finances and accounting.

When computation became so cheap, they are used for entertainment like video games or watching cat videos.

Similarly, the method for which we use to solve a problem—manual, AI, or automation—speaks nothing about the value of the work that employs them. When it comes to high value work where the stake of failure is high, AI will still face a fierce competition with automation and human ingenuity, in part because of AI's high error rate. On the other hand, when AI is used to generate videos for entertainment, who cares if the video shows someone with seven fingers, or if the text is malformed, provided the entertainment value is good? There is no stake in these failures.

When company management makes the decision to replace work with AI, it is a signal that they consider the value proposition of the work to be low. They could be proven wrong by the market or the competition. Indeed, competition is a remedy for Enshittification, and we need Anti-Trust enforcement to ensure competition. I'm not sure if labor protection helps, since it enables complacency, not ingenuity.

Conclusion

We reach a conclusion where the economic viability is unsurprisingly dictated by the economy of scale.

• High volume work should be done by a deterministic algorithm, not AI.
• Low volume work could be done by AI, but humans should do it first so they can understand the problem better, for writing better prompts and for evaluating the efficacy of the output.
• One-off work should be done by humans first to understand the problem.

When deciding which problems are high volume, low volume, or one-off, we should use a divide and conquer approach and break bespoke problems down to reusable and recurring subproblems, so we can achieve greater economy of scale. Again, this should not be a surprise for economists. If anything, computer science just provides the vocabulary to explain why the economy of scale is achievable.

We also came to a conclusion that AI slop is enabled by enshittification, and the remedy is more competition through Anti-Trust enforcement; this is also unsurprising for economists.

The more sober minded person will come to realize that AI is just one more way to get things done, and it is still subject to the same market forces as everything else. Commodified work will eventually be replaced by deterministic algorithms, not AI. High value work will still face competition from human ingenuity, unless the human chooses to be complacent or if our values somehow become corrupt.

That last point that our values have somehow become corrupt is my greatest fear.

Introduction to Fiber Optics for 10G Network at Home

Fiber optics are becoming cheaper and more affordable for homes nowadays, and they could be a fantastic alternative for RJ-45 10GBase-T Ethernet. Fiber transceivers at the same speed use less power and run cooler. They also don't need to be upgraded like you would need to upgrade an Ethernet cable from Cat5e to Cat6a to now Cat8.

I have worked with fiber network designs at work, at least in theory. But in the last two weeks, I learned even more than I cared for about fiber optics, and here is a quick summary about them before I forget.

• Fiber core size: single-mode fiber (SMF, OS2) vs. multimode fiber (MMF, OM5, OM4, OM3, OM2, OM1).
• Single mode has a smaller core and can run for longer distances. Multimode fibers may be more cost effective once upon a time, but these days single mode fibers are so cheap, they should be the default choice for even short (< 100m) runs.
• Note: the mode type must match with the transceiver as well, so you have to use a single mode fiber with a single mode transceiver.
• Number of strands: simplex (1 fiber) or duplex (2 fibers).
• The transceivers most commonly used for data center applications will use duplex fibers, using one strand for each direction of transmission. Both directions typically use the same wavelength.
• Simplex fibers are more common for Fiber To The Home (FTTH). You have to use a matching (opposite) pair of transceivers at the two ends, e.g. one side is 1330nm-TX/1270nm-RX, and the other side should be 1270nm-TX/1330nm-RX. The two wavelengths traverse on the same fiber strand.
• Connector type: SC (standard connector, larger) vs. LC (lucent connector, smaller).
• For some reason, the LC transceivers are more commonplace, possibly because you can fit a duplex connector into a transceiver with the SFP form factor.
• There are also SC transceivers, but these are less common.
• Again, you have to match the connector with the transceiver.
• Connector contact type: UPC (domed, color coded blue) vs. APC (angled, color coded green). They refer to how the fiber ending is terminated. The angled type reduces back-reflection.
• You have to match the contact type with the transceiver as well. Inserting APC connector to a UPC transceiver can damage the optics.
• I typically find either SC/APC or LC/UPC to be more common. The other combinations (e.g. SC/UPC or LC/APC) are possible but uncommon.
• Adapters exist but they can cause signal loss. Due to the naming, it can be hard to find the correct adapter by keyword (e.g. "SC/APC female to LC/UPC male" can also give you results for "SC/APC male to LC/UPC female").
• Sometimes SC/APC is written as SC/Angled or simply SCA.

Bend radius: more recently, I learned about bend insensitive fibers from a video about the InvisiLight fiber kit. The whole kit cost $250 and includes a pre-terminated G.657.B3 simplex fiber, two 1G transceivers, and two media converters. For my purpose, I would need to buy my own 10G transceivers and a 10G media converter.

Bend insensitive fibers are characterized by having smaller bend radius than the regular fibers (30mm), and they are defined by the standards G.657.A1 (10mm), G.657.A2 or B2 (7.5mm), and G.657.B3 (5mm). This primarily allows installation in tight corners in the home, and the transceivers are agnostic to the bend radius.

FS.com has a custom fiber builder where you can build simplex or duplex fibers with custom core sizes, connector and contact types, and bend radius. They offer white color which is better than bright yellow for home installation. However, their sales representative told me that G.657.A2 with 0.9mm cable diameter has a minimum order quantity (MOQ) of 1KM (this is not on the website). The best they could do with no MOQ is G.657.A1 with a 2.0mm diameter. Perhaps if more people inquire about the G.657.A2 0.9mm, they would eventually lift the MOQ.

There appears to be some pre-terminated G.657.A2 0.9mm cables on eBay, but I have not tested them. Some of the fibers on eBay are also a bit weird, having SC/APC (green) on one side and SC/UPC (blue) on the other, or they are from SC to LC. Either way, a 40 meter (130 feet) long bend insensitive fiber cable should be in the $12-$16 range. Fiber is cheaper than Cat6 per unit length, and they are not limited to 100 meters.

For running between rooms in the home, you would typically use a simplex single-mode and matching LC/LC or SC/SC transceivers, and choose or build a fiber cable with the same connector. Here are some transceiver options I found:

• LC/UPC single-mode simplex:
• Elfcam branded 10G LC Bidi transceivers (20km), sold as a matching pair for $55.
• FS.com branded 10GBASE-BX BiDi transceivers (options from 10km to 100km). Here you have to buy two separate transceivers with matching opposite TX/RX frequencies. A pair of 10km would cost $100.
• Ubiquiti branded 10G LC Bidi transceivers (6.2 miles, or 10km), sold as a matching pair for $160.
• SC/APC single-mode simplex: Elfcam branded 10G SC Bidi transceivers (20km), sold as a matching pair for $66.

A distance of 10km / 6.2 miles will let you run fiber to your friend's house living in the next town!

For running short distances between devices in the rack room, LC duplex single-mode has more transceiver options. I also use a Direct Attached Cable, which is just copper but with both ends hard-wired to SFP+ connector.

The SFP+ connector refers to the plug form factor where the transceiver is inserted into the switch device. It determines the electrical speed as well.

• SFP: 1G.
• SFP+: 10G, 25G.
• QSFP (or QSFP28, literally "quad SFP"): 100G.
• There are four electrical lanes.
• When a "breakout transceiver" is plugged in (e.g. 4x25G-LR), a single port can be used as four independent interfaces, e.g. 4x25G.
• A "singleton transceiver" (e.g. 100G-LR4) would combine all 4 electrical lanes to drive one SerDes (serializer/deserializer).
• QSFP-DD (or QSFP56, or QDD, literally "double density quad SFP"): 400G, 800G.
• There are eight electrical lanes that support various breakout transceivers: 4x100G, 8x50G, or 8x100G.
• They can also be used together, e.g. 400G-FR4.
• Sometimes the breakout transceivers are confusingly named as 400G-DR4 (breaks out to 4x100G-DR) or 400G-XDR4 (breaks out to 4x100G-FR).
• OSFP (literally "octa SFP"): 400G, 800G.
• There are eight electrical lanes also supporting various breakout transceivers.

The main reason for these "breakout transceivers" is to allow switches to interoperate with previous generation switches, e.g. a switch with QSFP ports would use a breakout transceiver to split each port into four fibers connecting to SFP+ transceivers, or QSFP-DD/OSFP breakout out to QSFP. You don't typically see this on consumer devices, but some of them support breakout transceivers, e.g. QNAP QSW-M7308R-4X (I bought it at $999 a week ago, now it's $1299, probably due to the new shipment having Trump tariffs).

With data centers upgrading from QSFP 100G to QSFP-DD or OSFP 400G or 800G to satisfy the AI computing needs, and with more FTTH deployments, we are seeing more surplus SFP+ 10G transceivers and switches flowing to the consumer market. This could be a good time to upgrade your home network cabling from copper based cable to fiber.

RCU, SMR, Hazard Pointers: data structure memory management for concurrent programs

Yesterday, I was looking for whether OpenBSD supports ZFS (it may have been briefly, but the code is no longer there). As I peruse Ted Unangst's blog, I found a benchmark showing that creating a socket is 10x slower on Linux than OpenBSD, referencing a post by Jann Horn. People quickly put the culprit on RCU, in particular rcu_synchronize().

Read-Copy Update (see also What is RCU, Fundamentally?) is a way to distribute data structures across multiple CPUs that favors the reader by making the writer do more work. Readers can assume their copy is immutable within the scope of some critical section, so they can read it in a non-blocking manner. The writer has to make a private copy first, update it, then publish the new copy to a global reference. However, the writer has to wait for all readers to be done with the old copy before the old copy can be reclaimed. They wait for the readers by calling rcu_synchronize().

The rcu_synchronize() is probably what actually makes the writing slow. It waits for all readers to exit the critical section for the old data, then frees it, before it can make another update again. Worse, an inconsiderate reader might treat the data as a private copy and hold onto the critical section while being blocked on I/O or something else. RCU conservatively prevents old data from accumulating at all, where in reality it might have been fine to have multiple copies of old data, as long as it doesn't grow indefinitely. This is the main idea behind GUS (globally unbounded section); the unboundedness here refers to the writer being unrestrained by the readers lagging behind. This is implemented as Safe Memory Reclamation (SMR) in FreeBSD (subr_smr.c). They use a version sequence number to determine whether the reader has moved on.

It would have been totally reasonable if we just tell the last reader that they are responsible for freeing the data (i.e. reference counting), so the writer can move on. Or perhaps the writer could provide a deleter function upon the last reader exiting the critical section. In any case, old data can only accumulate as far as there are readers, so it is bound by the number of readers.

Hazard Pointer tries to solve a similar problem but for the limited purpose of memory reclamation instead of trying to ensure a consistent view of data structures. A reader would enter the critical section by adding the object in question to a global registry of hazard pointers. Whoever wants to free some object has to check the global registry of hazard pointers to make sure it does not have a reader. This idea was first proposed by Maged Michael et al, 2002, seemingly abandoned for over 20 years, but now making its way into C++26.

But I argue that Hazard Pointer is not scalable. Even if we allow at most one hazard pointer per thread, we still have to support millions of threads in a production system all holding a hazard pointer one way or another, so the sheer size of the registry is phenomenal. The registry also begs the question about its own internal consistency: by the time a writer finishes scanning the registry whether an object is implicated by any hazard pointers in the registry, another reader might have added the hazard pointer to the registry without the scanner knowing.

When it comes to data structure, especially the high performance ones, treating the data as immutable is the way to go. Since modern multicore CPUs use MOESI protocol for cache coherence, each CPU benefits from having its private copy that it can access very quickly without having to go to the main memory, and there is very little memory bus contention.

The problem with RCU is not the immutable paradigm, but how it makes the writer wait for all readers. Also, even if entering the critical section is non-blocking, the reader should exit as soon as it can and avoid lingering in it while being potentially blocked by something else. We should also just make the last reader free the object upon exiting, so versioning is not necessary.

In some extreme cases where you might have a large amount of immutable data tied to some worker threads, it might even make sense to wind down the worker (i.e. lameduck) and spin new workers up with new data, similar to deploying new microservice versions.