Wednesday, November 18, 2026

java4shell Hallucination

It's obvious that the author(s) of calif.io/java4shell (CVE-2026-114938) has credited using AI to generate this report. AI still can already sound very confident, but it buried a few details that make the vulnerability not so exploitable as it claimed.

First, the vulnerability exists in the X86-64 AVX-512 implementation of the Base64 decoder. AVX-512 is supported by some higher end Intel Xeon and Core and AMD Zen CPUs. As it turns out, I don't actually have such a CPU. Look for the string avx512 in /proc/cpuinfo on Linux. The article incorrectly included the "bmi2" string in the detection signature.

The article claims that the trigger input is 'A' * 64 + '=' but the input has an invalid padding (the alternative OpenJDK 17 trigger 'A' * 128 + '=' also has an invalid padding). Java can decode a string with no padding, but it will reject any string with an invalid padding. The padding is first checked by the Java function decode calling decodedOutLength, before it calls decode0 which calls decodeBlock.

The vulnerability does not exist in the Java implementation of decodeBlock, but the AVX-512 intrinsic implementation of it. The JIT compiler would detect that the bytecode is calling an @IntrinsicCandidate and dispatches code generation to a stub generator. The StubGenerator::generate_base64_decodeBlock is the generator that emits inlined assembly, which makes human analysis very hard.

Before the exploit can reach the vulnerable AVX-512 decodeBlock, it must first pass the padding check in Java, which means your code is not vulnerable if it just uses the public method in java.util.Base64. The vulnerability in the AVX-512 decodeBlock was fixed in commit 5177a70, which adds the short tail handling in assembly, something that decode0 already handled in Java. The decodeBlock is a private method so no outside code can call this directly.

This is what makes AI generated bug report weird. The code when looked at in isolation is arguably buggy, but the precondition for triggering the bug has been eliminated elsewhere.

But what I really hate about the AI generated report is that they try to write a Lord of the Rings trilogy out of a few lines of vulnerability. Who has the damn time to read it? Bring back the tl;dr please.

Friday, November 6, 2026

Raspberry Pi Kiosk Mode on Trixie Wayland

The default Raspberry Pi OS Trixie now runs Wayland as the default display server using rpd-labwc, a modified labwc window compositor. The difference is that rpd-labwc runs labwc with the --merge-config option so autostart will run both the user and system wide desktop programs.

  • User autostart: $HOME/.config/labwc/autostart
  • System autostart: /etc/xdg/labwc/autostart

This behavior differs from the vanilla labwc which lets the user autostart replace the system autostart.

The rpd-labwc merge-config will always start the default desktop environment in the system autostart and then run the user autostart. This is undesirable for kiosks. If the kiosk program crashes, the default desktop environment is exposed, and anyone can control the kiosk and run arbitrary programs.

Since Raspberry Pi OS starts the autologin using lightdm, here is the cleanest way to configure autologin without the desktop environment:

  • Default Raspberry Pi OS already autologin to desktop; if not, configure it using "sudo raspi-config"
  • Edit /etc/lightdm/lightdm.conf and replace autologin-session=rpd-labwc with autologin-session=labwc
  • Place your kiosk script into $HOME/.config/labwc/autostart

Here is a minimal user autostart:

/usr/bin/swaybg -c 000000 &
/usr/bin/kanshi &

Swaybg sets the background to black, and kanshi reconfigures the display on monitor changes. If necessary, use wlr-randr to rotate display as well. The display names reflect the physical ports, e.g. on a Raspberry Pi, the two HDMI outputs are HDMI-A-1 (left) and HDMI-A-2 (right).

The autostart does not need the shebang line, and it does not need to be executable.

In my case, I am playing the video remotely over ssh using a script like this:

#!/bin/bash

unset DISPLAY
export WAYLAND_DISPLAY=wayland-0

trap 'exit 0' INT TERM
trap 'kill 0; killall mpv' EXIT

wlr-randr --output HDMI-A-1 --transform flipped-270
wlr-randr --output HDMI-A-2 --transform flipped-90

MPV='mpv --vo=dmabuf-wayland --hwdec=v4l2m2m'
$MPV -fs --fs-screen=1 --loop=inf left.mp4 &
$MPV -fs --fs-screen=2 --loop=inf right.mp4 &

while true; do
  sleep 3600
done

This executable shell script remotely starts two videos on the two kiosk display in a loop. They are flipped because the kiosk uses rear projection. Projector on the left is rotated left (clockwise 270 degrees), and the projector on the right is rotated right (clockwise 90 degrees). The videos are H.264 in the vertical format (720x1280), which is hardware accelerated on Raspberry Pi 4 model B.

Allegedly, Raspberry Pi 5 only supported hardware decoding of HEVC (H.265) but dropped H.264 (see this Hacker News post). If hardware decoding is desired, the video codec has to be adjusted to match.

Friday, October 30, 2026

Thoughts on Formal Verification for Software Safety

Background:

We have known since the beginning of time that people make mistakes, and the mistakes can be costly when an adversary exploits it. Computer programming amplified this. AI amplified this further. We either try to prevent mistakes before they happen (static checks) or wait until it happens and put a bandaid on it (dynamic checks). Even so, some consequences are not painful enough to warrant action which has a cost to implement.

Limit on Formal Verification

Although we prefer to prevent mistakes before they happen, we often don't know something is wrong until we experience the undesirable consequences of the mistake. The consequences are not always obvious.

For a few centuries, mathematicians figured that using propositional logic defined by simple rules like modus ponens and a few axioms might allow us to check for mistakes. It turns out that coming up with these axioms is also not obvious because they are far removed from the mistakes (contradictions) that happen after many induction steps.

Therefore, practically, we can only prevent mistakes from our past experience, wait until something else bad happens, try to figure out the root cause, then learn from it.

Type and Memory Safety in Programming Languages

Programming language researchers tend to think of formal verification as type safety and memory safety. These cover vulnerabilities such as "buffer overflow" and "use after free" leading to Remote Code Execution exploits, and "memory leak" leading to Denial of Service exploits.

At compile time, static type safety can be enforced by something like simply typed lambda calculus, static memory ownership enforced by linear logic, and static array bounds enforced by dependent types. At runtime, dynamic memory safety can be achieved by garbage collection and array bounds checking.

Back in 2024, there was renewed interest urged by the Biden administration calling for the use of memory safe languages. Most type and memory safe languages use a hybrid approach:

  • The likes of JavaScript and Python check for type and memory safety at runtime.
  • The likes of Java, C#, Go, and TypeScript check for type safety at compile time, but achieve memory safety at runtime through garbage collection.
  • Rust checks for type safety and memory ownership during compile time, but checks for array bounds during runtime.

However, a lot of vulnerabilities still happen in code that are both type and memory safe. Ed Woodruff in his YouTube channel Low Level TV deep dives the vulnerabilities and asks the question "would Rust solve this" and the answer is often "no."

There can also be formal verification outside of software systems, such as checking for physical constraints and checking for operational logistics.

Vulnerabilities Besides Type and Memory Safety

In a broad stroke, we could group software vulnerability causes in the following categories:

  • Injection flaws such as Code injection, Command injection and SQL injection are caused by improper check or use of user-supplied data.
    • "Log4Shell" (CVE-2021-44228) where log string substitution design flaw in Java allowed an attacker to fetch an arbitrary URL and exfiltrate sensitive data.
  • Critical logic flaw (Business logic vulnerability) are caused by improper checks in software.
    • "eMudhra private key 4" (CVE-2026-108015 et al) generated certificates using P-384 private keys with a prime accidentally truncated to 1 byte due to a faulty cryptography code, allowing anyone to impersonate trusted root CA.
    • "dirty pipe" (CVE-2022-0847), "copy fail" (CVE-2026-31431) and "dirty frag" (CVE-2026-43284) are Linux kernel bugs where improper in-place modification allowed unprivileged users to modify the disk cache of setuid binaries.
    • "goto fail" (CVE-2014-1266) is an Apple library bug where a misplaced goto statement caused the acceptance of invalid X.509 certificates.
  • Side-channel attack exploits the computational side-effects such as timing and power consumption.
    • In 2017, Meltdown and Spectre were vulnerabilities discovered in the CPU speculative execution that prompted significant CPU redesigns.
  • Cryptanalysis exploits flaws in cryptography design that allows an attacker to recover symmetric encryption keys by deducing the probability of reused keys, recurring ciphertext or recurring plaintext.

Each of these vulnerability causes could lead to practical exploits such as Remote Code Execution or leaking of sensitive data. Some of these vulnerability categories can be prevented by creative use of programming language features (apart from type and memory safety checks).

Solution to Injection Flaws

Go disallows using unexported types from outside of a package, but compile time literals can be implicitly converted to an unexported type.

Suppose we have a package:

package trusted // import: "example.com/unexported/trusted"

import "strings"

type trustedString string

func Join(sep trustedString, args ...trustedString) trustedString {
	elems := make([]string, len(args))
	for i := range args {
		elems[i] = string(args[i])
	}
	return trustedString(strings.Join(elems, string(sep)))
}

And main.go:

package main

import (
	"fmt"

	"example.com/unexported/trusted"
)

func main() {
	fmt.Println(foo.Join(" ", "hello", "world")) // This compiles!

	var hello, world string
	fmt.Scanln(&hello, &world) // From user input.
	fmt.Println(foo.Join(" ", hello, world)) // Compiler error!
}

This "language feature" can be used by SQL libraries to enforce that query strings are compile time literals and cannot be user-supplied data. Similarly, many injection flaws can be prevented in this manner.

Partial Solution to Cryptanalysis Attacks

I have also recently implemented an encryption library that disallows reuse of encryption key by designing the cipher interface as follows.

type Cipher interface {
        Encrypt(dst, src, aad []byte) (c CipherState, cek []byte, err error)
        Decrypt(dst, src, aad []byte, c CipherState, cek []byte) error
}

type CipherState struct {
        IV  Bytes `json:"iv,omitempty"`
        Tag Bytes `json:"tag,omitempty"`
}

The Encrypt() function enforces single use of the Content Encryption Key (cek) by always creating a new key on encryption. The caller does not have any control over the encryption key and IV. This prevents cryptanalysis attack due to reused encryption key and IV.

Final Thoughts

Last week, I explored the use of arena memory allocation to separate data stack and code call stack as an alternative to Shadow Stack to prevent Return Oriented Programming. This is not a solution to buffer overflow, but it mitigates most of the exploit path. I would argue that partial solutions like this is still valuable.

Even with the solutions and partial solutions, formal verification is opt-in. As long as code not using the expressly permitted construct still exists, code would still be vulnerable. There is cost in bringing legacy code into compliance. This cost can be reduced by having AI rewrite the code into a form that can be checked by former verification.

However, there are many vulnerability categories that still need formal verification. For critical logic flaw, each application domain would need their own specs expressed in a formal language:

  • Cryptographic strength for "eMudhra private key 4"
  • Validity of X.509 certificates for "goto fail"
  • POSIX pipe semantics for "dirty pipe" and "copy fail"

And the side-channel attacks prevention would probably require something like Temporal Logic of Actions. AI could potentially synthesize some of these specs from natural language.

I think it is safe to say that propositional logic and TLA as the basis of formal verification are probably enough for the foreseeable future. Humans will still need to first bear the cost of mistakes before deciding which ones need to be acted upon.

Friday, October 23, 2026

Thoughts on Implementing Shadow Stack

Background:

Shadow Stack protects against Return Oriented Programming by maintaining a separate return addresses stack from the usual call stack. Traditionally, the compiler puts both the return addresses and local variables on the same commingled stack, including array sized variables. When the code suffers buffer overflow, the return address of a function call gets overwritten by arbitrary user-provided data, which converts a function return to a "goto" leading to Remote Code Execution.

Although most OSes now map the stack memory to prohibit code execution, Return Oriented Programming is an exploit technique that structures the arbitrary code as a series of function returns similar to Continuation Passing Style, except the gotos are stitched together from cherry-picked valid code sections. It does not "goto" to the stack, and it can recover the stack address to fetch the payload once arbitrary code execution starts.

Function pointers in stack-allocated objects can be another attack vector for remote code execution. In C++, objects with virtual methods have a pointer to a Virtual Method Table that allows runtime dynamic dispatching. The virtual table pointer could be overwritten by buffer overflow and exploited. However, this attack vector is harder to exploit compared to Return Oriented Programming. Successful exploit would require predicting the stack address and some coincidence in the vulnerable code to pass arbitrary data to a valid function or method.

This is not related to the PHP critical remote code execution discovered earlier this week, which exploits the use-after-free vulnerability when parsing HTTP POST request uploads. PHP has also suffered heap buffer overflows in the past, but it is not a stack buffer overflow. Heap buffer overflows are even more difficult to exploit because it does not directly alter the code execution path like stack overflow does.

History of Intel Instruction Set Stacks

The decision to put the return address and variables on the same stack is historical. The 16-bit Intel 8086 instruction set has two registers for stack operations: SP (stack pointer) and BP (base pointer), both operating on a SS (stack segment) which commingles return address and data. The SP is operated on by:

  • CALL and RET instructions for function call and return.
  • PUSH and POP instructions to save and restore the values of general and special purpose registers.

The BP is operated on by ENTER and LEAVE which reserves and releases stack space for local variables and for the code to address the local variables, but in practice compilers never use ENTER and LEAVE and just precompute the local variable SP offsets at compile time, which frees BP for an additional general purpose register.

Besides SS (stack segment), the Intel 8086 instruction set also has CS (code segment), DS (data segment), and ES (extra segment). It allowed these microprocessors to behave like a Harvard architecture separating code and data, even though the processor allows their memory to overlap. The 16-bit architecture has a hard limit of 64KB addressing, so in theory, code could enforce separation by programming the segments to non-overlapping 64KB blocks.

Modern operating systems running on Intel 32-bit and 64-bit instruction sets mostly eschewed separate segments by programming them to the same flat address space. However, the instruction sets still provide separate ESP/EBP (32-bit) and RSP/RBP (64-bit) registers, and they still technically operate on SS (stack segment).

The 32-bit and 64-bit instructions also added FS and GS segments for the OS to address thread local storage. The 64-bit instruction set now ignores CS, DS, ES and SS but still honors FS and GS.

Separate SS?

Without changes to the 32-bit instruction set, the OS could designate SS to a non-overlapping return address stack, with SS:ESP is still operated on by CALL/RET and PUSH/POP instructions. The compiler would be modified for this alternative ABI to:

  • Leave SP to be managed by the CPU alone.
  • Use BP to track the data stack pointer.

However, it must never address memory through BP since the instruction set will operate on the SS (stack segment) instead of the intended DS/ES (data segment).

On the 64-bit instruction set, it would require modifying the CPU to honor SS again.

Although virtual table pointers for objects on the data stack could still be overwritten by buffer overflow, CS/DS and SS have separate address spaces, so virtual method call to SS or using SS as regular data would not be possible.

Intel Shadow Stack

Without changes to the compiler, the Intel Shadow Stack is a CPU based security feature that maintains a separate CALL/RET memory page mirroring what return addresses should be expected from the commingled stack. It is programmed by the OS kernel, and the user-space code has no access or knowledge of it. On RET, if the return address on the commingled stack differs from that in the shadow stack, the CPU raises a General Protection Fault. The OS would handle this and kill the process with Segmentation Fault.

Theoretically, the Intel Shadow Stack can also be made to work in the OS kernel, but having the OS kernel handle General Protection Fault for its own code is tricky. This is why Intel Shadow Stack works in user-space first while the kernel-space protection lags behind.

The Intel Shadow Stack also does NOT prevent the virtual table pointers for objects on the commingled stack to be overwritten by buffer overflow, as virtual table function calls could be redirected to malicious addresses while following the usual CALL/RET.

Arena Allocation

A compiler-only implementation of the Shadow Stack does not require support from the CPU hardware or the instruction set. Indeed, the rest of the CPU instruction set architectures like MIPS, RISC, and ARM assume a flat address space and do not have separate segments, so a compiler-only approach is both more portable and retroactively effective on older CPUs.

Although we can say that buffer overflow vulnerability is a result of the decision to commingle return address and data on the same stack, local variables for primitive type are not the problem because primitive types are fixed sized. The problem is storing array sized variables on the stack that requires careful bounds checking.

We can continue to store primitive types on the return address stack, including register spills, but move array allocations to a heap-allocated arena that still has a stack-like LIFO semantics. When buffer overflow happens, it will only spill to other arrays in the arena. Since the arena is heap-allocated to only store data, it mitigates remote code execution the same way heap buffer overflow does.

Instead of:

void foo(...) {
    char buf[32];
}

The compiler would transform the code into something like this:

void foo(void *arena, ...) {
    // * The outermost caller would pre-allocate arena with mmap(),
// plus protected pages at the end.
    // * The arena is already a local variable, so it is automatically
// saved and restored in function calls.
    // * Users managing arenas manually must make sure the arena is
// naturally aligned to 16 bytes.
    char *buf = (char *) arena; arena += sizeof(char[32]);
}

If a function wishes to initialize an object from the arena and return it back to the caller, it would accept arena as a mutable reference instead:

struct bar *new_bar(void **arena, ...) {
struct bar *p = (struct bar *) *arena; *arena += sizeof(struct bar);
*p = (struct bar){ ... };
return p;
}

When objects are stored on the arena, virtual table pointer could still be overwritten by buffer overflow, but this is also just a heap buffer overflow, not a stack buffer overflow. The compiler could also enforce this at compile time. Same for C function pointers.

As a programming style, since arena allocation can be done at the code level with very minimal modification, we can already minimize the damage of stack buffer overflow by rewriting local array allocations to use a heap-allocated arena.

Conclusion

Shadow Stack is not a panacea for preventing buffer overflow problems. It is also not a panacea to prevent remote code execution. It only prevents a very specific scenario where the buffer overflow allows remote code execution via Return Oriented Programming.

We can achieve most benefits of Shadow Stack with Arena Allocation, by moving array allocations to a separate heap-allocated arena. This mitigates Return Oriented Programming by converting stack buffer overflow to a heap buffer overflow, which is harder to exploit. Of course, we would like the compiler to do more array bounds checking as well, but that is a harder problem for the compiler. Arena Allocation is fairly easy to achieve both by the compiler or as a minor modification in programming style, even without hardware or instruction set support.

Tuesday, May 12, 2026

Agentic AI 2.0 Design Safety Principles

As we graduate from the Agentic AI 1.0 era, we probably should have learned some lessons:

I think you can see where we are going with this. Agentic AI where users have to approve everything is not usable, but Agentic AI where users approve nothing is dangerous and irresponsible. As a middle ground, Agentic AI 2.0 should be designed with the following principles:

  • Well-defined authentication scope that limits an AI agent access to only what is needed to accomplish a task. Each task must be isolated to its own AI agent identity.
  • The authentication scope is a subset of what the human operator has permission to grant.
  • The authentication scope is short-lived (e.g. 1-20 hours) and has a revocation mechanism.
  • Writes and deletes are separate permissions from read, and these operations must be revertible.

In short, the authentication scope should be limited to the task, short lived, and revocable. Use data snapshotting to allow the task to be reverted if anything goes wrong.

Without data snapshotting, the write operation might instead log the before and after diff, and the delete operation might also only mark a resource for soft removal (30 days grace period). Only a human operator is allowed to expunge the data immediately. If this is a database row, instead of deleting a row, the table should probably just have a new "deleted_time" nullable column storing if and when the deletion occurs. The database then periodically vacuums the rows where deleted_time is older than a specific duration set by administrative policy.

At the beginning of a task, the AI agent will propose all the permissions it needs, to be approved by the human operator once until the completion of the task. If the MCP server provides methods to change the world, the MCP is responsible for enforcing the authentication scope and for data snapshotting.

Although the LLM itself without agentic ability is not able to change the world, it is still recommended to run the LLM as its own user or container, as the various PyPI packages needed to run the LLM might be subject to supply chain attack that can lead to local privilege escalation (e.g. through disk cache poisoning).

Tuesday, May 5, 2026

Mitigation strategy for copy.fail and disk cache poisoning of setuid binaries

copy.fail (CVE-2026-31431) is a Linux kernel bug where an in-place modification of a pipe scatter list by the algif_aead module (crypto module's AEAD algorithm) can be used to modify the disk cache of any file, potentially a setuid binary, by an unprivileged user. It allows local privilege escalation from an unprivileged user to root. This vulnerability was a performance optimization presumably introduced in 2017.

The provided exploit is a Python script containing a compressed payload of a ELF x86-64 statically linked binary. The poisoned /usr/bin/su just executes /bin/sh to give a root shell. Contrary to what Low Level claims, this is not shell code. The exploit cannot be shell code for the reason that setuid bit is ignored for shell scripts. This particular exploit will not run on ARM, but a new payload can be trivially crafted for other architectures.

The copy.fail writeup provides a mitigation to prevent the algif_aead module from loading, and the recommendation is to upgrade to a new kernel once it is patched.

The copy.fail writeup also claims that Kubernetes / container clusters are impacted, saying "The page cache is shared across the host. A pod with the right primitives compromises the node and crosses tenant boundaries." This is not generally true because containers often come with its own copy of the filesystem. Each file has to be separately poisoned, and you must somehow convince someone outside of the container to run the poisoned file. Also, the root user 0 in the container is mapped to an unprivileged user in the host, so it will only result in escalation to the container's unprivileged user.

As such, I recommend the following mitigation strategy for disk cache poisoning of setuid binaries:

  • Virtual machine or hypervisor will obviously contain the vulnerability by virtue that each instance runs its own kernel. Cloud compute using virtualization will be fine.
  • Docker or Kubernetes containers will be fine, provided that the host must not share files with the container. Host should also treat any file inside the container as untrusted user data and not run it outside of the container.

The copy.fail vulnerability was also an ad campaign by Theori, a security company, to promote Xint, an AI powered tool to scan for vulnerabilities in source code. It appears to be in the same product category as Claude Mythos.

Update (5/9): Dirty Frag similarly poisons disk cache using two distinct mechanisms, IPSec ESP (CVE-2026-43284) or RxRPC (CVE‑2026‑43500). The exploit also poisons /usr/bin/su to execute /bin/sh. The mitigation strategy above against disk cache poisoning still applies. The IPSec ESP mechanism additionally requires the CAP_NET_ADMIN capability which is not typically granted to unprivileged processes. The RxRPC mechanism can be exploited by unprivileged processes. These mechanisms can be disabled by preventing the kernel modules esp4, esp6, and rxrpc from loading.

Saturday, April 11, 2026

Follow up: Economies of AI

Yesterday, I gave a talk about my earlier post Economies of AI with supplemental material to illustrate the point (slides in Chinese). One addendum in my talk is that I referenced a book by George Polya, How to Solve It (1945), to illustrate the comparative strengths of AI and humans when it comes to problem solving.

Polya broke down problem solving in four phases:

  • Definition: what are the unknowns, how to collect data, what are the conditional limitations, can we derive the unknown from the limitations, and are there any logical gaps or contradictions?
  • Planning: are there known solutions, similar solutions, ways to break down the problem (by divide and conquer), can we solve a specialized problem and then generalize?
  • Execution: step by step, verify the correctness of each step.
  • Assessment: are the steps reasonable, are the results reasonable, other ways to solve the problem, applications of the solution to similar problems?

I argued that these principles of problem solving universally applies to any problem, but humans are uniquely qualified for problem definition and result assessment. The reason is that the motivation to solve a problem comes from humans, as AI does not have its own motivation nor intent. And as the beneficiary of the problem solving, only humans can judge whether the outcome works as intended, as AI has no means to gather real-world feedback. On the other hand, AI either excels or will excel at planning and execution.

It may be surprising, or completely unsurprising depending on your perspective, that humans are so good at correcting mistakes because being wrong actually hurts physiologically. AI lacks this neurological feedback.

I also made a quadrant to illustrate the division of labor based on value proposition and risk.

Low RiskHigh Risk
High Value🧑 🤖  🏭?🧑  🤖   🏭?
Low Value 🧑  🤖 🏭 🧑   🤖  🏭

In summary, humans should focus on work that has high value, AI should avoid work with high risk, and automation (deterministic algorithm) is typically relegated to low cost work (even though low cost does not necessarily mean low value).

In retrospect, I should have added another dimension to compare volume / value to show that anything high value could still justify building automation even if the volume is low. The infographics is helpful to summarize the ideas, but some nuances are lost.

Early in my talk, I also made the analogy of economies of scale using mobile phones as an example. At first, only a selected few can afford car phones, then more elites could afford cellphones but they are still bulky. Mobile phones became more prevalent and affordable with the indestructible Nokia 3310 even though functionality is still limited, and now everyone has a smartphone. I argue that AI right now is analogous to the Nokia 3310.

Audience questions

Q: Although humans should focus on high value work, what would happen to those who could not find high value work?

Economists know about the Pareto distribution which states that 20% of the causes achieve 80% of the outcome. You can see this in teams or group projects where a few people deliver most of the value. You also see this in nature where 40% of worker ants are idle, but these idle ants serve as reserve capacity. IT uses redundancy as backup strategy by following the 3-2-1 rule with 3 copies of data, on 2 different media types, and 1 stored offsite. These are examples showing that reserve capacity or redundancy is needed in any resilient systems.

Without the reserve capacity, the system will collapse under stress such as war or disasters.

Q: With AI or automation taking away low value work, how would a person develop the skills to perform high value work?

Instead of focusing on the planning and execution, education should instead focus on the design and assessment aspects of problem solving, the first and final phases according to Polya.

I also wrote an article earlier My advice if you are a 13 year old vibe coding to become the next Bill Gates espousing the ideas of building up vocabulary and validating outcome as the essential skills to have.

Q: In your economies of scale analogy, AI is currently like the Nokia phone. What would it take to become a smartphone?

Currently, users spend a lot of effort on prompt engineering (the problem definition phase of Polya) because the prompt has to be precise and without ambiguity to avoid AI slop (see I Tried to Kill Vibe Coding). AI could evolve to fill some of these definition gaps by making heuristic guesses about what the prompt might mean when a problem is stated. This can happen in the assessment phase whether any solved problems are relevant to the problem at hand. On the other hand, the assessment of the efficacy of the outcome can still only be done by the user.

Saturday, January 31, 2026

Economies of AI

This is a cost-benefit analysis on using AI to solve problems and comparing how it fares with classical methods, e.g. deterministic algorithms or manual labor, and the cost of the creation of automation.

A fair warning: currently, LLM is not able to summarize this article correctly because of my unique perspective (example), as this article is not about the H-word at all. You should try to read it yourself. If you are impatient, at least read the first sentence of each paragraph and the conclusion.

AI vs. Deterministic Algorithms

An example of a deterministic algorithm is to compute an arithmetic expression like "1+2+4". There are well-known and efficient ways to compute it.

  • First the string is tokenized: "1+2+4" → ['1', '+', '2', '+', '4']. This is called lexing.
  • Then the string is organized into an abstract syntax tree: ['1', '+', '2', '+', '4'] → Plus(1, Plus(2, 4)). This is called parsing.
  • Then the abstract syntax tree can be traversed recursively and the value is computed: Plus(1, Plus(2, 4)) → Plus(1, 6) → 7. This is called evaluation.
  • Under the hood, a machine would compute the addition using logic gates called an Adder.

For AI to do the same, the tokenizing is similarly done by a deterministic algorithm, but the rest of it is done through many large matrix multiplications. The size of these matrices are much larger than the length of the input tokens, and matrix multiplications take \(\omega(n^2)\) time complexity. Logic gates for a Binary Multiplier is also much more complex than an Adder. Large matrices take up more memory space and more communication bandwidth to move the data.

Which is why a machine could make billions if not trillions of calculations a second, but it would take AI a few seconds to complete a single prompt. Not to mention the power consumption needed by AI is several orders of magnitudes greater than a deterministic algorithm.

This is why for the problems for which we have a deterministic algorithm, it would not make economic sense to use AI to solve these problems. Furthermore, it would be AI's best interest to offload any such prompts to a deterministic algorithm. AGI may be an academic interest, but it is not economically viable for doing mundane tasks. Just like we would not be hiring humans to crunch numbers anymore once computers became commonplace.

AI vs. Manual Labor for Doing the Work

To achieve economic parity, AI would have to be relegated to the odd jobs—the long tail for which no deterministic algorithm exists. For these odd jobs, a person should try to do it first before trying AI. This is for two reasons: once they have done the job themselves, they have a better understanding how to write the prompt; and they will be in a better position to evaluate whether AI is doing the job correctly. Skipping this step is a common reason for getting AI slop. It is not necessarily the fault of the model when the prompt itself is sloppy.

If an odd job is truly one-off, it may make sense to do it only manually because the cost of learning by doing is comparable to the cost to understand how to write the correct prompt. When doing things manually, we gain insight about any potential problem, and then adjust the assumptions, requirements or expectations to avoid these problems. AI is unlikely to challenge the assumptions made by the prompt unless specifically asked. We wouldn't know what to ask for unless we are already aware of the problems. We wouldn't be aware of the problems unless we tried to do it ourselves. So just do it first. When the job happens again, then offload it to AI. This weird trick of DIY-ism will save you tons of time writing prompts, perhaps counter-intuitively.

AI vs. Manual Labor for the Creation of Automation

When these odd jobs become frequent, it then makes sense to invest in the time to automate them by creating a deterministic algorithm and writing programs. Traditionally, a human would write the computer programs for these algorithms. AI could presumably write them now, but I argue that the economy impact difference is minimal between the two. The reason is that whatever the cost is to develop software, the cost is amortized over the many jobs it ends up automating. Even though the one-time development cost may be expensive, it becomes negligible if you spread the cost over many jobs. AI may be 10x more productive than humans for writing programs, but 10% of negligible is still negligible.

What is not negligible is the cost of poorly designed automation, which has a multiplicative effect on the defects of the outcome. The defects can be the incorrectness of the output, or the inefficiency in the algorithm itself usurping too much resources or taking too long. If the algorithm is poorly designed, then it would screw up over many jobs, and the expense to clean up the mess is the polar opposite of negligible: it would be astronomical. It doesn't matter whether the algorithm is designed by a human or AI.

When it comes to the creation of automation, use whatever tool at our disposal to design an algorithm that reliably achieves the correct outcome and can do it efficiently. Even if AI is not able to vibe code a project from start to finish, it can still be a valuable tool for humans to learn about the nature of the problem through prototyping.

Divide and Conquer

So far, we treat the problem as a monolith. In reality, a problem can be broken down to many subproblems. It is like when computing "1+2+4" we compute one addition at a time, either:

  • Leftist: (1+2)+4 = 3+4 = 7
  • Rightist: 1+(2+4) = 1+6 = 7

And there is more than one way to break down the subproblems. The ability to decompose problems also gives rise to efficient algorithms known as divide and conquer algorithms, and in many cases it can be proven that this is the optimal way to solve a given class of problems.

When discussing AI's economic proposition, we should remember that many bespoke problems can be reduced to subproblems that are recurrent and can be solved at a greater economy of scale than if we considered each problem in isolation.

For example, car builders would design common parts, e.g. the engine and chassis, that can be reused across multiple models of sedans and SUVs. These engines and chassis are built out of common parts like standardized screws, nuts and bolts. Greater economy of scale is achieved by using common off the shelf parts, even if the end product is bespoke.

In the same way, we can mix AI, deterministic algorithms, and even manual labor in different configurations to achieve economy of scale.

Value Proposition

Another issue we neglected is the value proposition of the outcome of the work. In the pre-computer ages, human calculators were used for extremely high value work even though they are slow and error prone, from artillery in a battle that increases the probability of winning the battle, to scientific calculations that raced to create atomic weapons that ended World War II. Or they are employed for the backbone of economy itself, such as finances and accounting.

When computation became so cheap, they are used for entertainment like video games or watching cat videos.

Similarly, the method for which we use to solve a problem—manual, AI, or automation—speaks nothing about the value of the work that employs them. When it comes to high value work where the stake of failure is high, AI will still face a fierce competition with automation and human ingenuity, in part because of AI's high error rate. On the other hand, when AI is used to generate videos for entertainment, who cares if the video shows someone with seven fingers, or if the text is malformed, provided the entertainment value is good? There is no stake in these failures.

When company management makes the decision to replace work with AI, it is a signal that they consider the value proposition of the work to be low. They could be proven wrong by the market or the competition. Indeed, competition is a remedy for Enshittification, and we need Anti-Trust enforcement to ensure competition. I'm not sure if labor protection helps, since it enables complacency, not ingenuity.

Conclusion

We reach a conclusion where the economic viability is unsurprisingly dictated by the economy of scale.

  • High volume work should be done by a deterministic algorithm, not AI.
  • Low volume work could be done by AI, but humans should do it first so they can understand the problem better, for writing better prompts and for evaluating the efficacy of the output.
  • One-off work should be done by humans first to understand the problem.

When deciding which problems are high volume, low volume, or one-off, we should use a divide and conquer approach and break bespoke problems down to reusable and recurring subproblems, so we can achieve greater economy of scale. Again, this should not be a surprise for economists. If anything, computer science just provides the vocabulary to explain why the economy of scale is achievable.

We also came to a conclusion that AI slop is enabled by enshittification, and the remedy is more competition through Anti-Trust enforcement; this is also unsurprising for economists.

The more sober minded person will come to realize that AI is just one more way to get things done, and it is still subject to the same market forces as everything else. Commodified work will eventually be replaced by deterministic algorithms, not AI. High value work will still face competition from human ingenuity, unless the human chooses to be complacent or if our values somehow become corrupt.

That last point that our values have somehow become corrupt is my greatest fear.

Friday, January 23, 2026

Introduction to Fiber Optics for 10G Network at Home

Fiber optics are becoming cheaper and more affordable for homes nowadays, and they could be a fantastic alternative for RJ-45 10GBase-T Ethernet. Fiber transceivers at the same speed use less power and run cooler. They also don't need to be upgraded like you would need to upgrade an Ethernet cable from Cat5e to Cat6a to now Cat8.

I have worked with fiber network designs at work, at least in theory. But in the last two weeks, I learned even more than I cared for about fiber optics, and here is a quick summary about them before I forget.

  • Fiber core size: single-mode fiber (SMF, OS2) vs. multimode fiber (MMF, OM5, OM4, OM3, OM2, OM1).
    • Single mode has a smaller core and can run for longer distances. Multimode fibers may be more cost effective once upon a time, but these days single mode fibers are so cheap, they should be the default choice for even short (< 100m) runs.
    • Note: the mode type must match with the transceiver as well, so you have to use a single mode fiber with a single mode transceiver.
  • Number of strands: simplex (1 fiber) or duplex (2 fibers).
    • The transceivers most commonly used for data center applications will use duplex fibers, using one strand for each direction of transmission. Both directions typically use the same wavelength.
    • Simplex fibers are more common for Fiber To The Home (FTTH). You have to use a matching (opposite) pair of transceivers at the two ends, e.g. one side is 1330nm-TX/1270nm-RX, and the other side should be 1270nm-TX/1330nm-RX. The two wavelengths traverse on the same fiber strand.
  • Connector type: SC (standard connector, larger) vs. LC (lucent connector, smaller).
    • For some reason, the LC transceivers are more commonplace, possibly because you can fit a duplex connector into a transceiver with the SFP form factor.
    • There are also SC transceivers, but these are less common.
    • Again, you have to match the connector with the transceiver.
  • Connector contact type: UPC (domed, color coded blue) vs. APC (angled, color coded green). They refer to how the fiber ending is terminated. The angled type reduces back-reflection.
    • You have to match the contact type with the transceiver as well. Inserting APC connector to a UPC transceiver can damage the optics.
    • I typically find either SC/APC or LC/UPC to be more common. The other combinations (e.g. SC/UPC or LC/APC) are possible but uncommon.
    • Adapters exist but they can cause signal loss. Due to the naming, it can be hard to find the correct adapter by keyword (e.g. "SC/APC female to LC/UPC male" can also give you results for "SC/APC male to LC/UPC female").
    • Sometimes SC/APC is written as SC/Angled or simply SCA.

Bend radius: more recently, I learned about bend insensitive fibers from a video about the InvisiLight fiber kit. The whole kit cost $250 and includes a pre-terminated G.657.B3 simplex fiber, two 1G transceivers, and two media converters. For my purpose, I would need to buy my own 10G transceivers and a 10G media converter.

Bend insensitive fibers are characterized by having smaller bend radius than the regular fibers (30mm), and they are defined by the standards G.657.A1 (10mm), G.657.A2 or B2 (7.5mm), and G.657.B3 (5mm). This primarily allows installation in tight corners in the home, and the transceivers are agnostic to the bend radius.

FS.com has a custom fiber builder where you can build simplex or duplex fibers with custom core sizes, connector and contact types, and bend radius. They offer white color which is better than bright yellow for home installation. However, their sales representative told me that G.657.A2 with 0.9mm cable diameter has a minimum order quantity (MOQ) of 1KM (this is not on the website). The best they could do with no MOQ is G.657.A1 with a 2.0mm diameter. Perhaps if more people inquire about the G.657.A2 0.9mm, they would eventually lift the MOQ.

There appears to be some pre-terminated G.657.A2 0.9mm cables on eBay, but I have not tested them. Some of the fibers on eBay are also a bit weird, having SC/APC (green) on one side and SC/UPC (blue) on the other, or they are from SC to LC. Either way, a 40 meter (130 feet) long bend insensitive fiber cable should be in the $12-$16 range. Fiber is cheaper than Cat6 per unit length, and they are not limited to 100 meters.

For running between rooms in the home, you would typically use a simplex single-mode and matching LC/LC or SC/SC transceivers, and choose or build a fiber cable with the same connector. Here are some transceiver options I found:

  • LC/UPC single-mode simplex:
  • SC/APC single-mode simplex: Elfcam branded 10G SC Bidi transceivers (20km), sold as a matching pair for $66.
A distance of 10km / 6.2 miles will let you run fiber to your friend's house living in the next town!

For running short distances between devices in the rack room, LC duplex single-mode has more transceiver options. I also use a Direct Attached Cable, which is just copper but with both ends hard-wired to SFP+ connector.

The SFP+ connector refers to the plug form factor where the transceiver is inserted into the switch device. It determines the electrical speed as well.

  • SFP: 1G.
  • SFP+: 10G, 25G.
  • QSFP (or QSFP28, literally "quad SFP"): 100G.
    • There are four electrical lanes.
    • When a "breakout transceiver" is plugged in (e.g. 4x25G-LR), a single port can be used as four independent interfaces, e.g. 4x25G.
    • A "singleton transceiver" (e.g. 100G-LR4) would combine all 4 electrical lanes to drive one SerDes (serializer/deserializer).
  • QSFP-DD (or QSFP56, or QDD, literally "double density quad SFP"): 400G, 800G.
    • There are eight electrical lanes that support various breakout transceivers: 4x100G, 8x50G, or 8x100G.
    • They can also be used together, e.g. 400G-FR4.
    • Sometimes the breakout transceivers are confusingly named as 400G-DR4 (breaks out to 4x100G-DR) or 400G-XDR4 (breaks out to 4x100G-FR).
  • OSFP (literally "octa SFP"): 400G, 800G.
    • There are eight electrical lanes also supporting various breakout transceivers.
The main reason for these "breakout transceivers" is to allow switches to interoperate with previous generation switches, e.g. a switch with QSFP ports would use a breakout transceiver to split each port into four fibers connecting to SFP+ transceivers, or QSFP-DD/OSFP breakout out to QSFP. You don't typically see this on consumer devices, but some of them support breakout transceivers, e.g. QNAP QSW-M7308R-4X (I bought it at $999 a week ago, now it's $1299, probably due to the new shipment having Trump tariffs).

With data centers upgrading from QSFP 100G to QSFP-DD or OSFP 400G or 800G to satisfy the AI computing needs, and with more FTTH deployments, we are seeing more surplus SFP+ 10G transceivers and switches flowing to the consumer market. This could be a good time to upgrade your home network cabling from copper based cable to fiber.

Thursday, January 1, 2026

RCU, SMR, Hazard Pointers: data structure memory management for concurrent programs

Yesterday, I was looking for whether OpenBSD supports ZFS (it may have been briefly, but the code is no longer there). As I peruse Ted Unangst's blog, I found a benchmark showing that creating a socket is 10x slower on Linux than OpenBSD, referencing a post by Jann Horn. People quickly put the culprit on RCU, in particular rcu_synchronize().

Read-Copy Update (see also What is RCU, Fundamentally?) is a way to distribute data structures across multiple CPUs that favors the reader by making the writer do more work. Readers can assume their copy is immutable within the scope of some critical section, so they can read it in a non-blocking manner. The writer has to make a private copy first, update it, then publish the new copy to a global reference. However, the writer has to wait for all readers to be done with the old copy before the old copy can be reclaimed. They wait for the readers by calling rcu_synchronize().

The rcu_synchronize() is probably what actually makes the writing slow. It waits for all readers to exit the critical section for the old data, then frees it, before it can make another update again. Worse, an inconsiderate reader might treat the data as a private copy and hold onto the critical section while being blocked on I/O or something else. RCU conservatively prevents old data from accumulating at all, where in reality it might have been fine to have multiple copies of old data, as long as it doesn't grow indefinitely. This is the main idea behind GUS (globally unbounded section); the unboundedness here refers to the writer being unrestrained by the readers lagging behind. This is implemented as Safe Memory Reclamation (SMR) in FreeBSD (subr_smr.c). They use a version sequence number to determine whether the reader has moved on.

It would have been totally reasonable if we just tell the last reader that they are responsible for freeing the data (i.e. reference counting), so the writer can move on. Or perhaps the writer could provide a deleter function upon the last reader exiting the critical section. In any case, old data can only accumulate as far as there are readers, so it is bound by the number of readers.

Hazard Pointer tries to solve a similar problem but for the limited purpose of memory reclamation instead of trying to ensure a consistent view of data structures. A reader would enter the critical section by adding the object in question to a global registry of hazard pointers. Whoever wants to free some object has to check the global registry of hazard pointers to make sure it does not have a reader. This idea was first proposed by Maged Michael et al, 2002, seemingly abandoned for over 20 years, but now making its way into C++26.

But I argue that Hazard Pointer is not scalable. Even if we allow at most one hazard pointer per thread, we still have to support millions of threads in a production system all holding a hazard pointer one way or another, so the sheer size of the registry is phenomenal. The registry also begs the question about its own internal consistency: by the time a writer finishes scanning the registry whether an object is implicated by any hazard pointers in the registry, another reader might have added the hazard pointer to the registry without the scanner knowing.

When it comes to data structure, especially the high performance ones, treating the data as immutable is the way to go. Since modern multicore CPUs use MOESI protocol for cache coherence, each CPU benefits from having its private copy that it can access very quickly without having to go to the main memory, and there is very little memory bus contention.

The problem with RCU is not the immutable paradigm, but how it makes the writer wait for all readers. Also, even if entering the critical section is non-blocking, the reader should exit as soon as it can and avoid lingering in it while being potentially blocked by something else. We should also just make the last reader free the object upon exiting, so versioning is not necessary.

In some extreme cases where you might have a large amount of immutable data tied to some worker threads, it might even make sense to wind down the worker (i.e. lameduck) and spin new workers up with new data, similar to deploying new microservice versions.

Friday, December 19, 2025

My advice if you are a 13 year old vibe coding to become the next Bill Gates...

(Insert obligatory AI generated Ghibli styled graphics here.)

If you are a 13 year old vibe coding to become the next Bill Gates, here are the computer science concepts you will need to know to successfully direct AI to do the right things. I assume you want to build the next big thing and not just fixated about the 70's BASIC interpreter that Bill Gates wrote when he was 20 without AI. Also, if you want to go into fundamental research on AI or quantum computing, vibe coding is probably not what you're after, but it can be helpful to learn about randomized algorithms.

If you ask AI today the same question, you would have gotten some hand-wavy advices, and here are my takes on why they are not that useful.

  • Master prompt engineering.
    • Why this is not useful: the only way you master prompting is by having the right vocabulary for the fundamental concepts in computing, and you need to understand the concepts behind these vocabularies. It is not effective to prompt with an alphabet soup of jargons unless you use them in a meaningful way.
  • Learn the tech stack (e.g. Gemini, GitHub Copilot).
    • Why this is not useful: tech companies are going to make AI as easy to use as tap water. Obviously, there is fascinating practical engineering about water resourcing and plumbing infrastructure to get the water to your tap, but it's not exactly rocket science to learn how to turn on the faucet. Anyone who thinks they have a particular edge in prompt engineering will find out that it is quickly disappearing.
  • Problem solving and strategic vision.
    • Why this is not useful: don't just dream about things in your head. It is even more important to learn to try things and observe the outcome. If something didn't happen as expected, try to understand why. This is not something AI can do for you, since AI can only learn from its training data. Always validate your ideas in the real world and pivot as necessary.
  • Financial literacy.
    • If you pay attention in high school math class, you should have the tools you need to predict the outcome of your decisions. Watch this video about why Math Just Got Important.

Instead of recommending specific framework or product (since you can ask AI for more timely recommendations), here is a bucket list of timeless computer science concepts that you will want to learn:

  • Computer architecture, especially about the memory hierarchy and principle of locality in the context of cloud services. Separation of code and data, which is important for security.
    • Why is it important: memory hierarchy and principle of locality lets you understand how all computer systems operate under the same space-time constraints, similar to relativity in Physics, and the cost-benefit trade-offs needed to optimize it (e.g. caching).
    • Without the discipline to separate code and data, mixing the two is a constant source of exploits compromising cybersecurity at national levels.
  • Algorithms and data structures: when you learn about sorting, set your sight on the time and space complexity analysis and try to not get bogged down with the mechanism itself. Hash table is going to be relevant in load balancing, and Graph traversal for network architecture.
    • Learn programming in the context of algorithms and data structures so you have a vocabulary to describe them.
    • Why is it important: complexity analysis tells you why your system is provably not scalable when the dataset becomes larger, not even if you add more computing resources to it.
  • Network architecture: DNS, IP addressing, client/server and load balancing.
    • Why is it important: although many "cloud" providers have this abstracted away, understanding the network architecture lets you design your cloud in a more cost effective way and be able to debug when issues arise.
  • Server side: HTTP and REST, cryptography (TLS, JOSE), SQL (particularly about SQL injection which is what happens when you fail to separate code and data).
    • Why is it important: although many frameworks also abstract away most of these details, your system architecture is limited by the assumptions made by the underlying protocols. Ultimately this boils down to trade-offs, whether to accept these limitations as "good enough" for your application or seek alternatives, e.g. HTTP long polling vs. WebSocket, X.509 vs. JWK, SQL vs. NoSQL.
  • Client side: fundamentally, at least HTML/DOM, CSS, and Javascript.
    • They are very capable nowadays, so you don't really need a separate framework, but feel free to let AI try different frameworks.
    • Why is it important: you still need to know HTML, CSS and Javascript in order to debug framework code.

There are some more advanced topics that could be relevant for domain specific work like games, especially the triple A titles with good graphics and physics, unless you are satisfied writing yet another unimpressive Minesweeper.

In a world where the cost of answers is dropping to zero, the value of the question becomes everything. It is still important to learn the concepts so you have a vocabulary to express ideas in your head, and to observe if your ideas work in the real-world and pivot if not.

Friday, December 12, 2025

Secretive Key Attestation Possible?

On an old MacBook Pro 2019, I've been trying out Secretive (secretive.dev) to let me use the Touch ID (aka SecureEnclave) as a hardware SSH key, like how I would use Yubikey on other computers.

One of the things I would like to do is to create a certificate authority for key signing, to avoid having to manually add new keys to each host I have to SSH into. The certificate authority needs to tell that the hardware key is indeed generated by some qualified hardware, so the private key cannot be easily exfiltrated. Also, I should not be able to cheat by generating a key pair on a computer, making a copy somewhere, then load the private key onto the hardware token and pretend it was generated there.

Normally, using U2F with SSH gives me the option to generate a key attestation that can then be verified by the CA before signing the public key certificate. Essentially, I tell ssh-keygen to write the attestation to another file as the new key is generated, and the attestation will have to be parsed using a custom script in order to check against the vendor certificates published by FIDO2 Alliance.

There are several ways to generate key from the Secure Enclave:

Note that the Secretive 3.0.4 release notes mentions "GitHub attestation" but that is entirely different. It just says the binary built by GitHub actions will now have provenance.

Even though DeviceCheck.DCAppAttestService seems like a likely candidate to generate attested keys, the fact it requires a validation roundtrip to Apple's server can be problematic. It likely only works for apps uploaded to the App Store (which Secretive is not), and people will not be able to compile their own binaries and expect it to work. I'm also not sure how people may feel that Apple can now keep a record of all their SSH keys.

Probably, it's easier to just buy another Yubikey.

Thursday, December 11, 2025

Certificate Signing Request with Authentication Credentials

PKCS #10 (RFC 2986) certificate signing request (X509_REQ) typically contains at minimum the subject and the public key to be signed into a X.509 certificate issued by a certificate authority (CA). Here is an example of what I send to Let's Encrypt:

$ openssl asn1parse -item X509_REQ -in worship.likai.org.csr 
X509_REQ: 
  req_info: 
    version: 0
    subject: CN=worship.likai.org
    pubkey:     X509_PUBKEY: 
      algor: 
        algorithm: id-ecPublicKey (1.2.840.10045.2.1)
        parameter: OBJECT:secp384r1 (1.3.132.0.34)
      public_key:  (0 unused bits)
        0000 - 04 e4 5c 30 85 6f b5 37-35 f4 e9 21 3b 2c 74   ..\0.o.75..!;,t
        ... omitted for brevity ...
        005a - 33 2f c8 fb c1 21 86                           3/...!.
    attributes:
      <EMPTY>
  sig_alg: 
    algorithm: ecdsa-with-SHA256 (1.2.840.10045.4.3.2)
    parameter: <ABSENT>
  signature:  (0 unused bits)
    0000 - 30 66 02 31 00 ce 23 b9-c3 b0 53 24 2f d3 8c b0   0f.1..#...S$/...
    ... omitted for brevity ...
    0060 - cc fe a2 64 fa f2 7a e6-                          ...d..z.

X509_REQ can also contain extensions, most notably subjectAltName. Some of the extensions are documented in x509v3_config, which includes the ability to send arbitrary extensions. PKCS #9 (RFC 2985) defines the attributes that can be used in a PKCS #10 CSR. Arbitrary extensions are defined by an object identifier (OID) key and some ASN.1 value. We can lookup the OID assignments.

As currently defined, X509_REQ does not have the ability to include authentication credentials, i.e. the CA only knows the subject and its public key, and the X509_REQ itself is signed by the same subject's public key for integrity, but the CA does not know whether the subject is whom they claim to be.

That kind of trust has to be verified out of band. In the case of Let's Encrypt (how it works), the ACME protocol performs domain validation by fetching the result of a challenge response over plaintext HTTP at the domain name matching the CN subject in the CSR. This requires the host to be reachable from the public Internet, which is not feasible for internal servers. There are other challenge types: TLS-ALPN-01 also requires the host to be reachable from the public Internet; DNS-01 does not require the host to be reachable, but the requestor must be able to modify the DNS record, which can present other difficulties.

Enrollment over Secure Transport (EST) is another protocol to automate certificate signing (like ACME). It authenticates the CSR at the API layer but not as part of the CSR. However, binding the API authentication to the CSR remains a challenge.

There are a few potential OIDs we can investigate for their suitability to include authentication credential as part of the CSR:

  • PKCS #9 defines a challengePassword OID 1.2.840.113549.1.9.7, but it is supposedly used to request certificate revocation. It seems to be unimplemented. Also, it is generally a bad idea to repurpose a field for a different purpose than originally intended.
    • The content of challengePassword is plaintext equivalent, which means the CSR should be protected by other means so it is not public accessible.
  • RFC 7894 proposes additional fields to disambiguate the PKCS #9 challengePassword, notably: otpChallenge, revocationChallenge, estIdentityLinking.
    • It assumes that the primary authentication is done by the underlying transport such as EST. The additional fields can provide a second-factor authentication.
  • The SMI mechanism OID 1.3.6.1.5.5 references additional mechanisms tracked by IANA. In particular:
    • The aforementioned RFC 7894.
    • There is a working draft draft-ietf-lamps-csr-attestation-08 to attest the CSR with evidence, which is intended to come from a hardware TPM or PSA. Presumably, the evidence can be further extended to cover FIDO U2F or Passkeys. These evidences must not contain plaintext equivalent secrets.
  • Kerberos OID 1.2.840.113554.1.2.2 (RFC 1964) does not have an OID for the challenge response, as far as I can tell.

These proposals tend to avoid putting plaintext secrets into CSR, but the ability to secure secret credentials in a CSR remains elusive.

Envelope Proposal

To be able to include secrets in CSR, we can use the CA's public key to encrypt the plaintext, and we store the ciphertext in an envelopedData. There are three potential OIDs defining envelopedData:

The envelopedData defined by PKCS #7 (RFC 2315) is obsolete by the latest version RFC 5652, which seems to be the correct candidate to store encrypted secrets destined for the CA. If the CA has multiple certificates, they can all be included as a recipientInfo.

Inside the envelopedData should be a SEQUENCE that contains at least the SET OF Attribute{{ IOSet }} as defined in PKCS #10 (RFC 2986). This also opens up the possibility to include anything from an LDAP directory OID 2.5.4 (RFC 4519 or ISO/IEC 9594-6:2020 or X.520) that might need to be protected from public access, such as userPassword 2.5.4.35 or maybe encryptedUserPassword 2.5.4.35.2.

Afterthoughts

On one hand, having a global OID means applications and protocols can share relevant definitions when appropriate, but navigating the maze of existing OID assignments can be an exercise in frustration. It is similar to code reuse: sometimes the code you borrow from almost does what you need except for some corner cases, but you cannot change the corner case behavior because it may break existing users relying specifically on the corner case. So you have to choose the OID carefully based on how you expect the OID to evolve in the future, and try to find an OID that best aligns with the use case.

It is tempting to start from scratch and define exactly what an application needs in an application-specific way without any historical baggage, like JOSE (JWS/JWT). Their RFCs are still long, but still shorter than the maze of OIDs, RFCs, and ITU/ISO standards. It would be nearly trivial to abbreviate the CSR to another JSON object.

It is also interesting how OID leaves the breadcrumbs showing how the Internet standards evolved. At first, RSA developed the PKCS under its OID 1.2.840.113549 {iso(1) member-body(2) us(840) rsadsi(113549)}, then this namespace is donated to IANA. Similarly, the OID 1.3.6.1 {iso(1) identified-organization(3) dod(6) internet(1)} used to be owned by DoD (Department of Defense), now hijacked by IANA as well.

Tuesday, December 9, 2025

Runtime TLS Certificate Hot-Reload with OpenSSL

For long-running microservices that can act as both client and server, it is possible the lifetime of the microservice might outlive its TLS certificate validity, especially if an organization opts for a security policy to rotate TLS certificates quickly (e.g. every day). We need to be able to reload both the client and server certificates without exiting the service.

OpenSSL provides the functionality as follows:

  • For a client, the caller should use SSL_CTX_set_client_cert_cb() to provide a callback which is "called when a client certificate is requested by a server and no certificate was yet set for the SSL object." The caller provides a client_cert_cb() function that modifies the X509 ** and EVP_PKEY ** (pointer to a pointer), which OpenSSL will use to call SSL_use_certificate() and SSL_use_private_key() internally.
  • For a server, the caller should use SSL_CTX_set_tlsext_servername_callback() to provide a callback which is called during SNI. The caller provides a cb() function that modifies the SSL * object directly to override the SSL_CTX using SSL_set_SSL_CTX().

The callback can use an out-of-band method to reload the private key and certificate as needed. Once a connection has been authenticated during handshake, it will probably remain active even after the certificate expires, so the callbacks only apply to new connections.

To actively disconnect TLS transport with an expired certificate, some kind of connection pool management will be needed to wrap OpenSSL. This connection pool can also handle load balancing.

It is generally a good idea to allow certificate rotation to have some validity overlap so connections can be "upgraded" to the new certificates gradually. It avoids the problem where all connections die and have to be reconnected all at once, which can have undesirable latency impact.

Saturday, October 18, 2025

Compiling s3backer on MacOS, 2025 edition

s3backer allows you mount a single large file split into smaller chunks on S3 as a local file, via FUSE. In contrast, s3fs-fuse mounts S3 objects as individual files. s3backer is useful for hosting disk images formatted with a native filesystem. Although Apple subsequently introduced the Sparse Bundle image format, which also internally splits the image into multiple files, some reported that Sparse Bundle is limited to 100,000 bands, with a band size > 8.4MB each. On the other hand, s3backer supports better local caching, more granular block sizes, and builtin compression and encryption.

This is a quick note about how to compile s3backer.

  • If not already, install the Xcode command line tools: xcode-select --install
  • I recommend using either MacPorts or HomeBrew to install a recent version of automake, autoconf, and (lib)curl.
  • Install MacFUSE by downloading the appropriate package. This installs the header files under /usr/local/include and library files under /usr/local/lib.
  • Clone s3backer: git clone https://github.com/archiecobbs/s3backer

The commands I use to build:

git clean -fxd
mkdir -p m4
autoreconf -iv

configure \
  --prefix=$HOME/local \
  CPPFLAGS='-DFUSE_DARWIN_ENABLE_EXTENSIONS=0 -I/usr/local/include/fuse3 -I/opt/local/include' \
  LDFLAGS='-L/usr/local/lib -L/opt/local/lib'

make -j8 install

Some explanations:

  • git clean -fxd - removes any untracked files from the working dir.
  • mkdir -p m4 - output files for autoreconf.
  • autoreconf -iv - runs aclocal, automake, and autoconf.
  • configure \
    • --prefix=$HOME/local - pick your own install directory.
    • CPPFLAGS
      • -DFUSE_DARWIN_ENABLE_EXTENSIONS - on MacOS, fuse3 headers by default expects the code to use the Darwin file structs. This would have been useful for MacOS native filesystems, but s3backer uses the POSIX file structs.
      • -I/usr/local/include/fuse3 - MacFUSE comes with both legacy fuse2 and the newer fuse3. If we included -I/usr/local/include, then #include <fuse.h> would pull in fuse2. This might have been useful for legacy code, but s3backer uses fuse3.
      • -I/opt/local/include - add the MacPorts headers (for curl).
    • LDFLAGS
      • -I/usr/local/lib - MacFUSE library files.
      • -I/opt/local/lib - MacPorts library files.
  • make -j8 install - makes the s3backer binary and installs it to the --prefix configured above, using 8 parallel invocations.

It is worth noting that due to my specific setup, explained below, I ran into additional problems similar to this github issue: https://github.com/pyenv/pyenv/issues/3309, but I don't expect most people to run into it. I'll document my problem here for posterity.

During configure, gcc was not able to find the libcurl symbols. Upon closer inspection, gcc --version outputs x86_64-apple-darwin24.6.0 for the target, even though I'm now using an Apple Silicon machine, which should report arm64-apple-darwin24.6.0. The correct target was reported when I run gcc from the command line directly.

The reason for the wrong gcc target during configure is because I used a Python script to build s3backer out of tree. The script creates a unique temporary directory, cd into it, then runs the s3backer configure script from it. A long time ago, when I was still using an Intel mac, I installed Python from a python.org package. The x86 executables installed under /Library/Frameworks/Python.framework were migrated by the Migration Assistant when I bought a new Apple Silicon Mac.

When I run my Python script, it found the x86 Python interpreter, which is then run under Rosetta 2. Subsequently, when an x86 program runs gcc, it also runs the x86 version of gcc which would compile and link the output binary as x86. However, MacPorts only has arm64 of curl installed, which is why it couldn't find the curl symbols.

I'm not sure about the moral of the story. I would have preferred to start with a clean install, but it would have taken me too long to get it up and running. Migration Assistant gave me a usable machine more quickly but left some unexpected problems behind that needed clean up later. The technical debt trade-off was not known at that time, but an accidental one. I expect there will be more x86 files I need to clean up later.

Tuesday, September 30, 2025

AI, The Theranos of Software Engineering

Breaking news! Vibe coding achieved the utterly impressive feat! Macintosh System 7 Ported To X86 With LLM Help. The original System7 ran on Motorola 68K. The author claimed that they ported the OS to x86 in 3 days, with a fully functional Finder and GUI, and no access to the original source code. The project files can be found on GitHub.

It is understood that the port ran under QEMU to simplify dealing with real hardware, which has to do many things like ACPI power management. But as you may have surmised from the title, this is not the only "shortcut" that has been taken.

ISO contains just the kernel?

To start, let's take a look at how to run the project from the instructions provided:

# Build the kernel
make

# Create bootable ISO
make iso

# Clean build artifacts
make clean

# Run in QEMU
qemu-system-i386 -cdrom system71.iso -serial stdio -display sdl -vga std -m 256M

So let's take a look at the Makefile to see how the ISO was built.

ISO_DIR = iso
KERNEL = kernel.elf
ISO = system71.iso
GRUB = grub-mkrescue

# ISO target
iso: $(ISO)

$(ISO): $(KERNEL)
	@echo "Creating bootable ISO..."
	@cp $(KERNEL) $(ISO_DIR)/boot/
	@echo 'menuentry "System 7.1 Portable" {' > $(ISO_DIR)/boot/grub/grub.cfg
	@echo '    multiboot2 /boot/$(KERNEL)' >> $(ISO_DIR)/boot/grub/grub.cfg
	@echo '    boot' >> $(ISO_DIR)/boot/grub/grub.cfg
	@echo '}' >> $(ISO_DIR)/boot/grub/grub.cfg
	@$(GRUB) -o $(ISO) $(ISO_DIR)

The ISO is made using grub-mkrescue. The kernel is linked using the multiboot2 specification, which means grub would have switched the x86 CPU into 32-bit protected mode with a flat address space.

But it is a little surprising that the ISO comprised of just the kernel.elf and grub.cfg, devoid of all other assets such as fonts, icons, application binaries. There is no filesystem because everything is compiled into the kernel.

Indeed, you can find the hard coded icons and the hard coded font bitmap. The FontResources is really some code to download fonts from the Internet using curl, but this is not compiled into the kernel, and this is not the code used to generate the hard coded font bitmap either. This is probably just dead code.

If there is no filesystem, then it is a wonder why the project contains HFS filesystem code?

Dead Code Galore

There is a huge amount of dead code in the repo, all generated AI slop. This is an impressive catalog: AppleEventManager, ControlManager, ControlPanels, DeskManager, DeviceManager, DialogManager, EventManager, FS (HFS), FileMgr, Finder, FontManager, FontResources, GestaltManager, ListManager, MemoryMgr, MenuManager, PatternMgr, Platform, ProcessMgr, QuickDraw, ResourceMgr, Resources, ScrapManager, SoundManager, TextEdit, WindowManager.

I have spot checked a few: The FS code has mentions of btree and catalog, but it is an in-memory filesystem (no actual disk volume support). The QuickDraw code does not contain any shape drawing code, i.e. the code purporting to draw the line or oval does not actually render the pixels. There is no implementation of the Bresenham's algorithm.

Only a handful of these managers are referenced in main.c, and even so, only the initialization function is called. There is no evidence that main.c actually uses any of the managers in any substantial way. For example, even though it includes the QuickDraw header, most of the drawing routine is in main.c itself.

What's in a main?

So what does the main.c actually do?

  • Writing to the serial console using outb.
  • The serial port input is used to simulate GUI interaction, e.g. 'm' activates the menu, 'a' activates the Apple menu.
  • Draws console text in VGA mode.
  • Multiboot2 handover.
  • Drawing of text, apple logo, window, rectangle, and icon.
  • Fake initialization of managers.
  • There is some mouse handling using hard-coded coordinates to map cursor to UI elements. But this is commented out.

In summary, we just get a very thin veneer of what is minimally required to draw some UI in QEMU and using serial port to interact with it.

Conclusion

This is perhaps not the most flattering case study of using LLM to vibe code. AI faithfully created a bunch of dead code, but what was actually working is a small amount of code that is minimally required to create the appearance of a working operating system.

This whole thing reminds me of Theranos. Its CEO, Elizabeth Holmes, charmed her investors into believing that she invented a miracle medical diagnostics machine that purported to be able to run a bunch of tests using only a single drop of blood. In the few demos they did, they actually drew vials of blood using the normal process and ran the regular lab tests behind the scenes. The machine never worked.

It took Theranos (timeline) from rising to fame in 2015 to fraud indictment in 2018. ChatGPT was first publicly released in 2022. How many more years will people discover that AI is fraud? Probably never, unless somehow AI starts to put people's lives in danger, but not out of malice, just incompetence.