Consider this hypothetical scenario. One day, a coworker suddenly becomes hyper-productive. He is simultaneously handling production on-call duties yet cranking out pull requests for 100s lines of code every 10 minutes. He has an IT background, not computer science.
But when you look at his code, something seems off.
- It looks messy and unnecessarily complicated for what it tries to do, and contains incoherent style every few lines of code.
- People can write code that is unnecessarily complicated, but that's typically coming from someone who is dogmatic about their style and thought process.
- Even though the code obeys coding style rules, the style guide still gives a lot of leeway for personal discretion.
- The comments mansplain what the code is doing, rather than why it is done like that.
- This is typically a beginner's mistake. As software engineers become more experienced, they write the code to be self-explanatory but only write comments about the rationale.
- Each pull request makes a lot of unnecessary changes, such as renaming variables.
- Renaming a variable is rarely done by the same person because people feel attached to the names they have given, to a point where they might feel defensive about it.
- It is also a pain because you might break existing variable references. Automated tools exist, but it is still work, and people don't like doing work unless it is absolutely necessary.
What am I saying? It is strongly suggestive that your coworker is using AI to write code.
You are a competent software engineer, and on a good day, you can probably only write 100s lines in an hour. But not simultaneously when you might have to monitor production alerts and supervise rollouts. Furthermore, even though you can read the pull requests at the speed they are coming, you cannot type your review comments as fast.
What do you do? You try the AI code review tool, but they only spot minor surface level issues and generally say the code is okay to merge.
One of the pull requests supposedly fixes some data truncation issue. You know this typically means the tail end of the data was not flushed properly from the buffer before a file is closed. But the proposed fix would split one step in the data pipeline into two steps, with extra copies of the data being made as a result. It does not explain how it fixes the data truncation. It just happens to work.
Looking deeply, you realize that even the code appears to finalize the data, it defers the finalization to run when that multi-hundred lines long function returns. The function uploads the data to a server before it has a chance to finalize it.
You comment your finding on the pull request, and figure you would try the AI code review tool again. This time, it magically figures out the same problem that you commented on, without crediting your work.
There are still some obvious issues about the incoherent coding style, the complexity, the unhelpful comments, and the unnecessary changes, but you have no energy to deal with that right now. You just merge the rest of the pull requests because it would have taken too long to write the comments to point them out.
After all, you still have your own pull requests that you hope to send out by the end of the day. You were informed the day before that your position will be eliminated next month due to organizational strategy. Your coworker gets to keep his job.
There is a lot to unpack here, but there are some obvious predictions we can make when this scenario becomes more prevalent.
- Organizations who value productivity will encourage more use of AI without considering the impact on its code base.
- The code base will become increasingly unmaintainable by manual effort, and it will require more use of AI, as a self-fulfilling prophecy.
- The institutional knowledge why the code was written will be lost, resulting in a collapse of human knowledge.
- Increased complexity of code also means they run less efficiently, consuming more electricity yet run more slowly on fast hardware.
- More electricity consumption means higher carbon footprint from the power generation, resulting in the worsening of global warming and climate change.
- Data centers running AI operated by powerful conglomerates consume so much electricity that makes electricity more expensive.
This case study provides a concrete example how the indiscriminate use of AI causes problems. However, AI is not inherently bad. It just exacerbates existing societal problems.
- Many organizations already measure productivity by lines of code, the number of pull requests or issues solved, or other artificial metrics that are easy to measure.
- There are metrics that measure the efficacy of the code: latency, throughput, cycle/ops counting, cost (or any sort of resource usage). But they are not as easy to measure, so organizations that focus on growth will often deprioritize them.
- Human knowledge is already susceptible to degradation over a game of telephones, not to mention fake news or the desire to chase glory and fame over the desire to seek the truth.
- Externalities from economic activities result in consequences on a third-party, whose cost is not captured by the economic transaction.
- Unfair distribution of resources that disproportionally impacts the poor.
There are things we can work on:
- Organizational leaders need to be more thoughtful about their organizational values and how to reward them. You reap what you sow.
- We need to figure out how to have AI contribute to the pool of scientific knowledge and humanities insight, not just leech off existing ones.
- Computer science has focused too much on generative problems. We need to solve more optimization problems, like how compilers optimize code, but it needs to be done at an architectural level.
- Laws should continuously address economic externalities and unfair distribution of resources.
I think it is encouraging that, at least for the moment, courts are catching legal proceedings that have been hallucinated by AI and have reprimanded lawyers for submitting them. This means that when it comes to the matter of legal rights, the judges are still holding them to the highest level.
The day AI takes over humanity is when humans stop caring.
