With the rise of the internet came the need to find information more quickly. The concept of search engines came into this space to fill this need, with a relatively basic initial design.
This is the basis of the giant megacorp Google, whose claim to fame was they made the best one of these. Into this stack they inject ads, both ads inside the sites themselves and then turning the search results themselves into ads.
As time went on, what we understood to be "Google search" was actually a pretty sophisticated machine that effectively determined what websites lived or died. It was the only portal that niche websites had to get traffic. Google had the only userbase large enough for a website dedicated to retro gaming or VR headsets or whatever to get enough clicks to pay their bills.
Despite the complexity, the basic premise remained. Google steers traffic towards your site, the user gets the answer from your site and then everyone is happy. Google showed some ads, you showed some ads, everyone showed everyone on Earth ads.
This incredibly lucrative setup was not enough, however, to drive endless continous growth, which is now the new expectation of all tech companies. It is not enough to be fabulously profitable, you must become Weyland-Yutani. So now Google is going to break this long-standing agreement with the internet and move everything we understand to be "internet search" inside their silo.
Zero-Click Results
In March 2024 Google moved to embed LLM answers in their search results (source). The AI Overview takes the first 100 results from your search query, combines their answers and then returns what it thinks is the best answer. As expected, websites across the internet saw a drop in traffic from Google. You started to see a flood of smaller websites launch panic membership programs, sell off their sites, etc.
It became clear that Google has decided to abandon the previous concept of how internet search worked, likely in the face of what it considers to be an existential threat from OpenAI. Maybe the plan was always to bring the entire search process in-house, maybe not, but OpenAI and its rise to fame seems to have forced Google's hand in this space.
This is not a new thing, Google has been moving in this direction for years. It was a trend people noticed going back to 2019.
It appears the future of Google Search is going to be a closed loop that looks like the following:
Google LLM takes the information from the results it has already ingested to respond to most questions.
Companies will at some point pay for their product or service to be "the answer" in different categories. Maybe this gets disclosed, maybe not, maybe there's just a little i in the corner that says "these answers may be influenced by marketing partners" or something.
Google will attempt to reassure strategic partners that they aren't going to kill them, while at the same time turning to their relationship with Reddit to supply their "new data".
This is all backed up by data from outside the Google ecosystem confirming that the ratio of scrapes to click is going up. Basically it's costing more for these services to make their content available to LLMs and they're getting less traffic from them.
This new global strategy makes sense, especially in the context of the frequent Google layoffs. Previously it made strategic sense to hold onto all the talent they could, now it doesn't matter because the gates are closing. Even if you had all the ex-Google engineers money could buy, you can't make a better search engine because the concept is obsolete. Google has taken everything they need from the internet, it no longer requires the cooperation or goodwill of the people who produce that content.
What happens next?
So the source of traffic for the internet is going to go away. My guess is there will be some effort to prevent this, some sort of alternative Google search either embraced or pushed by people. This is going to fail, because Google is an unregulated monopoly. Effectively because the US government is so bad at regulating companies and so corrupt with legalized bribery in the form of lobbying, you couldn't stop Google at this point even if you wanted to.
Android is the dominant mobile platform on Earth
Chrome is the dominant web browser
Apple gets paid to make the other mobile platform default to Google
Firefox gets paid to make the other web browser default to Google
Even if you wanted to and had a lot of money to throw at the problem, it's too late. If Apple made their own search engine and pointed iOS to it as the default and paid Firefox to make it the default, it still wouldn't matter. The AI Overview is a good enough answer for most questions and so convincing consumers to:
switch platforms
and go back to a two/three/four step process compared to a one step process is a waste of time.
I'm confident there will still be sites doing web searching, but I suspect given the explosion in AI generated slop it's going to be impossible to use them even if you wanted to. We're quickly reaching a point where it would be possible to generate a web page on demand, meaning the capacity of the slop-generation exceeds the capacity of humans to fight it.
Because we didn't regulate the internet, we're going to end up with an unbreakable monopoly on all human knowledge held by Microsoft and Google. Then because we didn't learn anything we're going to end up with a system that can produce false data on demand and make it impossible to fact check anything that the LLM companies return. Paid services like Kogi will be the only search engines worth trying.
Impact down the line
So I think you are going to see a rush of shutdowns and paywalls like you've never seen before. In some respects, it is going to be a return to the pre-Google internet, where it will once again be important that consumers know your domain name and go directly to your site. It's going to be a massive consolidation of the internet down and I think the ad-based economy of the modern web will collapse. Google was the ad broker, but now they're going to operate like Meta and keep the entire cycle inside their system.
My prediction is that this is going to basically destroy any small or medium sized business that attempts to survive with the model of "produce content, get paid per visitor through ads". Everything instead is going to get moved behind aggressive paywalls, blocking archive.org. You'll also see prices go way up for memberships. Access to raw, human produced information is going to be a premium product, not something for everyday people. Fake information will be free.
Anyone attempting to make an online store is gonna get mob-style shakedown. You can either pay Amazon to let consumers see your product or you can pay Google to have their LLM recommend your product or you can (eventually) pay OpenAI/Microsoft to do it. I also think these companies will use this opportunity to dramatically reprice their advertising offerings. I don't think it'll be cheap to get the AI Summary to recommend your frying pan.
I suspect there will be a brief spike in other forms of marketing spend, like podcasts, billboards, etc. When companies see the sticker shock from Google they're going to explore other avenues like social media spend, influencers, etc. But all those channels are going to be eaten by the LLM snake at the same time.
If consumers are willing to engage with an LLM-generated influencer, that'll be the direction companies go in because they'll be cheaper and more reliable. Podcast search results are gonna be flooded with LLM-generated shows and my guess is that they're going to take more of the market share than anyone wants to admit. Twitch streaming has already moved from seeing the person to seeing an anime-style virtual overlay where you don't see the persons face. There won't be a reason for an actual human to be involved in that process.
End Game
My prediction is that a lot of the places that employ technical people are going to disappear. FAANG isn't going to be hiring at anywhere near the same rate they were before, because they won't need to. I don't need 10,000 people maintaining relationships with ad sellers and ad buyers or any of the staff involved in the maintenance or improvement of those systems.
The internet is going to return to more of its original roots, which are niche fan websites you largely find through social media or word of mouth. These sites aren't going to be ad driven, they'll be membership driven. Very few of them are going to survive. Subscription fatigue is a real thing and the math of "it costs a lot of money to pay people to write high quality content" isn't going to go away.
In a relatively short period of time, it will go from "very difficult" to absolutely impossible to launch a new commercially viable website and have users organically discover that website. You'll have to block LLM scrapers and need a tremendous amount of money to get a new site bootstrapped. Welcome to the future, where asking a question costs $4.99 and you'll never be able to find out if the answer is right or not.
Around 2012-2013 I started to hear a lot in the sysadmin community about a technology called "Borg". It was (apparently) some sort of Linux container system inside of Google that ran all of their stuff. The terminology was a bit baffling, with something called a "Borglet" inside of clusters with "cells" but the basics started to leak. There was a concept of "services" and a concept of "jobs", where applications could use services to respond to user requests and then jobs to complete batch jobs that ran for much longer periods of time.
Then on June 7th, 2014, we got our first commit of Kubernetes. The Greek word for 'helmsman' that absolutely no one could pronounce correctly for the first three years. (Is it koo-ber-NET-ees? koo-ber-NEET-ees? Just give up and call it k8s like the rest of us.)
Microsoft, RedHat, IBM, Docker join the Kubernetes community pretty quickly after this, which raised Kubernetes from an interesting Google thing to "maybe this is a real product?" On July 21st 2015 we got the v1.0 release as well as the creation of the CNCF.
In the ten years since that initial commit, Kubernetes has become a large part of my professional life. I use it at home, at work, on side projects—anywhere it makes sense. It's a tool with a steep learning curve, but it's also a massive force multiplier. We no longer "manage infrastructure" at the server level; everything is declarative, scalable, recoverable and (if you’re lucky) self-healing.
But the journey hasn't been without problems. Some common trends have emerged, where mistakes or misconfiguration arise from where Kubernetes isn't opinionated enough. Even ten years on, we're still seeing a lot of churn inside of ecosystem and people stepping on well-documented landmines. So, knowing what we know now, what could we do differently to make this great tool even more applicable to more people and problems?
What did k8s get right?
Let's start with the positive stuff. Why are we still talking about this platform now?
Containers at scale
Containers as a tool for software development make perfect sense. Ditch the confusion of individual laptop configuration and have one standard, disposable concept that works across the entire stack. While tools like Docker Compose allowed for some deployments of containers, they were clunky and still required you as the admin to manage a lot of the steps. I set up a Compose stack with a deployment script that would remove the instance from the load balancer, pull the new containers, make sure they started and then re-added it to the LB, as did lots of folks.
K8s allowed for this concept to scale out, meaning it was possible to take a container from your laptop and deploy an identical container across thousands of servers. This flexibility allowed organizations to revisit their entire design strategy, dropping monoliths and adopting more flexible (and often more complicated) micro-service designs.
Low-Maintenance
If you think of the history of Operations as a sort of "naming timeline from pets to cattle", we started with what I affectionately call the "Simpsons" era. Servers were bare metal boxes set up by teams, they often had one-off names that became slang inside of teams and everything was a snowflake. The longer a server ran, the more cruft it picked up until it became a scary operation to even reboot them, much less attempt to rebuild them. I call it the "Simpsons" era because among the jobs I was working at the time, naming them after Simpsons characters was surprisingly common. Nothing fixed itself, everything was a manual operation.
Then we transition into the "01 Era". Tools like Puppet and Ansible have become common place, servers are more disposable and you start to see things like bastion hosts and other access control systems become the norm. Servers aren't all facing the internet, they're behind a load balancer and we've dropped the cute names for stuff like "app01" or "vpn02". Organizations designed it so they could lose some of their servers some of the time. However failures still weren't self-healing, someone still had to SSH in to see what broke, write up a fix in the tooling and then deploy it across the entire fleet. OS upgrades were still complicated affairs.
We're now in the "UUID Era". Servers exist to run containers, they are entirely disposable concepts. Nobody cares about how long a particular version of the OS is supported for, you just bake a new AMI and replace the entire machine. K8s wasn't the only technology enabling this, but it was the one that accelerated it. Now the idea of a bastion server with SSH keys that I go to the underlying server to fix problems is seen as more of a "break-glass" solution. Almost all solutions are "destroy that Node, let k8s reorganize things as needed, make a new Node".
A lot of the Linux skills that were critical to my career are largely nice to have now, not need to have. You can be happy or sad about that, I certainly switch between the two emotions on a regular basis, but it's just the truth.
Running Jobs
The k8s jobs system isn't perfect, but it's so much better than the "snowflake cron01 box" that was an extremely common sight at jobs for years. Running on a cron schedule or running from a message queue, it was now possible to reliably put jobs into a queue, have them get run, have them restart if they didn't work and then move on with your life.
Not only does this free up humans from a time-consuming and boring task, but it's also simply a more efficient use of resources. You are still spinning up a pod for every item in the queue, but your teams have a lot of flexibility inside of the "pod" concept for what they need to run and how they want to run it. This has really been a quality of life improvement for a lot of people, myself included, who just need to be able to easily background tasks and not think about them again.
Service Discoverability and Load Balancing
Hard-coded IP addresses that lived inside of applications as the template for where requests should be routed has been a curse following me around for years. If you were lucky, these dependencies weren't based on IP address but were actually DNS entries and you could change the thing behind the DNS entry without coordinating a deployment of a million applications.
K8s allowed for simple DNS names to call other services. It removed an entire category of errors and hassle and simplified the entire thing down. With the Service API you had a stable, long lived IP and hostname that you could just point things towards and not think about any of the underlying concepts. You even have concepts like ExternalName that allow you to treat external services like they're in the cluster.
What would I put in a Kubernetes 2.0?
Ditch YAML for HCL
YAML was appealing because it wasn't JSON or XML, which is like saying your new car is great because it's neither a horse nor a unicycle. It demos nicer for k8s, looks nicer sitting in a repo and has the illusion of being a simple file format. In reality. YAML is just too much for what we're trying to do with k8s and it's not a safe enough format. Indentation is error-prone, the files don't scale great (you really don't want a super long YAML file), debugging can be annoying. YAML has so many subtle behaviors outlined in its spec.
I still remember not believing what I was seeing the first time I saw the Norway Problem. For those lucky enough to not deal with it, the Norway Problem in YAML is when 'NO' gets interpreted as false. Imagine explaining to your Norwegian colleagues that their entire country evaluates to false in your configuration files. Add in accidental numbers from lack of quotes, the list goes on and on. There are much better posts on why YAML is crazy than I'm capable of writing: https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell
Why HCL?
HCL is already the format for Terraform, so at least we'd only have to hate one configuration language instead of two. It's strongly typed with explicit types. There's already good validation mechanisms. It is specifically designed to do the job that we are asking YAML to do and it's not much harder to read. It has built-in functions people are already using that would allow us to remove some of the third-party tooling from the YAML workflow.
I would wager 30% of Kubernetes clusters today are already being managed with HCL via Terraform. We don't need the Terraform part to get a lot of the benefits of a superior configuration language.
The only downsides are that HCL is slightly more verbose than YAML, and its Mozilla Public License 2.0 (MPL-2.0) would require careful legal review for integration into an Apache 2.0 project like Kubernetes. However, for the quality-of-life improvements it offers, these are hurdles worth clearing.
Why HCL is better
Let's take a simple YAML file.
# YAML doesn't enforce types
replicas: "3" # String instead of integer
resources:
limits:
memory: 512 # Missing unit suffix
requests:
cpu: 0.5m # Typo in CPU unit (should be 500m)
Even in the most basic example, there are footguns everywhere. HCL and the type system would catch all of these problems.
replicas = 3 # Explicitly an integer
resources {
limits {
memory = "512Mi" # String for memory values
}
requests {
cpu = 0.5 # Number for CPU values
}
}
Take a YAML file like this that you probably have 6000 in your k8s repo. Now look at HCL without needing external tooling.
# Need external tools or templating for dynamic values
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
# Can't easily generate or transform values
DATABASE_URL: "postgres://user:password@db:5432/mydb"
API_KEY: "static-key-value"
TIMESTAMP: "2023-06-18T00:00:00Z" # Hard-coded timestamp
Loops and Iteration: Simplifying repetitive configurations
Better Comments: Improving documentation and readability
Error Handling: Making errors easier to identify and fix
Modularity: Enabling reuse of configuration components
Validation: Preventing invalid configurations
Data Transformations: Supporting complex data manipulations
Allow etcd swap-out
I know, I'm the 10,000 person to write this. Etcd has done a fine job, but it's a little crazy that it is the only tool for the job. For smaller clusters or smaller hardware configuration, it's a large use of resources in a cluster type where you will never hit the node count where it pays off. It's also a strange relationship between k8s and etcd now, where k8s is basically the only etcd customer left.
What I'm suggesting is taking the work of kine and making it official. It makes sense for the long-term health of the project to have the ability to plug in more backends, adding this abstraction means it (should) be easier to swap in new/different backends in the future and it also allows for more specific tuning depending on the hardware I'm putting out there.
What I suspect this would end up looking like is much like this: https://github.com/canonical/k8s-dqlite. Distributed SQlite in-memory with Raft consensus and almost zero upgrade work required that would allow cluster operators to have more flexibility with the persistence layer of their k8s installations. If you have a conventional server setup in a datacenter and etcd resource usage is not a problem, great! But this allows for lower-end k8s to be a nicer experience and (hopefully) reduces dependence on the etcd project.
Beyond Helm: A Native Package Manager
Helm is a perfect example of a temporary hack that has grown to be a permanent dependency. I'm grateful to the maintainers of Helm for all of their hard work, growing what was originally a hackathon project into the de-facto way to install software into k8s clusters. It has done as good a job as something could in fulfilling that role without having a deeper integration into k8s.
All that said, Helm is a nightmare to use. The Go templates are tricky to debug, often containing complex logic that results in really confusing error scenarios. The error messages you get from those scenarios are often gibberish. Helm isn't a very good package system because it fails at some of the basic tasks you need a package system to do, which are transitive dependencies and resolving conflicts between dependencies.
What do I mean?
Tell me what this conditional logic is trying to do:
# A real-world example of complex conditional logic in Helm
{{- if or (and .Values.rbac.create .Values.serviceAccount.create) (and .Values.rbac.create (not .Values.serviceAccount.create) .Values.serviceAccount.name) }}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: {{ template "myapp.fullname" . }}
labels:
{{- include "myapp.labels" . | nindent 4 }}
{{- end }}
Or if I provide multiple values files to my chart, which one wins:
Ok, what if I want to manage my application and all the application dependencies with a Helm chart. This makes sense, I have an application that itself has dependencies on other stuff so I want to put them all together. So I define my sub-charts or umbrella charts inside of my Chart.yaml.
But assuming I have multiple applications, it's entirely possible that I have 2 services both with a dependency on nginx or whatever like this:
Helm doesn't handle this situation gracefully because template names are global with their templates loaded alphabetically. Basically you need to:
Don't declare a dependency on the same chart more than once (hard to do for a lot of microservices)
If you do have the same chart declared multiple times, has to use the exact same version
The list of issues goes on and on.
Cross-Namespace installation stinks
Chart verification process is a pain and nobody uses it
Let's just go to the front page of artifacthub:
I'll grab elasticsearch cause that seems important.
Seems pretty bad for the Official Elastic helm chart. Certainly ingress-nginx will be right, it's an absolute critical dependency for the entire industry.
Nope. Also how is the maintainer of the chart "Kubernetes" and it's still not marked as a verified publisher. Like Christ how much more verified does it get.
No metadata in chart searching. You can only search by name and description, not by features, capabilities, or other metadata.
Helm doesn't strictly enforce semantic versioning
# Chart.yaml with non-semantic version
apiVersion: v2
name: myapp
version: "v1.2-alpha"
If you uninstall and reinstall a chart with CRDs, it might delete resources created by those CRDs. This one has screwed me multiple times and is crazy unsafe.
I could keep writing for another 5000 words and still wouldn't have outlined all the problems. There isn't a way to make Helm good enough for the task of "package manager for all the critical infrastructure on the planet".
What would a k8s package system look like?
Let's call our hypothetical package system KubePkg, because if there's one thing the Kubernetes ecosystem needs, it's another abbreviated name with a 'K' in it. We would try to copy as much of the existing work inside the Linux ecosystem while taking advantage of the CRD power of k8s. My idea looks something like this:
The packages are bundles like a Linux package:
There's a definition file that accounts for as many of the real scenarios that you actually encounter when installing a thing.
Like how great would it be to have something where I could automatically update packages without needing to do anything on my side.
apiVersion: kubepkg.io/v1
kind: Installation
metadata:
name: postgresql-main
namespace: database
spec:
packageRef:
name: postgresql
version: "14.5.2"
# Configuration values (validated against schema)
configuration:
replicas: 3
persistence:
size: "100Gi"
resources:
limits:
memory: "4Gi"
cpu: "2"
# Update policy
updatePolicy:
automatic: false
allowedVersions: "14.x.x"
schedule: "0 2 * * 0" # Weekly on Sunday at 2am
approvalRequired: true
# State management reference
stateRef:
name: postgresql-main-state
# Service account to use
serviceAccountName: postgresql-installer
What k8s needs is a system that meets the following requirements:
True Kubernetes Native: Everything is a Kubernetes resource with proper status and events
First-Class State Management: Built-in support for stateful applications
Enhanced Security: Robust signing, verification, and security scanning
Declarative Configuration: No templates, just structured configuration with schemas
Lifecycle Management: Comprehensive lifecycle hooks and upgrade strategies
Dependency Resolution: Linux-like dependency management with semantic versioning
Audit Trail: Complete history of changes with who, what, and when, not what Helm currently provides.
Policy Enforcement: Support for organizational policies and compliance.
Simplified User Experience: Familiar Linux-like package management commands. It seems wild that we're trying to go a different direction from the package systems that have worked for decades.
IPv6 By Default
Try to imagine, across the entire globe, how much time and energy has been invested in trying to solve any one of the following three problems.
I need this pod in this cluster to talk to that pod in that cluster.
There is a problem happening somewhere in the NAT traversal process and I need to solve it
I have run out of IP addresses with my cluster because I didn't account for how many you use. Remember: A company starting with a /20 subnet (4,096 addresses), deploys 40 nodes with 30 pods each, and suddenly realizes they're approaching their IP limit. Not that many nodes!
I am not suggesting the entire internet switches over to IPv6 and right now k8s happily supports IPv6-only if you want and a dualstack approach. But I'm saying now is the time to flip the default and just go IPv6. You eliminate a huge collection of problems all at once.
Flatter, less complicated network topology inside of the cluster.
The distinction between multiple clusters becomes a thing organizations can choose to ignore if they want if they want to get public IPs.
Easier to understand exactly the flow of traffic inside of your stack.
Built-in IPSec
It has nothing to do with driving IPv6 adoption across the entire globe and just an acknowledgement that we no longer live in a world where you have to accept the weird limitations of IPv4 in a universe where you may need 10,000 IPs suddenly with very little warning.
The benefits for organizations with public IPv6 addresses is pretty obvious, but there's enough value there for cloud providers and users that even the corporate overlords might get behind it. AWS never needs to try and scrounge up more private IPv4 space inside of a VPC. That's gotta be worth something.
Conclusion
The common rebuttal to these ideas is, "Kubernetes is an open platform, so the community can build these solutions." While true, this argument misses a crucial point: defaults are the most powerful force in technology. The "happy path" defined by the core project dictates how 90% of users will interact with it. If the system defaults to expecting signed packages and provides a robust, native way to manage them, that is what the ecosystem will adopt.
This is an ambitious list, I know. But if we're going to dream, let's dream big. After all, we're the industry that thought naming a technology 'Kubernetes' would catch on, and somehow it did!
We see this all the time in other areas like mobile developer and web development, where platforms assess their situation and make radical jumps forward. Not all of these are necessarily projects that the maintainers or companies would take on but I think they're all ideas that someone should at least revisit and think "is it worth doing now that we're this nontrivial percentage of all datacenter operations on the planet"?
So awhile ago I purchased a Tp-link AX3000 wireless router as a temporary same-day fix to a dying AP. Of course, like all temporary fixes, this one ended up being super permanent. It's a fine wireless router, nothing interesting to report, but one of the features I stumbled upon when I was clicking around the webUI seemed like a great solution for a place to stick random files.
Inside of Advanced Settings, you'll see this pane:
You have a few options for how to expose this USB drive:
I actually didn't find the SMB to work that well in my testing, seemingly disconnecting all the time. But FTP works pretty well. So that's what I ended up using, which was fine except seemingly randomly files were getting corrupted when I moved them over.
FAT32 will never die
Looking at the failing files, I realized they were all over 4 GB and thought "there's no way in 2025 they are formatting this external drive in FAT32, right?" To be clear, I didn't partition this drive. The router offered to wipe it when I plugged it in and I said sure.
However that is exactly what they are doing, which means we have a file size limit of 4 GB per file. This explained the transfer problems and, while still annoying, is not a complicated thing to work around.
Script to transfer stuff to the local FTP
Notes:
LOCAL_DIR will obviously need to get changed
FTP_HOST has a different IP range than the default router range because of specific stuff for me. You'll need to check that.
FTP_PASS required an email address format. I don't know why.
The directory of "G" was assigned to me by the router, so I assume this is a common convention with these routers. I don't know why it puts a directory inside of the drive instead of writing out to the root of the drive. Presumably some Windows convention.
#!/usr/bin/env python3
import os
import ftplib
import hashlib
import sys
import time
# --- Configuration ---
# Adjust these settings to match your environment
LOCAL_DIR = "/mnt/usb/test"
FTP_HOST = "192.168.86.1"
FTP_USER = "anonymous"
FTP_PASS = "[email protected]"
FTP_TARGET_DIR = "G"
# Do not transfer files larger than this size (4 GiB - 1 byte).
MAX_FILE_SIZE = 4294967295
# The size of chunks to use for reading and hashing files.
CHUNK_SIZE = 8192
# --- Helper Functions ---
def get_file_hash(file_path):
sha256 = hashlib.sha256()
try:
with open(file_path, "rb") as f:
# Read the file in chunks to handle large files efficiently
for byte_block in iter(lambda: f.read(CHUNK_SIZE), b""):
sha256.update(byte_block)
return sha256.hexdigest()
except IOError as e:
print(f" - Error reading file for hashing: {e}")
return None
def ftp_makedirs(ftp, path):
"""Recursively creates a directory structure on the FTP server."""
parts = path.strip('/').split('/')
current_dir = ''
for part in parts:
current_dir += '/' + part
try:
ftp.mkd(current_dir)
print(f" - Created remote directory: {current_dir}")
except ftplib.error_perm as e:
# Error 550 often means the directory already exists.
if "550" in str(e):
pass
else:
print(f" - FTP error while creating directory {current_dir}: {e}")
raise
def upload_and_verify(ftp, local_path, remote_filename):
"""
Uploads a file, verifies its integrity via hashing, and deletes
the local file upon successful verification.
"""
print(f" - Calculating hash for local file: {local_path}")
local_hash = get_file_hash(local_path)
if not local_hash:
return False
print(f" - Uploading to '{ftp.pwd()}/{remote_filename}'...")
try:
with open(local_path, 'rb') as f:
ftp.storbinary(f'STOR {remote_filename}', f, CHUNK_SIZE)
print(" - Upload complete.")
except ftplib.all_errors as e:
print(f" - !!! Upload failed: {e}")
return False
print(" - Verifying remote file integrity...")
remote_hash = ""
try:
sha256_remote = hashlib.sha256()
ftp.retrbinary(f'RETR {remote_filename}', sha256_remote.update, CHUNK_SIZE)
remote_hash = sha256_remote.hexdigest()
except ftplib.all_errors as e:
print(f" - !!! Verification failed. Could not download remote file: {e}")
return False
# 4. Compare hashes and delete local file if they match
print(f" - Local Hash: {local_hash}")
print(f" - Remote Hash: {remote_hash}")
if local_hash == remote_hash:
print(" - ✅ Integrity check PASSED. Hashes match.")
try:
os.remove(local_path)
print(f" - Successfully deleted local file: {local_path}")
return True
except OSError as e:
print(f" - !!! Error deleting local file: {e}")
return False
else:
print(" - ❌ Integrity check FAILED. Hashes do not match.")
print(" - Deleting corrupt file from FTP server...")
try:
ftp.delete(remote_filename)
print(f" - Remote file '{remote_filename}' deleted.")
except ftplib.all_errors as e:
print(f" - !!! Could not delete corrupt remote file: {e}")
return False
def main():
print("Starting FTP transfer process...")
if not os.path.isdir(LOCAL_DIR):
print(f"Error: Local directory '{LOCAL_DIR}' does not exist.")
sys.exit(1)
print("\n--- Pass 1: Scanning for oversized files ---")
dirs_to_skip = set()
for root, _, files in os.walk(LOCAL_DIR):
for filename in files:
file_path = os.path.join(root, filename)
try:
if os.path.getsize(file_path) > MAX_FILE_SIZE:
print(f"Oversized file found: {file_path}")
print(f" -> Marking directory for skipping: {root}")
dirs_to_skip.add(root)
break
except OSError as e:
print(f"Could not access {file_path}: {e}")
print("--- Pass 1 Complete ---")
# --- Pass 2: Connect to FTP and transfer valid files ---
print("\n--- Pass 2: Connecting to FTP and transferring files ---")
try:
with ftplib.FTP(FTP_HOST, timeout=30) as ftp:
ftp.login(FTP_USER, FTP_PASS)
print(f"Successfully connected to {FTP_HOST}.")
# Create and change to the base target directory
ftp_makedirs(ftp, FTP_TARGET_DIR)
ftp.cwd(FTP_TARGET_DIR)
print(f"Changed to remote base directory: {ftp.pwd()}")
for root, dirs, files in os.walk(LOCAL_DIR, topdown=True):
# Check if the current directory is in our skip list
if any(root.startswith(skip_dir) for skip_dir in dirs_to_skip):
print(f"\n⏭️ Skipping directory and its contents: {root}")
dirs[:] = [] # Prune subdirectories from the walk
continue
print(f"\nProcessing directory: {root}")
# Create the corresponding directory structure on the FTP server
remote_subdir = os.path.relpath(root, LOCAL_DIR)
remote_path = ftp.pwd()
if remote_subdir != '.':
remote_path = os.path.join(ftp.pwd(), remote_subdir).replace("\\", "/")
ftp_makedirs(ftp, remote_path)
# Process each file in the current valid directory
for filename in files:
local_file_path = os.path.join(root, filename)
if os.path.getsize(local_file_path) > MAX_FILE_SIZE:
continue
print(f"\n📂 Processing file: {filename}")
ftp.cwd(remote_path)
upload_and_verify(ftp, local_file_path, filename)
except ftplib.all_errors as e:
print(f"\nFTP Error: {e}")
except Exception as e:
print(f"\nAn unexpected error occurred: {e}")
print("\nProcess complete.")
if __name__ == "__main__":
main()
Now I have an easy script that I can use to periodically offload directories full of files that I don't know if I want to delete yet, but I don't want on my laptop.
In general, Ghost CMS has been a good tool for me. I've been pleased by the speed and reliability of the platform, with the few problems I have run into being fixed by the Ghost team pretty quickly. From the very beginning though I've struggled with the basic approach of the Ghost platform.
At its core, the Ghost CMS tool is a newsletter platform. This makes sense, it's how small content creators actually generate revenue. But I don't need any of that functionality, as I don't want to capture a bunch of users email addresses. I'm lucky enough to not need the $10 a month it costs to host this website on my own and I'd rather not have to think about who I would need to notify if my database got breached.
But it means that most of the themes for Ghost are completely cluttered with junk I don't need. I started working on my own CMS, but other than the more simplistic layout, I couldn't think of anything my CMS did that was better than Ghost or Wordpress. There was less code, but it was code I was going to have to maintain. After going through the source for a bunch of Ghost themes, I realized I could probably get where I wanted to go through the theme work alone.
I didn't find a ton of resources on how to actually crank out a theme, so I figured I would write up the base outline I sketched out as I worked.
Make your own Ghost theme
So Ghost uses the Handlebars library to make templates. Here's the basic layout:
package.json(required): The theme's "ID card." This JSON file contains metadata like the theme's name, version, author, and crucial configuration settings such as the number of posts per page.
default.hbs(optional but probably required): The main base template. Think of it as the master "frame" for your site. It typically contains the , , tags, your site-wide header and footer, and the crucial {{ghost_head}} and {{ghost_foot}} helpers. All other templates are injected into the {{{body}}} tag of this file.
index.hbs(required): The main template for listing your posts. It's used for your homepage by default and will also be used for tag and author archives if tag.hbs and author.hbs don't exist. It uses the {{#foreach posts}} helper to loop through and display your articles.
post.hbs(required): The template for a single post. When a visitor clicks on a post title from your index.hbs page, Ghost renders the content using this file. It uses the {{#post}} block helper to access all the post's data (title, content, feature image, etc.).
/partials/ (directory): This folder holds reusable snippets of template code, known as partials. It's perfect for elements that appear on multiple pages, like your site header, footer, sidebar, or a newsletter sign-up form. You include them in other files using {{> filename}}
/assets/ (directory): This is where you store all your static assets. It's organized into sub-folders for your CSS stylesheets, JavaScript files, fonts, and images used in the theme's design. You link to these assets using the {{asset}} helper (e.g., {{asset "css/screen.css"}}).
page.hbs (Optional): A template specifically for static pages (like an "About" or "Contact" page). If this file doesn't exist, Ghost will use post.hbs to render static pages instead.
tag.hbs (Optional): A dedicated template for tag archive pages. When a user clicks on a tag, this template will be used to list all posts with that tag. If it's not present, Ghost falls back to index.hbs.
author.hbs (optional): A dedicated template for author archive pages. This lists all posts by a specific author. If it's not present, Ghost falls back to index.hbs
How It All Fits Together: The Template Hierarchy
Ghost uses a logical hierarchy to decide which template to render for a given URL. This allows you to create specific designs for different parts of your site while having sensible defaults.
The Request: A visitor goes to a URL on your site (e.g., your homepage, a post, or a tag archive).
Context is Key: Ghost determines the "context" of the URL. Is it the homepage? A single post? A list of posts by an author?
Find the Template: Ghost looks for the most specific template file for that context.
Visiting /tag/travel/? Ghost looks for tag.hbs. If it doesn't find it, it uses index.hbs.
Visiting a static page like /about/? Ghost looks for page.hbs. If it's not there, it uses post.hbs.
Inject into the Frame: Once the correct template is found (e.g., post.hbs), Ghost renders it and injects the resulting HTML into the {{{body}}} helper inside your default.hbs file.
This system provides a clean separation of concerns, making your theme easy to manage and update. You can start with just the three required files (package.json, index.hbs, post.hbs) and add more specific templates as your design requires them.
Source code for this theme
You are more than welcome to use this theme as a starting point. The only part that was complex was the "Share with Mastodon" button that you see, which frankly I'm still not thrilled with. I wish there was a less annoying way to do it than prompting the user for their server, but I can't think of anything.
So Ghost actually has an amazing checking tool for seeing if your theme will work available here: https://gscan.ghost.org/. It tells you all the problems and missing pieces from your theme and really helped me iterate quickly on the design. Just zip up the theme, upload it and you'll get back a nicely formatted list of problems.
Anyway I found the process of writing my own theme to be surprisingly fun. Hopefully folks like how it looks, but if you hate it I'm still curious to hear why.
I generate a lot of CSVs for my jobs, mostly as a temporary storage mechanism for data. So I make report A about this thing, I make report B for that thing and then I produce some sort of consumable report for the organization at large. Part of this is merging the CSVs so I don't need to overload each scripts to do all the pieces.
For a long time I've done this in Excel/LibreOffice, which totally works. But I recently sat down with the pandas library and I had no idea how easy it is use for this particular use case. Turns out this is a pretty idiot-proof way to do the same thing without needing to deal with the nightmare that is Excel.
Steps to Run
Make sure Python is installed
Run python3.13 -m venv venv
source venv/bin/activate
pip install pandas
Change file_one to the first file you want to consider. Same with file_two
The most important thing to consider here: I only want the output if the value in the column is in BOTH files. If you want all the output from file_one and then enrich it with the values from file_two if it is present, change how='inner' to how='left'
import pandas as pd
import os
# Define the filenames
file_one = 'one.csv'
file_two = 'two.csv'
output_file = 'combined_report.csv'
# Define the column names to use for joining
# These should match the headers in your CSVs exactly
deploy_join_col = 'Deployment Name'
stacks_join_col = 'name'
try:
# Check if input files exist
if not os.path.exists(file_one):
raise FileNotFoundError(f"Input file not found: {file_one}")
if not os.path.exists(file_two):
raise FileNotFoundError(f"Input file not found: {file_two}")
# Read the CSV files into pandas DataFrames
print(f"Reading {file_one}...")
df_deploy = pd.read_csv(file_one)
print(f"Read {len(df_deploy)} rows from {file_one}")
print(f"Reading {file_two}...")
df_stacks = pd.read_csv(file_two)
print(f"Read {len(df_stacks)} rows from {file_two}")
# --- Data Validation (Optional but Recommended) ---
if deploy_join_col not in df_deploy.columns:
raise KeyError(f"Join column '{deploy_join_col}' not found in {file_one}")
if stacks_join_col not in df_stacks.columns:
raise KeyError(f"Join column '{stacks_join_col}' not found in {file_two}")
# ----------------------------------------------------
# Perform the inner merge based on the specified columns
# 'how="inner"' ensures only rows with matching keys in BOTH files are included
# left_on specifies the column from the left DataFrame (df_deploy)
# right_on specifies the column from the right DataFrame (df_stacks)
print(f"Merging dataframes on '{deploy_join_col}' (from deployment) and '{stacks_join_col}' (from stacks)...")
df_combined = pd.merge(
df_deploy,
df_stacks,
left_on=deploy_join_col,
right_on=stacks_join_col,
how='inner'
)
print(f"Merged dataframes, resulting in {len(df_combined)} combined rows.")
# Sort the combined data by the join column for grouping
# You can sort by either join column name as they are identical after the merge
print(f"Sorting combined data by '{deploy_join_col}'...")
df_combined = df_combined.sort_values(by=deploy_join_col)
print("Data sorted.")
# Write the combined and sorted data to a new CSV file
# index=False prevents pandas from writing the DataFrame index as a column
print(f"Writing combined data to {output_file}...")
df_combined.to_csv(output_file, index=False)
print(f"Successfully created {output_file}")
except FileNotFoundError as e:
print(f"Error: {e}")
except KeyError as e:
print(f"Error: Expected column not found in one of the files. {e}")
print(f"Please ensure the join columns ('{deploy_join_col}' and '{stacks_join_col}') exist and are spelled correctly in your CSV headers.")
except Exception as e:
print(f"An unexpected error occurred: {e}")
Just a super easy to hook up script that has saved me a ton of time from having to muck around with Excel.
The impact of Large Language Models (LLMs) on the field of software development is arguably one of the most debated topics in developer circles today, sparking discussions at meetups, in lunchrooms, and even during casual chats among friends. I won't attempt to settle that debate definitively in this post, largely because I lack the foresight required. My track record for predicting the long-term success or failure of new technologies is, frankly, about as accurate as a coin flip. In fact, if I personally dislike a technology, it seems destined to become an industry standard.
However, I do believe I'm well-positioned to weigh in on a much more specific question: Is GitHub Copilot beneficial for me within my primary work environment, Vim? I've used Vim extensively as my main development tool for well over a decade, spending roughly 4-5 hours in it daily, depending on my meeting schedule. My work within Vim involves a variety of technologies, including significant amounts of Python, Golang, Terraform, and YAML. Therefore, while I can't provide a universal answer to whether an LLM is right for you, I can offer concrete opinions based on my direct experience with GitHub Copilot as a dedicated Vim user today.
Testing
So just to prove I really set it up:
It's a real test, I've been using it every day for this time period. I have it set up in what I believe to be the "default configuration".
The plugin uses Vimscript to capture the current state of the editor. That includes stuff like:
The entire content of the current buffer (the file being edited).
The current cursor position within the buffer.
The file type or programming language of the current buffer.
The Node.js language server receives the request from the Vim/Neovim plugin. It processes the provided context and constructs a request to the actual GitHub Copilot API running on GitHub's servers. This request includes the code context and other relevant information needed by the Copilot AI model to generate suggestions.
The plugin receives the suggestions from the language server. It then integrates these suggestions into the Vim or Neovim interface, typically displaying them as "ghost text" inline with the user's code or in a separate completion window, depending on the plugin's configuration and the editor's capabilities.
How it feels to use
As you can tell from the output of vim --startuptime vim.log the plugin is actually pretty performant and doesn't add a notable time to my initial launch.
In terms of the normal usage, it works like it says on the box. You start typing and it shows the next line it thinks you might be writing.
The suggestions don't do much on their own. Basically the tool isn't smart enough to even keep track of what it has already suggested. So in this case I've just tab completed and taken all the suggestions and you can tell it immediately gets stuck in a loop.
Now you can use it to "vibe code" inside of Vim. That works by writing a comment describing what you want to do and then just tab accepting the whole block of code. So for example I wrote Write a new function to check if the JWT is encrypted or not. It produced the following.
So I made a somewhat misleading comment on purpose. I was trying to get it to write a function to see if a JWT was actually a JWE. Now this python code is (obviously) wrong. The code is_jwt_encrypted assumes the token will always have exactly three parts separated by dots (header, payload, signature). This is the structure of a standard JSON Web Token (JWT). However, a JSON Web Encryption (JWE), which is what a wrapped encrypted JWT is, has five parts:
Protected Header
Encrypted Key
Initialization Vector
Ciphertext
Authentication Tag
So this gives you a rough idea of the quality of the code snippets it produces. If you are writing something dead simple, the autogenerate will often work and can save you time. However go even a little bit off the golden path and, while Copilot will always give it a shot, the quality is all over the place.
Scores Based on Common Tasks
Reviewing a product like this is extremely hard because it does everything all the time and changes daily with no notice. I've had weeks where it seems like the Copilot intelligence gets cranked way up and weeks where its completely brain dead. However I will go through some common tasks I have to do all the time and rank it on how well it does.
Parsing JSON
90/100
This is probably the thing Copilot is best at. You have a JSON that you are getting from some API and then Copilot helps you fill in the parsing for that so you don't need to type the whole thing out. So just by filling in my imports it already has a good idea of what I'm thinking about here.
So in this example I write the comment with the example JSON object and then it fills in the rest. This code is....ok. However I'd like it to probably check the json_data to see if it matches the expectation before it parses. Changing the comment however changes the code.
This is very useful for me as someone who often needs to consume JSONs from source A and then send JSONs on to target B. Saves a lot of time and I think the quality looks totally acceptable to me. Some notes though:
Python Types greatly improve the quality of the suggestions
You need to check to make sure it doesn't truncate the list. Sometimes Copilot will "give up" like 80% through writing out all the items. It doesn't often make up ones, which is nice, but you do need to make sure everything you expected to be there ends up getting listed.
Database Operations
40/100
I work a lot with databases, like everyone on Earth does. Copilot definitely understands the concepts of databases but your experience can vary wildly depending on what you write and the mood it is in.
I mean this is sort of borderline making fun of me. Obviously I don't want to just check if the file named that exists?
This is better but it's still not good. If there is a file sitting there with the right name that isn't a database, sqlite3.connect will just make it. The except sqlite3.Error part is super shitty. Obviously that's not what I want to do. I probably want to at least log something?
Let me show another example. I wrote Write a method to create a table in the SQLite database if it does not already exist with the specified schema. Then I typed user_ID UUID and let it fill in the rest.
Not great. What it ended up making was even worse.
We're missing error handling, no try/finally blocks with the connection cursor, etc. This is pretty shitty code. My experience is it doesn't get much better the more you use. Some tips:
If you write out the SQL in the comments then you will have a way better time.
CREATE TABLE users (
contact_id INTEGER PRIMARY KEY,
first_name TEXT NOT NULL,
last_name TEXT NOT NULL,
email TEXT NOT NULL UNIQUE,
phone TEXT NOT NULL UNIQUE
);
Just that alone seems to make it a lot happier.
Still not amazing but at least closer to correct.
Writing Terraform
70/100
Not much to report with Terraform.
So why the 70/100? I've had a lot of frustrations with Copilot hallucinations with Terraform where it will simply insert arguments that don't exist. I can't reliably reproduce it, but this is something that can really burn a lot of time when you hit it.
My advice with Terraform is to run something like terrascan after which will often catch weird stuff it inserts. https://github.com/tenable/terrascan
However I will admit it saves me a lot of time, especially when writing stuff that is mind-numbing like 1000 DNS entries. So easily worth the risk on this one.
Tips:
Make sure you use the let g:copilot_workspace_folders = ['/path/to/my/project1', '/path/to/another/project2']
That seems to ground the LLM with the rest of the code and allows it to detect things like "what is the cloud account you are using".
Writing Golang
0/100
This is a good summary of my experience with Copilot with Golang.
I don't know why. It will work fine for awhile and then at some point, roughly when the golang file hits around 300-400 lines, seems to just lose it. Maybe there's another plugin I have that's causing a problem with Copilot and Golang, maybe I'm holding it wrong, I have no idea.
There's nothing in the logs I can find that would explain why it seems to break on Golang. I'm not going to file a bug report because I don't consider this my job to fix.
Summary
Is Copilot worth $10 a month? I think that really depends on what your day looks like. If you are someone who is:
Writing microservices where the total LoC rarely exceeds 1000 per microservice
Spends a lot of your time consuming and producing JSONs for other services to receive
Are capable of checking SQL queries and confirming how they need to be fixed
Has good or great test coverage
Then I think this tool might be worth the money. However if your day looks like this:
Spends most of your day inside of a monolith or large codebase carefully adding new features or slightly modifying old features
Doesn't have any or good test coverage
Doesn't have a good database migration strategy.
I'd say stay far away from Copilot for Vim. It's going to end up causing you serious problems that are going to be hard to catch.
My first formal IT helpdesk role was basically "resetting stuff". I would get a ticket, an email or a phone call and would take the troubleshooting as far as I could go. Reset the password, check the network connection, confirm the clock time was right, ensure the issue persisted past a reboot, check the logs and see if I could find the failure event, then I would package the entire thing up as a ticket and escalate it up the chain.
It was effectively on the job training. We were all trying to get better at troubleshooting to get a shot at one of the coveted SysAdmin jobs. Moving up from broken laptops and desktops to broken servers was about as big as 22 year old me dreamed.
This is not what we looked like but how creepy is this photo?
Sometimes people would (rightfully) observe that they were spending a lot of time interacting with us, while the more senior IT people were working quietly behind us and they could probably fix the issue immediately. We would explain that, while that was true, our time was less valuable than theirs. Our role was to eliminate all of the most common causes of failure then to give them the best possible information to take the issue and continue looking at it.
There are people who understand waiting in a line and there are people who make a career around skipping lines. These VIPs encountered this flow in their various engineering organizations and decided that a shorter line between their genius and the cogs making the product was actually the "secret sauce" they needed.
Thus, Slack was born, a tool pitched to the rank and file as a nicer chat tool and to the leadership as a all-seeing eye that allowed them to plug directly into the nervous system of the business and get instant answers from the exact right person regardless of where they were or what they were doing.
My job as a professional Slacker
At first Slack-style chat seemed great. Email was slow and the signal to noise ratio was off, while other chat systems I had used before at work either didn't preserve state, so whatever conversation happened while you were offline didn't get pushed to you, or they didn't scale up to large conversations well. Both XMPP and IRC has the same issue, which is if you were there when the conversation was happening you had context, but otherwise no message history for you.
There were attempts to resolve this (https://xmpp.org/extensions/xep-0313.html) but support among clients was all over the place. The clients just weren't very good and were constantly going through cycles of intense development only to be abandoned. It felt like when an old hippie would tell you about Woodstock. "You had to be there, man".
Slack brought channels and channels bought a level of almost voyeurism into what other teams were doing. I knew exactly what everyone was doing all the time, down to I knew where the marketing team liked to go for lunch. Responsiveness became the new corporate religion and I was a true believer. I would stop walking in the hallway to respond to a DM or answer a question I knew the answer to, ignoring the sighs of frustration as people walked around my hoodie-clad roadblock of a body.
Sounds great, what's the issue?
So what's the catch? Well I first noticed it on the train. My daily commute home through the Chicago snowy twilight used to be a sacred ritual of mental decompression. A time to sift through the day's triumphs and (more often) the screw-ups. What needed fixing tomorrow? What problem had I pushed off maybe one day too long?
But as I got further and further into Slack, I realized I was coming home utterly drained yet strangely...hollow. I hadn't done any actual work that day.
My days had become a never-ending performance of "work". I was constantly talking about the work, planning the work, discussing the requirements of the work, and then in a truly Sisyphean twist, linking new people to old conversations where we had already discussed the work to get them up to speed on our conversation. All the while diligently monitoring my channels, a digital sentry ensuring no question went unanswered, no emoji not +1'd. That was it, that was the entire job.
Look I helped clean up (Martin Parr)
Show up, spend eight hours orchestrating the idea of work, and then go home feeling like I'd tried to make a sandcastle on the beach and getting upset when the tide did what it always does. I wasn't making anything, I certainly wasn't helping our users or selling the product. I was project managing, but poorly, like a toddler with a spreadsheet.
And for the senior engineers? Forget about it. Why bother formulating a coherent question for a team channel when you could just DM the poor bastard who wrote the damn code in the first place? Sure, they could push back occasionally, feigning busyness or pointing to some obscure corporate policy about proper channel etiquette. But let's be real. If the person asking was important enough (read: had a title that could sign off on their next project), they were answering. Immediately.
So, you had your most productive people spending their days explaining why they weren't going to answer questions they already knew the answer to, unless they absolutely had to. It's the digital equivalent of stopping a concert pianist to teach you "Twinkle Twinkle Little Star" 6 times a day.
It's a training problem too
And don't even get me started on the junior folks. Slack was actively robbing them of the chance to learn. Those small, less urgent issues? That's where the real education happens. You get to poke around in the systems, see how the gears grind, understand the delicate dance of interconnectedness. But why bother troubleshooting when Jessica, the architect of the entire damn stack, could just drop the answer into a DM in 30 seconds? People quickly figured out the pecking order. Why wait four hours for a potentially wrong answer when the Oracle of Code was just a direct message away?
You think you are too good to answer questions???
Au contraire! I genuinely enjoy feeling connected to the organizational pulse. I like helping people. But that, my friends, is the digital guillotine. The nice guys (and gals) finish last in this notification-driven dystopia. The jerks? They thrive. They simply ignore the incoming tide of questions, their digital silence mistaken for deep focus. And guess what? People eventually figure out who will respond and only bother those poor souls. Humans are remarkably adept at finding the path of least resistance, even if it leads directly to someone else's burnout.
Then comes review time. The jerk, bless his oblivious heart, has been cranking out code, uninterrupted by the incessant digital demands. He has tangible projects to point to, gleaming monuments to his uninterrupted focus. The nice person, the one everyone loves, the one who spent half their day answering everyone else's questions? Their accomplishments are harder to quantify. "Well, they were really helpful in Slack..." doesn't quite have the same ring as "Shipped the entire new authentication system."
It's the same problem with being the amazing pull request reviewer. Your team appreciates you, your code quality goes up, you’re contributing meaningfully. But how do you put a number on "prevented three critical bugs from going into production"? You can't. So, you get a pat on the back and maybe a gift certificate to a mediocre pizza place.
Slackifying Increases
Time marches on, and suddenly, email is the digital equivalent of that dusty corner in your attic where you throw things you don't know what to do with. It's a wasteland of automated notifications from systems nobody cares about. But Slack? There’s no rhyme or reason to it. Can I message you after hours with the implicit understanding you'll ignore it until morning? Should I schedule the message for later, like some passive-aggressive digital time bomb?
And the threads! Oh, the glorious, nested chaos of threads. Should I respond in a thread to keep the main channel clean? Or should I keep it top-level so that if there's a misunderstanding, the whole damn team can pile on and offer their unsolicited opinions? What about DMs? Is there a secret protocol there? Or is it just a free-for-all of late-night "u up?" style queries about production outages?
It felt like every meeting had a pre-meeting in Slack to discuss the agenda, followed by an actual meeting on some other platform to rehash the same points, and then a post-meeting discussion in a private channel to dissect the meeting itself. And inevitably, someone who missed the memo would then ask about the meeting in the public channel, triggering a meta-post-meeting discussion about the pre-meeting, the meeting, and the initial post-meeting discussion.
The only way I could actually get any work done was to actively ignore messages. But then, of course, I was completely out of the loop. The expectation became this impossible ideal of perfect knowledge, of being constantly aware of every initiative across the entire company. It was like trying to play a gameshow and write a paper at the same time. To be seen as "on it", I needed to hit the buzzer and answer the question, but come review time none of those points mattered and the scoring was made up.
I was constantly forced to choose: stay informed or actually do something. If I chose the latter, I risked building the wrong thing or working with outdated information because some crucial decision had been made in a Slack channel I hadn't dared to open for fear of being sucked into the notification vortex. It started to feel like those brief moments when you come up for air after being underwater for too long. I'd go dark on Slack for a few weeks, actually accomplish something, and then spend the next week frantically trying to catch up on the digital deluge I'd missed.
Attention has a cost
One of the hardest lessons for anyone to learn is the profound value of human attention. Slack is a fantastic tool for those who organize and monitor work. It lets you bypass the pesky hierarchy, see who's online, and ensure your urgent request doesn't languish in some digital abyss. As an executive, you can even cut out middle management and go straight to the poor souls actually doing the work. It's digital micromanagement on steroids.
But if you're leading a team that's supposed to be building something, I'd argue that Slack and its ilk are a complete and utter disaster. Your team's precious cognitive resources are constantly being bled dry by a relentless stream of random distractions from every corner of the company. There are no real controls over who can interrupt you or how often. It's the digital equivalent of having your office door ripped off its hinges and replaced with glass like a zoo. Visitors can come and peer in on what your team is up to.
Turns out, the lack of history in tools like XMPP and IRC wasn't a bug, it was a feature. If something important needed to be preserved, you had to consciously move it to a more permanent medium. These tools facilitated casual conversation without fostering the expectation of constant, searchable digital omniscience.
Go look at the Slack for any large open-source project. It's pure, unadulterated noise. A cacophony of voices shouting into the void. Developers are forced to tune out, otherwise it's all they'd do all day. Users have a terrible experience because it's just a random stream of consciousness, people asking questions to other people who are also just asking questions. It's like replacing a structured technical support system with a giant conference call where everyone is on hold and told to figure it out amongst themselves.
My dream
So, what do I even want here? I know, I know, it's a fool's errand. We're all drowning in Slack clones now. You can't stop this productivity-killing juggernaut. It's like trying to un-ring a bell, or perhaps more accurately, trying to silence a thousand incessantly pinging notifications.
But I disagree. I still think it's not too late to have a serious conversation about how many hours a day it's actually useful for someone to spend on Slack. What do you, as a team, even want out of a chat client? For many teams, especially smaller ones, it makes far more sense to focus your efforts where there's a real payoff. Pick one tool, one central place for conversations, and then just…turn off the rest. Everyone will be happier, even if the tool you pick has limitations, because humans actually thrive within reasonable constraints. Unlimited choice, as it turns out, is just another form of digital torture.
Try to get away with the most basic, barebones thing you can for as long as you can. I knew a (surprisingly productive) team that did most of their conversation on an honest-to-god phpBB internal forum. Another just lived and died in GitHub with Issues. Just because it's a tool a lot of people talk about doesn't make it a good tool and just because it's old, doesn't make it useless.
As for me? I'll be here, with my Slack and Teams and Discord open trying to see if anything has happened in any of the places I'm responsible for seeing if something has happened. I will consume gigs of RAM on what, even ten years ago, would have been an impossibly powerful computer to watch basically random forum posts stream in live.
So one of the more annoying things about moving from the US to Europe is how much of the American communication infrastructure is still built around the idea that you have a US phone number to receive text messages from. While some (a vanishingly small) percentage of them allow me to add actual 2FA and bypass the insane phone number requirement, it's a constant problem to need to get these text messages.
There are services like Google Voice, but they're impossible to set up abroad. So you need to already have it set up when you land. Then if you forgot to use it, you'll lose the number and start the entire nightmare over again. Also increasingly services won't let you add a Google Voice number to get these text messages, I assume because of fraud. I finally got tired of it and did what I should have done when I first moved here, which is just buy a DID number.
Buy the DID Number
So I'm going to use voip.ms for this because it seems to work the best of the ones I tried.
Password is the account password we set before, username is the 6 digit number from the account information screen.
Ta-dah! We're all done. Make sure you can call a test number: 1-202-762-1401 returns the time. Text messages should now be coming in super easily and it will cost you less than $1 a month to keep this phone number forever.
Anybody who has worked in a tech stack of nearly any complexity outside of Hello World is aware of the problems with the current state of the open-source world. Open source projects, created by individuals or small teams to satisfy a specific desire they have or problem they want to solve, are adopted en masse by large organizations whose primary interest in consuming them are saving time and/or money. These organizations rarely contribute back to these projects, creating a chain of critical dependencies that are maintained inconsistently.
Similar to if your general contractor got cement from a guy whose hobby was mixing cement, the results are (understandably) all over the place. Sometimes the maintainer does a great job for awhile then gets bored or burned out and leaves. Sometimes the project becomes important enough that a vanishingly small percentage of the profit generated by the project is redirect back towards it and a person can eek out a meager existence keeping everything working. Often they're left in a sort of limbo state, being pushed forward by one or two people while the community exists in a primarily consumption role. Whatever stuff these two want to add or PRs they want to merge is what gets pushed in.
In the greater tech community, we have a lot of conversations about how we can help maintainers. Since a lot of the OSS community trends towards libertarian, the vibe is more "how can we encourage more voluntary non-mandated assistance towards these independent free agents for whom we bare no responsibility and who have no responsibility towards us". These conversations go nowhere because the idea of a widespread equal distribution of resources based on value without an enforcement mechanism is a pipe dream. The basic diagram looks like this:
+---------------------------------------------------------------+
| |
| "We need to support open-source maintainers better!" |
| |
+---------------------------------------------------------------+
|
v
+---------------------------------------------------------------+
| |
| "Let's have a conference to discuss how to help them!" |
| |
+---------------------------------------------------------------+
|
v
+---------------------------------------------------------------+
| |
| "We should provide resources without adding requirements." |
| |
+---------------------------------------------------------------+
|
v
+---------------------------------------------------------------+
| |
| "But how do we do that without more funding or time?" |
| |
+---------------------------------------------------------------+
|
v
+---------------------------------------------------------------+
| |
| "Let's ask the maintainers what they need!" |
| |
+---------------------------------------------------------------+
|
v
+---------------------------------------------------------------+
| |
| Maintainers: "We need more support and less pressure!" |
| |
+---------------------------------------------------------------+
|
v
+---------------------------------------------------------------+
| |
| "Great! We'll discuss this at the next conference!" |
| |
+---------------------------------------------------------------+
|
v
+---------------------------------------------------------------+
| |
| "We need to support open-source maintainers better!" |
| |
+---------------------------------------------------------------+
I've already read this post a thousand times
So we know all this. But as someone who uses a lot of OSS and (tries) to provide meaningful feedback and refinements back to the stuff I use, I'd like to talk about a different problem. The problem I'm talking about is how hard it is to render assistance to maintainers. Despite endless hours of people talking about how we should "help maintainers more", it's never been less clear what that actually means.
I, as a person, have a finite amount of time on this Earth. I want to help you, but I need the process to help you to make some sort of sense. It also has to have some sort of consideration for my time and effort. So I'd like to propose just a few things I've run into over the last few years I'd love if maintainers could do just to help me be of service to you.
If you don't want PRs, just say that. It's fine, but the number of times I have come across projects with a ton of good PRs just sitting there is alarming. Just say "we don't merge in non-maintainers PRs" and move on.
Don't automatically close bug reports. You are under zero ethical obligation to respond to or solve my bug report. But at the very least, don't close it because nobody does anything with it for 30 days. Time passing doesn't make it less real. There's no penalty for having a lot of open bug reports.
If you want me to help, don't make me go to seven systems. The number of times I've opened an issue on GitHub only to then have to discuss it on Discord or Slack and then follow-up with someone via an email is just a little maddening. If your stuff is on GitHub do everything there. If you want to have a chat community, fine I guess, but I don't want to join your tech support chat channel.
Archive when you are done. You don't need to explain why you are doing this to anyone on Earth, but if you are done with a project archive it and move on. You aren't doing any favors by letting it sit forever collecting bug reports and PRs. Archiving it says "if you wanna fork this and take it over, great, but I don't want anything to do with it anymore".
Provide an example of how you want me to contribute. Don't say "we prefer PRs with tests". Find a good one, one that did it the right way and give me the link to it. Or make it yourselves. I'm totally willing to jump through a lot of hoops for the first part, but it's so frustrating when I'm trying to help and the response is "well actually what we meant by tests is we like things like this".
If you have some sort of vision of what the product is or isn't, tell me about it. This comes up a lot when you go to add a feature that seems pretty obvious only to have the person close it with an exhausted response of "we've already been over this a hundred times". I understand this is old news to you, but I just got here. If you have stuff that comes up a lot that you don't want people to bother you with, mention it in the README. I promise I'll read it and I won't bother you!
If what you want is money, say that. I actually prefer when a maintainer says something like "donors bug reports go to the front of the line" or something to that effect. If you are a maintainer who feels unappreciated and overwhelmed, I get that and I want to work with you. If the solution is "my organization pays you to look at the bug report first", that's totally ethnically acceptable. For some reason this seems icky to the community ethos in general, but to me it just makes sense. Just make it clear how it works.
If there are tasks you think are worth doing but don't want to do, flag them. I absolutely love when maintainers do this. "Hey this is a good idea, it's worth doing, but it's a lot of work and we don't want to do it right now". It's the perfect place for someone to start and it hits that sweet spot of high return on effort.
I don't want this to read like "I, an entitled brat, believe that maintainers owe me". You provide an amazing service and I want to help. But part of helping is I need to understand what is it that you would like me to do. Because the open-source community doesn't adopt any sort of consistent cross-project set of guidelines (see weird libertarian bent) it is up to each one to tell me how they'd like to me assist them.
But I don't want to waste a lot of time waiting for a perfect centralized solution to this problem to manifest. It's your project, you are welcome to do with it whatever you want (including destroy it), but if you want outside help then you need to sit down and just walk through the question of "what does help look like". Tell me what I can do, even if the only thing I can do is "pay you money".
I’ve always marveled at people who are motivated purely by the love of what they’re doing. There’s something so wholesome about that approach to life—where winning and losing don’t matter. They’re simply there to revel in the experience and enjoy the activity for its own sake.
Unfortunately, I am not that kind of person. I’m a worse kind of person.
For much of my childhood, I invented fake rivalries to motivate myself. A popular boy at school who was occasionally rude would be transformed into my arch-nemesis. “I see I did better on the test this week, John,” I’d whisper to myself, as John lived his life blissfully unaware of my scheming. Instead of accepting my lot in life—namely that no one in my peer group cared at all about what I was doing—I transformed everything into a poorly written soap opera.
This seemed harmless until high school. For four years, I convinced myself I was locked in an epic struggle with Alex, a much more popular and frankly nicer person than me. We were in nearly all the same classes, and I obsessed over everything he said. Once, he leaned over and whispered, “Good job,” then waited a half beat before smiling. I spent the rest of the day growing increasingly furious over what he probably meant by that.
“I think you need to calm down,” advised Sarah, the daughter of a coworker at Sears who read magazines in our break room.
“I think you need to stay out of this, Sarah,” I fumed, furiously throwing broken tools into the warranty barrel—the official way Sears handled broken Craftsman tools: tossing them into an oil drum.
The full extent of my delusion didn’t become clear until junior year of college, when I ran into Alex at a bar in my small hometown in Ohio. Confidently, I strode up to him, intent on proving I was a much cooler person now.
“I’m sorry, did we go to school together?” he asked.
Initially, I thought it was a joke—a deliberate jab to throw me off. But then it dawned on me: he was serious. He asked a series of questions to narrow down who exactly I was.
“Were you in the marching band? Because I spent four years on the football team, and I didn’t get to know a lot of those kids. It looked fun, though.”
That moment taught me a valuable lesson: no more fake rivals.
So imagine my surprise when a teenage grocery store checkout clerk emerges in my 30s to become my greatest enemy—a cunning and devious foe who forced me to rethink everything about myself.
Odense
Odense, Denmark, is a medium-sized town with about 200,000 people. It boasts a mall, an IKEA, a charming downtown, and a couple of beautiful parks. It also has a Chinese-themed casino with a statue of H.C. Andersen out front and an H.C. Andersen museum, since Odense is where the famous author was born.
Amusingly, Andersen hated Odense—the place where he had been exposed to the horrors of poverty. Yet now the city has formed its entire identity around him.
I moved here from Chicago, lured by the promise of a low cost of living and easy proximity to Copenhagen Airport (just a 90-minute train ride away). I had grand dreams of effortlessly exploring Europe. Then COVID hit, and my world shrank dramatically.
For the next 12 months, I rarely ventured beyond a three-block radius—except for long dog walks and occasional supply runs to one of the larger stores. One such store was a LokalBrugsen, roughly the size of a gas station. I’d never shopped there before COVID since it had almost no selection.
The actual store in question
But desperate times called for desperate measures, and its emptiness made it the better option. My first visit greeted me with a disturbing poster taped to the door.
The Danish word for hoarding is hamstre, a charming reference to stuffing your cheeks like a hamster. Apparently, during World War II, people were warned against hoarding food. The small grocery store had decided to resurrect this message—unfortunately using a German wartime poster, complete with Nazi imagery. I got the point, but still.
Inside, two Danish women frantically threw bread-making supplies into their cart, hamstering away. They had about 40 packets of yeast, which seemed sufficient to weather the apocalypse. Surely, at a certain point, two people have enough bread.
It was during this surreal period that I met my rival: Aden.
Before COVID, the store had been staffed by a mix of Danes and non-Danes. But during the pandemic, the Danes seemingly wanted nothing to do with the poorly ventilated shop, leaving it staffed entirely by non-Danes.
Aden was in his early 20s, tall and lean, with a penchant for sitting with his arms crossed, staring at nothing in particular, and directing random comments at whoever happened to be nearby.
The first thing I noticed about him was his impressive language skills. He could argue with a Frenchman, switch seamlessly to Danish for the next dispute, and insult me in near-perfect California-accented English.
My first encounter with him came when I tried to buy Panodil from behind the counter.
In my best Danish, I asked, “Må jeg bede om Panodil?” (which literally translates to “May I pray for Panodil?” since Danish doesn’t have a word for “please”).
Aden laughed. “Right words, but your accent’s way off. Try again, bro.”
He stared at me expectantly.
So I tried again.
“Yeah, still not right. You gotta get lower on the bede.”
The line behind me grew as Aden, seemingly with nothing but time on his hands, made me repeat myself.
Eventually, I snapped. “You understand me. Just give me the medicine.”
He handed it over with a grin. “We’ll practice again later,” he said as I walked out.
As my sense of time dissolved and my sleep became increasingly erratic, this feud became the only thing happening in my life.
Each visit to the store turned into a linguistic duel. Aden would ask me increasingly bizarre questions in Danish. “Do you think the Queen’s speech captured the mood of the nation in this time of uncertainty?” It would take me several long seconds to process what he’d said.
Then I’d retaliate with the most complex English sentence I could muster. “It’s kismet that a paragon of virtue such as this Queen rules and not a leader who acts obsequiously in the face of struggle. Why are you lollygagging around anyway?”
Aden visibly bristled at my use of obscure American slang like lollygag, bumfuzzle, cattywampus, and malarkey. Naturally, I made it my mission to memorize every regionalism I could find. My wife shook her head as I scrolled through websites with titles like “Most Unusual Slang in the Deep South.”
Increasingly Deranged
As weeks turned into months, my life settled into a bizarrely predictable pattern. After logging into my work laptop and finding nothing to do, I’d take my dog on a three-to-four-hour walk. His favorite spot was a stone embankment where H.C. Andersen’s mother supposedly washed clothes—a fact so boring it seems fabricated, yet somehow true.
If I was lucky, I’d witness the police breaking up gatherings of too many people. The fancy houses along the river were home to richer Danes who simply couldn’t follow the maximum group size rule. I delighted in watching officers disperse elderly tea parties.
My incredibly fit Corgi, whose fur barely contained his muscles after daily multi-hour walks, and I would eventually head home, where I wasted time until the “workday” ended. Then it was time for wine, news, and my trip to the store.
On the way, I passed the Turkish Club—a one-room social club filled with patio furniture and a cooler full of beer no one seemed to drink. It reminded me of a low-rent version of the butcher shop from The Sopranos, complete with men smoking all the cigarettes in the world.
Then I’d turn the corner and peek around to see if Aden was there. He usually was.
As the pandemic wore on, even the impeccably dressed Danes began to look unhinged, with home haircuts and questionable outfits. The store itself devolved into a chaotic mix of token fruits and vegetables, along with soda, beer, and wine bearing dubious labels like “Highest Quality White Wine.” People had stopped hamstering but this had been replaced with daytime drinking.
Sadly, Aden had become somewhat diminished too. His reign of terror ended when a very tough-looking Danish man verbally dismantled him in front of everyone. I was genuinely worried for my petite rival, who was clearly outmatched. Aden has said something about him buying "too many beers today" that had set the guy off. In Aden's defense it was a lot of beers, but still, probably not his place.
Our last conversation didn’t take place in the store but at a bus stop. I asked him where he’d learned English, as it was remarkably good. “The show Friends. I had the DVDs,” he said, staring forward. He seemed uncomfortable seeing me outside his domain, which wasn’t helped by my bowl haircut and general confusion about what day it was.
Then, on the bus, something heartwarming happened. The driver, also seemingly from Somalia, said something to Aden that I didn’t understand. Aden’s response was clearly ruder than expected, prompting the driver to turn around and start a heated argument.
It wasn’t just me—everyone hated him.
In this crazy, mixed-up world, some things can bring people together across language and cultural barriers.
Teenage boys being rude might just be the secret to world peace.
