In my hometown in Ohio, church membership was a given for middle-class people. With a population of 8,000 people, somehow 19 churches were kept open and running. A big part of your social fabric were the kids who went to the same church that you did, the people you would gravitate towards in a new social situation. The more rural and working class your family was, the less likely you actually went to a church on a regular basis. You'd see them on Christmas and Easter, but they weren't really part of the "church group".

My friend Mark was in this group. His family lived in an underground house next to their former motorcycle shop that had closed down when his dad died. It was an hour walk from my house to his through a beautiful forest filled with deer. I was often stuck within eyesight of the house as a freight train slowly rolled through in front of me, sitting on a tree stump until the train passed. The former motorcycle shop had a Dr. Pepper machine in front of the soaped windows I had a key to and we'd sneak in to explore the shop on sleepovers while his mom worked one of her many jobs.

She was a chain-smoker who rarely spoke, often lighting one cigarette with the burning cherry of the first as she drove us to the local video store to rent a videogame. Mark's older brother also lived in the house, but rarely left his room. Mostly it was the two of us bouncing around unsupervised, watching old movies and drinking way too much soda as his German Shepard wheezed and coughed in the cigarette smoke filled house.

Families like this were often targeted by Evangelical Christians, groups that often tried to lure families in with offers of youth groups that could entertain your kids while you worked. At some point, one of the youth pastors convinced Mark's mom that she should send both of us to his church. Instead of Blockbuster, we got dropped off in front of an anonymous steel warehouse structure with a cross and a buzzing overhead light on a dark country road surrounded by cornfields. Mark hadn't really been exposed to religion, with his father and mother having been deep into biker culture. I had already seen these types of places before and was dreading what I knew came next.

When we walked in, we were introduced to "Pastor Michael", who looked a bit like if Santa Claus went on an extreme diet and bought serial killer glasses. Mark bounced over and started asking him questions, but I kept my distance. I had volunteered the year before to fix up an old train station which involved a large crew of youth "supervised" by the fundamentalist Christian church that wanted to turn the train station into a homeless shelter. We slept on the floors of the middle school in this smaller town neighboring mine, spending our days stripping paint and sanding floors and our evenings being lectured about the evils of sex and how America was in a "culture war". In retrospect I feel like there should have been more protective gear in removing lead paint as child labor, but I guess that was up to God.

After one of these long sessions where we were made to stand up and promise we wouldn't have sex before we got married, I made a joke in the walk back to our assigned classroom and was immediately set upon by the senior boy in the group. He had a military style haircut and he threw me against a locker hard enough that I saw stars. I had grown up going to Catholic school and mass every Sunday, spending my Wednesday nights going to CCD (Confraternity of Christian Doctrine), which was like Sunday school for Catholics. All of this was to say I had pretty established "Christian" credentials. This boy let me know he thought I was a bad influence, a fake Christian and that I should be careful since I'd be alone with him and his friends every night in the locked classroom. The experience had left me extremely wary of these Evangelical cults as I laid silently in my sleeping bag on the floor of a classroom, listening to a hamster running in a wheel that had clearly been forgotten.

To those of you not familiar with this world, allow me to provide some context. My Catholic education presented a very different relationship with holy figures. God spoke directly to very few people, saints mostly. There were many warnings growing up about not falling into the trap of believing you were such a person, worthy of a vision or a direct conversation with a deity. It was suggested softly this was more likely mental illness than divine intervention. He enters your heart and changes your behavior and gives you peace, but you aren't in that echelon of rare individuals for whom a chat was justified. So to me these Evangelicals claiming they could speak directly with God was heresy, a gross blasphemy where random "Pastors" were claiming they were saints.

The congregation started to file in and what would follow was one of the most surreal two hours of my life. People I knew, the woman who worked at the library and a local postal worker started to scream and wave their arms, blaming their health issues on Satan. Then at one point Scary Santa Claus started to shake and jerk, looking a bit like he was having a seizure. He started to babble loudly, moving around the room and I stared as more and more people seemed to pretend this babbling meant something and then doing it themselves. The bright lights and blaring music seemed to have worked these normal people into madness.

In this era before cell phones, there wasn't much I could do to leave the situation. I waited and watched as Mark became convinced these people were channeling the voice of God. "It's amazing, I really felt something in there, there was an energy in the room!" he whispered to me as I kept my eyes on the door. One of the youth pastors asked me if I felt the spirit moving through me, that I shouldn't resist the urge to join in. I muttered that I was ok and said I had to go to the bathroom, then waited in the stall until the service wrapped up almost two hours later.

In talking to the other kids, I couldn't wrap my mind around the reality that they believed this. "It's the language of God, only a select few can understand what the Holy Spirit is saying through us". The language was all powerful, allowing the Pastor to reveal prophesy to select members of the Church and assist them with their financial investments. This was a deadly serious business that these normal people completely believed, convinced this nonsense jabbering that would sometimes kind of sound like language was literally God talking through them.

I left confident that normal, rational people would never believe such nonsense. These people were gullible and once I got out of this dead town I'd never have to be subjected to this level of delusion. So imagine my surprise when, years later, I'm sitting in an giant conference hall in San Francisco as the CEO of Google explains to the crowd how AI is the future. This system that stitched together random words was going to replace all of us in the crowd, solve global warming, change every job. This was met with thunderous applause by the group, apparently excited to lose their health insurance. All this had been kicked off with techno music and bright lights, a church service with a bigger budget.

Every meeting I went to was filled with people ecstatic about the possibility of replacing staff with this divine text generator. A French venture capitalist who shared a Uber with me to the original Google campus for meetings was nearly breathless with excitement. "Soon we might not even need programmers to launch a startup! Just a founder and their ideas getting out to market as fast as they can dream it." I was tempted to comment that it seemed more likely I could replace him with an LLM, but it felt mean. "It is going to change the world" he muttered as we sat in a Tesla still being driven by a human.

It has often been suggested by religious people in my life that my community, the nonreligious tech enthusiasts, use technology as a replacement for religion. We reject the fantastical concept of gods and saints only to replace them with delusional ideas of the future. Self-driving cars were inevitable until it became clear that the problem was actually too hard and we quietly stopped talking about it. Establishing a colony on Mars is often discussed as if it is "soon", even if the idea of doing so far outstrips what we're capable of doing by a factor of 10. We tried to replace paper money with a digital currency and managed to create a global Ponzi scheme that accelerated the destruction of the Earth.

Typically I reject this logic. Technology, for its many faults, also produces a lot of things with actual benefits which is not a claim religion can make most of the time. But after months of hearing this blind faith in the power of AI, the comparisons between what I was hearing now and what the faithful had said to me after that service was eerily similar. Is this just a mass delusion, a desperate attempt by tech companies to convince us they are still worth a trillion dollars even though they have no new ideas? Is there anything here?

Glossolalia

Glossolalia, the technical term for speaking in tongues, is an old tradition with a more modern revival. It is a trademark of the Pentecostal Church, usually surrounded by loud music, screaming prayers and a leader trying to whip the crowd into a frenzy. Until the late 1950s it was confined to a few extreme groups, but since then has grown into a more and more common fixture in the US. The cultural interpretation of this trend presents it as a “heavenly language of the spirit” accessible only to the gifted ones. Glossolalists often report an intentional or spontaneous suspension of will to convey divine messages and prophecies.

In the early 1900s W. J. Seymour, a minister in the US, started to popularize the practice of whipping his congregation into a frenzy such that they could speak in tongues. This was in Los Angeles and quickly became the center of the movement. For those who felt disconnected from religion in a transplant city, it must have been quite the experience to feel your deity speaking directly through you.

It's Biblical basis is flimsy at best however. Joel 2:28-9 says:

And afterwards I will pour out my Spirit on all people. Your sons and daughters will prophesy, your old men will dream dreams, young men will see visions. Even on my servants, both men and women, I will pour out my Spirit in those days.

A lot of research has been done into whether this speech is a "language", with fascinating results. In the Psychology of Speaking in Tongues, Kildahl and Qualben attempted to figure that out. Their conclusions were that while it could sound like a language, it was a gibberish, closer to the fake language children use to practice the sounds of speaking. To believers though, this presented no problems.

He argued that glossolalia is real and that it is a gift from the Holy Spirit. He argued that a person cannot fake tongues. Tongues are an initial evidence of the spirit baptism. It is a spiritual experience. He observed that tongues cannot be understood by ordinary people. They can only be understood spiritually. He noted that when he speaks in tongues he feels out of himself. The feeling is very strange. One can cry, get excited, and laugh. As our respondent prayed, he uttered: Hiro---shi---shi---sha---a---karasha. He jumped and clapped his hands in excitement and charisma. He observed that if God allows a believer to speak in tongues there is a purpose for that. One can speak in tongues and interpret them at the same time. However, in his church there is no one who can interpret tongues. According to our respondent, tongues are intended to edify a person. Tongues are beneficial to the person who speaks in tongues. A person does not choose to pray in tongues. Tongues come through the Spirit of God. When speaking in tongues, it feels as if one has lost one's memory. It is as if one is drunk and the person seems to be psychologically disturbed. This is because of the power of the influence of the Holy Spirit. Tongues are a special visitation symbolising a further special touch of the Holy Spirit. Source

In reality glossolalic speech is not a random and disorganized production of sounds. It has specific accents, intonations and word-like units that resembles the original language of the speaker. [source] That doesn't make it language though, even if the words leave the speaker feeling warm and happy.

What it actually is, at its core, is another tool the Evangelical machine has at its disposal to use the power of music and group suggestion to work people into a frenzy.

The tongue-speaker temporarily discards some of his or her ego functioning as it happens in such times as in sleep or in sexual intercourse.⁴¹ This phenomenon was also noticed in 2006 at the University of Pennsylvania, USA, by researchers under the direction of Andrew Newburg, MD who completed the world's first brain-scan study of a group of Pentecostal practitioners while they were speaking in tongues. The researchers noticed that when the participants were engaged in glossolalia, activity in the language centres of the brain actually decreased, while activity in the emotional centres of the brain increased. The fact that the researchers observed no changes in any language areas, led them to conclude that this phenomenon suggests that glossolalia is not associated with usual language function or usage.

Source

It's the power of suggestion. You are in a group of people and someone, maybe a plant, kicks it off. You are encouraged to join in and watch as your peers enthusiastically get involved. The experience has been explained as positive, so of course you remember it as a positive experience, even if in your core you understand that you weren't "channeling voices". You can intellectually know it is a fake and still feel moved by the experience.

LLMs

LLMs, large language models, which have been rebranded as AI, share a lot with the Evangelical tool. AI was classically understood to be a true artificial intelligence, a thinking machine that actually processed and understood your request. It was seen as the Holy Grail of computer science, the ability to take the best of human intellect and combine it into an eternal machine that could guide and help us. This definition has leaked from the sphere of technology and now solidly lives on in Science Fiction, the talking robot who can help and assist the humans tasked with something.

If that's the positive spin, then there has always been a counter argument. Known as the "Chinese room argument" as shorthand, it says that a digital computer running code cannot have a mind, understanding or consciousness. You can create a very convincing fake though. The thought experiment is as follows:

You've made a computer that behaves as if it understands Chinese. It takes Chinese characters as input and returns Chinese characters as output. It does so at such a high level that it passes the Turing test in that a native Chinese speaker believes the thing it is speaking to is a human being speaking Chinese. But the distinction is that the machine doesn't understand Chinese, it is simulating the idea of speaking Chinese.

Searle suggests if you put him into a room with an English version of the program he could receive the same characters through a slot in the door, process them according to the code and produce Chinese characters as output, without understanding anything that is being said. However he still wouldn't speak or understand Chinese.

This topic has been discussed at length by experts, so if you are interested in the counterarguments I suggest the great site by Stanford: https://plato.stanford.edu/entries/chinese-room/

What Is An AI?

The Neural Networks powering AI at a high level look as follows:

The magic part of AI is the Transformer neural network, which uses self-attention to process not just the elements of text on their own but the way the data relates to each other. It has been fed enough examples to be able to take the input and produce a likely output. Effectively by ingesting every work spoken by humans on video or written on the internet, it can predict with some level of accuracy what a common collection of words to return to a query is.

The end result is a box that consumes a frightening amount of electricity to mimic the responses of an all-knowing eternal AI. The believers can commune directly with this machine spirit, asking it questions and getting back blocks of text or pictures that sometimes look like something a human would make. You cannot question the model or work backwards to determine how specifically it came to the conclusion it did. There is an element of the divine in the process, a specter of magic.

Hallucination

As AI was introduced, we were cautioned to be on the lookout for "hallucinations", where the model would produce convincing-looking fakes. The problem with this warning is it both flies in the face of how humans work (we're not amazing at questioning authoritative-sounding answers) and also suggested this was a temporary problem. But the more you read about AI, you realize it can never solve the hallucination problem because it would need to understand what is "real" and "fake" in order to do that.

What do I mean on a practical level? So I thought I would test out the Google Gemini AI with some easy questions. I had recently written a bash script to query GCP cloud deployments to get their status and write them to markdown. This should be an easy test of the system, right? It's Google AI writing a simple script using the Google CLI to interact with the Google Cloud Platform. Closed ecosystem, high degree of confident, should have definite sources of truth.

Can you write a bash script using the gcloud CLI to query the status of all deployments using cloud deploy, a GCP service, in the GCP project test123 and output the name and current name to a markdown file for each one?

#!/bin/bash

# Set project ID
PROJECT_ID="test123"

# Output directory for markdown files
OUTPUT_DIR="deployment_statuses"

# Create output directory if it doesn't exist
mkdir -p "$OUTPUT_DIR"

# Get all deployments
deployments=$(gcloud deploy deployments list --project "$PROJECT_ID" --format="value(NAME)")

# Loop through each deployment
for deployment in $deployments; do
  # Get deployment details
  deployment_details=$(gcloud deploy deployments describe "$deployment" --project "$PROJECT_ID" --format="yaml")

  # Parse name and state from YAML output
  name=$(echo "$deployment_details" | yq .name)
  state=$(echo "$deployment_details" | yq .status.state)

  # Create markdown file for deployment
  output_file="$OUTPUT_DIR/$name.md"
  echo "# Deployment: $name" > "$output_file"
  echo "Current State: $state" >> "$output_file"

  # Add additional information to markdown file as needed (e.g., rollout strategy, create time)
  # ...

done

echo "Deployment statuses written to markdown files in $OUTPUT_DIR"

Looks great right? Exactly what I wanted. Except it doesn't work.

gcloud doesn't have a gcloud deploy deployments argument.

NAME
    gcloud deploy - create and manage Cloud Deploy resources

SYNOPSIS
    gcloud deploy GROUP | COMMAND [GCLOUD_WIDE_FLAG ...]

DESCRIPTION
    Create and manage Cloud Deploy resources.

GCLOUD WIDE FLAGS
    These flags are available to all commands: --help.

    Run $ gcloud help for details.

GROUPS
    GROUP is one of the following:

     automation-runs
        Manages AutomationRuns resources for Cloud Deploy.

     automations
        Manages Automations resources for Cloud Deploy.

     custom-target-types
        Create and manage Custom Target Type resources for Cloud Deploy.

     delivery-pipelines
        Create and manage Delivery Pipeline resources for Cloud Deploy.

     job-runs
        Manages job runs resources for Cloud Deploy.

     releases
        Create and manage Release resources for Cloud Deploy.

     rollouts
        Create and manage Rollout resources for Cloud Deploy.

     targets
        Create and manage Target resources for Cloud Deploy.

Now I know this because I wrote it, but there's no way someone looking at this with no knowledge of the gcloud CLI would understand why this wouldn't work.

I tried again, this time with a simpler question. Maybe because not that many people use gcloud it doesn't have a big enough sample size to construct a real answer. Let's ask a more basic question, but let's keep it on Google technology. Kubernetes has been documented to death, books written about it, millions of articles and blog posts. Certainly that will work.

How do I, in hosted Kubernetes on GCP, write a deployment configuration yaml which sets nodeAffinity using preferredDuringSchedulingIgnoredDuringExecution. The nodepool should be nodes02 with a weight of one.

What I'm trying to prompt it here is to give me back a configuration file not with nodeSelector but with Node Affinity, the softer preference level for assigning pods to nodes. But there's a small trick in the question. I want a deployment not a Pod spec. This is a distinction which requires a more nuanced comprehension of the subject matter, the ability to not pull the most common example but the more specific example. What we want is this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template: 
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
          preference:
            matchExpressions:
            - key: cloud.google.com/gke-nodepool
            operator: In
            values:
            - nodes02
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

What I got back was this:

  spec:
    nodeSelector: 
      cloud.google.com/gke-nodepool: nodes02
    affinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        matchExpressions:
        - key: cloud.google.com/gke-nodepool
          operator: In
          values:
          - nodes02

The problem with this response is it does a very different thing from the thing I was trying to do. nodeSelector is a stricter approach, ensuring pods are only scheduled on nodes that match the label. nodeAffinity is a much softer preference, telling k8s I'd like the pods to go there if possible but if that isn't possible, put them where you would normally do.

Both of these examples seem reasonable. The machine responded with something that could be construed as the answer, a clever parody of human intelligence, but ultimately it is more like a child playing. It doesn't understand the question, but understands how to construct convincing looking fakes.

To the faithful though, this isn't a problem.

However, if the training data is incomplete or biased, the AI model may learn incorrect patterns. This can lead to the AI model making incorrect predictions, or hallucinating.

For example, an AI model that is trained on a dataset of medical images may learn to identify cancer cells. However, if the dataset does not include any images of healthy tissue, the AI model may incorrectly predict that healthy tissue is cancerous. This is an example of an AI hallucination.

[Source]

This creates a false belief that the problem lies with the training data, which for both of my examples simply cannot be true. Google controls both ends of that equation and can very confidently "ground" the model with verifiable sources of information. In theory this should tether their output and reduce the chances of inventing content. It reeks of a religious leader claiming while that prophecy was false, the next one will be real if you believe hard enough. It also moves the responsibility for the problem from the AI model to "the training data", which for these LLMs represents a black box of information. I don't know what the training data is, so I can't question whether its good or bad.

Is There Anything Here?

Now that isn't to say there isn't amazing work happening here. LLMs can do some fascinating things and the transformer work has the promise to change how we allow people to interact with computers. Instead of an HTML form with strict validation and obtuse error messages, we can instead help explain to people in real-time what is happening, how to fix problems, just in general provide more flexibility when dealing with human inputs. We can have a machine look at less-sorted data and find patterns, there are lots of ways for this tech to make meaningful differences in human life.

It just doesn't have a trillion dollars worth of value. This isn't a magic machine that will replace all human workers, which for some modern executives is the same as being able to talk directly to God in terms of the Holy Grail of human progress. Finally all the money can flow directly to the CEO himself, cutting out all those annoying middle steps. The demand of investors for these companies to produce something new has outstripped their ability to do that, resulting in a dangerous technology being unleashed upon the world with no safeties. We've made a lying machine that doesn't show you its work, making it even harder for people to tell truth from fiction.

If LLMs are going to turn into actual AI, we're still years and years from that happening. This represents an interesting trick, a feel-good exercise that, unless you look too closely, seems like you are actually talking to an immortal all-knowing being that lives in the clouds. But just like everything else, if your faith is shaken for even a moment the illusion collapses.

Questions/comments/concerns: https://c.im/@matdevdug

I like Python. I've had a lot of success with it on projects large and small. It is fast enough for most of the things I need and when it isn't, migrating from it to a more performant language isn't challenging. The depth of the standard library has been incredible for stable long-living code. However the one thing I hear often when discussing Python with younger programmers is well the dependency management is so bad I wouldn't bother with the language. Lately it seems the narrative is now evolving into "it is so broken that we need a new package system to fix it", which to me is the programming version of Spock dying in the warp core. Let's make absolutely sure we have no other options.

The problem here isn't one of engineering. We have all the solutions to solve this problem for the majority of users. The issue is an incorrect approach to managing defaults. Pip, like many engineering-led projects, doesn't account for the power of defaults. Engineers tend towards maintaining existing behavior and providing the tooling for people to do it correctly. That's the wrong mentality. Experts who drive the project should be adjusting the default behavior to follow best practices.

Defaults are so important and I think so intimidating to change that this decision has been pushed back for years and years. If we have a better user experience for people and we know this is what they should be using, we should not expect users to discover that best way on their own. You have to make them opt out of the correct flow, not discover and opt in to the right way to do things. Change is scary though and maintainers don't have a massive corporate structure to hide behind. Whatever ire the change generates isn't directed at Faceless Corporation PR, it's directly at the people who make the decision.

Golang taught us this lesson. I work a lot with Golang at work and on some side projects. It is an amazing language to show the power of defaults and the necessity of experts pushing users forward. Golang code at every job looks like code at every other job, which is the direct result of intentional design. Shipping gofmt bundled in with the language increased the quality and readability of golang everywhere. Decentralizing dependency management became a "of course" moment for people when they tried it. Keep the language simple in the face of demands for increased complexity has preserved the appeal. The list goes on and on.

Pypa needs to push the ecosystem forward or give up on the project and officially endorse a new approach. Offering people 400 options is destroying confidence in the core language. The design mentality has to change from "it isn't a problem if there is a workaround" to the correct approach which is for most users the default is the only option they'll ever try.

Why it isn't that broken

Why do I think that we don't need to start fresh? Here's the workflow I use, which is not unique to me. I start a new Python repo and immediately make a venv with python -m venv venv. Then I activate it with source /venv/bin/activate and start doing whatever I want. I write all my code, feel pretty good about it and decide to lock down my dependencies.

I run pip freeze > requirement.in which gives me all the packages I have installed with their versions. It's 2024 so I need more security and confidence than a list of packages with a version number. The easiest way to get that is with package hashes, which is easy to do with pip-tools. pip-compile --generate-hashes requirements.in outputs a requirements.txt with the hashes I want along with the dependencies of the packages.

build==1.0.3 \
    --hash=sha256:538aab1b64f9828977f84bc63ae570b060a8ed1be419e7870b8b4fc5e6ea553b \
    --hash=sha256:589bf99a67df7c9cf07ec0ac0e5e2ea5d4b37ac63301c4986d1acb126aa83f8f
    # via
    #   -r requirements.in
    #   pip-tools
cachetools==5.3.2 \
    --hash=sha256:086ee420196f7b2ab9ca2db2520aca326318b68fe5ba8bc4d49cca91add450f2 \
    --hash=sha256:861f35a13a451f94e301ce2bec7cac63e881232ccce7ed67fab9b5df4d3beaa1
    # via
    #   -r requirements.in
    #   google-auth
certifi==2023.11.17 \
    --hash=sha256:9b469f3a900bf28dc19b8cfbf8019bf47f7fdd1a65a1d4ffb98fc14166beb4d1 \
    --hash=sha256:e036ab49d5b79556f99cfc2d9320b34cfbe5be05c5871b51de9329f0603b0474
    # via
    #   -r requirements.in
    #   aioquic
    #   aioquic-mitmproxy
    #   mitmproxy
    #   requests

Now I know all the packages I have, why I have the packages I have and also the specific hashes of those packages so I don't need to worry about supply chain issues. My Dockerfile is also pretty idiot-proof.

FROM python:3.12-slim

# Create a non-root user
RUN groupadd -r nonroot && useradd -r -g nonroot nonroot
WORKDIR /app

COPY requirements.txt .

RUN pip3 install -r requirements.txt

COPY . .

RUN chown -R nonroot:nonroot /app

USER nonroot

ENTRYPOINT ["./gunicorn.sh"]

Yay its running. I can feel pretty confident handing this project over to someone new and having them run into minimal problems getting all of this running. Need to check for updates? Not a big deal.

pip-review
Flask==3.0.2 is available (you have 3.0.0)
Jinja2==3.1.3 is available (you have 3.1.2)
MarkupSafe==2.1.5 is available (you have 2.1.3)
pip==24.0 is available (you have 23.3.1)

Basically if you know the happy path, there are no serious problems here. But you need to know all these steps, which are documented in random places all over the internet. How did we get here and what can be done to fix it?

Why do people think it is so bad

What is the combination of decisions that got us to this place? Why is the average users opinion so low? I think its everything below.

• Bare pip sucks for daily tasks. We can't declare a minimum version of Python, we don't get any information as to dependency relationships in the file, we don't have a concept of developer dependency vs production dependency, we don't have hashes so we're very open to upstream attacks, it's slow, it's not clear how to check for updates, there's no alert for a critical update, the list goes on and on.
  • What pip-compile does should always be the minimum. It should have been the minimum years and years ago.
  • Where pip shines is the range of scenarios it covers and backwards compatibility. We don't want to throw out all that work if we can avoid it to switch to a new package manager unless the situation is unfixable. To me the situation is extremely fixable, but we need to change defaults.
• People used Python as a bash replacement. This was a weird period where, similar to Perl, there was an assumption that Python would be installed on any system you were working with and so you could write Python scripts to do things and then package them up as Linux packages. If your script had dependencies, you would also pull those in as Linux packages.
  • To be blunt, this dependency management system never should have been allowed. It caused no end of confusion for everyone and ended up with people using super old packages. If your Python application had dependencies, you should have included them.
  • Starting to write Python in Linux and then running apt-get install requests but then later being told to use pip and remove the package even though packages are how you get software in Linux has thrown off beginners as long as I have been doing this job.
• The nature of dependencies has changed and how we think of including third-party software has evolved. I was shocked when I started working with NodeJS teams at how aggressively and (frankly) recklessly they would add dependencies to a project. However NPM and Node are designed around that model of lots of external dependencies and they've adopted a lot of things that people have come to expect
  • The package.json, package-lock.json and node_modules directory as a consistent design across all projects is huge. It completely eliminated confusion and ensures you can have super-easy project switching along with reproducible environments.
  • Node defaulting to per-project and not global is what Python should have switched over to years ago. Again, this is just what people expect when they're talking about having lots of dependencies.
  • People have a lot more dependencies. When I started in this field, the idea of every project adding a 66 MB dependency with boto would have been unthinkable. Not because of disk space, but because its just so much code to bring into a project. However now people don't even blink at adding more libraries. pip was designed in a world where requirements.txt were 10 lines. Now we could easily be talking 200.
  • If we're not going to switch over to per-project dependencies, then at the very least you need to switch to venv as a default. I don't care how you do it. Make a file that sits at the top level of a directory that tells Python we're using a venv. Have it check for the existence of a folder and if it exists use it by default, you gotta have something a bit easier here.
• Why this isn't a crisis is this is effectively a basic .profile fix

   cd() {
       builtin cd "$@"
       if [ -f "venv/bin/activate" ]; then
           source venv/bin/activate
       fi
   }

• Finally people think its bad because Golang and Rust exist. Expectations for how dependency management evolved in the space. Work has been done to expand pip to meet more of these expectations but we're still pretty far.

Where to go from here

Anyone familiar with the Apple ecosystem will know the term "Sherlocking". It's where Apple monitors the ecosystem of third-party apps and will periodically copy one and make it part of the default OS. While unfair at times to those third parties, it's a clever design from Apple's perspective. They can let someone else do all the work of figuring out what users like and don't like, what designs succeed or fail on their platform and then swoop in when there is general consensus.

pip needs to do some Sherlocking. Pypa has already done a ton of hard engineering work. We have the ability to create a more secure, easier to debug dependency management system with existing almost-stock tooling. It doesn't require any fundamental changes to the ecosystem, or the investment of a lot of engineering effort.

What it requires is being confident enough in your work to make a better experience for everyone by enduring a period of some complaints. Or its time to give up and endorse something like uv. Sitting and waiting for the problem to resolve itself through some abstract concept of community consensus has been a trainwreck. Either make the defaults conform to modern expectations or warn users when they run pip this is a deprecated project and they should go install whatever else.

Questions/comments/concerns: https://c.im/@matdevdug

Since folks seemed to like the first one, I figured I would do another one. These are just interesting stories from my whole time doing IT-type work. Feel free to subscribe via RSS but know that this isn't the only kind of writing I do.

Getting Started

I grew up in a place that could serve as the laboratory calibration small town USA. It had a large courthouse, a diner, one liquor store infamous for serving underage teens and a library. When I turned 12 my dad asked if I wanted to work for free for a local computer shop. My parents, like girlfriends, friends and a spouse in the future would be, were worried about the amount of time I was spending in the basement surrounded by half-broken electronics.

The shop was on the outskirts of town, a steel warehouse structure converted into offices. It was a father and son business, the father running the counter and phones with the son mostly doing on-site visits to businesses. They were deeply religious, members of a religion where church on Sunday was an all-day affair. Despite agreeing to let me work there for free, the son mostly didn't acknowledge that I was there. He seemed content to let me be and focus on his dream of setting up a wireless ISP with large microwave radio links.

Bill was put in charge of training me. He had been a Vietnam veteran who had lost a leg below the knee in the war. His favorite trick was to rotate the fake leg 180 degrees up and then turn his chair around when kids walked in, laughing as they ran away screaming. He had been a radio operator in the war and had spent most of his career working on radio equipment before getting this computer repair job as "something to keep myself busy". I was put to work fixing Windows 98 and later XP desktop computers.

This was my introduction to "Troubleshooting Theory" which Bill had honed over decades of fixing electronics. It effectively boiled down to:

• Ask users what happened before the failure.
• Develop a theory of the failure and a test to confirm.
• Record the steps you have taken so you don't repeat them
• Check the machine after every step to ensure you didn't make it worse.
• Software is unreliable, remove it as a factor whenever possible.
• Document the fix in your own notes.
• If you make the problem worse in your testing, walk away for a bit and start from the top. You are probably missing something obvious.

Nothing here is revolutionary but the quiet consistency of his approach is still something I use today. He was someone who believed that there was nothing exceptional in fixing technology but that people are too lazy to read the instruction manual. I started with "my PC is slow" tickets, which are basically "every computer that comes in". Windows 98 had a lot of bizarre behavior that was hard for normal users to understand. This was my first exposure to "the Registry".

The Registry

For those of you blessed to have started your exposure to Windows after the era of the registry, it was a hierarchical database used in older Windows versions that stored the information necessary to configure the system. User profiles, what applications are installed, what icons are for what folders, what hardware is in the system, it was all in this database. This database became the source of truth for everything in the system and also the only way to figure out what the system actually thought the value of something was.

The longer a normal person used a Windows device, the more cluttered this database becomes. Combined with adding and deleting files creating fragmentation on the spinning rust drives and you would get a constant stream of people attesting that their machine was slower than it was before. The combination of some Registry Editor work to remove entries and de-fragmentation would buy you some time, but effectively there was a ticking clock hanging over every Windows install before you would need to reinstall.

In short order I learned troubleshooting Windows was a waste of time. Even if you knew why 98 was doing something, you rarely could fix it. So I would just run assembly lines of re-installs, backing up all the users files to a file-share and then clicking through the 98 install screen a thousand times. It sounds boring now but I was thrilled by the work, even though copying lots of files off of bogged down Windows 98 machines was painfully, hilariously slow.

Since Bill believed in telling people they were (effectively) stupid and had broken their machines through an inability to understand simple instructions, I took over the delicate act of lying to users. A lot of Windows IT work is lying to people on the phone trying to walk a delicate line. You can't blame the software too much because we still want them to continue buying computers, but at the same time you don't want to tell the truth which was almost always "you did something wrong and broke it". I felt the lying in this case was practically a public service.

As time went on I graduated to hardware repairs, which was fascinating in that era. Things like "getting video to output onto a monitor" or "getting sound to come out of the sound card" were still minor miracles that often didn't work. Hardware failures were often showing up with blown capacitors. I lived on Bill's endless supply of cups of noodles, sparkling water bottles and his incredibly collection of hot sauce. The man loved hot sauce, buying every random variation he could find. His entire workstation was lined with little bottles of threatening-sounding sauces.

The hardware repairs quickly became my favorite. Bill taught me how to solder and I discovered most problems were pretty easy to fix. Capacitors of this time period were, for whatever reason, constantly exploding. Often even expensive components could be fixed right up by replacing a fan, soldering a new capacitor on or applying thermal paste correctly. Customers loved it because they didn't need to buy totally new components and I loved it because it made me feel like a real expert (even though I wasn't and this was mostly visual diagnosis of problems).

When Windows XP started to be a thing was the first time I felt some level of frustration. XP was so broken when it came out that it effectively put us underwater as a team. After awhile I felt like there wasn't much else for me to do in this space. Windows just broke all the time. I wasn't really getting better at fixing them, because there wasn't anything to fix. As Dell took over the PC desktop market in the area, everything from the videocard to the soundcard were on the motherboard, meaning all repairs boiled down to "replace the motherboard".

That was the end of my Windows career. I sold my PC gear, bought an iBook and from then on I was all-in on Mac. I haven't used Windows in any serious way since.

High School CCNA

While I was in high school, Cisco offered us this unique course. You could attend the Cisco Academy inside of high school, where you would study and eventually sit for your CCNA certification. It was a weird era where everyone understood computers were important to how life was going to work in the future but nobody understood what that meant. Cisco was trying to insert the idea that every organization on earth was going to need someone to configure Cisco routers and switches.

So we went, learned how to write Cisco configurations, recover passwords, reset devices and configure ports. Switches at this point were painfully simple, with concepts like VPNs working but not fully baked ideas. These were 10/100 port switches with PoE and had most of the basic features you would expect. It was a lot of fun to have a class where we would go down there and effectively mess with racks of networking equipment to try and get stuff to work. We'd set up DHCP servers and try to get anything to be able to talk to anything else on our local networks.

We mostly worked with the Cisco Catalyst 1900 which are models I would see well past their end of life in offices throughout my career. This class introduced me to a lot of the basic ideas I still use today. Designing network topology, the OSI model, having VLANs span switches, how routing tables work, IPv4 subnetting, all these concepts were introduced to me here and laid a good foundation for all the work I was to do later. More than the knowledge though, I appreciated the community.

This was the first time I discovered a group of people with the same interests and passions as me. Computer nerd was still very much an insult during this period of time, when admitting you enjoyed this sort of stuff opened you up to mocking. So you kinda didn't mention how much time you spent taking apart NESs from garage sales or you'd invite just a torrent of abuse. However here was a place where we could chat, compare personal projects and troubleshoot. I looked forward to it the 2 days a week I had the class.

To be clear, this was not a rich school. I grew up in a small town in Ohio whose primary industries were agriculture and making the Etch-A-Sketch. Our high school was full of asbestos to the extent that we were warned not to touch the ceiling tiles lest the dust get on us. The math teacher organized a prayer circle around the flagpole every morning as close to violating the Supreme Court ruling on prayer in school as he could get without actually breaking it. But somehow they threw this program together for a few years and I ended up benefiting from it.

The teacher also had contacts with lots of programmers and tech workers, which was the first time I had ever had contact with people in the tech field. They would come into this class and tell us what it was like to be a programmer or a network engineer at this time. It really opened my eyes to what was possible, since people in my life still made fun of the idea of "working with computers". Silicon Valley to people in the Midwest was long-haired hippies playing hacky sack, not doing actual work. These people looked way too tired to be accused of not doing real work.

Mostly though I appreciated our teacher, Mr. Bohnlein. The teacher was an old-school nerd who had been programming since the 70s. He had been a high school teacher for decades but a very passionate Mac user in his personal life. I remember he was extremely skilled at letting us fail for a long time while still giving us hints towards the correct solution. When it came time to take the test, I sailed through it thanks to his help. The students in the class used to make fun of him for his insistence on buying Apple stock. We all thought the company was going to be dead in the next 5 years. "The iPod is the inferior MP3 player" I remember stating very confidently.

He retired comfortable.

Playboy

One call I would get from time to time was to the Chicago Playboy office. This office was beautiful, high up overlooking the water with a very cool "Mad Men" layout. The writers and editors were up on a second level catwalk, with little "pod" offices that had glass walls. They dressed great and were just very stylish people. I was surprised to discover so many of the photographers were female, but I mostly didn't interact with them.

The group I did spend time with was the website team, which unfortunately worked in conventional cubicles facing a giant monitor showing a dashboard of the websites performance and stats. I remember that the carpet was weirdly shaggy and purple, which made finding screws when I dropped them tricky. Often I had to wave a magnet over the ground and hope it sucked up the screw I had lost. The website crew was great to work with, super nice, but the route to their offices involved going by just mountains of Playboy branded stuff.

It was just rack after rack of Playboy clothes, lighters, swimsuits, underwear, water bottles. Every single item you could imagine had that rabbit logo on it. You see it a lot around, but I've never seen it all piled up together. Beyond that was a series of photo studios, with tons of lighting and props. I have no idea if they shot the content for the magazine there (I never saw anyone naked) but it seemed like a lot of the merchandise promo photos were shot there. The photo pipeline was a well-oiled machine, with SD cards almost immediately getting backed up onto multiple locations. They had editing stations right by the photo shooting areas and the entire staff was zero-nonsense.

The repairs were pretty standard stuff, mostly iMac and Mac Pro fixes, but the reason it stood out to me was the weird amount of pornography they tried to give me. Magazines, posters, a book once (like an actual hardcover photo book) which was incredibly nice of the IT guy I worked with, but felt like a strange thing to end a computer repair session with. He would give these to me in a cubicle filled with things made of animals. He had an antler letter opener, wore a ring that looked like it was made out of bone or antler along with a lot of photos of him holding up the heads of dead animals.

The IT field and the gun enthusiast community has a lot of overlap. It makes sense, people who enjoy comparing and shopping for very specific equipment that has long serial number-type names along with weirdly strong brand allegiances. I had no particularly strong stance on hunting guns, having grown up in a rural area where everyone had a shotgun somewhere in the house. As a kid it was common for every visit to a new house to involve being warned to stay away from the gun cabinet. However hunting stories are a particular kind of boring, often beginning with a journey to somewhere I would never want to go and a lot of details I don't need. "I was debating between bringing the Tikka T3 and the Remington 700 but you know the recoil on the T3x is crazy". "Obviously it's a three-day drive from the airport to the hunting area in nowhere Texas but we passed the time talking about our favorite jerky".

I often spent this time trapped in cubicles or offices thinking about these men suddenly forced to fight these animals hand to hand. Are deer hard to knock out with your fists? Presumably they have a lot of brain protection from all the male jousting. I think it would quickly become the most popular show on television, just middle-aged IT managers trying to choke a white-tailed deer as it runs around an arena. We'd sell big steins of beer and turkey legs, like Medieval Times, for spectators. You and a date would crash the giant glasses together and cheer as people run for their lives from a moose.

Once after a repair session, while waiting for the L, I tripped and some of the stuff in my bag spilled out. This woman on the platform looked down at just a thick stack of porn magazines sliding over the platform and then at me. I still think about what she must have thought about me. It's not just that I had a Playboy, but like 6, as if I was one of the secret sexual deviants you read about on the news. "He looked like a normal person but everywhere he went he had a thick stack of porn."

Shedd Aquarium

One of my favorite jobs in the entire city was the Shedd aquarium. I would enter around the side by the loading dock, which is also where many of the animals would come in through. Almost every morning there would be just these giant containers of misc seafood for the animals packed into the loading dock. It was actually really nice to see how high quality it was, like I've eaten dodgier seafood than what they serve the seals at Shedd.

It did make me laugh when you'd see the care and attention that went into the food for the animals and then you'd go by the cafeteria and see kids sucking down chicken nuggets and diet coke. But it was impossible not to be charmed by the intense focus these people had for animals. I used to break some of the rules and spy on the penguins, my favorite animals. There is something endlessly amusing about seeing penguins in non-animal places. Try not to smile at penguins walking down a hallway, it's impossible.

The back area of the aquarium feels like a secret world, with lots of staircases going behind the tanks. Often I would be in a conversation and look through the exhibit, making eye contact with a guest on the other side of the water. It was a very easy place to get lost, often heading down a series of catwalks and down a few stairs to a random door. Even after going there a few times, I appreciated an escort to ensure I didn't head down a random hallway and into an animal area or accidentally emerge in front of a crowd of visitors.

I worked with the graphic design team that was split between the visuals inside the aquarium and their ad campaigns. It was my introduction to concepts like color calibration and large format printing, The team was great and a lot of fun to work with, very passionate about their work. However one part of their workflow threw me off a lot at first. Fonts.

So I, like many people, had not spent a lot of time thinking about the existence of fonts. In school I wrote papers exclusively in Times New Roman for some reason that was never explained to me. However in design teams buying and licensing fonts for each employee and project was a big deal. The technology that most places used at the time to manage these fonts were FontExplorer X Pro, which had a server and client side.

Quickly I learned a lot about fonts because debugging font issues is one of the stranger corners of technical troubleshooting. First some Adobe applications hijacked the system font directories, meaning even if you had injected the right font in the user directory they might not appear. Second fonts themselves were weird. TrueType fonts, which is the older format and the one a lot of these companies still dealt with, are at their lowest level "a sequence of points on a grid". You combine curves and straight lines into what we call a glyph.

Most of the fonts I worked with had started out with the goal of printing on paper. Now many of those were being repurposed for digital assets as well as printing on paper, which introduced no end of weirdness. Here are just a few of the things I tried to help with:

• Print and screen use different color modules
• DPI for print and PPI for digital aren't the same
• No screen is the same. The differences between how a digital asset looked on a nice screen vs a cheap screen wasn't trivial, even if we tried to color calibrate both

In general though I liked working with designers. They often knew exactly what they wanted to get out of my technical assistance, providing me with a ton of examples of what was wrong. Their passion for the graphic design work they were doing inside the aquarium and outside was clear with everyone I spoke with. It's rare to find a group of people who truly enjoys their jobs.

My primary task though was managing and backing up the Mac servers onto tape. For those who haven't used tape backups, it's a slow way to backup a lot of data that requires a lot of testing of backups (along with a good storage system for not confusing people). I quickly came to despise running large-scale tape backups. The rate of errors discovered when attempting to restore backups as a test was horrifying.

The tape backup was overall a complete fucking disaster. There were two tape drives from IBM and way too often a tape written by one drive wouldn't be readable by the other one. The sticker system used to track the tape backups got messed up when I went on vacation and when I came back I couldn't make heads or tails over what had happened. Every week I stopped by and basically tried anything I could think of to get the tape backup to work correctly every time.

Then I did something I'm not proud of. The idea of them calling me and telling me all their hard work was gone was keeping me up at night. So without telling them, I stuck an external 3.5 drive with as much storage as I could afford behind the server tucked away and started copying everything to the tapes and the drive. The IT department had vetoed this idea before but I did it without their permission and basically bet the farm that if the server drives failed and the tape didn't restore, they'd forgive me for making another onsite backup.

I found out years later that their IT found the drive, assumed they had installed it and switched over to backing up on disks in a Drobo since it was much easier to keep running.

United Airlines

Another frequent customer of mine was United Airlines. They had a suburban office which remains the strangest designed office I've ever been in. There was a pretty normal lobby, with executive offices upstairs, a cafeteria and meeting room on the ground floor and then a nuclear bunker style basement. Most of the offices I went to were in the basement along cement corridors so long that they had those little carts with the orange flashing lights zooming down there. It sort of felt like you were at the airport. You could actually ask for a ride on the carts and get one, which I only did once but it was extremely fun.

The team that asked for technical support the most was the design team for the in-flight magazine, Hemispheres. They were all-Mac and located in a side room with no windows in this massive basement complex. So you'd go into just this broiling hot little room with Mac Pros humming along and zero airflow. The walls were brown, the carpet was brown, it was one of the least pleasant offices I've ever been in. Despite working for an in-flight magazine, these people were deadly serious about their work and had frequent contact with Apple about suggested improvements to workflows or tooling.

It was, to be frank, a baffling job. The United Airlines IT didn't want anything to do with the Macs, so I was there to do extremely normal things. I'm talking about applying software updates, install Adobe products, things that anyone is capable of doing without any help. I'd often be asked to wait in a conference room for hours until someone remembered I was there and would ask me to do something. Their internet was so slow I would download the Mac updates at home and bring them into the compound on a hard drive. I've never seen corporate internet this slow in my life.

It wasn't the proudest I've ever been of a job but I was absolutely broke. So I would spend hours watching the progress bar tick by on Mac updates and bill them for it. I tried to do anything to fill the time. I wrapped cables in velcro, refilled printers, reorganized ethernet cables. It was too lucrative for me to walk away but it was also the most bored I've ever been in my life. I once emptied the recycling for everyone just to feel like I had done something that day, only to piss off the janitor. "What, is this your job?" he shouted as I handed him back the recycling bin.

The thing I remember the most was how impossibly hard it was to get paid. You would need to go to the end of this hallway, which had an Accounts Payable window slot with an older woman working there. Then you would physically submit the invoice to her, she would take it and put it in an old-timey wooden invoice tracking system. I'm talking sometimes months from when I submitted the invoice to when I got paid. I would borderline harass this woman, asking her on the way to the bathroom like "hey any chance I could get paid before Christmas? I gotta get the kids presents this year."

I didn't have kids, but I figured it sounded more convincing. I shouldn't have bothered with the lie, she looked at me with zero expression and resumed reading a magazine. At this point I was so poor that I had a budget of $20 a day, so waiting months to get paid by United put me in serious risk of not being able to pay my rent. In the end I learned a super valuable lesson about working for giant corporations. It's a great way to get paid as long as time was no object, but it's a dangerous waiting game to play.

Schools

Colleges hiring me to come out and do specific jobs wasn't uncommon. Setting up a media lab was probably the most common request, where I would show up, set up a bunch of Mac Pros with fiber and an Xserve somewhere to store the files. This was fine work, but it wasn't very exciting and typically involved a lot of unboxing stuff and figuring out how to run fiber. The weirdest niche I found myself in was somehow I became the go-to person for Jewish schools in the Chicago suburbs.

It started with Ida Crown Jewish Academy in Skokie, IL. I went in to fix a bunch of white MacBooks and iMacs and while I was there I showed the teachers how to automate some of their tasks with Automator.

Automator was a drag and drop automation tool that let you effectively write scripts to do certain tasks. I showed them how to automate some of the grading process with CSVs and after that I became the person they always called. Soon after, I started getting calls for all the Jewish schools in the area. To be clear there are not a lot of these schools and they are extremely small.

On average I'd say somewhere around 200-300 students in each school. Also they had pretty intense security, probably the most I'd seen at a high school before or since. To be honest I don't know why they picked me as the Mac IT guy, I don't have any particular feelings about the Jewish faith. The times when the schools staff would ask questions about my faith, they seemed pleased by my complete lack of interest in the topic. As someone who grew up with Christian fundamentalist cults constantly trying to recruit me, I appreciated them dropping it and never mentioning it again.

I loved these jobs because the schools were well organized, the staff knew everyone and they had a list of specific tasks for me when I showed up. Half my life doing independent IT was sitting in waiting rooms while the person who hired me to show up actually came and got me, so this was delightful. I started doing more "teacher automation" work, which was mostly AppleScript or Automator doing the repetitive tasks that these people were staying late to get done.

It wasn't until one of the schools offered me a full-time job that I realized my time in IT was coming to a close. The automation and writing AppleScript was so much more fun than anything I was doing related to Active Directory or printers. It had started to become more clear with the changes Apple was making that they were less and less interested in the creative professional space, which was my bread and butter. This school was super nice, but I knew if I started working here I would be here forever and it was too boring to do forever.

That's when I started transitioning to more traditional Linux sysadmin work. But I still think back fondly of a lot of those trips around Chicago.

Questions/comments/concerns: https://c.im/@matdevdug