matduggan.com

I was talking to a friend recently, a non-technical employee at a large company. He was complaining about his mouse making strange noises. Curious, I asked what he had been doing. "Oh well I need to fill in these websites every quarter with information I get off an Excel sheet. It takes about 30 hours to do and I need to click around a lot to do it." I was a bit baffled and asked him to elaborate, which he did, explaining that there was an internal business process where he got an Excel spreadsheet full of tags. He had to go into an internal website and make sure the tags on their internal app matched the spreadsheet.

"Surely you all have a development team or someone who can help automate that?" I asked, a little shocked at what he was saying. "No well we asked but they just don't have the spare capacity right now, so we're gonna keep doing it this year." He went on to explain this wasn't even the only one of these he had to do, in fact there were many web pages where he needed to manually update information, seemingly only possible through his browser. I thought of the thousands of people at just this company who are spending tens of thousands of hours copy and pasting from a spreadsheet.

When I was a kid and first introduced to computers this was exactly the sort of tedium they were supposed to help with. If you did all the work of getting the data into the computer and had everything set up, part of the sales pitch of a computer in every workplace was the incredible amount of time saved. But this didn't seem any better as compared to him pulling paper files and sorting them. What happened? Weren't we supposed to have fixed this already?

My First Automation

My first introduction to the concept of programming was a technology called HyperCard. HyperCard was a very basic GUI scripting program, which allowed you to create stacks of black and white cards. Apple gave you a series of GUI controls that you could drag and drop into the cards and you linked everything together with HyperTalk, a programming language that still structurally looks "correct" to me.

put the value of card field "typehere" into theValue is an example of a HyperTalk command, but you actually had a lot of interesting options available to you. This was my first introduction to the idea of basically prompting the user for input with commands like:

onmouseUp
answer "This is an example of the answer command" with "Reply 1" or -. "Reply 2" or "Reply 3"
endmouseUp

One of the things I enjoyed doing a lot was trying to script art, making it seem like I had drawn something on the computer that in fact had been done through HyperTalk. Those commands looked something like this:

choose spray can tool
drag from 300,100 to 400, 200 wait 1 second
choose eraser tool
drag from 300,100 to 400, 200 choose browse tool
endmouseUp

The syntax was very readable and easy to debug along with being very forgiving. HyperCard stacks weren't presented as complicated or difficult, they were supposed to be fun. What I remember about HyperCard vs the programming tools I would be exposed to later was it was impossible to get into an unrecoverable state. What I mean is backing out of a card while keeping the stack intact didn't require any special knowledge, a kid could figure it out with very little instruction.  

For those who never got the chance to try it out: check out this great emulator here.

Automator and AppleScript

Years went by and my interest in IT continued, but for the most part this didn't involve a lot of scripting or programming. In the late 90s and early 2000s the focus for IT was all about local GUI applications, trying to transition power tools out of the terminal and into a place where they looked more like normal applications. Even things like server management and LDAP configuration mostly happened through GUI tools, with a few PowerShell or Bash scripts used on a regular basis.

The problem with this approach is the same problem with modern web applications, which is its great for the user who is doing this for the first time to have all this visual information and guides on what to do. For someone doing it for the 5000th time, there's a lot of slowness inherit in the design. The traditional path folks took in IT was to basically abandon all GUI tooling at that point if possible, leaving people lower on the totem pole to suffer through clicking through screens a thousand times.

I was primarily a Mac user supporting other Mac users, meaning I was very much considered a niche. Apple was considered all but dead at the time, with IT teams preferring to silo off the few departments they couldn't force to adopt windows onto a junior helpdesk person. Mostly my job was setting up the odd Mac server, debugging why Active Directory logins weren't working, nothing too shocking. However it was here when I started to encounter this category of non-programmers writing programs to solve surprisingly important problems.

You would discover these Macs running in higher education or in design studios with a post-it note on it saying "Don't turn off". Inevitably there would be some sort of AppleScript and/or Automator process running, resizing files or sending emails, sometimes even connecting to databases (or most often using Excel as a database). I saw grading tools that would take CSVs, convert them into SQL and then put them into a database as a common example.

For those who haven't used Apple Automator, part of the reason I think people liked it for these tasks in Mac-heavy environments is because they're difficult to mess up. You drag and drop a series of commands into a Workflow and then run the Workflow.

After discussing it with users over the years, a similar story emerged about how these workflows came to exist.

1. Someone technical introduced the user to the existence of Automator because they started using it to automate a task.
2. The less-technical user "recorded" themselves doing something they needed to do a thousand times. The Automator record basically just captured what you did with a mouse and ran it again when you hit play.
3. Eventually they'd get more comfortable with the interface and move away from recording and move towards the workflows since they were more reliable.
4. Or if they did batch file processing, they'd go with a Folder Action, which triggered something when an item was placed in a folder.
5. At some point they'd run into the limit of what that could to and call out to either a Bash script or an AppleScript depending on whatever the more technical person they had access to knew.

Why do this?

For a lot of them, this was the only route to automation they were ever going to have. They weren't programmers, they didn't want to become programmers but they needed to automate some task. Automator isn't an intimidating tool, the concept of "record what my mouse does and do it again" makes sense to users. As time went on and more functionality was added to these homegrown programs, they became mission-critical, often serving as the only way that these tasks were done.

From an IT perspective that's a nightmare. I found myself in an uncomfortable place being asked to support what were basically "hacks" around missing functionality or automation in applications. I decided early on that it was at least going to be a more interesting part of my job and so I steered into it and found myself really enjoying the process. AppleScript wasn't an amazing language to write in but the documentation was pretty good and I loved how happy it made people.

For many of you reading I'm sure you rolled your eyes a bit at that part, how happy it made people. I cannot stress enough some of the most impactful programming I maybe have ever done was during this period. You are, quite literally, freeing people to do other things. I had teachers telling me they were staying late 5+ hours a week to do the things we could automate and run. This form of basic programming with the end customer right there providing you direct feedback on whether it works is incredibly empowering for both parties.

It was also very easy to turn into a feature request. Think of all the bad internal tooling feature requests you get, with complicated explanations or workflows that make no sense when they're written down. Instead you had the full thing, from beginning to end, with all the steps laid out. I noticed a much shorter pipeline from "internal request" to "shipped feature" with these amateur automations I think in part to the clarity of the request that comes hand-in-hand with already seeing how the request works end to end.

As time went on, less and less work happened in desktop applications. Platform specific tooling like Automator and AppleScript became less relevant as the OS hooks they need to operate aren't present or can't be guaranteed. You can't really depend on mouse placement on a web page where the entire layout might change any second.

Now we're back at a place where I see people often wasting a ton of time doing exactly the sort of thing computers were supposed to prevent. Endless clicking and dragging, mind-numbing repetition of selecting check-boxes and copy/pasting values in form fields. The ideas present in technology like HyperCard and AppleScript have never been more needed in normal non-technical peoples lives, but there just doesn't seem to be any tooling for it in a browser world. Why?

What about RPA?

RPA, or "robotic process automation", is the attempted reintroduction of these old ideas into the web space. There are a variety of tools and the ones I've tried are UIPath and OpenRPA. Both have similar capabilities although UIPath is certainly more packed with features. The basic workflow is similar, but there are some pretty critical problems with it that make it hard to put into a non-technical users hands.

For those unaware the basic workflow of an RPA tool is mostly browser-based, often with some "orchestrator" process and a "client" process. A user installs a plugin to their browser that records their actions and allows them to replay it. You can then layer in more automation, either using built-in plugins to call APIs or by writing actions that look for things like CSS tags on the page.

Here are some of the big issues with the RPA ecosystem as it exists now:

1. There isn't a path for non-technical users to take from "Record" to building resilient workflows that functions unattended for long periods of time. The problem is again that websites are simply not stable enough targets to really program against using what you can see. You need to, at the very least, understand a bit about CSS in order to "target" the thing you want to hit reliably. Desktop applications you control the updates, but websites you don't.
2. The way to do this stuff reliably is often through that services API but then we've increased the difficulty exponentially. You now need to understand what an API is, how they work and be authorized by your organization to have a key to access that API. It's not a trivial thing in a large workplace to get a Google Workspaces key, so it unfortunately would be hard to get off the ground to begin with.
3. The tools don't provide good feedback to non-technical users about what happened. You just need too much background information about how browsers work, how websites function, what APIs are, etc in order to get even minimal value out of this tooling.
4. They're also really hard to setup on your computer. Getting started, running your first automation, is just not something you can do without help if you aren't technically inclined.

I understand why there has been an explosion of interest in the RPA space recently and why I see so many "RPA Developer" job openings. There's a legitimate need here to fill in the gaps between professional programming and task automation that is consuming millions of hours of peoples lives, but I don't think this is the way forward. If your technology relies on the use of existing APIs and professional tech workers to function, then its bringing minimal value to the table over just learning Python and writing a script.

The key secret for a technology like this has to be that the person who understands the nuance of the task has to be able to debug it. If it doesn't work, if the thing doesn't write to the database or whatever, maybe you don't know exactly why that happened, but because you have some context on how the task is done manually there exists the possibility of you fixing it.

You see this all the time in non-technical workplaces, a "black box" understanding of fixes. "Oh if you close that window, wait five minutes and open it again, it'll work" is one I remember hearing in a courthouse I was doing contract work in. Sure enough, that did fix the issue, although how they figured it out remains a mystery to me. People are not helpless in the face of technical problems but it also isn't their full-time job to fix them.

Why not just teach everyone to program?

I hate this ideology and I always have. Human civilization progresses in part due to specialization, that we as a society don't have to learn to do everything in order to keep things functional. There's a good argument to be made that we possibly have gone too far in the other direction, that now we understand too little about how the world around us functions, but that's outside the scope of this conversation.

There is a big difference between "having a task that would benefit from automation" and "being interested in investing the hundreds of hours to learn how to program". For most people it has nothing to do with their skills or interest and would be something they would need to learn in their off-time. But I love to program, won't they? My parents both love being lawyers but I can't think of anything I would want to do less right now.

Giving people a path from automation to "real programming", whatever that means, is great and I fully support it. But creating the expectation that the only way for people to automate tasks is to become an expert in the space is selfish thinking on the part of the people who make those services. It is easier for us if they take on the cognitive load vs if we design stuff that is easier to use and automate.

Is there a way forward to help people automate their lives?

I think there is, but I'm not sure if the modern paradigm for how we ship website designs works with it. Similar to website scraping, automation based on website elements and tags is not a forever solution and it can be difficult, if not impossible, for a user to discover that it isn't working anymore without adding manual checks.

There are tools users can use, things like Zapier that work really well for public stuff. However as more and more internal work moves to the browser, this model breaks down for obvious reasons. What would be great is if there was a way to easily communicate to the automation tool "these are the hooks on the page we promise aren't going to go away", communicated through the CSS or even some sort of "promise" page.

If some stability could be added to the process of basically scraping a webpage, I think the tooling could catch up and at least surface to the user "here are the stable targets on this page". As it exists now though, the industry treats APIs as the stable interface for their services and their sites as dynamic content that changes whenever they want. While understandable to some extent, I think it misses this opportunity to really expand the possibility of what a normal user can do with your web service.

It's now become very much the norm for employees to get onboarded and handed a computer where the most used application is the web browser. This has enabled incredible velocity when it comes to shipping new features for companies and has really changed the way we think about the idea of software. But with this has come cost incurred on the side of the actual user of this software, removing the ability to reliably automate their work.

If there was a way to give people some agency in this process, I think it would pay off hugely. Not only in the sheer hours saved of human life, but also with the quality of life for your end users. For the people who found and embraced desktop automation technology, it really allowed normal people to do pretty incredible things with their computers. I would love to see a future in which normal people felt empowered to do that with their web applications.

An opinionated take on the tool I use the most

Like many of you, my terminal emulator is probably my most used piece of software. My day begins with getting a cup of coffee, opening up Slack and iTerm 2, my terminal emulator for years. iTerm 2 has an incredible number of features, almost too many to list. Here are some of my most-used features just off the top of my head:

• Hotkey global terminal dropdown, meaning I can get into the terminal from any application I'm in
• Really good search, including support for regex
• Paste history, which like come on who doesn't use that 100 times a day
• Password manager. With the death of sudolikeaboss I've come to rely on this functionality just to deal with the mess of passwords that fill my life. Also if you know a replacement for sudolikeaboss that isn't the 1Password CLI let me know.
• Triggers, meaning you can write basic actions that fire when text matching a regex pattern is encountered. Nice for when you want the icon to bounce in the dock when a job is done in a dock or when you want the password manager to automatically open when a certain login prompt is encountered.

With all this flexibility comes complexity, which smacks you in the face the second you open the Preference pane inside of iTerm 2. I don't blame the developers for this at all, they've done a masterful job of handling this level of customization. But I've seen new users jaw drop when they click around this preference pane:

I have very few complaints with iTerm 2, but I'm always open to try something new. Someone on Twitter told me about Warp, a new terminal emulator written in Rust with some very interesting design patterns. I don't know if its the right terminal for me but it definitely solves problems in a new way.

What makes a good terminal emulator?

This is a topic that can stir a lot of feelings for people. Terminal emulators are a tool that people invest a lot of time into, moving them from job to job. However in general I would say these are the baseline features I would expect from a modern terminal emulator:

• Control over color, people don't all have the same setups
• Tabs, they're great in browsers and even better with terminals
• Profiles. Different setups for different terminals when you are doing totally isolated kinds of work. I like a visual indicator I'm working in production vs testing, for instance.
• Access to command history through the tool itself
• Bookmarks, while not a must-have are nice so you don't need to define a ton of bookmarks in your bash profile.
• Notifications of some sort.
• Control over fonts. I love fonts, it's just one of those things.

So why am I reviewing a terminal emulator missing most of these features or having them present in only limited configurations? Because by breaking away from this list of commonly agreed-upon "good features" they've managed to make something that requires almost no customization to get started. Along the way, they've added some really interesting features I've never seen before.

I requested an invite on their site and a few weeks later got the email inviting me to download it. First, huge credit to the Warp team. I respect the hell out of software with an opinion and Warp has a strong point of view. The default for development tools is to offer options for everything under the sun and to see someone come to the conversation with a tool that declares "there is a right way to do this" is intriguing. Here is what you see when you open warp:

From launch it wants you to know this is not your normal terminal emulator. It is trying to get you to do things the warp way from minute 1, which is great. The Command Palette is a lightning fast dropdown of all the Warp commands you might need. Search commands is just bringing up the previous commands from your history.

Executing commands in Warp is unlike anything I've ever seen before. Every command is broken into a Block which is a total rethink of the terminal. Instead of focusing primarily on the manipulation of text, you are focused on each command run as an independent unit you can manipulate through the UI. This is me trying to show what it looks like:

You'll notice the space and blocking between each command. Clicking those 3 dots gives you this dropdown:

All this functionality is available on your local machine but they are also available on machines you SSH (if the remote host is using bash). This opens up a massive collection of power text editing functionality on remote machines that might not be configured to be used as a "development machine". Check out that list here and think of how much time this might have saved you in your life.

Sharing

I think the primary sales point of Warp is the Sharing functionality, which allows for some very interesting workflows. It appears at some point you'll be able to add things like approval before you run commands (which I think is kind of a weird anti-pattern but still I applaud the idea). Right now though you can generate links to your specific block and share them with folks.

I made an example link you can see here. However I don't know how long the links last so here is a quick screenshot.

I love this for a million uses:

• it beats sharing snippets of text in Slack all the time
• it's a game changer for folks trying to do coding meetups or teaching a class
• adding in concepts like approval or review to commands would be mind-blowing for emergency middle of the night fixes where you want a group of people to review it. It doesn't seem to have this functionality yet but appears to be coming.

Daily Usage

Alright so I love a lot of the concepts but how much do I like using it as a daily driver? Let's focus on the positive stuff on Mac. I'm testing this on a 16 inch MacBook Pro with 32 GB of RAM, so about as powerful as it gets.

Pros

• Warp is just as fast as iTerm 2, which is to say so fast I can't make it choke on anything I tried.
• Steps into an area of the market that desperately needs more options, which is the multi-platform terminal emulator space. Warp is going to eventually be on Linux, Windows and Mac which right now is something only a handful of emulators can say, the biggest being alacritty.
• All this functionality comes out of the box. No giant configuration screens to go through, this all works from launch.

• I can't stress how "baked" this software feels. Everything works pretty much like they promise every time. Even with weird setups like inside of a tmux or processing tons of text, it kept working.
• The team is open to feedback and seems to be responsive. You can see their repo here.

Cons

• I don't love how much space the blocks end up taking up, even with "compact mode" turned on. I often take my laptop to my balcony to work and I miss the screen real estate with Warp that I get with iTerm 2.
• Missing profiles is a bummer. I like to be able to tweak stuff per workflow.
• I'd love some concept of bookmark if I'm going to lose so much space to the "Block" concept.
• My workflow is heavily invested in tmux and Vim, meaning I already have a series of shortcuts for how to organize and search my data into distinct blocks. I can change it for Warp, but right now that would be more of a lateral move than something that gives me a lot of benefits today.
• You really don't get a lot of customization. You can change the theme to one of their 7 preset themes. In terms of fonts, you have one of 11 options. My favorite themes and fonts weren't on this list.
• I would love some documentation on how I might write a plugin for Warp. There's stuff I would love to add but I couldn't really see how I might do that.
• A lot of the game-changing stuff is still in the pipeline, things like real-time collaboration and shared environmental variables.
• I wish they would share a bit more about how the app works in general. Even opening up the app bundle didn't tell me a lot. I have no reason to not trust this program, but anything they would be willing to share would be appreciated.

A Rust Mac App?

I'm very curious how they managed to make a Rust GUI application on the Mac. I'd love to see if there is some Swift UI or AppKit code in there or if they managed to get it done with the referenced Rust library. If they managed to make an application that feels this snappy without having to write Swift or Objective-C, all the more credit to this team. I'd love more information on how the app is constructed and specifically how they wrote the client front-end.

This does not feel like a "Mac app" though. Immediately you'll notice the lack of Preference pane underneath the "Warp" header on the menu bar. Help is not a drop-down but a search and in general there aren't a lot of MacOS specific options in the menu bar. This likely fits with their model of a common work platform across every OS, but if feeling "Mac-like" is important to you, know this doesn't. It's just as fast as a native application, but it doesn't have the UI feel of one.

Summary

If you are just starting out on the Mac as a development machine and want to use a terminal emulator, this is maybe the fastest to start with. It comes with a lot of the quality of life improvements you normally need to install a bunch of different pieces of software for. Also if you teach or end up needing to share a lot of code as you go, this "Sharing" functionality could be a real game-changer.

However if you, like me, spend your time mostly editing large blocks of text with Vim in the terminal, you aren't going to get a ton out of Warp right now. I just don't have a workflow that is going to really benefit from most of this stuff and while I appreciate their great tab completion, most of the commands I use are muscle memory at this point and have been for years.

I wish this team all the success though and cannot stress enough how great it is to see someone actually experimenting in this space. There are great ideas here, well executed with an eye for stability and quality. I'm going to keep a close eye on this product and will definitely revisit as things like "command reviews" are added down the line.

Like this? Think I'm out of my mind? Ping me on Twitter.

There is a lot of downtime in a modern tech workers life. From meetings to waiting for deployments to finish, you can spend a lot of time watching a progress bar. Or maybe you've been banging your head against a problem all day, getting nowhere. Perhaps the endless going from one room of your apartment to another that is modern work has driven you insane, dreading your "off-time" of sitting in a different room and watching a different screen.

I can't make the problems go away, but I can give you a little break from the monotony. Take advantage of one of the perks of forever WFH and read for 5-10 minutes. I find the context switching to be a huge mental relief and it makes me feel like I'm still getting something out of the time. I've tried to organize them for a variety of moods, but feel free to tell me on Twitter if I'm missing some.

Watching a progress bar

Got something compiling, deploying or maybe just running your tests? Here's some stuff to read while you wait.

• Ignition!: An informal history of liquid rocket propellants

A funny and insightful look into the history of how modern rocket fuel was developed. The writing style is very light and you can pick it up, read a few pages, check on your progress bar and get back to it. This is a part of the Space race story I didn't know much about.

• Why Fish Don't Exist: A Story of Loss, Love, and the Hidden Order of Life

A history of Taxonomy in the United States and the story of one of its stars, David Starr Jordan. It is a wild story, touching on the founding of Stanford University, the history of sterilization programs in the US and everything between. The primary theme is attempting to impose order among chaos, something I think many technology workers can relate to. I was hooked from the intro on.

Picture the person you love the most. Picture them sitting on the couch, eating cereal, ranting about something totally charming, like how it bothers them when people sign their emails with a single initial instead of taking those four extra keystrokes to just finish the job-

Chaos will get them. Chaos will crack them from the outside - with a falling branch, a speeding car, a bullet - or unravel them from the inside, with the mutiny of their very own cells. Chaos will rot your plants and kill your dog and rust your bike. It will decay your most precious memories, topple your favorite cities, wreck any sanctuary you can ever build.

• Blacktop Wasteland

A mistake is a lesson, unless you make the same mistake twice.

This is a great thriller, crime novel that is easy to pick up and burn through. The first chapter might be some of the tightest writing I've seen in awhile, introducing everything you need and not wasting your time. It's all about the last job a wheel-man needs to pull, a classic in these kinds of novels. Great little novel to pick up, read for 15 minutes and put back down.

• Forensics: What Bugs, Burns, Prints, DNA and More Tell Us About Crime

This is the history of Forensics as told through a series of cases, making each section digestable in a limited amount of time. While there is an understandable amount of skepticism about some areas of forensics, this book really sticks to the more established scientific practices. Of particular interest to me is why you might use one tool over another in different situations.

You are frustrated by a problem and want a break

Maybe you've thrown everything you have at a problem and somehow made it worse. Is that project you inherited from the person who no longer works here an undocumented rats nest of madness? Go sit in the beanbag chairs your office provides but nobody ever intended for you to sit in. You know the ones, by the dusty PlayStation that serves as a prop of how fun your office is. Take a load off and distract yourself with one of these gems.

• Trainspotting

I know, you saw the movie. The book is a classic for a good reason. It's funny, it is sad and the characters are incredibly authentic. However the biggest reason I recommend it for people needing a short mental break from a problem is it is written in a Scottish accent. You'll need to focus up a bit to get the jokes, which for me helps push problems out of my head for a few minutes.

• A Confederacy of Dunces

"Oh, Fortuna, blind, heedless goddess, I am strapped to your wheel," Ignatius belched. "Do not crush me beneath your spokes. Raise me on high, divinity."

If you have never had the pleasure of reading this book, I'm so jealous of you. Ignatius J. Reilly is one of the strangest characters I've ever been introduced to in a book. His journey around New Orleans is bizarre and hilarious, with a unique voice I've never read from an author before or since. You'll forget what you were working on in seconds.

• I Feel Bad About My Neck

“One of my favorite things about New York is that you can pick up the phone and order anything and someone will deliver it to you. Once I lived for a year in another city, and almost every waking hour of my life was spent going to stores, buying things, loading them into the car, bringing them home, unloading them, and carrying them into the house. How anyone gets anything done in these places is a mystery to me.”

Written by the hilarious Nora Ephron, it is an honest and deeply funny commentary about being a woman of a certain age. She's done it all, from writing hits like When Harry Met Sally to interning in the Kennedy White House. If you are still thinking about your problem 5 pages in, you aren't holding it right.

• Three Men in a Boat

It's from the late 1800s, so you will need to focus up a bit to follow the story. But there is a reason this comedic gem hasn't been out of print since it was introduced. It's the story of three men and a dog, but on a bigger level is about the "clerking class" of London, a group of people who if they lived today would likely be working in startups around the world. So sit back and enjoy a ride on the Thames.

You hate this work

We've all been there. You realize that whatever passion you had for programming or technology in general is gone, replaced with a sense of mild dread every time you open a terminal and go to the git repository. Maybe it was a boss telling you that you need to migrate from one cloud provider to another. Or maybe you found yourself staring out a window at someone hauling garbage and think well at least they can say they did something at the end of the day. I can't solve burnout, but I can allow you to indulge those feelings for awhile. Then back to the git repo, you little feature machine! We need a new graph for that big customer.

• New Maps - deindustrial fiction

A successor to the much-enjoyed Into the Ruins, this quarterly journal of short stories explores a post-industrial world. However this isn't Star Trek, but instead stories of messy futures with people making do. When it all feels too much in the face of climate change and the endless cycle of creation and destruction in technology, I like to reach for these short stories. They don't fill you with hope necessarily, but they are like aloe cream for burnout.

• An Archdruid's Tales: Fiction From The Archdruid Report

This is an anthology of the short stories published the now-gone Archdruid Report. They're a little strange and out there, but it's a similar energy to New Maps.

• The Golden Apples

If you want to escape modern life for a few hours and go to the 1950s South, this will do it. It's here, the good and bad, not really short stories but more a loosely combined collection of stories. If you've never been exposed to Eudora Welty’s writing, get ready for writing that is both light and surprisingly dense, packed full of meaning and substance.

• The Fortnight in September

This is a story of normal people living ordinary lives in England, after World War One. There is a simplicity to it that is delightful because it's not a trick. You will start to care about what happens to this community and the entire thing has a refreshing lack of gravity to it. It's a beach read, something to enjoy with low stakes but will stick with you days after you finish it.

• The Baron in the Trees

What's not to enjoy about an Italian noble who leaves everything behind to live in a tree for the rest of his life? About as right to the point as you can get, but it is also about the passing of an age in civilization, which feels appropriate right now.

For those looking for a fast overview of containers click here.

Containers have quickly become the favorite way to deploy software, for a lot of good reasons. They have allowed, for the first time, developers to test "as close to production" as possible. Unlike say, VMs, containers have a minimal performance hit and overhead. Almost all of the new orchestration technology like Kubernetes relies on them and they are an open standard, with a diverse range of corporate rulers overseeing them. In terms of the sky-high view, containers have never been in a better place.

I would argue though that in our haste to adopt this new workflow, we missed some steps. To be clear, this is not to say containers are bad (they aren't) or that they aren't working correctly (they are working mostly as advertised). However many of the benefits to containers aren't being used by organizations correctly, resulting in a worse situation than before. While it is possible to use containers in a stable and easy-to-replicate workflow across a fleet of servers, most businesses don't.

We're currently in a place where most organizations relying on containers don't use them correctly. At the same time, we also went back 10+ years in terms of the quality of tools Operation teams have for managing servers as defined broadly as "places where our code runs and accepts requests". There has been a major regression inside of many orgs who now tolerate risks inside containers that never would have been allowed on a fleet of virtual machines.

For me a lot of the blame seems to rest with Dockerfiles. They aren't opinionated enough or flexible enough, forcing people into workflows where they can make catastrophic mistakes with no warning, relying too much on brittle bash scripts and losing a lot of the tools we gained in Operations over the last decade.

What did containers replace?

In the beginning, there were shell scripts and they were bad. The original way a fleet of servers was managed when I started was, without a doubt, terrible. There was typically two physical machines for the databases, another 4 physical machines for the application servers, some sort of load balancer, and then networking gear at the top of the rack. You would PXE boot the box onto an install VLAN and it would kind of go from there.

There was a user with an SSH key added, usually admin. You would then run a utility to rsync a directory of bash scripts that would run. Very quickly, you would run into problems. Writing bash scripts is not "programming light", it's just real programming. But it's programming with both hands tied behind your back. You still need to be writing functions, encapsulate the logic, handling errors, etc. But bash doesn't want to help you do this.

You can still get thrown with undefined variables, comparisons vs assignment is a constant issue when people start writing bash ( foo=bar vs foo = bar), you might not check to make sure bash is the shell you are running and a million other problems. Often you had these carefully composed scripts with raw sh just in case the small things bash does to make your life better were not there. I have worked with people who are expert bash programmers and can do it correctly, but it is not a safer, easier or more reliable programming environment.

Let's look at a basic example I see all the time.

for f in $(ls *.csv); do    
    some command $f         
done

I wish this worked like I assumed it did for years. But it doesn't. You can't treat ls like a stable list and iterate over it. You'll have to account for whitespace in the file name, checking for glob characters, ls can mangle filenames. This is just a basic example of something that everyone assumes they are doing right until it causes a massive problem.

The correct way I know to do this looks like this:

while IFS= read -r -d '' file; do
  some command "$file"
done < <(find . -type f -name '*.csv' -print0)

Do you know what IFS is? It's ok if you don't, I didn't for a long time. My point is that this requires a lot of low-level understanding of how these commands work in conjunction with each other. But for years around the world, we all made the same mistakes over and over. However, things began to change for the better.

As time went on new languages became the go-to for Sys Admin tasks. We started to replace bash with python, which was superior in every imaginable way. Imagine being able to run a debugger with a business-critical bootstrapping script? This clearly emerged as the superior paradigm for Operations. Bash still has a place, but it couldn't be the first tool we reached for every time.

So we got new tools to match this new understanding. While there are a lot of tools that were used to manage fleets of servers, I'm going to focus on the one I used the most professionally: Ansible. Ansible is a configuration management framework famous for a minimal set of dependencies (Python and SSH), being lightweight enough to deploy to thousands of targets from a laptop and having a very easy-to-use playbook structure.

Part of the value of Ansible was its flexibility. It was very simple to write playbooks that could be used across your organization, applying different configurations to different hosts depending on a variety of criteria, like which inventory they were in or what their hostnames were. There was something truly magical about being able to tag a VLAN at the same time as you stood up a new database server.

Ansible took care of the abstraction between things like different Linux distributions, but its real value was in the higher-level programming concepts you could finally use. Things like sharing playbooks between different sets of servers, writing conditionals for whether to run a task or not on that resource, even writing tests on information I could query from the system. Finally, I could have event-driven system administration code, an impossibility with bash.

With some practice, it was possible to use tools like Ansible to do some pretty incredible stuff, like calling out to an API with lookups to populate information. It was a well-liked, stable platform that allowed for a lot of power. Tools like Ansible Tower allowed you to run Ansible from a SaaS platform that made it possible to keep a massive fleet of servers in exact configuration sync. While certainly not without work, it was now possible to say with complete confidence "every server in our fleet is running the exact same software". You could even do actual rolling deploys of changes.

This change didn't eliminate all previous sources of tension though. Developers could not just upgrade to a new version of a language or install random new binaries from package repositories on the system. It created a bottleneck, as changes had to be added to the existing playbooks and then rolled out. The process wasn't too terrible but it wasn't hands-off and could not be done on-demand, in that you could not decide in the morning to have a new cool-apt-package in production by that afternoon.

Then containers appeared

When I was first introduced to Docker, I was overjoyed. This seemed like a great middle step between the two sets of demands. I could still rely on my mature tooling to manage the actual boxes, but developers would have control and responsibility for what ran inside of their containers.  Obviously, we would help but this could be a really good middle ground. It certainly seemed superior to developers running virtual machines on their laptops.

Then I sat down and started working with containers and quickly the illusion was shattered. I was shocked and confused, this was the future? I had to write cron jobs to clean up old images, why isn't this a config file somewhere? Why am I managing the docker user and group here? As it turns out installing docker would be the easy part.

Application teams began to write Dockerfiles and my heart started to sink. First, because these were just the bash scripts of my youth again. The learning curve was exactly the same, which is to say a very fast start and then a progressively more brutal arc. Here are some common problems I saw the first week I was exposed to Dockerfiles that I still see all the time:

• FROM: ubuntu:latest Already we have a problem. You can pull that down to your laptop, work for a month, deploy it to production, and be running a totally different version of Ubuntu. You shouldn't use latest but you also shouldn't be using other normal tags. The only tool Docker gives you to ensure everyone is running the exact same thing is the SHA. Please use it. FROM ubuntu@sha256:cf25d111d193288d47d20a4e5d42a68dc2af24bb962853b067752eca3914355e is less catchy but it is likely what you intended. Even security updates should be deliberate.
• apt-get is a problem. First, don't run apt-get upgrade otherwise we just upgraded all the packages and defeated the point. We want consistent, replicable builds. I've also seen a lot of confusion between users on apt vs apt-get.
• COPY yourscript.py before RUN install dependencies breaks the caching functionality.
• Running everything as root. We never let your code run as root before, why is it now suddenly a good idea? RUN useradd --create-home cuteappusername should be in there.
• Adding random Linux packages from the internet. I understand it worked for you, but please stick to the official package registry. I have no idea what this package does or who maintains it. Looking at you, random curl in the middle of the Dockerfile.
• Writing brittle shell scripts in the middle of the Dockerfile to handle complicated operations like database migrations or external calls, then not accounting for what happens if they fail.
• Please stop putting secrets in ENV. I know, we all hate secrets management.
• Running ADD against unstable URL targets. If you need it, download it and copy it to the repo. Stop assuming random URL will always work.
• Obsessing about container size over everything else. If you have a team of Operations people familiar with Debian, following Debian releases, plugging into the ecosystem, why throw all that expertise in the trash for a smaller container?
• Less && and && \please. This one isn't your fault but sometimes looking at complicated Dockerfiles makes my eyes hurt.
• Running a full Linux container for a script. Thankfully Google has already solved this one.

This is not your fault

You may be looking at this list and be like "I know all this because I read some article or book". Or maybe you are looking at this list and thinking "oh no I do all of that". I'm not here to judge you or your life. Operations people knew this was a problem and it was a problem we had come to the conclusion could not be fixed by assuming people would magically discover this information.

My frustration is that we already went through this learning. We know that the differences between how distros handle packages throw people off. We know bash scripts are hard to write and easy to mess up. The entire industry learned through years of pain that it was essential you be able to roll back not just your application, but the entire infrastructure that the application is running on. Creating endless drift in infrastructure worked until it didn't when suddenly teams had to spend hours trying to reverse engineer what of the dozens of changes introduced with the latest update caused a problem.

In our rush to get to a place where obstacles were removed from developers, we threw away years of hard-earned experience. Hoping for the best and having absolutely no way to recover if it doesn't work isn't a plan. It isn't even really a philosophy. Saying "well the Linux part isn't the important part of my application" is fine until that is very much not the case. Then you are left in an extremely difficult position, reaching for troubleshooting skills your organization might not even have anymore.

Stuff we can do now

• Start running a linter against our Dockerfiles: https://github.com/hadolint/hadolint
• Look at alternatives to conventional Dockerfiles. Below is an example of combining Ansible and the Dockerfile template.

FROM debian@sha256:47b63f4456821dcd40802ac634bd763ae2d87735a98712d475c523a49e4cc37e

# Install Ansible
RUN apt-get update && apt-get install -y wget gcc make python python-dev python-setuptools python-pip libffi-dev libssl-dev libyaml-dev
RUN pip install -U pip
RUN pip install -U ansible

# Setup environment
RUN mkdir /ansible
COPY . /ansible
ENV ANSIBLE_ROLES_PATH /ansible/roles
ENV ANSIBLE_VAULT_PASSWORD_FILE /ansible/.vaultpass

# Launch Ansible playbook
RUN cd /ansible && ansible-playbook -c local -v example.yml

# Cleanup
RUN rm -rf /ansible
RUN apt-get purge -y python-dev python-pip
RUN apt-get autoremove -y && apt-get autoclean -y && apt-get clean -y

# Final steps
ENV HOME /home/test
WORKDIR /
USER test

CMD ["/bin/bash"]

• Better than this would be to use Packer. It allows for developers to string together Docker as a builder and Ansible or Puppet as a provisioner! It's the best of all possible worlds. Here are the details. Plus you can still run all the Dockerfile commands you want.

Place I would love to get to

I would love for some blessed alternative to Dockerfiles to emerge. We don't want to break backwards compatibility but I would love a less brittle tool to work with. Think like Terraform or Packer, something sitting between me and the actual build. It doesn't need to be a full programming language but some guardrails around me making common mistakes is desperately needed, especially as there are fewer and fewer restrictions between developers and production.

Questions/comments/does this tool already exist and I don't know about it? Hit me up on twitter.

My code doesn't compile. Why?

The number of times in my career I have been asked a variation on "why doesn't my application work" is shocking. When you meet up with Operations people for drinks, you'll hear endless variations on it. Application teams attempting to assign ownership of a bug to a networking team because they didn't account for timeouts. Infrastructure teams being paged in the middle of the night because an application suddenly logs 10x what it did before and there are disk space issues. But to me nothing beats the developer who pings me being like "I'm getting an error message in testing from my application and I'd like you to take a look".

It is baffling on many levels to me. First, I am not an application developer and never have been. I enjoy writing code, mostly scripting in Python, as a way to reliably solve problems in my own field. I have very little context on what your application may even do, as I deal with many application demands every week. I'm not in your retros or part of your sprint planning. I likely don't even know what "working" means in the context of your app.

Yet as the years go on the number of developers who approach me and say "this worked on my laptop, it doesn't work in the test environment, why" has steadily increased. Often they have not even bothered to do basic troubleshooting, things like read the documentation on what the error message is attempting to tell you. Sometimes I don't even get an error message in these reports, just a development saying "this page doesn't load for me now but it did before". The number of times I have sent a full-time Node developer a link to the Node.js docs is too high.

Part of my bafflement is this is not acceptable behavior among Operations teams. When I was starting out, I would never have wandered up to the Senior Network Administrator and reported a bug like "I sometimes have timeouts that go away. Do you know why?" I would have been politely but sternly told to do more troubleshooting. Because in my experience Operations is learned on the job, there was a culture of training and patience with junior members of the team. Along with that was a clear understanding that it was my obligation to demonstrate to the person I was reporting this error to:

1. What specifically the error was.

2. Why that error was something that belonged to them.

Somehow in the increased lack of distinction between Development and Operations, some developers, especially younger ones, have come to see Operations as their IT department. If a problem wasn't immediately recognizable as one resulting from their work, it might be because of "the servers" or "the network", meaning we could stop what we were doing and ask Operations to rule that out before continuing.

How did we get here?

First they came for my QA team...

When I started my career in Operations, it was very different from what exists today. Everything lived in or around datacenters for us, the lead time on new servers was often measured in months not minutes and we mostly lived in our own world. There were network engineers, managing the switches and routers. The sysadmins ruled over the boxes and, if the org was large enough, we had a SAN engineer managing the massive collection of data.

Our flow in the pipeline was pretty simple. The QA team approved some software for release and we took that software to our system along with a runbook. We treated software mostly like a black box, with whatever we needed to know contained inside of the runbook. Inside were instructions on how to deploy it, how to tell if it was working and what to do if it wasn't working. There was very little expectation that we could do much to help you. If a deployment went poorly in the initial rollout, we would roll back and then basically wait for development to tell us what to do.

There was not a lot of debate over who "owned" an issue. If the runbook for an application didn't result in the successful deployment of an application, it went back to development. Have a problem with the database? That's why we have two DBAs. Getting errors on the SAN? Talk to the SAN engineer. It was a slow process at times, but it wasn't confusing. Because it was slower, often developers and these experts could sit down and share knowledge. Sometimes we didn't agree, but we all had the same goal: ship a good product to the customer in a low-stress way.

Deployments were events and we tried to steal from more mature industries. Runbooks were an attempt to quantify the chaotic nature of software development, requiring at least someone vaguely familiar with how the application worked to sit down and write something about it. We would all sit there and watch error logs, checking to see if some bash script check failed. It was not a fast process as compared to now but it was simple to understand.

Of course this flow was simply too straightforward and involved too many employees for MBAs to allow it to survive. First we killed QA, something I am still angry about. The responsibility for ensuring that the product "worked as intended" was shifted to development teams, armed with testing frameworks that allowed them to confirm that their API endpoints returned something like the right thing. Combined with the complete garbage fire that is browser testing, we now had incredibly clunky long running testing stacks that could roughly approximate a single bad QA engineer. Thank god for that reduced headcount.

With the removal of QA came increased pressure to ship software more often. This made sense to a lot of us as smaller more frequent changes certainly seemed less dangerous than infrequent massive changes of the entire codebase. Operations teams started to see more and more pressure to get stuff out the door quickly. New features attract customers, so being the first and fastest to ship had a real competitive advantage. Release windows shrunk, from cutting a release every month to every week. The pressure to ship also increased as management looked at the competitive landscape growing more aggressive with cycles.

Soon the runbook was gone and now developers ran their own deployment schedule, pushing code out all the time. This was embraced with a philosophy called DevOps, a concept that the two groups, now that QA was dead and buried, would be able to tightly integrate to close this gap even more. Of course this was sold to Development and Operations as if it would somehow "empower" better work out of them, which was of course complete nonsense.

Instead we now had a world where all ownership of problems was muddled and everyone ended up owning everything.

DevOps is not a decision made in isolation

When Operations shifted focus to the cloud and to more GitOps style processes, there was an understanding that we were all making a very specific set of tradeoffs. We were trading tight cost control for speed, so never again would a lack of resources in our data centers cause a single feature not to launch. We were also trading safety for speed. Nobody was going to sit there and babysit a deploy, the source of truth was in the code. If something went wrong or the entire stack collapsed, we could "roll back", a concept that works better in annoying tech conference slide decks then in practice.

We soon found ourselves more pressed than ever. We still had all the responsibilities we had before, ensuring the application was available, monitored, secure and compliant. However we also built and maintained all these new pipelines, laying the groundwork for empowering development to get code out quickly and safely without us being involved. This involved massive retraining among operations teams, shifting from their traditional world of bash scripts and Linux to learning the low-level details of their cloud provider and an infrastructure as code system like Terraform.

For many businesses, the wheels came off the bus pretty quickly. Operations teams struggled to keep the balls in the air, shifting focus between business concerns like auditing and compliance to laying the track for Development to launch their products. Soon many developers, frustrated with waiting, would attempt to simply "jump over" Operations. If something was easy to do in the AWS web console on their personal account, certainly it was trivial and safe to do in the production system? We can always roll back!

In reality there are times when you can "roll back" infrastructure and there are times you can't. There are mistakes or errors you can make in configuring infrastructure that are so catastrophic it is difficult to estimate their potential impact to a business. So quickly Operations teams learned they needed to install rails to infrastructure as code, guiding people to the happy safe path in a reliable and consistent way. This is slow though and after awhile started to look a lot like what was happening before with datacenters. Businesses were spending more on the cloud than on their old datacenters but where was the speed?

Inside engineering, getting both sides of the equation to agree in the beginning "fewer blockers to deploying to production is good" was the trivial part. The fiercer fights were over ownership. Who is responsible in the middle of the night if an application starts to return errors? Historically operations was on-call, relying on those playbooks to either resolve the problem or escalate it. Now we had applications going out with no documentation, no clear safety design, no QA vetting and sometimes no developers on-call to fix it. Who owns an RDS problem?

Tools like Docker made this problem worse, with developers able to craft perfect application stacks on their laptops and push them to production with mixed results. As cloud providers came to provide more and more of the functionality, soon for many teams every problem with those providers also fell into Operations lap. Issues with SQS? Probably an Operations issue. Not sure why you are getting a CORS error on S3? I guess also an Operations problem!

The dream of perfect harmony was destroyed with the harsh reality that someone has to own a problem. It can't be a community issue, someone needs to sit down and work on it. You have an incentive in modern companies to not be the problem person, but instead to ship new features today. Nobody gets promoted for maintenance or passing a security audit.

Where we are now

In my opinion the situation has never been more bleak. Development has been completely overwhelmed with a massive increase in the scope of their responsibilities (RIP QA) but also with unrealistic expectations by management as to speed. With all restrictions lifted, it is now possible and expected that a single application will get deployed multiple times a day. There are no real limiters except for the team itself in terms of how fast they can ship features to customers.

Of course this is just a fundamental misunderstanding about how software development works. It isn't a factory and they aren't "code machines". The act of writing code is a creative exercise, something that people take pride in. Developers, in my experience, don't like shipping bad or rushed features. What we call "technical debt" can best be described as "the shortcuts taken today that have to be paid off later". Making an application is like building a house, you can take shortcuts but they aren't free. Someone pays for them later, but probably not the current executive in charge of your specific company so who cares.

Due to this, developers are not incentivized or even encouraged to gain broader knowledge of how their systems work. Whereas before you might reasonable be expected to understand how RabbitMQ works, SQS is "put message in, get message out, oh no message is not there, open ticket with Ops". This situation has gotten so bad that we have now seen the adoption of widespread large-scale systems like Kubernetes who attempt to abstract away the entire stack. Now there is a network overlay, a storage overlay, healthchecks and rollbacks all inside the stack running inside of the abstraction that is a cloud provider.

Despite the bullshit about how this was going to empower us to do "our best work faster", the results have been clear. Operations is drowning, forced to learn both all the fundamentals their peers had to learn (Linux, networking, scripting languages, logging and monitoring) along with one or more cloud providers (how do network interfaces attach to EC2 instances, what are the specific rules for how to invalidate caches on Cloudfront, walk me through IAM Profiles). On top of all of that, they need to understand the abstraction on top of this abstraction, the nuance of how K8 and AWS interact, how storage works with EBS, what are you monitoring and what is it doing. They also need to learn more code than before, now often expected to write relatively complicated internal applications which manage these processes.

With this came monitoring and observability responsibilities as well. Harvesting the metrics and logs, shipping them somewhere, parsing and shipping them, then finally making them consumable by development. A group of engineers who know nothing about how the application works, who have no control over how it functions or what decisions it makes, need to own determining whether it is working or not. The concept makes no sense. Nuclear reactor technicians don't ask me if the reactor is working well or not, I have no idea what to even look for.

Developers simply do not have the excess capacity to sit down and learn this. They are certainly intellectually capable, but their incentives are totally misaligned. Every meeting, retro and sprint is about getting features out the door faster, but of course with full test coverage and if it could be done in the new cool language that would be ideal. When they encounter a problem they don't know the answer to, they turn to the Operations team because we have decided that means "the people who own everything else in the entire stack".

It's ridiculous and unsustainable. Part of it is our fault, we sell tools like Docker and Kubernetes and AWS as "incredibly easy to use", not being honest that all of them have complexity which matter more as you go. That testing an application on your laptop and hitting a "go to production" button works, until it doesn't. Someone will always have to own that gap and nobody wants to, because there is no incentive to. Who wants to own the outage, the fuck up or the slow down? Not me.

In the meantime I'll be here, explaining to someone we cannot give full administrator IAM rights to their serverless application just because the internet said that made it easier to deploy. It's not their fault, they were told this was easy.

Thoughts/opinions? @duggan_mathew on twitter