Skip to content

Easier alternative to Nginx + Lets Encrypt with Caddy Docker Proxy

So this is a request I get probably 4-5 times a year. "I'm looking to host a small application in docker and I need it to be easy to run through a GitLab/GitHub CICD pipeline, it needs SSL and I never ever want to think about how it works." Up until this point in my career the solution has been pretty consistent: Nginx with Let's Encrypt. Now you might think "oh, this must be a super common request and very easy to do." You would think that.

However the solution I've used up to this point has been frankly pretty shitty. It usually involves a few files that look like this:

services:
    web: 
        image: nginx:latest
        restart: always
        volumes:
            - ./public:/var/www/html
            - ./conf.d:/etc/nginx/conf.d
            - ./certbot/conf:/etc/nginx/ssl
            - ./certbot/data:/var/www/certbot
        ports:
            - 80:80
            - 443:443

    certbot:
        image: certbot/certbot:latest
        command: certonly --webroot --webroot-path=/var/www/certbot --email [email protected] --agree-tos --no-eff-email -d domain.com -d www.domain.com
        volumes:
            - ./certbot/conf:/etc/letsencrypt
            - ./certbot/logs:/var/log/letsencrypt
            - ./certbot/data:/var/www/certbot

This sets up my webserver with Nginx bound to host ports 80 and 443 along with the certbot image. Then I need to add the Nginx configuration to handle forwarding traffic to the actual application which is defined later in the docker-compose file along with everything else I need. It works but its a hassle. There's a good walkthrough of how to set this up if you are interested here: https://pentacent.medium.com/nginx-and-lets-encrypt-with-docker-in-less-than-5-minutes-b4b8a60d3a71

This obviously works but I'd love something less terrible. Enter Caddy Docker Proxy: https://github.com/lucaslorentz/caddy-docker-proxy. Here is an example of Grafana running behind SSL:

services:
  caddy:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    ports:
      - 80:80
      - 443:443
    environment:
      - CADDY_INGRESS_NETWORKS=caddy
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy_data:/data
    restart: unless-stopped
    
  grafana:
    environment:
      GF_SERVER_ROOT_URL: "https://GRAFANA_EXTERNAL_HOST"
      GF_INSTALL_PLUGINS: "digiapulssi-breadcrumb-panel,grafana-polystat-panel,yesoreyeram-boomtable-panel,natel-discrete-panel"
    image: grafana/grafana:latest-ubuntu
    restart: unless-stopped
    volumes:
      - grafana-storage:/var/lib/grafana
      - ./grafana/grafana.ini:/etc/grafana/grafana.ini
    networks:
      - caddy
    labels:
      caddy: grafana.example.com
      caddy.reverse_proxy: "{{upstreams 3000}}"

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - prometheus-data:/prometheus
      - ./prometheus/config:/etc/prometheus
    ports:
      - 9090:9090
    restart: unless-stopped
    networks:
      - caddy

How it works is super simple. Caddy listens on the external ports and proxies traffic to your docker applications. In return, your docker applications tell Caddy Proxy what url they need. It goes out, generates the SSL certificate for grafana.example.com as specified above and stores it in its volume. That's it, otherwise you are good to go.

Let's use the example from this blog as a good test case. If you want to set up a site that is identical to this one, here is a great template docker compose for you to run.

services:

  ghost:
    image: ghost:latest
    restart: always
    networks:
      - caddy
    environment:
      url: https://matduggan.com
    volumes:
      - /opt/ghost_content:/var/lib/ghost/content
    labels:
      caddy: matduggan.com
      caddy.reverse_proxy: "{{upstreams 2368}}"

  caddy:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    ports:
      - 80:80
      - 443:443
    environment:
      - CADDY_INGRESS_NETWORKS=caddy
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy_data:/data
    restart: unless-stopped

networks:
  caddy:
    external: true

volumes:
  caddy_data: {}

So all you need to do in order to make a copy of this site in docker-compose is:

  1. Install Docker Compose.
  2. Run docker network create caddy
  3. Replace matduggan.com with your domain name
  4. Run docker-compose up -d
  5. Go to your domain and set up your Ghost credentials.

It really couldn't be more easy and it works like that for a ton of things like Wordpress, Magento, etc. It is, in many ways, idiot proof AND super easy to automate with a CICD pipeline using a free tier.

I've dumped all the older Let's Encrypt + Nginx configs for this one and couldn't be happier. Performance has been amazing, Caddy as a tool is extremely stable and I have not had to think about SSL certificates since I started. Strongly recommend for small developers looking to just get something on the internet without thinking about it or even larger shops where you have an application that doesn't justify going behind a load balancer.


Download Mister Rogers Neighborhood with Python

A dad posted on a forum I frequent in Denmark asking for some help. His child loves Mister Rogers, but he was hoping for a way to download a bunch of episodes that didn't involve streaming them from the website to stick on an iPad. I love simple Python projects like this and so I jumped on the chance. Let me walk you through what I did.

If you just want to download the script you can skip all this and find the full script here.

Step 1: Download Youtube-DL

My first thought was of youtube-dl for the actual downloading and thankfully it worked great. This is one of those insanely useful utilities that I cannot recommend highly enough. You can find the download instructions here: http://ytdl-org.github.io/youtube-dl/download.html

Step 2: Install Python 3

You shouldn't need a super modern version of python. I wrote this with Python 3.7.3, so anything that number or newer should be good. We are using f strings because I love them, so you will need 3.6 or newer.

Download Python here.

I'm checking the version here but only to confirm that you are running Python 3, on the assumption that if you have 3 you have a relatively recent version of 3.

version = platform.python_version_tuple()
if version[0] != "3":
    print("You are not running Python 3. Please check your version.")
    sys.exit(1)

Step 3: Decide where you are going to download the files

I have my download location in the script here:

path = "/mnt/usb/television/mister-rogers-neighborhood/"

However if you just want them to download into the Downloads folder, uncomment the line above this one by removing the # and delete the line I show above. So you'll want path = str(Path.home() / "Downloads") to not have a # in front of it.

Step 4: Run the script

Not sure how to run a Python script? We got you taken care of. Click here for Windows. Here are some Mac tips.

You can find the script on Gitlab here: https://gitlab.com/-/snippets/2100082

Download the script and run it locally. The script checks if it is the first or third Monday of the month and only runs the download if it is. This is to basically keep us from endlessly spamming the servers hosting this great free content.

The first Monday of every month will feature programs from the early years 1968-1975. The third Monday of every month will feature programs from the “Theme Weeks” library 1979-2001.

NOTE: If you just want to download 5 episodes right now, delete these lines:

today = date.today().isocalendar()
if today[2] == 1 and (today[1] == 1 or 3):
    logging.info("There is a new download available.")
else:
    logging.info("There are no new downloads today.")
    sys.exit(0)

Step 5: Set the script to run every day

This script is designed to be run every day and only go out to the servers if there is a new file to get.

Here is how to run a python script every day on Windows.

For Linux and Mac open up your terminal, run crontab -e and enter in the frequency you want to run the script at. Here is a useful site to generate the whole entry.

File Formatting

Here is the metadata formatting I followed for the Infuse iOS app, my favorite app. You may want a different format for the filename depending on your application.

Questions?

If people actually use this script I'll rewrite it to use celery beat to handle the scheduling of the downloads, but for my own use case I'm comfortable writing cron jobs. However if you run into issues running this, either add a comment on the GitLab link or shoot me an email: mat at matduggan.com.


This Bloomberg Story Makes No Sense

Someone Is Making Something Up

One of the biggest stories right now in the tech world is the bombshell dropped by Bloomberg that the biggest names in tech have been hacked. Not through a software exploit or through some gap in a firewall, but by the Chinese government infiltrating their server supply chain. You can read the story here.

This resonates with tech professionals for a lot of good reasons. First, the people who know how to make these servers do something useful for an end user are rarely the same people who understand the specifics of how they work. I might know how to bootstrap and set up a bunch of Dell servers, but if asked to explain in intricate detail how iDRAC functions or what specifically can be done to harden the server itself, I would have to plead some ignorance. There's just so much to learn in this field that one person can't do it all. At some point I have to trust the thing I just unboxed from Dell or Supermicro or whomever is safe to use.

So when the assertion came out that the supply chain itself for the hardware we all rely on may have been breached, it shook us all to our cores. First we rely on these companies to provide us with services not easily duplicated elsewhere. I don't know a lot of stacks anymore that don't rely on some AWS functionality, so saying all of that information may have been released to the Chinese government is akin to your bank saying it might have lost the keys to the safety deposit boxes. There's also the knowledge that we cannot individually all vet these vendors. I lack the training to disable every PCB inside of every model of device in my datacenter and determine whether it's safe and up to spec. I don't even have the diagrams to do this assuming I could! Basically Bloomberg asserted everyone's worse fears.

Then cracks started to show up. Not small cracks, not companies covering their butts kind of problems. Amazon denies the story 100% in a way where there is no room for debate as to what they mean.. Amazon wasn't alone in these strong denials. Apple denied it as well in a letter to Congress. I would challenge anyone to read these statements and come away thinking these were parties attempting to leave room for doubt. Which is incredibly strange!

As time goes on we start to hear more problems with the actual reporting. One of the few named sources in the story, who provided technical background and context, admits it feels strange that almost everything he told the reporter as to how these attacks might happen is later apparently confirmed by different sources. I would encourage folks to listen to Joe Fitzpatrick here. Then the people who WERE supporting the narrative started to come out of the woodwork and they raised more questions than they answered. Yossi Appleboum, CEO of Sepio Systems (a firm with deep ties to the intelligence community based on their companies "About" page) comes out swinging that this is a much bigger problem! The scope is indeed even higher than Bloomberg asserts. You can read his take here.

Someone is lying, clearly. Either this hack didn't happen or it did happen and companies are willing to lie on the record to everyone involved. The later scenario feels unlikely for a number of reasons. One, because we know from recent events like the Facebook hack and the Google+ hack, the penalty for being caught leaking user data isn't that high. It would be a stock-price hit for Apple and Amazon if indeed this was true, but certainly one they could recover from. However the PR hit would appear to be relatively minor and you could bury the entire thing in enough technical details to throw off the average reader. Like I doubt if I told a random person on the street who uses and enjoys their iPhone that some of the servers making up the iCloud infrastructure might have been compromised by Chinese, they would swear off the company forever.

If it isn't "this hack happened and everyone involved is attempting to cover it up", then what is it? A false flag attack by the intelligence community on China? Sure maybe, but why? I've never seen anything like this before where a major news outlet that seemingly specializes in business news gets shut down by major industry players and continues writing the story. Whatever this story is, it feels like these companies coming out strongly against it is sending a message. They're not going to play along with whatever this narrative is. It also doesn't appear that foreign nationals would even need to go through all this trouble. Turns out the Supermicro firmware wasn't the best to start with on its own.

At the end of this saga I think either some folks at Bloomberg need to be let go or we need to open investigations into being deceived by officers of publicly traded companies. I don't see any other way for this to resolve.


Git Tips and Tricks

Git is an amazing tool that does a lot of incredible things. It's also a tool that is easy to screw up. Here is a list of tips I've found useful for when working with git.

Git Up

So when you run git pull chances are what you actually want to do is rebase over them. You also are probably working in a "dirty workspace" which means you want to pull the stuff ahead of you and rebase over it but until you are ready to commit, you don't want your work to be reflected. (I'm saying all this like its a certainty, I'm sure for a lot of people it's not. However I work in an extremely busy repo all day).

We're gonna set a new git alias for us that will do what we actually want to do with git.
git config --global alias.up 'pull --rebase --autostash'

Autostash basically makes a temporary stash before you run the rest of the rebase and then applies it back after the operation ends. This lets you work out of a dirty directory. Make sure you sanity check your changes though since the final stash application has the potential to do some weird things.

Approving Stuff As You Git Add

Been working for awhile and ready to add stuff to a commit but maybe don't remember all the changes you've made? Not a problem.

git add -p lets you step through your changes one by one and only add the ones to this commit you need. I use this as a final sanity check to ensure some fix I didn't mean to go out with this deploy doesn't sneak out.

Go Back In Commit History

git reflog will show you a list of everything done across all branches along with the index HEAD@{index}

git reset HEAD@{index} will let you fix it

Commited to the wrong branch

Go to the branch you want: git checkout right-branch

Grab your commit: git cherry-pick SHA of commit (get this from git log) or if its the last commit on a branch just the name of the branch

If needed, delete the commit from the wrong branch: git checkout wrong-branch then git reset HEAD~ --hard to knock out the last commit

You can pass an argument to HEAD to undo multiple commits if needed, knucklehead: git reset HEAD~3 to undo three

Squash Commits Down

First just do a soft reset passing HEAD the argument for how many commits you want to go back: git reset --soft HEAD~3

Then take your previous commit messages and make them your new unified commit: git commit --edit -m"$(git log --format=%B --reverse HEAD..HEAD@{1})"

Fix Messed Up Commit Message

git commit --amend and then just follow the prompts


MegaCLI Cheat Sheet

Everyone who has worked with specific vendors servers for awhile has likely encountered megacli. This is the comman line tool to manage your RAID controller and disks. Since often I'll only end up doing this every six months or so, I usually forget the syntax and decided to write it down here.

Install megacli

sudo apt-get install megacli

Common Parameters

Controller Syntax: -aN

This is telling MegaCLI the adapter ID. Don't use the ALL flag, always use this because you'll get in a bad habit and forget which boxes have multiple controllers

Physical drive: -PhysDrv [E:S]

I hate this format. E is the enclosure drive ID where the drive is and S is the slot number starting at 0. You get this info with the megacli -EncInfo -aALL

Virtual drive: -Lx

Used for specifying the virtual drive (where x is a number starting with zero or the string all).

Information Commands

Get Controller Information:

  • megacli -AdpAllInfo -aALL
  • megacli -CfgDsply -aALL
  • megacli -adpeventlog -getevents -f lsi-events.log -a0 -nolog

Common Operations

Replace drive:

  • Set the drive offline with megacli -PDOffline -PhysDrv[E:S] -aN
  • Mark the drive as missing: megacli -PDMarkMissing -PhysDrv [E:S] -aN
  • Get the drive ready to yank out: megacli -PDPrpRmv -PhysDrv [E:S] -aN
  • Change/replace the drive: megacli -PdReplaceMissing -PhysDrv[E:S] -ArrayN -rowN -aN
  • Start the drive: megacli -PDRbld -Start -PhysDrv [E:S] -aN

Fix Drive marked as "foreign" when inserted:

This happens when you steal a drive from a existing RAID to put in your more critical array. I know, you've never done anything so evil. Anyway when you need to here's how you do it.

  • Flip to good: megacli -PDMakeGood -PhysDrv [E:S] -aALL
  • Clear the foreign (that feels offensive but we're gonna move on): megacli -CfgForeign -Clear -aALL

Shut the stupid alarm off for good

  • megacli -AdpSetProp AlarmDsbl -aALL