I'll admit I am late to learning these exist. About once a month, maybe more, I need to spin up some sort of debug container inside of Kubernetes. It's usually for something trivial like checking networking or making sure DNS isn't being weird and it normally happens inside of our test stack. Up until this point I've been using a conventional deployment, but it turns out there is a better option. kukectl debug
It allows you to attach containers to a running pod, meaning you don't need to bake the troubleshooting tools into every container (because you shouldn't be doing that). The Kubernetes docs go into a lot more detail.
In combination with this, I found some great premade debug containers. Lightrun makes these Koolkits in a variety of languages. I've really found them to be super useful and well-made. You do need Kubernetes v1.23 or above, but the actual running is super simple.
kubectl debug -it --image=lightrun-platform/koolkits/koolkit-node --image-pull-policy=Never --target=
In addition to node they have Golang, Python and JVM. With Python it's possible to debug most networking problems with simple commands. For example, a DNS lookup is just:
import socket
socket.getaddrinfo("matduggan.com")`
I regularly use the socket library for tons of networking troubleshooting and ensuring ports and other resources are not blocked by a VPC or other security policy. You can find those docs here.
For those in a datacenter consider using the excellent netmiko library.