Bringing all of my projects together under one umbrella
What is the issue anyway?
Over the years, I have accumulated a bunch of virtual servers on my DigitalOcean account for small experimental projects I dabble in. And this has resulted in quite a bill. I mean, I wouldn't care if these projects were actually being used. But there were just being there unused and wasting resources. Which makes this an unnecessary burden for me.
Most of them are just small HTML pages that have an endpoint or two to read data from or to, and for that reason I wrote servers left and right. To be honest, all of those things could have been done with CGI scripts and that would have been more than enough.
Recently, I decided to stop language hopping and focus on a simpler stack which includes C, Go and Lua. And I can accomplish all the things I am interested in.
Finding a web server replacement
Usually I had Nginx in front of these small web servers and I had to manage SSL certificates and all that jazz. I am bored with these things. I don't want to manage any of this bullshit anymore.
So the logical move forward was to find a solid alternative for this. I have ended up on Caddy server. I've used it in the past but kind of forgotten about it. What I really like about it is an ease of use and a bunch of out of the box functionalities that come with it.
These are the pitch points from their website:
- Secure by Default: Caddy is the only web server that uses HTTPS by default. A hardened TLS stack with modern protocols preserves privacy and exposes MITM attacks.
- Config API: As its primary mode of configuration, Caddy's REST API makes it easy to automate and integrate with your apps.
- No Dependencies: Because Caddy is written in Go, its binaries are entirely self-contained and run on every platform, including containers without libc.
- Modular Stack: Take back control over your compute edge. Caddy can be extended with everything you need using plugins.
I had just a few requirements:
- Automatic SSL
- Static file server
- Basic authentication
- CGI script support
And the vanilla version does all of it, but CGI scripts. But that can easily be fixed with their modular approach. You can do this on their website and build a custom version of the server, or do it with Docker.
This is a Dockerfile
I used to build a custom server.
FROM caddy:builder AS builder
RUN xcaddy build \
--with github.com/aksdb/caddy-cgi
FROM caddy:latest
RUN apk add --no-cache nano
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
Getting rid of all the unnecessary virtual machines
The next step was to get a handle on the number of virtual servers I have all over the place.
I decided to move all the projects and services into two main VMs:
- personal server (still Nginx)
- git server
- static file server
- personal blog
- projects server (Caddy server)
- personal experiments
- other projects
I will focus on projects' server in this post since it's more interesting.
Testing CGI scripts
The first thing I tested was how CGI scripts work under Caddy. This is particularly import to me because almost all of my experiments and mini projects need this to work.
To configure Caddy server, you must provide the server with a configuration
file. By default, it's called Caaddyfile
.
{
order cgi before respond
}
examples.mitjafelicijan.com {
cgi /bash-test /opt/projects/examples/bash-test.sh
cgi /tcl-test /opt/projects/examples/tcl-test.tcl
cgi /lua-test /opt/projects/examples/lua-test.lua
cgi /python-test /opt/projects/examples/python-test.py
root * /opt/projects/examples
file_server
}
- The order is very important. Make sure that
order cgi before respond
is at the top of the configuration file. - Also, when you run with Caddy v2, make sure you provide
adapter
argument like this/usr/bin/caddy run --watch --environ --config /etc/caddy/Caddyfile --adapter caddyfile
. Otherwise, Caddy will try to use a different format for config file.
I did a small batch of tests with Bash, Tcl, Lua and Python. Here is a cheat sheet if you need it.
Let's get Bash out of the way first.
#!/usr/bin/bash
printf "Content-type: text/plain\n\n"
printf "Hello from Bash\n\n"
printf "PATH_INFO [%s]\n" $PATH_INFO
printf "QUERY_STRING [%s]\n" $QUERY_STRING
printf "\n"
for i in {0..9..1}; do
printf "> %s\n" $i
done
exit 0
This one is for Tcl script.
#!/usr/bin/tclsh
puts "Content-type: text/plain\n"
puts "Hello from Tcl\n"
puts "PATH_INFO \[$env(PATH_INFO)\]"
puts "QUERY_STRING \[$env(QUERY_STRING)\]"
puts ""
for {set i 0} {$i < 10} {incr i} {
puts "> $i"
}
And for all you Python enjoyers.
#!/usr/bin/python3
import os
print("Content-type: text/plain\n")
print("Hello from Python\n")
print("PATH_INFO [{}]".format(os.environ['PATH_INFO']))
print("QUERY_STRING [{}]".format(os.environ['QUERY_STRING']))
print("")
for i in range(10):
print("> {}".format(i))
And for the final example, Lua.
#!/usr/bin/lua
print("Content-type: text/plain\n")
print("Hello from Lua\n")
print(string.format("PATH_INFO [%s]", os.getenv("PATH_INFO")))
print(string.format("QUERY_STRING [%s]", os.getenv("QUERY_STRING")))
print()
for i = 0, 9 do
print(string.format("> %d", i))
end
Basic authentication
One thing was also to have an option for some sort of authentication, and something like Basic access authentication would be more than enough.
Thankfully, Caddy supports this out of the box already. Below is an updated example.
{
order cgi before respond
}
examples.mitjafelicijan.com {
cgi /bash-test /opt/projects/examples/bash-test.sh
cgi /tcl-test /opt/projects/examples/tcl-test.tcl
cgi /lua-test /opt/projects/examples/lua-test.lua
cgi /python-test /opt/projects/examples/python-test.py
root * /opt/projects/examples
file_server
basicauth * {
bob $2a$14$/wCgaf9oMnmQa20txB76u.nI1AldGMBT/1J7fXCfgOiRShwz/JOkK
}
}
basicauth *
matches everything under this domain/sub-domain and protects it
with Basic Authentication.
bob
is the usernamehash
is the password
To generate these passwords, execute caddy hash-password
and this will prompt
you to insert a password twice and spit out a hashed password that you can put
in your configuration file.
Restart the server and you are ready to go.
Making Caddy a service with systemd
After the tests were successful, I copied caddy
to /usr/bin/caddy
and copied
Caddyfile
to /etc/caddy/Caddyfile
.
Now off to the systemd. Each systemd service requires you to create a service file.
- I created a
/etc/systemd/system/caddy.service
and put the following content in the file.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=root
Group=root
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile --adapter caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force --adapter caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
- You might need to reload systemd with
systemctl daemon-reload
. - Then I enabled the service with
systemctl enable caddy.service
. - And then I started the service with
systemctl start caddy.service
.
This was about all that I needed to do to get it running. Now I can easily add new subdomains and domains to the main configuration file and be done with it. No manual Let's Encrypt shenanigans needed.
Other posts
- My journey from being an internet über consumer to being a full hominum again
- Golang profiling simplified
- Software development and my favorite pitfalls
- Using DigitalOcean Spaces Object Storage with FUSE
- What I've learned developing ad server
- Trying to build a New kind of terminal emulator for the modern age
- Trying out Helix code editor as my main editor
- My love and hate relationship with Node.js
- Elitist attitudes are sapping all the fun from programming
- Replacing Dropbox in favor of DigitalOcean spaces
- Using Digitalocean Spaces to sync between computers
- Wireless Application Protocol and the mobile web before the web
- Running Golang application as PID 1 with Linux kernel
- Debian based riced up distribution for Developers and DevOps folks
- Who knows what the world will look like tomorrow