Proxies cannot peer into your HTTPS calls, and cache, unless you let them.
Install on Ubuntu a build of squid that is capable of bump/splice packet inspection to permit HTTPS caching
At the time of writing 2020-04-23 the build of squid that was available was built without --with-openssl and --enable-ssl-crtd “./configure” directives. Those are required to turn squid into a very powerful packet inspection engine that can inspect https traffic routed through it. This is conditional on the client permitting the, most often, self signed certificate to “trust” the proxy endpoint to break open the packets
This is a rather large two part post on HTTPS with Proxy, as we pass through several technologies and interleave encryption through each stage. This continues on from previous posts regarding Docker and proxy cache in the past. As I’ll use a docker container to later to demonstrate flushing of cache and re-requesting from Maven external repos. Key, again, to this is reading Squid’s access.log, to not only determine that Squid was accessed by the Maven process, but how it was accessed. Decoding the following Status Codes for squid logs is key — you will likely refer to this often. Specifically: our work is not complete until we witness GET calls to HTTPS packages, resulting in a TCP_HIT or TCP_MEM_HIT outcome in access.log.
This process requires:
Most distributions, Ubuntu 19.10 included compile the squid server binary without –with-openssl or –enable-ssl-crtd libraries compiled into it. The first directive openssl provides squid’s capability to use openssl generate certificates to “spoof” endpoint certificates — you’re effectively going to make squid make a man-in-the-middle attach on your packets. The second directive ssl-crtd provides a certificate storage daemon, to place certificates into a lookup database, more about that later.
Other very useful directives already come with your existing version that you’ve already installed. I recommend you keep these details, as your distro has probably thought hard about these, particularly the performance optimisations for your hardware (x86_64 etc).
On your existing squid instance run squid -v - for me this produced:
configure options: ‘–build=x86_64-linux-gnu’ ‘–disable-arch-native’ ‘–disable-dependency-tracking’ ‘–disable-maintainer-mode’ ‘–disable-silent-rules’ ‘–disable-translation’ ‘–enable-async-io=8’ ‘–enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB’ ‘–enable-auth-digest=file,LDAP’ ‘–enable-auth-negotiate=kerberos,wrapper’ ‘–enable-auth-ntlm=fake,SMB_LM’ ‘–enable-build-info=Ubuntu linux’ ‘–enable-cache-digests’ ‘–enable-delay-pools’ ‘–enable-ecap’ ‘–enable-esi’ ‘–enable-eui’ ‘–enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group’ ‘–enable-follow-x-forwarded-for’ ‘–enable-icap-client’ ‘–enable-icmp’ ‘–enable-inline’ ‘–enable-linux-netfilter’ ‘–enable-removal-policies=lru,heap’ ‘–enable-security-cert-validators=fake’ ‘–enable-storeid-rewrite-helpers=file’ ‘–enable-storeio=ufs,aufs,diskd,rock’ ‘–enable-url-rewrite-helpers=fake’ ‘–enable-zph-qos’ ‘–’–srcdir=.’ ‘–with-build-environment=default’ ‘–with-default-user=proxy’ ‘–with-filedescriptors=65536’ ‘–with-gnutls’ ‘–with-large-files’ ‘BUILDCXX=x86_64-linux-gnu-g++’ ‘BUILDCXXFLAGS=-g -O2 -fdebug-prefix-map=/build/squid-4.11=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,–as-needed’ ‘CC=x86_64-linux-gnu-gcc’ ‘CFLAGS=-g -O2 -fdebug-prefix-map=/build/squid-4.11=. -fstack-protector-strong -Wformat -Werror=format-security -Wall’ ‘CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2’ ‘CXX=x86_64-linux-gnu-g++’ ‘CXXFLAGS=-g -O2 -fdebug-prefix-map=/build/squid-4.11=. -fstack-protector-strong -Wformat -Werror=format-security’ ‘build_alias=x86_64-linux-gnu’ –enable-ltdl-convenience
At the time of writing, Squid 4.11 is the latest, freshly built three days earlier, and published here. Download the source, and unpack it.
VERSION='4.11'
$ wget --directory-prefix=/tmp/ \
"http://www.squid-cache.org/Versions/v4/squid-${VERSION}.tar.gz"
$ tar --directory=/tmp/ --extract --gunzip --verbose \
--file="/tmp/squid-${VERSION}.tar.gz"
$ cd /tmp/squid-${VERSION}
We will be building from source code, and there are three stages to compiling a Squid binary. configure, make & make install. Squid is written in C++ so you’ll require the GNU C++ compiler g++` and GNU Make make fortunately these are preinstalled in Ubuntu 19.10. More complicated to diagnose is what other packages are required to be installed before compilation.
In Ubuntu package extensions to permit building against your OS libraries are suffixed with *-dev These include “shared object” (*.so) library files that permit linking (hence don’t required to be compiled into your binary), and C/C++ Header files (*.h) , those familiar with C/C++ will have used these, but for now lets say there are required files in addition to your source program and need to be present to permit compilation of the source code.
The above statements placed you in the root of the source code for Squid. you should see an executable file called configure . This is step is where you consider all aspects of how you want this binary to be placed on your distro. Think carefully about this, particularly if you want to have multiple versions co-existing on your system. Furthermore, monitoring the configure output is critical to confirming that your binary will “glue” to all the existing libraries on your distro. Notably, if you add another directive to your configure command statement, it’s likely that your disro does not have the required *-dev packages to support the additional functionality and will need to be installed before you will succeed with an error free configure execution.
For example: I got the following error:
configure: Log daemon helpers to be built: DB file … checking db.h usability… no checking db.h presence… no checking for db.h… no configure: error: external acl helper session … found but cannot be built
Now in Fedora, the dnf package manager has a dnf provides command that states the packages that provide the file/program you search for. Ubuntu/Debian’s APT package manager requires installation of apt-file to do this using sudo apt install apt-file; sudo apt-file update. Thence you can search for db.h and discover what package is required to step past this.
$ apt-file find 'db.h' #this spits out 2550 lines of *db.h files$ apt-file find db.h | grep '\/db.h\b'
freebsd-glue: /usr/include/freebsd/db.h
htdig: /usr/include/htdig_db/db.h
libbind-dev: /usr/include/dns/db.h
libbind-export-dev: /usr/include/bind-export/dns/db.h
libdb5.3-dev: /usr/include/db.h
libgroonga-dev: /usr/include/groonga/groonga/db.h
libgwenhywfar-core-dev: /usr/include/gwenhywfar4/gwenhywfar/db.h
libknot-dev: /usr/include/libknot/db/db.h
libleveldb-dev: /usr/include/leveldb/db.h
libnbcompat-dev: /usr/include/nbcompat/db.h
librocksdb-dev: /usr/include/rocksdb/db.h
mlocate: /usr/include/mlocate/db.h$ sudo apt install --yes libdb5.3-dev
In the end I discovered 7 missing packages by trial and error:
sudo apt install --yes \
libecap3-dev libghc-gnutls-dev libldap2-dev \
libpam0g-dev libsasl2-dev libkrb5-dev libdb5.3-dev
For me my combined configure statement became:
VERSION='4.11'
./configure \
'--with-openssl' \
'--enable-ssl-crtd' \
"--prefix=/opt/squid-${VERSION}" \
'--build=x86_64-linux-gnu' \
'--datadir=${prefix}/data' \
'--includedir=${prefix}/include' \
'--infodir=${prefix}/info' \
'--libexecdir=${prefix}/lib' \
'--localstatedir=${prefix}/var/state' \
'--mandir=${prefix}/man' \
'--sysconfdir=${prefix}/etc' \
'--with-logdir=${prefix}/var/log' \
'--with-pidfile=${prefix}/var/run/squid.pid' \
'--with-swapdir=${prefix}/var/swap' \
'--disable-arch-native' \
'--disable-dependency-tracking' \
'--disable-maintainer-mode' \
'--disable-silent-rules' \
'--disable-translation' \
'--enable-async-io=8' \
'--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' \
'--enable-auth-digest=file,LDAP' \
'--enable-auth-negotiate=kerberos,wrapper' \
'--enable-auth-ntlm=fake,SMB_LM' \
'--enable-build-info=Ubuntu linux' \
'--enable-cache-digests' \
'--enable-delay-pools' \
'--enable-ecap' \
'--enable-esi' \
'--enable-eui' \
'--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group' \
'--enable-follow-x-forwarded-for' \
'--enable-icap-client' \
'--enable-icmp' \
'--enable-inline' \
'--enable-linux-netfilter' \
'--enable-ltdl-convenience' \
'--enable-removal-policies=lru,heap' \
'--enable-security-cert-validators=fake' \
'--enable-storeid-rewrite-helpers=file' \
'--enable-storeio=ufs,aufs,diskd,rock' \
'--enable-url-rewrite-helpers=fake' \
'--enable-zph-qos' \
'--srcdir=.' \
'--with-build-environment=default' \
'--with-default-user=proxy' \
'--with-filedescriptors=65536' \
'--with-gnutls' \
'--with-large-files' \
'BUILDCXX=x86_64-linux-gnu-g++' \
'BUILDCXXFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed' \
'CC=x86_64-linux-gnu-gcc' \
'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall' \
'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' \
'CXX=x86_64-linux-gnu-g++' \
"CXXFLAGS=-g -O2 -fdebug-prefix-map=/build/squid-${VERSION}=. -fstack-protector-strong -Wformat -Werror=format-security" \
'build_alias=x86_64-linux-gnu'
The resultant artifacts after ./compile … should be the following files:
config.status Makefile config.sh
This should be the least painful bit but it will take a few minutes, this uses g++ to compile all the source, the binaries will be staged locally first.
$ make
Next step requires sudo as I will place this into the /opt directory, written to only by a privileged user, but accessable by anybody for read or execute. Worthwhile noting the following:
./configure step above./usr/…,
it should be uninstalled using: sudo apt remove --purge squid .Those conditions out of thw way, let’s build:
$ sudo make install
Once this completes you can update the path value.
$ export PATH="/opt/squid-4.11/sbin:/opt/squid-4.11/bin:$PATH"
Thence we will be able to run squid --version and get the squid daemon installed at /opt/squid-4.11/sbin/squid.
$ squid --version
Squid Cache: Version 4.11
Service Name: squid
Ubuntu linuxThis binary uses OpenSSL 1.1.1c 28 May 2019. For legal restrictions on distribution see https://www.openssl.org/source/license.htmlconfigure options: '--with-openssl' '--enable-ssl-crtd' '--prefix=/opt/squid-4.11' '--build=x86_64-linux-gnu' '--datadir=${prefix}/data' '--includedir=${prefix}/include' '--infodir=${prefix}/info' '--libexecdir=${prefix}/lib' '--localstatedir=${prefix}/var/state' '--mandir=${prefix}/man' '--sysconfdir=${prefix}/etc' '--with-logdir=${prefix}/var/log' '--with-pidfile=${prefix}/var/run/squid.pid' '--with-swapdir=${prefix}/var/swap' '--disable-arch-native' '--disable-dependency-tracking' '--disable-maintainer-mode' '--disable-silent-rules' '--disable-translation' '--enable-async-io=8' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,SMB_LM' '--enable-build-info=Ubuntu linux' '--enable-cache-digests' '--enable-delay-pools' '--enable-ecap' '--enable-esi' '--enable-eui' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group' '--enable-follow-x-forwarded-for' '--enable-icap-client' '--enable-icmp' '--enable-inline' '--enable-linux-netfilter' '--enable-ltdl-convenience' '--enable-removal-policies=lru,heap' '--enable-security-cert-validators=fake' '--enable-storeid-rewrite-helpers=file' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-url-rewrite-helpers=fake' '--enable-zph-qos' '--srcdir=.' '--with-build-environment=default' '--with-default-user=proxy' '--with-filedescriptors=65536' '--with-gnutls' '--with-large-files' 'BUILDCXX=x86_64-linux-gnu-g++' 'BUILDCXXFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed' 'CC=x86_64-linux-gnu-gcc' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXX=x86_64-linux-gnu-g++' 'CXXFLAGS=-g -O2 -fdebug-prefix-map=/build/squid-4.11=. -fstack-protector-strong -Wformat -Werror=format-security' 'build_alias=x86_64-linux-gnu'
For my Ubuntu distribution, missing from a installation via make install, is a Systemd *.service file, to manage stopping and starting and reloading configuration changes for the squid server. This ended up being a non-trivial excersise, so it’s inclusion here is important.
VERSION=4.11
PREFIX="/opt/squid-${VERSION}"
cat <<EOF | sudo tee /etc/systemd/system/squid-${VERSION}.service
[Unit]
Description=Squid Web Proxy Server
Documentation=man:squid(8)
After=network.target network-online.target nss-lookup.target[Service]
Type=forking
PIDFile=/opt/squid-${VERSION}/var/run/squid.pid
ExecStartPre=/opt/squid-${VERSION}/sbin/squid --foreground -z
ExecStart=/opt/squid-${VERSION}/sbin/squid -sYC
#ExecReload=/bin/kill -HUP $MAINPID
ExecReload=sudo -u proxy /opt/squid-${VERSION}/sbin/squid -k reconfigure
KillMode=mixed[Install]
WantedBy=multi-user.target
EOF
#unfortunately, we have the wrong permissions the custom log
#directory. This needs to be updated for the squid "proxy" user.
sudo chgrp proxy /opt/squid-4.11/var/log
sudo chmod g+w /opt/squid-4.11/var/log
sudo systemctl start squid-${VERSION}.service
sudo systemctl status squid-${VERSION}.service
A few notes on this systemd service file: It is designed to use the process kill command to manage restart and reload, however I have chosen to use the native cli command to manage squid config reload, you’ll note it needs to be done as the proxy user too, so there’s a sudo to that user, otherwise files get written as root, and it breaks the program. Also I had to override the Type directive to forking as the default is simple and it disallows Squid’s multi-threaded architecture. KillMode allows signals SIGKILL to be executed against the processes to shutdown the main or kids (being child processes under the main instance). See below Parent/Children squid processes:
$ ps -fC squid
UID PID PPID C STIME TTY CMD
root 8530 1 0 10:07 ? /opt/squid-4.11/sbin/squid -sYC
proxy 8532 8530 0 10:07 ? (squid-1) --kid squid-1 -sYC
Great! You’ve now done one of the hardest things a sysadmin is required to do — managed “dependency hell” caused by installing required packages outside of a package dependency management tool like apt or yum or dnf that has binaries precompiled and dependency managed. This is a required task in Ubuntu 19.10, as this Linux distribution does not come with binary enable ment to inspect HTTPS and manage it’s own certs.