FreeBSD offers two basic technologies for installing third-party applications. The easiest way is installing precompiled packages from official FreeBSD's package repositories. Somewhat more complicated, but more flexible way is compiling and installing applications using ports. Main shortcoming of official precompiled packages is the fact that they are being built with default set of options, whereas advanced setups often need to enable non-default option (e.g. LDAP authentication), change target database support (e.g. PostgreSQL instead of MySQL), or to disable undesirable default option (e.g. X11 support on headless servers). Luckily, there's additional way which combines flexibility of ports with simplicity of packages provided by Poudriere. The following article will help you set your own FreeBSD package repository where you will be able to compile your own packages from ports, and serve them to your other FreeBSD boxes. In a jail, of course!

jail.conf Tweaking

Previous article in this series, Initial Jail Setup, gives instruction about setting up a jail. Now it's time to put it to good use - compiling packages for many more jails that are about to be set up in the future, using Poudriere. Poudriere uses jails to build packages, which means it will spawn additional hierarchical jails. Our current jail.conf on jailhost does not permit this, which is why we need to edit it as follows:

Make sure to always edit highlighted lines according to your environment.

You can safely use copy button in code blocks - it will copy only text and commands, not line numbers and terminal prompts .

# jailhost.example.org:/etc/jail.conf

path          = "/usr/jails/${host.hostname}";
exec.start    = "/bin/sh /etc/rc";
exec.stop     = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

pkg_example_org {
  host.hostname    = pkg.example.org;
  host.domainname  = example.org;
  ip4.addr         = 'lo1|127.0.1.11/32';
  ip4.addr        += 'em0|192.0.2.11/32';

  ip4.addr        += 'lo0|127.0.0.1/32';
  children.max     = 20;
  enforce_statfs   = 1; 
  sysvshm          = new;
  sysvsem          = new;
  persist;
  allow.chflags;
  allow.mount.devfs;
  allow.mount.procfs;
  allow.mount;
  allow.mount.devfs;
  allow.mount.procfs;
  allow.mount.nullfs;
  allow.mount.tmpfs;
  allow.socket_af;
}

Familiarize yourself with jail(8) and understand what all of the above options mean.

In first article of this series, Initial Jail Setup, I wrote that our jails will have their own loopback interfaces, in order to prevent them from communicating directly over jail host's loopback. This jail, pkg.example.org will be exception to the rule, because poudriere currently requires access to lo0 interface on 127.0.0.1.

Install Required Packages

After jail restart, we can ssh into it and install some packages from default official FreeBSD quarterly package repository.

pkg
pkg install \
  databases/memcached \
  devel/ccache-memcached-static \
  ports-mgmt/poudriere \
  sysutils/screen \
  www/apache24
Poudriere Configuration

We need to edit poudriere.conf as follows:

# pkg.example.org:/usr/local/etc/poudriere/poudriere.conf

NO_ZFS=yes
FREEBSD_HOST=https://download.FreeBSD.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=no
USE_TMPFS=yes
DISTFILES_CACHE=$BASEFS/distfiles
PKG_REPO_SIGNING_KEY=/etc/ssl/selfsigned/example.org/poudriere/poudriere-privkey.pem
CCACHE_DIR=$BASEFS/ccache
CCACHE_STATIC_PREFIX=/usr/local
CCACHE_DIR_NON_ROOT_SAFE=yes
RESTRICT_NETWORKING=no
PARALLEL_JOBS=2
PREPARE_PARALLEL_JOBS=6
NOLINUX=yes
ALLOW_MAKE_JOBS=yes
URL_BASE=http://pkg.example.org
MAX_EXECUTION_TIME=172800
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=5
BUILDER_HOSTNAME=pkg.example.org
BUILD_AS_NON_ROOT=yes
PORTBUILD_USER=nobody
HTML_TRACK_REMAINING=yes

Familiarize yourself with poudriere.conf.sample which gives detailed comments about all of the above settings, and more.

Poudriere Jail Management

We are now ready to create our poudriere jail:

poudriere jail -j 12_2:x86:64 -c -v 12.2-RELEASE

12.2+ jail creation currently gives awk: can't open file /sys/param.h. This appears to be harmless and should already be fixed in poudriere-devel.

If something went bad, we can start from scratch after deleting the jail:

poudriere jail -j 12_2:x86:64 -d

Always use above command to delete poudriere jails, don't just try to delete files with rm -rf like we do with standard jails.

It is possible to have multiple jails in poudriere which differ by OS version or architecture, also to fetch them using methods other than default http, but I am not going to cover them in this tutorial. For detailed information check poudriere-jail(8).

Poudriere Ports Tree Management

Next thing to do would be to fetch current snapshot of ports tree:

poudriere ports -c

If something went bad, we can start from scratch after deleting it:

poudriere ports -d

Port trees managed by poudriere should alway be deleted using above command, not with rm -rf.

After initial ports tree creation, consequent updates are done with:

poudriere ports -u

It is possible to have multiple port trees in poudriere, also to fetch them using methods other than default portsnap, but I am not going to cover them in this tutorial. For detailed information check poudriere-ports(8).

Tweaking Port Options

Poudriere has the ability to build multiple sets of ports. I find it useful for having separate sets for headless servers, whose ports don't need X11 port options, and workstations who do. Options are being configured by specifically named config files which are actually make.conf(5), where parts of their filenames have significance for poudriere.

In order for options to be applied to ports built in poudriere jail named 12_2:x86:64, from default ports tree, for server set, a file named 12_2:x86:64-default-server-make.conf should be placed under /usr/local/etc/poudriere.d/. Here is how ours look like:

# pkg.example.org:/usr/local/etc/poudriere.d/12_2:x86:64-default-server-make.conf

MAKE_JOBS_NUMBER= 6
OPTIONS_UNSET+= CUPS DOCS X11
devel_apr1_SET+= LDAP
www_apache24_SET+= AUTHNZ_LDAP LDAP

MAKE_JOBS_NUMBER instructs poudriere to use up to 6 CPU cores simultaneously to build single port. Product of MAKE_JOBS_NUMBER and PARALLEL_JOBS shouldn't exceed nuber of CPU cores on jail host. Building ports is RAM and CPU intensive process, so we need to take care not to exhaust all the resources needed for other jails that will reside on same jail host.. Any options we want to enable or disable globally for all the ports should be declared in OPTIONS_SET and OPTIONS_UNSET lines, respectively. Any options we want to enable or disable for specific port should be declared in category_port_SET and category_port_UNSET lines, respectively. Specific port options declared after global options will override global options.

FreshPorts website gives detailed information about FreeBSD ports.

Creating Required Folders and Files

We need to create folder where distfiles (source tarballs) are going to be stored, as declared earlier in poudriere.conf, as well as folder to store our lists of ports to be built:

mkdir /usr/local/poudriere/distfiles
mkdir /usr/local/etc/poudriere.d/pkglists/

Inside this folder we will create file named server containing port names in category/port origin format:

# pkg.example.org:/usr/local/etc/poudriere.d/pkglists/server

ports-mgmt/pkg
www/apache24

Feel free to add additional packages to your pkglists. Only top level ports need to be added, dependencies will be built automatically.

We need to create directory for storing private key to sign our packages, as well as public certificate which will need to be imported on all the hosts which will use this repository. We will also set appropriate permissions - private key should be readable only to root, while public certificate should be world-readable.

mkdir -p /etc/ssl/selfsigned/example.org/poudriere/
openssl genrsa -out /etc/ssl/selfsigned/example.org/poudriere/poudriere-privkey.pem 4096
openssl rsa -in /etc/ssl/selfsigned/example.org/poudriere/poudriere-privkey.pem \
  -pubout -out /etc/ssl/selfsigned/example.org/poudriere/poudriere-cert.pem
chmod 400 /etc/ssl/selfsigned/example.org/poudriere/poudriere-privkey.pem
chmod 444 /etc/ssl/selfsigned/example.org/poudriere/poudriere-cert.pem

Finally, we need to create ccache directory in order to cache compilation results and hopefully speed up consequent builds:

mkdir /usr/local/poudriere/ccache
chown nobody:nobody /usr/local/poudriere/ccache
A Few Words About screen

I prefer to build packages from screen session which gives me the ability to detach from console and leave build job running, as opposed to standard ssh session where closing session kills the job running inside of it. I always prefer to start screen as unprivileged user and get root with su once inside, not the other way around - get root with su and start screen. Actually I prefer to use sudo but we haven't installed and configured it so far - we will do it later in the series.

A short digression for crash course in screen:

  • screen session is started with screen command without any additional arguments.
  • Detaching from screen session is done by pressing first ctrl+a followed by d.
  • Listing screen sessions belonging to currently logged user is done by screen -l. Sessions will be listed as attached or detached.
  • If there's only one deteched screen we can attach to it with screen -r.
  • If there are multiple detached screen sessions we can attach to each one with screen -r <id>.
  • If we need to attach to already attached screen session (e.g. when we remained attached on another computer), we can do it with screen -d -r <id>. This will detach session that was previously attached.
Bulk Building

I like to symlink my pkglists to root's home dir so I don't need to type their full path.

ln -s /usr/local/etc/poudriere.d/pkglists/server ~/server

We can now start our first bulk build:

poudriere bulk -j 12_2:x86:64 -z server -f ~/server
Setting Up Apache Web Server

While packages are getting built, we need to set up Apache web server so we can monitor building process and check build logs in nice web inteface. Also, other FreeBSD hosts will acces package repository over http.

Setting up web server can admittedly be done in much easier manner, but I like to have apache preforked, and also to serve everything from name-based vhosts, not directly from main server.

We need a few config files for apache:

# pkg.example.org:/usr/local/etc/apache24/httpd.conf

ServerRoot "/usr/local"
Listen pkg.example.org:80

LoadModule authn_file_module libexec/apache24/mod_authn_file.so
LoadModule authn_core_module libexec/apache24/mod_authn_core.so
LoadModule authz_host_module libexec/apache24/mod_authz_host.so
LoadModule authz_groupfile_module libexec/apache24/mod_authz_groupfile.so
LoadModule authz_user_module libexec/apache24/mod_authz_user.so
LoadModule authz_core_module libexec/apache24/mod_authz_core.so
LoadModule access_compat_module libexec/apache24/mod_access_compat.so
LoadModule auth_basic_module libexec/apache24/mod_auth_basic.so
LoadModule reqtimeout_module libexec/apache24/mod_reqtimeout.so
LoadModule filter_module libexec/apache24/mod_filter.so
LoadModule mime_module libexec/apache24/mod_mime.so
LoadModule log_config_module libexec/apache24/mod_log_config.so
LoadModule env_module libexec/apache24/mod_env.so
LoadModule headers_module libexec/apache24/mod_headers.so
LoadModule setenvif_module libexec/apache24/mod_setenvif.so
LoadModule version_module libexec/apache24/mod_version.so
LoadModule unixd_module libexec/apache24/mod_unixd.so
LoadModule status_module libexec/apache24/mod_status.so
LoadModule autoindex_module libexec/apache24/mod_autoindex.so
LoadModule dir_module libexec/apache24/mod_dir.so
LoadModule alias_module libexec/apache24/mod_alias.so
LoadModule mpm_event_module libexec/apache24/mod_mpm_event.so

IncludeOptional etc/apache24/modules.d/[0-9][0-9][0-9]_*.conf

<IfModule unixd_module>
  User www
  Group www
</IfModule>

ServerAdmin webmaster@example.org
ServerName pkg.example.org

<Directory />
  AllowOverride none
  Require all denied
</Directory>

<Files ".ht*">
  Require all denied
</Files>

ErrorLog "/var/log/httpd-error.log"
LogLevel warn

<IfModule log_config_module>
  LogFormat "%v %h %t \"%r\" %>s %b" vhost_common
  CustomLog "/var/log/httpd-access.log" vhost_common
</IfModule>

<IfModule headers_module>
  RequestHeader unset Proxy early
</IfModule>

<IfModule mime_module>
  TypesConfig etc/apache24/mime.types
  AddType application/x-compress .Z
  AddType application/x-gzip .gz .tgz
</IfModule>

Include etc/apache24/extra/httpd-mpm.conf
Include etc/apache24/extra/httpd-vhosts.conf
Include etc/apache24/extra/httpd-default.conf
Include etc/apache24/Includes/*.conf
# pkg.example.org:/usr/local/etc/apache24/extra/httpd-mpm.conf

<IfModule !mpm_netware_module>
  PidFile "/var/run/httpd.pid"
</IfModule>
<IfModule mpm_event_module>
  StartServers             3
  MinSpareThreads         75
  MaxSpareThreads        250
  ThreadsPerChild         25
  MaxRequestWorkers      400
  MaxConnectionsPerChild   0
</IfModule>
<IfModule !mpm_netware_module>
  MaxMemFree            2048
</IfModule>
# pkg.example.org:/usr/local/etc/apache24/extra/httpd-defaut.conf

Timeout 60
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
UseCanonicalName Off
AccessFileName .htaccess
ServerTokens Prod
ServerSignature Off
HostnameLookups Off
<IfModule reqtimeout_module>
  RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
</IfModule>
# pkg.example.org:/usr/local/etc/apache24/extra/httpd-vhosts.conf

<VirtualHost _default_:80>
  ServerName pkg.example.org
  ServerAdmin webmaster@example.org
  <IfModule dir_module>
    DirectoryIndex index.html
  </IfModule>
  DocumentRoot "/usr/local/share/poudriere/html"
  <Directory "/usr/local/share/poudriere/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
  </Directory>
  Alias /data "/usr/local/poudriere/data/logs/bulk"
  <Directory "/usr/local/poudriere/data/logs/bulk">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
  </Directory>
  Alias /packages "/usr/local/poudriere/data/packages"
  <Directory "/usr/local/poudriere/data/packages">
    Options Indexes FollowSymLinks
    IndexOptions FancyIndexing FoldersFirst SuppressDescription \
                 SuppressColumnSorting NameWidth=50 VersionSort
    AllowOverride None
    Require all granted
  </Directory>
</VirtualHost>

Once all above config files are in place, we can set apache to start at boot time and start it:

sysrc apache24_enable=YES
service apache24 start

Poudriere web interface is now accessible at web URL we configured in Apache.

In order to access Poudriere web URL, some mechanism to resolve FQDN to IP address is needed on clients.

Next tutorial in this series will be about setting DNS server. Until then you can use hosts file for above purpose.

Changing pkg Repository On Clients

By default, FreeBSD is configured to use official quarterly repo for installing packages by means of /etc/pkg/FreeBSD.conf. In order to switch to our repo, we won't be editing this file directly, but overriding it in another config file, as instructed in comments of /etc/pkg/FreeBSD.conf.

First, we need to disable official quarterly repo:

mkdir -p /usr/local/etc/pkg/repos
echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf

Next, we need to create our config file which points to our repo:

# pkg.kappastar.com:/usr/local/etc/pkg/repos/example.org

example: {
        url: "http://pkg.example.org/packages/12_2:x86:64-default-server/",
        mirror_type: "http",
        signature_type: "pubkey",
        pubkey: "/etc/ssl/selfsigned/example.org/poudriere/poudriere-cert.pem",
        enabled: yes
}

Make sure to check whether pubkey points to correct location of poudriere-cert.pem we created in required folders and files section.

A Few Words About Transport Security

One could reasonably ask how come we serve our repositories over http, an insecure protocol, and not over https which is supposed to be more secure. I see three reasons:

  • We have cryptographically signed packages with poudriere-privkey.pem, and configured clients with poudriere-cert.pem. Thanks to this, clients would refuse to install packages if they were altered in transit.
  • With above authenticity mechanism in place, it would be redundant, but also quite time-consuming, to create self-signed keys and certificates for apache and to configure clients to trust them.
  • Even with globally trusted SSL certificates, until 12.2, FreeBSD did not trust any root CAs upon basic install. In order to access anything over SSL one would need to install third party package - security/ca_root_nss - Root certificate bundle from the Mozilla Project. Requiring package installation for accessing repo which provides that same packages creates chicken and egg problem.

I have actually tried to serve my pkg repos over SSL with Let's Encrypt globally trusted certificates. It is by all means possible, but requires additional unnecessary work. Perhaps I could try it once again now that FreeBSD 12.2_RELEASE introduced root CAs in base.

Wait, there's fourth reason. I had the pleasure to see Marc Espie of OpenBSD fame presenting his talk titled Advances in OpenBSD packages: https is a lie in person on EuroBSDcon 2018 in Bucharest, Romania. I admit I didn't understand all the nifty details, but what I did understand is that, talking about package management, SSL alone can't save us from bad guys, that additional protection mechanisms need to be implemented, and that SSL can sometimes bring more harm than good.

Increasing ccache Disk Space

Remember we have configured our poudriere to make use of ccache in order to speed up builds by caching previous compilations and detecting when the same compilation is being done again? The thing is, default size of cache is just 5Gb, which will get filled up quite soon once we start to build a lot of packages, and we will lose any benefits because of overwrites. In order to increase cache size we need to modify ccache.conf:

# pkg.example.org:/usr/local/poudriere/ccache/ccache.conf

max_size = 20.0G

20Gb should be enough for caching builds of hundreds of packages including big ones such as chromium.

RAM Based ccache Using memchached

Things can get even faster if you have plenty of RAM, as memcached can use RAM to cache compilation instead of disk. We have already istalled it at the beginning of the article, we just need to set it to start at boot time with appropriate flags. After that, we can immediately start it:

sysrc memcached_enable=YES
sysrc memcached_flags="-l 127.0.1.11 -m 20480"
service memcached start

My jail host have plenty of RAM so I have allocated 20G of RAM to memcached, which shouldn't be exhausted even when building hundreds of packages including huge ones such as chromium. Feel free to experiment with cache size while monitoring RAM usage of memcached process with top.

We also need to replace contents of ccache.conf with the following:

# pkg.example.org:/usr/local/poudriere/ccache/ccache.conf

memcached_conf = --SERVER=127.0.1.11:11211
memcached_only = true

Consequent builds should now be even faster.

Conclusion

Above article hopefully taught you all the basics needed for building your own ports from packages and serving them to other FreeBSD systems.

If you have any additional questions, comments or any kind of feedback, drop me a line.

Previous Post