Device Tree Overlays in FreeBSD

Hello! It's clearly time for my yearly post, so this one will be over a recent interest and project- device tree overlays in FreeBSD.


What are device tree overlays?

Device tree overlays are used to modify a device tree, generally (but not exclusively) used on Small Board Computers ("SBC"). The problem they usually solve is how to describe/list devices on a bus that doesn't support enumeration of attached devices. Your base DTB might describe an i2c/SPI bus or similar, then you can add overlays at runtime that describe peripherals attached to said i2c bus.

I won't go any further into describing overlays or how they are generated, because there are plenty of good resources that do so better than I could. The Raspberry Pi Foundation's Device Tree Documentation has some good information, and Adafruit has an article on device tree overlays; others can be found fairly easily.

Why do I care?

I've recently been working on Allwinner support on FreeBSD. A good amount of documentation for Allwinner SoCs can be found on the Sunxi Community Wiki, so they served as a pretty good starting point for getting into development on ARM platforms and kernel work in general. We recently flipped the switch to use DTS imported from mainline Linux releases directly, rather than our own DTS or modified DTS. This has had its ups and downs, but the results have been mostly positive.

However, this has also been somewhat inconvenient at times. Adding support for devices has to move with the pace of Linux releases, which is a roughly two month release cycle. We had some regressions in support because we weren't using stable bindings for some devices, and upstream (mainline Linux) hadn't adopted stable bindings yet, so the devices simply aren't in DTS. Regressions like ethernet going away on your SBC can be quite frustrating at times and were the main pain point, but all of this is happening in -HEAD so some breakage should be expected occasionally anyway.

This is where overlays come in. As a developer, I can easily pull the latest Linux -rc or patches going into Linux with the new bindings that will be coming in and build a new DTB from that. This is in fact necessary to try and keep ahead of the releases a little bit so we don't have major breakage when a new release comes in. The problem comes during any of the following scenarios:

  1. Testing compatibility with old bindings as well as new bindings
  2. Getting others to test the new binding support
  3. Helping users gain functionality of new bindings

Granted, none of these problems are particularly hard to solve. #1 is only a problem because it's inefficient to swap back and forth between old/new DTB frequently, and #2/#3 are basically the same problem from different perspectives. The latter problems are easily solved by just distributing new DTB as needed, but overlays can be pretty easily compiled and it's clear what's being changed from looking at an overlay.


Where we were

When I started working in this area a couple of months ago now, our overlay support was fairly limited. We supported overlays to an extent, but there's no real spec for these things and it showed in both our dtc(1) and our loader implementation for overlay application. You could specify a comma-delimited list of overlays to apply as fdt_overlays in loader.conf(5), and loader would load and attempt to apply them. However, we had the following bugs/limitations:

  • All fdt_overlays must be full filenames - this is in contrast to Linux land, where overlays were generally specified as the basename.
  • All fdt_overlays must appear in one of the current module paths, with /boot/dtb being the logical path to place them in.
  • fdt_overlays were not applied in all cases, including aarch64 boots and when U-Boot provided FDT on ARMv7 boots.
  • A bug in the loading bits would discard previously loaded dtbo upon loading another one, effectively only allowing one overlay- the last one specified.
  • When an overlay applied, /__symbols__ from the overlay would not get merged into the resulting FDT, so new symbols could not be referenced.
  • Most importantly, because there was no spec on these things, our implementation would only work with overlays generated by BSDL dtc(1). This was due to the format of /__local_fixups__, which we were building in the exact same way as /__fixups__. This made sense, but was not the winning implementation in libfdt and everywhere else.

None of these were particularly serious, but it did hinder my ability to adopt overlays as a solution. gonzo@'s initial implementation did me a lot of good, and I really appreciate the work he did, but eventually I hit a point where it bit me pretty badly as I was getting neck deep in overlays. gonzo had already said we should use libfdt's implementation for overlays rather than continuing to maintain our own, so this was a clear next step.

Where we are now

  • All but one (full filenames must be specified) of the above limitations are gone as of r328107 in -HEAD. Overlays may now be placed in /boot/overlays, which matches a convention from over in Linux land.
  • A notable change, we now use libfdt for applying overlays as of importing libfdt 1.4.6 in r328106.
  • All of the relevant changes should be MFC'd to stable/11 by 2018/01/25
  • Ironically, our dtc(1) in-tree cannot currently generate overlays usable by our loader implementation. To generate overlays currently, either sysutils/dtc (GPL dtc) or an up-to-date BSDL dtc is required (get this from its GitHub project, you'll need to compile it). I will be importing BSDL dtc following some manpage revision, so hopefully this will not be the case for long.

Where we are going

As of time of writing, I have two reviews in flight: allow ".dtbo" to be dropped from fdt_overlays, and check /compatible on the overlay if it exists. The latter functionality is an interesting case: most examples of overlays will have a "compatible" specified on the root node, and it's expected to be checked and honored. This review checks it against /compateible on the base node while allowing a developer or someone just wanting to rapidly develop/prototype overlays to leave it off and still have their overlays apply.

I also consider /compatible checking a blocker for another feature I've been wanting lately: auto-loading of overlays present in /boot/overlays. The idea being that one could specify something in loader.conf(5), perhaps 'fdt_overlays_autoload="YES"', and the loader will attempt to load all overlays in /boot/overlays. In my opinion, this cannot work effectively unless we have a way in the overlay to prevent it from being loaded on a board that it isn't compatible with. This is mainly because I want to drop all of my sunxi overlays into /boot/overlays and have the loader just do the right thing no matter which board I've booted at the time. Autoloading overlays will likely require a slight re-working how we load overlays to check compatible at load time, rather than right before we attempt to apply it.

The next import of BSDL dtc(1) will leave us in a great position. The current master not only generates overlays properly, but it also adds support for the syntactic sugar recently supported by GPL dtc(1), an example of which you can find in the unit tests. This allows a syntax more common to DTS- the same syntax used to modify previously defined nodes, usually from a .dtsi. Internally, dtc takes &emac { prop = "value"; }; and generates a fragment with target = <&emac>; and the contents of the node are the contents of the fragment's __overlay__. This produces a much cleaner looking DTS for the overlay that is a little bit more consumable by humans.

Radio Silence

In 2017 so far, I've written a couple of posts that have not yet made it to the publish stage. One entitled "Fixing BSDgrep," and another with general thoughts on Python. These will eventually get published. =) Here's a little bit of what I've been up to in 2017:

Ham Radio

I recently obtained my general license in the USA, and I have plans to shoot for extra in the short- to mid- term. I have a couple of Baofengs that I use, and I've also setup a Raspberry Pi (running FreeBSD, of course) with an RTL-SDR dongle plugged into it running Direwolf to capture APRS traffic.

Fixing BSDgrep

Also the subject of one of the articles I'm writing, I've spent a good chunk of time fixing BSDgrep in FreeBSD. There were many outstanding bugs open against it in Bugzilla, but I've slowly squashed each one and got a couple of improvements through in the process mostly so that I don't have to specify a working directory with -r. I've also been doing some work in other areas of the base system to help my workflow in some areas. I intend to continue trying to improve things in FreeBSD; mostly because I get a certain satisfaction out of doing so, but also because I converted al of my stuff to FreeBSD roughly two years ago now.

Server Migration

I've also managed to migrate my www droplet with DigitalOcean to another droplet, but this time with more jails. In the new setup, I've got four jails using iocage:

  • db0 - Postgres jail, runs postgres 9.6, houses website databases
  • vpn0 - OpenVPN server for my private network 
  • www0 - Nginx jail, houses acme/letsencrypt bits
  • xmpp0 - Jabber server, runs ejabberd

This is a marked increase from the one jail (xmpp0) I previously had, but SSL certs are still mounted into individual jails as required.

Future Plans

The year is still just under half-way over, and I have much to do. Here are some future plans I have for this year:

  • Help get HDMI up on pine64/pinebook
  • Continue FreeBSD development
  • Replace old SunFire build server with more cost effective solution
  • More programming

Not the most difficult lit of things to accomplish, but still worth formalizing. I reserve the right to accomplish more or less as needed.

Recent Projects

In my previous post, I gave a brief overview of what I'd done in the past year. Looking back, though, I completely omitted recent projects that I'd written to make my life a little easier. There's only two (2) of them, so this won't be a necessarily long post.

freebsd-pkgbase-build (GitHub)

The first most recent project that I've worked on was freebsd-pkgbase-build. The gist of it is: I have multiple machines that I've taken the time to trim down the GENERIC kernel config to only include what I really need, and my machines span multiple architectures (amd64, armv6, arm64) and sometimes multiple branches (stable/11, head, stable/10) .

I wanted to try out FreeBSD's PkgBase, which is still not quite ready for prime time, but doing the buildworld's and buildkernel's for everything was kind of time consuming, not a great process, and required some maintenance overhead that I didn't necessarily want to go through. Ideally, I could:

  1. Spin off all of my pkgbase builds
  2. Create the different repos that arise from different ABIs and different branches
  3. Combine the different repos into one that I could serve fairly easy, despite these things being scattered all over the place

And thus, freebsd-pkgbase-build was born. It is a make(1) based solution that uses a Makefile.local or /etc/make.conf for configuration. My Makefile.local looks like so:



Rather intense, yes? No, not really. My arm64 stuff is excluded right now, but it's a fairly simple addition. After inclusion of this, it's a simple `make packages` to produce the repo at /usr/local/pkgbase/repo. Easy peasy, yeah? Yeah.

py-rtprio (GitHub, PyPI)

py-rtprio is my other project. It is also nothing fancy, just a Python-based interface to rtprio(2) for setting realtime priority of the executing thread/process. Originating in Python 3, shortly ported to be Python 2 compatible just to say I tried to be friendly. Admittedly, I like the Python 3-only version better, but it didn't end up too terrible. It uses ctypes to load libc and expose rtprio(2) in a Python-ish way. When I decide to drop Python 2 support, I'll likely go back to the cleaner version that uses Enum, which I avoided for Python 2 support so that I didn't need to bring in a dependency just for that one small (yet nice) feature.

My motivation for this was that I have many dreams of doing things with FreeBSD that are more suited to an RTOS, but I think with setting real-time priorities I might be able to accomplish some of the things I'd like to do. My reasoning (likely naive) is that setting a real-time priority on a system that I already tightly control will limit the number of things that can pre-empt me to just the kernel. My hope is that will net me the responsive-ness I may require.

Desert Wasteland


Wow, it's been almost a full year since I've written anything. I guess it's time for the yearly update, then?

FreeBSD Shenanigans

In the past year, I've been involved in more FreeBSD shenanigans. Find enclosed a list of highlights.


In my last post, I'd migrated my DigitalOcean droplets to FreeBSD.  Since then, I've pretty much consolidated all of them into my one webserver that runs all of my websites and a jail for Jabber. Want to contact me? is just one more way to do so. =)

I've also stood up a Sun Fire X4150 (with dual Xeon E5440, 24GB RAM) at home with the following roles:

  • Poudriere Build System
  • IPCam Manager
  • OpenLDAP Server
  • Personal wiki (Running DokuWiki)
  • Personal Git server (Running my own git-shell wrapper, gitwrap-shell)
  • Jenkins Builder (For keeping tabs on ZDoom build status on FreeBSD as the maintainer)

It stays fairly well loaded, and I'm considering standing up a second server (ideal Sun Fire X4150) to operate as a backup server. Most of my stuff uses ZFS as it is, but getting some redundancy going would be quite nice.


I've also tried to stay fairly active with the ports tree this year. The things I've done here include:

  • Took maintainership of net/freerdp
  • Implemented arbitrary baud rate support for *BSD in comms/py-serial - upstream first, then back to the ports tree
  • Provided distfile mirror for games/bsp and games/heretic
  • Got fmodl aliasing issue on arm, mips, powerpc resolved
  • Introduced 16-ish ports total, most of them for Arduino 1.6 (devel/arduino16) support, which brings along SAM support (Arduino Due)
  • Introduced OctoPrint (www/py-octoprint, loosely based on the 1.3.0 release) for interfacing with my work-in-progress 3D printer (which I will likely talk about in a future post, with pictures!). I've contributed upstream some bits to update dependencies to bring OctoPrint to FreeBSD -- we'll see where my upstream PR ultimately goes. I remain hopeful.

You can review the list of ports that I currently maintain (under the assumed alias of -- sssh! Don't tell anyone!) if desired. Most of these were introduced for bringing in the new Arduino IDE.


Worth noting is that I've spent some amount of time playing around with different hardware. I've deployed two FreeBSD Raspberry Pis, one to go along with my work-in-progress 3D Printer and another that does...well, not much of anything but sit on my desk at work.

I also bought a Carambola 2 Dev Kit to hopefully replace my router or my AP/extender at home. I put FreeBSD on it using Adrian Chadd's freebsd-wifi-build, but ran into problems when attempting to isolate the WAN port on the chip. Attempts to setup VLANs ended up in interrupt storms and failing to TX on the port with a different VLAN. One day, I might find time to debug this.

Since that didn't work out so great, I decided to put FreeBSD (again, using freebsd-wifi-build) on my TP-Link TL-WDR4300. I ended up in pretty much the same situation as before -- the case physically separates one of the ports as the WAN port, but the hardware doesn't really make this distinction. There's 4 LAN  + 1 WAN port, but underneath it's really just a five port switch with some clever tricks to isolate the WAN port. I've consulted Adrian on this setup because I'm having difficulties again isolating the WAN port, I will update this post when I hear back from him. My current problem seems to be that I can't TX on any port again no matter the VLAN configuration I try. I assume I'm handling the cpu ports on the switch wrong, but we shall see.

My other experiment was with my IP Cameras. I have two of them, both Microseven models. I'm not very happy with the WiFi stability on them, so I intend to flash them with FreeBSD. I disassembled one of them to see if I can't get a serial console on it, but failed at this, too. My first road-block was that what I've identified as the serial pins are 1.27mm pitch, but all of my headers and cables are 2.54mm. I ordered 1.27mm headers from Alibaba, all OK. However, I don't actually have cables I can work with 1.27mm pitch with. =( It's a work in progress.

The other hurdle on that one is that the WiFi chip seems to be an SDIO chip that we don't currently have support for in FreeBSD. I have no further details on it until I can actually boot with a serial console, unfortunately, so we'll find out more once I can work with that.

Personal - Graduation / Work

I graduated from Kansas State University in May with a B.S. in Computer Science. I did not get a minor at this time, because I was just ready to be done. I took a full time position at Kansas State University after that, and continue to do, for the most part, what I was doing before I graduated. Student loans are coming due right about now, though, so that's not so great.


I think I've said all that I need to for this year. I can say that I really did do things this year, although they may not have been the things I wanted to accomplish. I hope to do a little more writing in the upcoming year than I did this year, and use more pictures to complement my writing. At the very least, I need to talk about my 3D Printer as I bring it to completion as well as documenting the journeys in getting any of my more exotic hardware running FreeBSD.

Servers, Ubuntu -> FreeBSD

Hello! There's actually a couple of intermediate steps that have happened in the past year and a half or so. Arch was falling out of favor with DigitalOcean, so I moved my servers over to Ubuntu. As one might expect, that migration was relatively painless. There were version differences between the two, but for the most part my configuration and setup could be copied almost as-is and everything worked pretty well. For the move to Ubuntu, I did alter my servers a little bit. My webserver picked up Varnish in front of it, and I believe this was also the point where I made the full switch over from Apache to NGINX, instead of maintaining a hybrid setup with my Drupal sites being on Apache and my non-Drupal sites being on NGINX. I put together a PHP-FPM setup that worked well to shove out my previous Apache/suPHP setup and separated out my NGINX configuration enough that adding new sites is a breeze. Around this time, I also dropped GitLab and developed my own "gitwrap" utility w/ Python 3. My util and git servers were merged into one, and I dropped my BYOND and area51 servers as they saw very little activity. I also dropped OpenLDAP for the time being, as I was no longer running Redmine. That brings us up to November 2015. As of September, I had switched my Desktop over to FreeBSD. I eventually switched over to Xfce, and I've yet to look back. In November, I decided to take the plunge and convert my servers from Ubuntu to FreeBSD. The initial step was to assess version differences between Ubuntu and FreeBSD. Much to my fortune, there was not much of a discrepancy in Varnish or NGINX versions-- nothing to significantly change my configuration, the following steps were then taken to migrate properly: At first, I put up two servers that I would then be moving to: www2 and util2. Only being two months into my FreeBSD experience, I hadn't actually setup any of the stuff I would need on my servers before in a FreeBSD-way. So, I installed www/nginx, lang/php56, databases/postgresql94-server, and www/varnish4 (FreshPorts is awesome, by the way). First observation: php-fpm actually comes with lang/php56...OK, that's interesting (and convenient)! I was already aware that most configuration of packages not in the base system occurs in /usr/local/(etc), so I was prepared for that. I wasn't quite prepared for php/fpm conf files to just be dropped into /usr/local/etc, but that's not exactly a killer. The stock php.ini-production was suitable for my needs, so I symlinked that over to php.ini and moved on to FPM. For FPM, life's not too difficult. Adding an "include=etc/fpm.d/pool.d/*.conf" is enough for me to go ahead and copy over my pool definitions. For each pool, very little ultimately needed changed. Listen socket as well as user/group and listen owner/group, then log locations to reflect that I'd now like to put logs in log/nginx (/usr/local/log/nginx). After that, FPM and PHP were happy enough that I went ahead and setup NGINX. This setup was also pretty easy. I commented out the server {} block in nginx.conf and added an include directive to make sure I was pulling in etc/nginx/sites-enabled/* -- which is where I prefer to store my server {} blocks. I pulled this one from Apache habits, and have a corresponding sites-available/ directory that holds all of my site configuration, and then sites-enabled/* is just a bunch of symlinks. Other than that, I was actually able to drop in my drupal.conf, php.conf, set-pool.conf, and ssl.conf to etc/nginx without changing anything except the path to my php-fpm sockets in my php.conf. SWEET! Now NGINX is happily serving my domains, although I hadn't quite switched over DNS at this point or stood up Postgres. Varnish was also relatively easy-- still using Varnish 4, so I could drop my 4.vcl from my Ubuntu configuration into etc/varnish and setup varnishd_config in /etc/rc.conf appropriately, and all was good. Postgres was a slightly different story. I was moving from 9.4 to 9.4, so things got a little bit sketchy. I've got 9 or 10 different databases in the same cluster, so I didn't necessarily want to backup/restore to a new cluster as I probably should have done (note: If it's feasible, this should be the recommended method- especially if you're moving between minor versions). Instead, I rsync'd my data/ directory over on a live/running instance of Postgres, and adjusted accordingly. I did not maintain symlinks to any of the *.conf files as I was in Ubuntu, but that didn't seem to be an issue. I created new local users for my corresponding database users, and then ran into an issue with postgres/pgsql differences. In Ubuntu-land, the user is named 'postgres' -- this seems to make more sense to me, because initializing the cluster seems to automagically created a databasee named 'postgres'. databases/postgresql94-server actually creates a 'pgsql' user, but cluster creation still automagically creates the 'postgres' database- so a `psql` as pgsql won't actually get you into a working database or command-line at first, which seems kind of odd. If I were first dealing with postgres on FreeBSD without any previous experience, I would probably be mildly confused by this. I tested this out on my laptop, just to make sure I wasn't crazy. Much to my happiness, though, this pretty much worked. All of my data happily carried over, I just needed to get in and rename the 'postgres' user so that my local authentication as 'pgsql' would still be happy. My webserver was up and running at this point, other than needing to install a couple of other extensions (all suitably named either php56-* or pecl-* -- this made things really easy to work with). My git server is not much to speak of, however. As it was a custom 'gitwrap' wrapper, all I really had to do was install lang/python35, rsync my gitwrap directories over, and then everything was pretty much happy. I suppose the point (or TL;DR, if that's your flavor) of this post is that it doesn't have to be painful to make the switch from Linux to *BSD. Granted, my setup isn't all that intense/extensive- I have Drupal (backed by Postgres) running w/ PHP, PHP-FPM on NGINX behind Varnish. I have a couple of other odds and ends running around on here, but nothing to speak seriously of. My webserver doesn't have an entirely uncommon setup, and I was still able to easily find comparable ports to suit my needs. Installing PHP extensions was also entirely straightforward, and these things are pretty well-named. The problems you run into in your own migrations will be different than the problems I experienced, but given what I've already seen and know- I don't suspect they'll be anything major or show-stopping. My migration for both servers was completed in under 3.5 hours.

<3 /var/backups

Long story short, I got tired of Windows 10 a couple of weeks ago. The performance on my five year old dual core laptop was less than satisfying, and I wanted to get more out of my laptop. Enter FreeBSD. I started out with 10.2-RELEASE and KDE4. It had some annoying issues, particularly in switching to a virtual console locking up the system. It didn't take long for me to decide there's not much reason to stay on -RELEASE, so I moved to -CURRENT. This background comes in now because it may be relevant in future posts about FreeBSD exploits. Anywho! In my weekly update to the latest -CURRENT && rebuilding world, I started running mergemaster. I was in a little bit of a hurry and managed to nuke my /etc/group and overwrite /etc/passwd. Cue a good six minutes figuring out an easy way to restore all of the passwords and groups I lost, until I came across a post mentioning that there should be backups of my previous revision in /var/backups. Restored all, happiness returns, thank you internet.


Shockingly enough, I have a very large interest in technology. From electronics to programming to system administration, I'd like to be diverse enough to have confidence enough in all three of these areas. The latest area I've been developing in is system administration.

At my current job, I do not get much exposure to doing system administration. I do, however, get to be exposed to a lot of the inner-workings that drives our network. This gives me a general idea of network architecture and how enterprise-level systems should really work. We have a lot of servers doing a lot of different things, and most of them are pretty much independent of each other and secure.

This gave me ideas for my personal setup, running this website and a couple of other services, causing me to land on the following servers:

  • www - The sole purpose of this server is to run any websites that I maintain. This includes the one you are currently viewing as well as [work in progress]
  • util - This is my utility server. It runs my LDAP server and it will eventually run my mail server. LDAP is only accessible over the private network, and mail will function the same.
  • area51 - This is my development server. It is for projects that I'm working on that are not quite ready for production. Once a project is ready for production, all trace of it is removed from area51 and it is migrated either to one of the other servers that suits its purpose or a new server will be spawned for it.
  • byond - This is my BYOND server. It is special because it is running a 32-bit OS as opposed to 64-bit. It will not only power any production BYOND applications I have, but it is currently setup to pull all of my BYOND projects that get served and builds them for me on the server every 30 minutes. Moving them to be hosted in production is then simply a matter of copying the necessary files over to the hosted directory and rebooting the server.
  • git - This is my git server. Its sole purpose is to run GitLab. It is special because is my only server that is NOT running Arch. It is instead running Debian, which is one of the recommended distros for GitLab.

All of my servers run 64-bit Arch, unless otherwise noted. In the future, I plan to introduce a Windows server (just to try it out), and servers for at least one or two more products that I currently have in miind.

Introduction and Statement of Purpose

Hi there, this is my personal website. It shall serve as an index for my published(-esque) projects, as well as a place for me to rant about some of the aspects of development that I find to be 'cool' or otherwise interesting. One of the first of such posts will probably be about version control and why it's so awesome.

As a bit of background about myself (some of which is on the /whoami page, so excuse the redundancy), I am a student in Electrical and Computer Engineering at Kansas State University. I have been programming since 2004 (~9 years, as of the time of writing) in various languages. These languages include, but are not limited to:

  • Visual Basic 6
  • C++
  • C
  • Java
  • C#
  • PHP
  • Javascript

This list is in the order in which I picked them up, rather than any order of preference. My favorite language of the above listed is C++, and my favorite platform is Linux. My favorite flavor is Arch, which I use everywhere- including my netbook and on the server running this website. I've been working with Linux in various capacities since 2002 or 2003, and I've been doing administration of Debian-based servers since 2006 or 2007, with a small hiatus from everything for a year or two starting in 2008 when I moved to Kansas and ending in early to mid 2009.

As far as C++ goes, i've been working with it since late 2004 or early 2005, and I've still got a lot to learn. Most of my experience in it has been with networking (mostly playing around with Berkeley sockets, because Winsock makes me cringe), but I've also dabbled in OpenGL and playing around with Qt/SDL/Fun stuff like that.

Well, that's all for now. Until next time.