In the first part, we created the Ansible project to manage Jails with shared IP. In this post, we will adapt our playbook to create vnet jails.

vnet gives to jails their own network stacks. Each Jail will have a specific network interface with epair connected to a bridge. Let’s quote something I read in a forum which resume well how it works:

Analogous to a physical network, a bridge interface works like a software switch, an epair works like a virtual network cable and a jail acts as a virtual computer.

The bridge creation will be automated with jib script.
Let’s adapt our playbook to create that kind of jail networking.

There is only few steps, but before all, reset our first configurations.

Clean environment

Remove the task which creates aliases, and remove them from your system.

$ ansible host-test -m shell -a "service jail stop bind && service jail stop nginx"
$ ansible host-test -m lineinfile -a 'path=/etc/rc.conf regexp=".*alias.*" state=absent'
$ ansible host-test -m file -a 'path=/etc/jail.conf state=absent'
$ ansible host-test -m zfs -a 'name=zroot/jails/bind state=absent'
$ ansible host-test -m zfs -a 'name=zroot/jails/nginx state=absent'
$ ansible host-test -m raw -a "service netif restart"
$ ansible host-test -m raw -a "service routing restart"

Install jib script

Add a task to copy the script in your $path with execution perms.

- name: install jib script
  copy:
    src: /usr/share/examples/jails/jib
    dest: /usr/local/bin/
    remote_src: yes
    mode: 0755

Configure network stack in jails

At jail startup, jib will create a bridge if non existent, create epairs and automatically attach them to the bridge. Stopping the jail will destroy interfaces but not the bridge.
Let’s change our task to declare to use that script in exec.prestart and exec.poststop.

 - name: declare jails
-  vars:
-    alias_ip: "{{ inet | ipmath(ansible_loop.index) }}"
   blockinfile:
     path: /etc/jail.conf
     marker: "# {mark} ANSIBLE MANAGED: {{ item }}"
     block: |
       {{ item }} {
           host.hostname = "{{ item }}.domain.local";
           path = "/usr/local/jails/{{ item }}";
           exec.consolelog = "/var/log/jail_{{ item }}.log";
-          ip4.addr = {{ alias_ip }};
+          vnet;
+          vnet.interface = "e0b_{{ item }}";
+          exec.prestart += "jib addm {{ item }} {{ ansible_default_ipv4.interface }}";
+          exec.poststop += "jib destroy {{ item }}";
       }
   loop: "{{ jails | sort | flatten(levels=1) }}"
-  loop_control:
-    extended: yes

Configure epair interface in jails template

Edit the bsdinstall template to add epair interface configuration and default gateway.

 - name: template bsdinstall script
+  vars:
+    jail_ip: "{{ inet | ipmath(ansible_loop.index) }}"
   copy:
     dest: "/usr/local/jails/{{ item }}.template"
     content: |
       DISTRIBUTIONS="base.txz"
       export nonInteractive="YES"
       #!/bin/sh
       sysrc sshd_enable="YES"
+      sysrc ifconfig_e0b_{{ item }}="inet {{ jail_ip }} netmask 255.255.255.0"
+      sysrc defaultrouter="{{ gateway }}"
       pkg install -y python37
       mkdir /root/.ssh
       chmod 600 /root/.ssh
   loop: "{{ jails | sort | flatten(levels=1) }}"
+  loop_control:
+    extended: yes

Run the playbook and test connectivity

$ ansible-playbook playbook.yml
[...]

TASK [jails : create zfs per jail dataset] ***************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : template bsdinstall script] ****************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : bsdinstall jails] **************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : authorize your ssh key] ********************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : permit root login] *************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : set default jails config] ******************************************
changed: [host-test]

TASK [jails : declare jails] *****************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : start jails at startup] ********************************************
ok: [host-test]

TASK [jails : start jails] *******************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

PLAY RECAP ***********************************************************************
host-test                  : ok=11   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

jib created a bridge and attached jails interfaces in it.

$ ansible host-test -m shell -a "ifconfig vtnet0bridge | grep member"
host-test | CHANGED | rc=0 >>
	member: e0a_nginx flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	member: e0a_bind flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

Let’s check if jails are running.

$ ansible host-test -m shell -a "jls"
host-test | CHANGED | rc=0 >>
   JID  IP Address      Hostname                      Path
     3                  bind.domain.local             /usr/local/jails/bind
     4                  nginx.domain.local            /usr/local/jails/nginx

As IP is configured in the Jail, host doesn’t know which one it is.
e0b_* interfaces are attached to jails.

$ ansible host-test -m shell -a "jexec bind ifconfig | grep e0b"
host-test | CHANGED | rc=0 >>
e0b_bind: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

$ ansible host-test -m shell -a "jexec nginx ifconfig | grep e0b"
host-test | CHANGED | rc=0 >>
e0b_nginx: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

$ ansible bind:nginx -m ping
nginx | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/local/bin/python3.7"
    },
    "changed": false,
    "ping": "pong"
}
bind | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/local/bin/python3.7"
    },
    "changed": false,
    "ping": "pong"
}

Jails are now exposed on my private subnet with their own network stack and reachable by Ansible.

We learned how to automate shared IP and vnet Jails provisionning with Ansible.
Managing Jails is more than just provision it, you’ll need to maintain it, by upgrading packages or releases.
In the network side, you would use a private subnet to isolate your jails from your local private subnet, and NAT to have the abilitiy to allow/deny access, or forward a port to the right Jail.

Full raw Ansible management is possible, but there already exists some clever wrapper with great features, like cbsd, iocage, or bastille.
In the next part, I’ll focus on bastille and how to automate our jails management with it, still wrapped by Ansible.