XenServer: Core architecture and critical components

boot_time_cpus

This is populated on each boot and contains information regarding the physical host’s socket and cores-per-socket information. For Linux users, if this seems familiar, it is a dump of cpuinfo that is found in the /proc/ runtime filesystem. Example 1-4 shows the command to view the contents of the boot_time_cpus file. This information is not only saved in this file but also known by dom0. In turn, the XAPI database is updated with this information and propagated to peers where the host is a member in a XenServer pool.

# cat /etc/xensource/boot_time_cpus

To compare the accuracy of this file against dom0’s Linux-based /proc/ filesystem (populated at runtime), issue the command listed in Example 1-5.

# diff /etc/xensource/boot_time_cpus /proc/cpuinfo

The normal output should be nothing, or that no difference exists between /proc/cpuinfo and /etc/xensource/boot_time_cpus. As the former is generated by dom0 after the host boots and the Xen kernel begins its boot processes, any difference between these files would be an indication of a serious misconfiguration.

/etc/xensource/bugtool/

The bugtool directory has a few subdirectories and XML files, which define scope, attributes, and limits for data output as it relates to XenServer’s built-in bugtool. Such files dictate tools, maximum output size, and other criteria necessary for the crash dump kernel (or server status report) to access, exploit, and store data needed under erroneous conditions.

/etc/xensource/db.conf

This file, along with a slightly longer version named db.conf.rio, describes the following XenServer database configuration information:

• Where to find the XAPI database
• The format to use when reading the database
• Sanity information relating to session usage and validity

/etc/xensource/installed-repos/

The installed-repos directory contains subdirectories that, either from a fresh installation or upgrade, list the repositories used during installation of the host. If supplemental packs are installed, they will also be listed under installed-repos; and upon upgrade, you will be alerted to the presence of the supplemental pack. This ensures that if an installation is dependent upon a specific supplemental pack, during upgrade, an administrator can verify and obtain a suitable replacement.

/etc/xensource/master.d/

This directory can contain one or more init-level scripts that were used during host initialization. In addition, an example init-level script is provided and for XenServer administrators is a good reference illustrating the flow of run-level activities, initialization sequencing, and resources called. This example script should be named 01-example, but if one is not present on your system, this is OK.

/etc/xensource/network.conf

The network.conf file describes to the administrator what type of virtual network fabric is being used by a XenServer host. Example 1-6 shows the command to view this file. There are two syntactical options: bridge or openvswitch. bridge corresponds to the legacy Linux network bridge, while openvswitch corresponds to the Open Virtual Switch or ovs. The ovs is the default network used for all installations starting with XenServer 6.0, but if an older system is upgraded, then the Linux bridge may still be in place. It’s important to note that new network functionality will only be developed for the ovs, while at some point, the Linux bridge may be retired.

# cat /etc/xensource/network.conf

The output will either state “bridge” or “openvswitch,” but this is a powerful piece of information to have—especially in diagnosing XenServer pool issues where the virtual network fabric, or “backend networking,” is in question. Furthermore, there are some third-party Citrix products that may require that the virtual networking fabric is of one type or another. If it is found that a particular host needs to have its virtual network fabric switched, ensure all Guest VMs are halted (or migrated to another XenServer host), and issue the command listed in Example 1-7.

# xe-switch-network-backend {bridge | openvswitch}

The host should reboot, and upon reboot will be conducting network operations via the mode specified.

/etc/xensource/pool.conf

Another key file for troubleshooting, pool.conf is used to determine if a host is a standalone server, pool master (primary), or slave (pool member reporting to a primary XenServer). The file can have one of two values: master or slave.

Using the cat command, as illustrated in Example 1-8, one can determine from the contents of pool.conf if the XenServer host thinks it is a pool master or a pool slave/member.

# cat /etc/xensource/pool.conf

If the result of the command is master, then the host has the role of pool master. If the result of the command includes the word slave, the host is a member server and the IP address of the pool master will also be provided. When working with a standalone XenServer host (i.e., one not part of a pool), the standalone host will assume the role of master.

Only in the event of serious configuration corruption should this file be manually edited, and then only under guidance from the XenServer support team. Examples of such corruption include:

• High Availability being enabled when it should be disabled, such as during maintenance mode
• A network communication problem among pool members
• dom0 memory exhaustion
• Timeout induced by latency when electing a new pool master
• A full, read-only root filesystem
• Stale or read-only PID file

/etc/xensource/ptoken

The ptoken file is used as a secret, along with SSL, in XenServer pools for additional security in pool member communications.  

This topic, along with XenServer’s use of SSL, is discussed in not available.

/etc/xensource/scripts/

The scripts directory contains one or more init-level scripts, much like the master.d directory.

/etc/xensouce/xapi-ssl.conf

The XenAPI is exposed over port 443, but proper authentication is required to invoke or query the API. In addition to authentication, a certificate private key has to be exchanged to a trusted source to secure further communications to and from a XenServer via SSL-based encryption. This is where xapi-ssl.conf is involved because it dictates the cipher, encryption method, and where to store the XenServer’s unique certificate. 

/etc/xensouce/xapi-ssl.pem

The xapi-ssl.pem is a private key that is generated on installation by /etc/init.d/xapissl, which references xapi-ssl.conf to produce a unique Privacy-Enhanced Mail (PEM) file for a XenServer host. This file is what, along with two-factor authentication, is exchanged amongst XenServer hosts, XenServer administration tools, etc., to secure communications back and forth to a unique, specific XenServer host.

Replacing the xapi-ssl.pem file with a self-signed or signed certificate is discussed in not available.

/etc/ntp.conf

In our experience, this is probably the most important of all configuration files that a XenServer administrator has liberty to modify at any time. In part, this is due to the nature of system and clock synchronization for precision. But it’s also due to the fact that any virtualization solution must facilitate time synchronization across Guest VMs. The Network Time Protocol configuration file—leveraged by ntpd—should never reference a virtual machine and should always reference a local, physical machine with no more than four entries.

While any virtualization solution will do its best to accommodate time drift, if the bare metal host’s clock drifts too far off, this can affect scheduling as well as eventually bring a Guest VM down. It is vital that /etc/ntpd.conf is set up accordingly, per time zone, so the host can appropriately update Guest VMs and their internal clocks. Additionally, if XenServer hosts are pooled, time drift between hosts must be minimized, and NTP settings be identical on all pool members.

/var/xapi/

This directory contains the complete XAPI database, which is a reference of hardware utilized within a standalone XenServer installation or pool installation.  

/var/patch/ and /var/patch/applied/

These two directories—/var/patch/ and /var/patch/applied/—are extremely important because between the two, they contain the footprint of hotfixes, patches, and other maintenance metadata that is essential for XAPI. Whether you have a standalone XenServer or a pool of XenServers, XAPI has to have a means to ensure that the following sequence has been successfully performed:

1. Patches have been applied to a standalone host.
2. Patches have been applied to all hosts in pool.
3. All post-patch actions, such as a reboot, have been accomplished.

Now, while both directories contain files with unique, UUID-based filenames, it is worth noting that the most important directory is that of /var/patch/applied/, because as the name implies, this contains a minimal footprint for patches that have been applied to a host.

This directory—/var/patch/applied/—should never be deleted because it is used by XAPI to check if a pool member needs a patch applied to it, if there are new hotfixes available for your XenServer host, and so on. As such, deletion of these files could lead to a host incorrectly seeing itself as requiring a patch.

pif

The pif object references the physical network hardware as viewed from within dom0. Because dom0 is CentOS based, the default network will be labeled “eth0” and will be assigned the IP address of the XenServer host. All additional physical networks will be represented as “ethX” but only physical interfaces with a “Management” or “Storage” role will have assigned IP addresses. The list of all physical network interfaces can be obtained from the command line as shown in Example 1-9.

# xe pif-list params=all

network

Because a XenServer will typically host many virtual machines that need to be connected to a network, a virtual switching fabric is created. Each fabric bridges all hosts in a XenServer resource pool and is implemented using the Open Virtual Switch or ovs. The number of virtual switches contained in a XenServer environment will vary and can often be dynamic. The list of all networks can be obtained from the command line as shown in Example 1-10.

# xe network-list params=all

The logical connection between network and pif is maintained by the PIF-uuids field.

vif

Each VM in a XenServer environment will typically have at least one virtual NIC assigned to it. Those virtual NICs are each called a vif and are plugged into the network fabric. All vifs are assigned a MAC address by XenServer. Often, network administrators look for virtual NICs to have a MAC vendor ID that they can then filter on. Because XenServer was designed to handle a large number of VMs, the vendor ID of a universally administered MAC address would artificially limit virtual networking so XenServer uses locally administered VMs to avoid the potential of MAC collision.  

The IP address a vif receives is set by the guest and communicated to XenServer through an interface in the XenServer tools. Because the XenServer network fabric implements a virtual switch, each vif plugs into a network, which in turn plugs into a pif representing a physical NIC that is then connected to a physical switch. The list of all virtual network interfaces can be found using the command line shown in Example 1-11. 

# xe vif-list params=all

bond

Optionally, administrators can configure redundant network elements known as “bonds.” NIC bonding, also referred to as “teaming,” is a method by which two or more NICs are joined to form a logical NIC. More on bonding can be found in not available. The list of all bonded networks can be found using the command line shown in Example 1-12.

# xe bond-list params=all

If a bond is defined, the “master,” “slaves,” and “primary-slave” configuration items all refer to pif objects.

pGPU

A physical graphical processor unit (pGPU) is an object that represents a physical video card, or graphics engine, that is installed as an onboard or PCI device on the physical host. Example 1-13 shows the command to get a list of all pGPUs installed for a host, as well as an example of results after running the command.

One example of a pGPU is an onboard VGA chipset, which is used by dom0 and its console. Example 1-14 shows the command to use to find the default GPU being used by dom0. Unless a dedicated graphic adapter or the host has a CPU with embedded GPU capabilities, the primary pGPU detected on boot is the only one that can be utilized by XenServer. Certain graphic adapters can be used in either GPU pass-through mode where the entire GPU is assigned to a VM, or in virtualized graphics mode where a portion of the GPU is assigned to different VMs.

# xe pgpu-list 
uuid ( RO)              : 47d8c17d-3ea0-8a76-c58a-cb1d4300a5ec
       vendor-name ( RO): NVIDIA Corporation
       device-name ( RO): G98 [Quadro NVS 295]
    gpu-group-uuid ( RW): 1ac5d1f6-d581-1b14-55f1-54ef6a1954b4

# lspci | grep "VGA"

gpu-group

A GPU group is simply a collection of graphics engines that are contained within a single, physical graphics card. If the graphics card is able to partition its resources into multiple GPU objects, then a GPU group will contain a reference to each object, as shown in Example 1-15.

# xe gpu-group-list

vgpu

A vgpu represents a virtual graphics adapter as defined by the graphics card. The vgpu-type-list command shown in Example 1-16, returned three possiblevGPU options for the given host.

# xe vgpu-type-list
uuid ( RO)            : ad32125b-e5b6-2894-9d16-1809f032c5bb    
     vendor-name ( RO): NVIDIA Corporation
      model-name ( RO): GRID K100
framebuffer-size ( RO): 268435456

uuid ( RO)            : ee22b661-4aa0-e6e6-5876-e316c3ea09fe
     vendor-name ( RO): NVIDIA Corporation
      model-name ( RO): GRID K140Q
framebuffer-size ( RO): 1006632960

uuid ( RO)            : 2025cc3e-c869-ef44-2757-a1994cc77c8e
     vendor-name ( RO):
      model-name ( RO): passthrough
framebuffer-size ( RO): 0