Gradle Beyond the Basics

Chapter 2. Custom Plug-Ins

Plug-In Philosophy

With its standard domain-specific language (DSL) and core plug-ins, Gradle intends to be a powerful build tool without the addition of any extensions or add-ons. Most common build tasks can be accomplished with these tools as configured by simple build files. Common builds are easy to write; however, common builds are not so common.^[7]

Projects that begin as simple collections of source files and a few static resources bundled into a standard archive format often evolve into complex multi-project hierarchies with requirements to perform database migration, execute repetitive transformations of static resources, perform and validate automated deployments, and accomplish still more build automation that doesn’t always easily conform to an existing standard or set of parameters.

Developing such a build is a specialized form of software development. The software in question is not the code that automates the business domain of the project itself, but code that automates the build domain of the project. This specialized code is software nevertheless, and it is precisely this kind of development that Gradle aims to facilitate.

To write this kind of code, an untutored Gradle developer might simply write a large amount of imperative Groovy code inside doLast() clauses of a build file. However, this code would be untestable, and would lead to large and unreadable build files of the kind other build tools are often criticized for creating. This practice is strongly discouraged. In its place, we offer the plug-in API.

The Plug-In API

A Gradle plug-in is a distributable archive that extends the core functionality of Gradle. Plug-ins extend Gradle in three ways. First, a plug-in can program the underlying Project object just as if an additional build file were mixed into the current build file. Tasks, SourceSets, dependencies, repositories, and more can be added or modified by applying a plug-in.

Second, a plug-in can bring new modules into the build to perform specialized work. A plug-in that creates WSDL files from an annotated Java web service implementation should not include its own code for scanning for annotations and generating content in a specialized XML vocabulary, but should instead declare a dependency on an existing library to do that work, and provide a mechanism for that library to be fetched from an online repository if it is not already present on the build system.

Finally, plug-ins can introduce new keywords and domain objects into the Gradle build language itself. There is nothing in the standard DSL to describe the servers to which a deployment might be targeted, the database schemas associated with the application, or the operations exposed by an underlying source control tool. Indeed, the standard DSL can’t possibly envision every scenario and domain that build developers may encounter. Instead, Gradle opts to provide a well-documented API that allows you, the build developer, to extend Gradle’s standard build language in ways that are entirely customized to your context. This is a core strength of Gradle as a build tool. It allows you to write concise, declarative builds in an idiomatic language backed by rich, domain-specific functionality. This is accomplished through plug-ins.

The Example Plug-In

In this chapter, we will create a Gradle plug-in to automate the use of the open-source database refactoring tool, Liquibase. Liquibase is a command-line tool written in Java whose purpose is to manage change in a relational database schema. It can reverse-engineer an existing database schema into its XML change log, and track the version of that change log against running instances of the database scheme to determine whether any new database refactorings must be applied. For users who prefer a Groovy syntax over XML, an open-source Groovy Liquibase DSL is available.

You can learn more about Liquibase online at the Liquibase Quick Start.

Liquibase is very good at what it does, but it is cumbersome to execute from the command line without a wrapper script of some kind. Moreover, since a high level of build and deployment automation is always an implicit goal, we would prefer to be able to wire Liquibase operations into our build lifecycle.

Our goals in this chapter will be to do the following:

• Create Gradle tasks corresponding to the generateChangeLog, changeLogSync, and update commands inside a Gradle build file.
• Make the Groovy DSL available to replace the default XML Changelog format.
• Refactor the Gradle tasks into a custom task type.
• Introduce Gradle DSL extensions to describe Changelogs and database configurations.
• Package the plug-in as a distributable JAR file.

The Liquibase plug-in will begin its life as a standard Gradle build file. This is an easy way to begin sketching out and prototyping code whose final form you do not yet know, which is a typical workflow in the development of new forms of build automation. As the plug-in takes shape, we will slowly refactor it into a distributable plug-in project with a lifecycle of its own. Evolving plug-in development in this manner is a perfectly appropriate, low-ceremony path to learning the API and discovering the requirements of your build extension.

Setup

To run the example code in this chapter, you’ll need a database for Liquibase to connect to. The book’s example code has a build file that sets up the H2 database for this purpose. Using Git, clone the http://github.com/gradleware/oreilly-gradle-book-examples repository, then change to the plugins/database-setup directory. Run the following two tasks:

$ gradle -b database.gradle createDatabaseScript
$ gradle -b database.gradle buildSchema

The first command will provide a platform-specific script called starth2 that will run the H2 embedded database administrative console for you to inspect the schema at any time during plug-in development. The second command will create a sample database schema in desperate need of refactoring—just the kind of test environment we’ll need for our plug-in development.

Note

You will have to move the database.gradle build file to the directory in which you are doing plug-in development, and execute the buildSchema task from there to ensure that the H2 database is in the right location for your plug-in to find it. Alternatively, you can place the database in a directory outside of your development directory and edit the JDBC URL to point to the correct path, but this step is left as an exercise for the reader.

Sketching Out Your Plug-In

Our Liquibase plug-in begins with the need to create tasks to perform Changlelog reverse engineering, Changelog synchronization, and updating of the Changelog against the database. Some digging into the Liquibase API shows that the best way to run these three commands is to call the liquibase.integration.commandline.Main.main() method. This method expects an array of command-line arguments indicating the database to connect to and which Liquibase sub-command to run. For each of its tasks that perform some Liquibase action, our plug-in will end up constructing this array and calling this method.

It’s worth thinking about precisely what those tasks might be. Given that we plan to support three Liquibase commands—generateChangeLog, changeLogSync, and update—we can plan on creating three tasks by those same names. In a different scenario, you might decide to “namespace” the task names by prefixing them with lb or liquibase to keep them from colliding with tasks from other plug-ins, but for our purposes here we can keep the task names short and simple.

We also know we’re going to introduce some custom DSL syntax to describe databases and ChangeLogs, but let’s keep that as a footnote for now. We’ll revisit the idea and decide what that syntax should look like as soon as we’re ready to implement it.

Custom Liquibase Tasks

Our plug-in will eventually introduce some fully-implemented tasks that call Liquibase with little or no declarative configuration. Before it can do that, though, we will need to build a custom task type. The purpose of this task is to convert task parameters into the required argument list for the Liquibase command line entry point and call the main() method. The implementation is shown in Example 2-1.

Example 2-1. The Liquibase task type prototype

import org.gradle.api.DefaultTask
import org.gradle.api.tasks.TaskAction
import liquibase.integration.commandline.Main

class LiquibaseTask extends DefaultTask {
  String command
  String url, password, username
  File changeLog

  @TaskAction
  def liquibaseAction() {
    def args = [
      "--url=${url}",
      "--password=${password}",
      "--username=${username}",
      "--changeLogFile=${changeLog.absolutePath}",
      command
    ]

    Main.main(args as String[])
  }
}

Remember, a custom task type is simply a class that implements the org.gradle.api.Task interface, or more commonly extends the org.gradle.api.DefaultTask base class. The LiquibaseTask provides a basic interface between the rest of the build and the core action of executing the Liquibase command-line driver through which all Liquibase operations are normally accessed. The properties of the LiquibaseTask will become task configuration parameters when the plug-in tasks are used in an actual build later on.

Having defined the custom task, we need only to create an actual task having that type, and to configure it. Note in Example 2-2 that we can use Gradle’s configuration syntax to set instance variables in the task class. We assign values to the url, username, password, changeLog, and command properties through a very standard assignment syntax.

Example 2-2. Instantiating the custom Liquibase task

task generateChangeLog(type: LiquibaseTask) {
  url = 'jdbc:h2:db/gradle_plugins'
  username = 'secret'
  password = 'sa'
  changeLog = file('changelog.xml')
  command = 'generateChangeLog'
}

Applying Yourself

Now that we’ve got a custom task type that makes it possible to run Liquibase from Gradle, let’s take a step back and begin building the plug-in proper. The simplest Gradle plug-in is a class that implements the org.gradle.api.Plugin<Project> interface. That interface defines a single method: void apply(Project project). We could begin with a method like what we see in Example 2-3.

Note

The Plugin interface is type-parameterized because plug-ins can theoretically be applied to any kind of Gradle object. Applying them to Project is by far the most common use case, and is the only one we’ll look at here.

Example 2-3. The apply() method of the first version of the Liquibase plug-in

class LiquibasePlugin implements Plugin<Project> {
  void apply(Project project) {
    project.task('generateChangeLog', type: LiquibaseTask) {
      group = 'Liquibase'
      command = 'generateChangeLog'
    }
    project.task('changeLogSync', type: LiquibaseTask) {
      group = 'Liquibase'
      command = 'changeLogSync'
    }
    project.task('update', type: LiquibaseTask) {
      group = 'Liquibase'
      command = 'update'
    }
  }
}

As a reminder, when you are first sketching out a plug-in as we are doing, you can code this class and the associated LiquibaseTask class directly in your build.gradle file. At this point in development, you are trying to learn the plug-in API and the scope and design of your plug-in itself. Deployment and packaging will eventually be very important concerns, but we can happily ignore them for now.

This example creates three new build tasks: generateChangeLog, changeLogSync, and update.^[8] Since the Liquibase plug-in is written in Groovy, we’re able to use a very Gradle-like syntax to declare new tasks; indeed, the code shown here would work verbatim inside a build file, apart from any plug-in definition. Build masters don’t have to write plug-ins in Groovy, but it’s a rewarding choice due to its similarity to the Gradle build file syntax and its productivity advantages over Java as a language.

Extensions

At this point our plug-in is starting to be able to do some work, but its configuration is rather pedantic (Example 2-4). We must configure each and every task with the database username, password, URL, and the changelog file.

Example 2-4. The Liquibase-enabled build so far

generateChangeLog {
  url = 'jdbc:h2:db/gradle_plugin'
  password = 'secret'
  username = 'sa'
  changeLog = file('changelog.xml')
}

changeLogSync {
  url = 'jdbc:h2:db/gradle_plugin'
  password = 'secret'
  username = 'sa'
  changeLog = file('changelog.xml')
}

update {
  url = 'jdbc:h2:db/gradle_plugin'
  password = 'secret'
  username = 'sa'
  changeLog = file('changelog.xml')
}

Clearly, we shouldn’t settle for this. The real power of Gradle plug-ins comes not just from the ability to hide a bunch of imperative code behind a plug-in declaration—custom Ant tasks and Maven plug-ins had already accomplished this a decade before Gradle had its 1.0 release—but rather from the ability to extend the domain model of the build. The Extension API is the primary means of doing so.

Design of plug-in extensions should begin with a sketch of what the desired build file syntax will look like. To design our build file syntax, we must first imagine what sorts of things our build will interact with in its now-expanding domain. In this case, this is simple: the build needs to know about databases and changelogs.

A database is a particular instance of a JDBC-connected database. A build automated with Liquibase database migrations will have separate domain objects representing the local database sandbox, a database instance on a staging server used for ad-hoc testing, a production database instance, and so on.

A Liquibase changelog is a file containing an ordered list of all of the refactorings performed on the database, expressed in an XML format. You can read more about Liquibase changelogs on the Liquibase site. Example 2-5 will have a single changelog, but real-world builds using Liquibase may break up their database refactorings into two, three, four, or more separate files. Our domain model must support any number of changelog files.

Example 2-5. The goal of our plug-in’s new DSL.

liquibase {
  changelogs {
    main {
      file = file('changelog.groovy')
    }
  }

  databases {
    sandbox {
      url = 'jdbc:h2:db/liquibase_workshop;FILE_LOCK=NO'
      username = 'sa'
      password = ''
    }
    staging {
      url = 'jdbc:mysql://staging.server/app_db'
      username = 'dev_account'
      password = 'ab87d24bdc7452e557'
    }
  }

  defaultDatabase = databases.sandbox
}

Note what we have done here: we have proposed new Gradle build syntax for describing new domain objects not originally envisioned by the designers of Gradle. All of this new syntax is contained within a fixed namespace (liquibase), but otherwise we have significant control over what it looks like and how it represents our domain. This deceptively simple observation is at the heart of the value of Gradle as a platform for creating customized builds. Gradle allows us not just to add custom code to our builds, but to add custom language as well. This is a key enabling feature for managing build complexity.

Plug-in extensions can hide complexity from build users by exposing a simple, idiomatic DSL in the build—and it isn’t even difficult to implement them. An extension takes the form of a class, usually written in Java or Groovy, that exposes the properties and methods accessed in the extension block. Our example, written in Groovy, is shown in Example 2-6.

Example 2-6. The Liquibase plug-in extension class

import org.gradle.api.NamedDomainObjectContainer

class LiquibaseExtension {
  final NamedDomainObjectContainer<Database> databases
  final NamedDomainObjectContainer<ChangeLog> changelogs
  Database defaultDatabase
  String context

  LiquibaseExtension(databases, changelogs) {
    this.databases = databases
    this.changelogs = changelogs
  }

  def databases(Closure closure) {
    databases.configure(closure)
  }

  def changelogs(Closure closure) {
    changelogs.configure(closure)
  }
}

The extension class defines two methods and two non-final properties.^[9] The two NamedDomainObjectContainer instances, databases and changelogs, will hold collections of domain objects created in the build syntax shown in Example 2-5. NamedDomainObjectContainer is a generic collection, and each instance holds domain objects of a different type. The Database and ChangeLog classes will have to be defined as shown in Example 2-7. The only feature that sets them apart from regular POJOs^[10] (or POGOs) is that they must have a property called name and a constructor that accepts a String and initializes the name property with it. Otherwise they do not extend any base class or implement any interface in the Gradle API.

The Liquibase-enabled Gradle build is able to maintain collections of databases and changelogs because of these two classes, and the way they are included in the extension class through the NamedDomainObjectContainer collection.

Example 2-7. The Database and ChangeLog classes

class ChangeLog
{
  def name
  def file
  def description

  ChangeLog(String name) {
    this.name = name
  }
}

class Database
{
  def name
  def url
  def username
  def password

  Database(String name) {
    this.name = name
  }
}

To apply this extension to the projects that use our plug-in, we will have to modify our LiquibasePlugin.apply() method. The enhanced apply() method can be seen in Example 2-8. The new functionality is in at the end, where the extensions.create() method is called. This method call indicates that the extension context will be named liquibase, and passes in the instances of the NamedDomainObjectContainers that will be held by the extension object.

Example 2-8. The apply() method with the plug-in extension included

class LiquibasePlugin implements Plugin<Project> {
  void apply(Project project) {
    // Create and install custom tasks
    project.task('generateChangeLog', type: LiquibaseTask) {
      group = 'Liquibase'
      command = 'generateChangeLog'
    }
    project.task('changeLogSync', type: LiquibaseTask) {
      group = 'Liquibase'
      command = 'changeLogSync'
    }
    project.task('update', type: LiquibaseTask) {
      group = 'Liquibase'
      command = 'update'
    }

    // Create the NamedDomainObjectContainers
    def databases = project.container(Database)
    def changelogs = project.container(ChangeLog)

    // Create and install the extension object
    project.configure(project) {
      extensions.create("liquibase",
                        LiquibaseExtension,
                        databases,
                        changelogs)
    }
  }
}

With all of that in hand, let’s take a look at the parts of the extension class itself, starting with the easiest part: the defaultDatabase property. This is an instance of the Database class that can simply be assigned as a normal property inside the liquibase extension block. In the DSL example we sketched out, you can see this in the defaultDatabase = databases.sandbox line. This indicates to the LiquibaseTasks that they should use the Database instance called sandbox if no other database configuration is provided.

The extension class has two methods: databases and changelogs, which accept a single parameter of type Closure. Passing this closure to the configure() method of the domain object collection class will do two things: first, it will create a new instance of the domain object managed by the collection (either Database or ChangeLog). Second, it will set fields on that domain object using the property names referenced in the configuration block in the build file itself. Let’s refer again to just this piece of the proposed build DSL as shown in Example 2-9.

Example 2-9. The creation and configuration of a Database domain object

liquibase {
  databases {
    sandbox {
      url = 'jdbc:h2:db/liquibase_workshop;FILE_LOCK=NO'
      username = 'sa'
      password = 'secret'
    }
  }
}

The inner sandbox block causes the new Database domain name object to be named “sandbox.” (Remember, this is a named domain object container, so all of the objects we create in it must have names.) The property assignments inside the curly braces of the sandbox block are converted into property assignments on the new instance of our Database class.

The outer set of curly braces wrapping the sandbox block form the Groovy closure that is passed to the databases() method of the extension class. If you wanted to define additional databases for the build to access—perhaps a staging server or a production instance—those would be placed as peers to the sandbox definition.

The complete source code of the sketched-out plug-in is available in a GitHub Gist. Remember, all of this code was typed directly into a plain Gradle build file, with no compilation or build steps required. This is a very efficient way to create a prototype, but of course a properly managed plug-in will need a build of its own, so it can be decorated with all of the appropriate testing, versioning, and release processes that attend mature software development. We’ll look at this next.

Packaging a Plug-In

Converting our quick-and-dirty plug-in implementation to an independent project with a build all its own is not difficult. The process consists mainly of breaking up the classes developed in the previous section into files of their own, adding a little bit of metadata to the JAR file, and fixing up our classpath with the Gradle API. We’ll look at these things one at a time.

The plug-in we built has seven separate classes packaged in the com.augusttechgroup.gradle.liquibase and com.augusttechgroup.gradle.liquibase.tasks packages. Since these are Groovy classes, we’ll place them under the src/main/groovy directory of the build. You can browse the source of this chapter’s version of the plug-in on GitHub.

After locating the classes there, we’ll need to provide one additional file in the build to give the plug-in its ID in builds in which it is applied. In the previously sketched-out plug-in, we applied the plug-in directly by its classname with the line apply plugin: LiquibasePlugin at the top of the build.^[11] A properly packaged Liquibase plug-in should be applied just like the core plug-ins are, with the line apply plugin: 'liquibase'. This plug-in ID is provided through a metadata file.

We will add a file called liquibase.properties in the src/resources/META-INF/gradle-plugins directory of the project. This directory will automatically be included in the jar file under its META-INF directory. The purpose of the file is to translate from the string ID of the plug-in to the fully qualified name of the class that implements the plug-in. The file’s brief contents are shown in Example 2-10.

Example 2-10. The contents of the liquibase.properties meta file

implementation-class=com.augusttechgroup.gradle.liquibase.LiquibasePlugin

Finally, the plug-in code must have a build. Naturally, we will use Gradle for this task. The full build shown in the example project contains many extra features for packaging and deployment to the Central Repository, but it can be much simpler. The main problem is solving the compile-time classpath dependency on the Gradle APIs themselves. (The plug-in also depends on Liquibase modules, but we already know how to bring in regular Maven-style dependencies from reading Buiding and Testing With Gradle.)

The answer to the Gradle API question is the gradleApi() convenience method. Declared as a compile-time dependency, it brings in all of the internal Gradle classes needed to build a project that extends Gradle itself. The completed minimum subset build is shown in Example 2-11.

Example 2-11. A simplified plug-in build file

apply plugin: 'groovy'

repositories {
  mavenCentral()
}

dependencies {
  groovy 'org.codehaus.groovy:groovy:1.8.6'
  compile gradleApi()
  compile 'org.liquibase:liquibase-core:2.0.3'
}

Conclusion

The Liquibase plug-in is a good example of a real-world product that introduces an external API to power new execution-time features and extends the build domain model in ways unique to its own domain. Hundreds of lines of build code are completely hidden inside the plug-in, and advanced functionality is introduced to the build tool as a native part of its vocabulary. And that vocabulary itself is expanded to encompass a new domain that isn’t a part of the core build tool itself.

When built and packaged as a project of its own, the plug-in takes on life as a software project. We can introduce automated tests, or ideally even bake them in while we code as an integrated part of the development process. We can deploy a continuous integration server. We can apply appropriate versioning, source repository tagging, and other release management protocols. In short, this extension to the build tool brings new functionality to bear on the system, is expressed in new build language, and is engineered within the context of contemporary software development practice. This is the essence of the Gradle plug-in story, which is a very powerful story indeed.

---

^[7] With apologies to Kurt Vonnegut.

^[8] In Liquibase, generateChangeLog reverse engineers a database schema, changeLogSync places a new database under Liquibase’s control, and update pushes new changes into a database.

^[9] In Java and Groovy, final fields can be initialized when the object is constructed, but can’t be changed thereafter. These two final fields are object collections, and the objects in the collections can be changed at runtime, but the collection instances themselves are fixed once the object is constructed.

^[10] POJO stands for Plain Old Java Object. It refers to a Java object whose type consists only of properties, methods, and an ordinary constructor, with no external requirement on a runtime container to create instances of it.

^[11] This line can be seen in the completed code.