You may be curious about how Gradle actually runs and executes our build.gradle script. What happens behind the scenes when we use Gradle APIs like apply, buildscript, repositories, and others in our script?And how are the plugins loaded?

If you’re interested in exploring these questions, let’s dive into the process of how Gradle loads Groovy scripts.

Unless otherwise stated, the analysis of Gradle in this article and subsequent chapters is based on the Gradle 8.0 source code.

Build scripts are code

First and foremost, it is important to understand that build scripts themselves are code. The configurations we write inside the build.gradle script are essentially invocations of APIs.

Scripts can be categorized into four types.

init.gradle
settings.gradle
build.gradle
precompiled script(buildSrc/includeBuild下面的script)

Let’s focus on build.gradle for now since it is the most commonly used and other types of scripts are relatively simpler compared to it.

The methods that can be used within a Gradle script primarily include:

1. gradle core api. like files, project, repository etc
2. blocks defined within the DSL. for example buildscript, plugins, pluginManagement, apply
3. extensions defined by Plugins. When applying plugins to the build script, they often provide additional extensions and configurations that can be used to customize the build, like implementation, api provided by java plugin

Groovy Script Execution Process

groovy.lang.Script

Gradle scripts are actually built using the capabilities of the Groovy programming language. Understanding the language features of Groovy can help facilitate better comprehension.

You can find information on how to compile Groovy scripts and integrate them into applications in Integrating Groovy into applications.

And to learn more about Groovy closures and their usage, you can refer to the following resource Closures.

The loading of scripts with the .gradle (groovy script) extension is supported by the inherent script capabilities of Groovy itself. In simple terms, the GroovyClassLoader provides the ability to parse script files and save their corresponding Class files locally. Then, through reflection, these Class files can be instantiated, and their run method can be invoked to execute the script.

There are three main concepts that need to be understood:

1. groovy.lang.Script
2. CompilerConfiguration
3. GroovyClassLoader

In order for Groovy to load a script, it must be a subclass of groovy.lang.Script. groovy.lang.Script is an abstract class that contains a single abstract method called run. After the script is compiled, all its contents are encapsulated within the run method. To execute the script, you simply need to invoke the run method.

The parseClass method of GroovyClassLoader allows us to parse the script file that we have written. The codeSource parameter of this method represents the content of our script and the name of the generated class.

The reason for setting a custom script base class using CompilerConfiguration instead of directly using groovy.lang.Script is that groovy.lang.Script itself doesn’t have many methods. Apart from the default imports provided by Groovy, there aren’t many methods available to be called directly. By extending groovy.lang.Script, we can create our own base class and add additional methods that we can directly invoke in our custom scripts. This is similar to how we can call methods like buildscript, apply, files, and others in Gradle scripts. Customizing the script base class allows us to provide a more tailored and convenient scripting experience with specific methods and functionality suited to our needs.

Here’s a simple example demonstrating the loading and execution of a Groovy script(You can find the complete code for this example, including additional details and error handling, in the Gradle Script Loading file on GitHub)

public abstract class DefaultScript extends Script {  
  
    public void buildscript(Closure configureClosure) {  
        println("DefaultScript:buildscript  parameters:" + configureClosure.getMaximumNumberOfParameters());  
        configureClosure.call();  
    }  
  
}

We are testing a very simple script with the following content:

buildscript {
 println 'test'
}

The execution of the script involves calling the buildscript method from DefaultScript and then outputting “test“. The following code demonstrates how to create a GroovyClassLoader to load this script. In the CompilerConfiguration, we set the ScriptBaseClass to DefaultScript.class and output it to the root directory of the current project. The name is set to “build_xxx” in GroovyCodeSource. When run, it will generate a class file with that name in the project’s root directory.

public static void main(String[] args) {  
    CompilerConfiguration configuration = createBaseCompilerConfiguration(DefaultScript.class);  
    ClassLoader systemClassLoader = ClassLoader.getSystemClassLoader();  
    GroovyClassLoader gcl = new GroovyClassLoader(systemClassLoader, configuration, false);  
    GroovyCodeSource codeSource = new GroovyCodeSource(scriptText, "build_xxx", "/groovy/script");  
    Class scriptClass = gcl.parseClass(codeSource, false);  
    try {  
        DefaultScript script = (DefaultScript) scriptClass.newInstance();  
        script.run();  
    } catch (Exception e) {  
        e.printStackTrace();  
    }  
}  
  
private static CompilerConfiguration createBaseCompilerConfiguration(Class<? extends Script> scriptBaseClass) {  
    CompilerConfiguration configuration = new CompilerConfiguration();  
    configuration.setScriptBaseClass(scriptBaseClass.getName());  
    configuration.setTargetBytecode(CompilerConfiguration.JDK8);  
    configuration.setTargetDirectory(new File("."));  
    return configuration;  
}

private static String scriptText =  
    "buildscript {" +  
    "    println 'test'" +  
    "}";

You can take a quick look at the generated code, which has been partially trimmed for simplicity. Please note that the code below is a decompiled version.

build_xxx.java

public class build_xxx extends DefaultScript {  
    public build_xxx() {  
        CallSite[] var1 = $getCallSiteArray();  
        super();  
    }  
  
    public static void main(String... args) {  
        CallSite[] var1 = $getCallSiteArray();  
        var1[0].callStatic(InvokerHelper.class, build_xxx.class, args);  
    }  
  
    public Object run() {  
        CallSite[] var1 = $getCallSiteArray();  
  
        final class _run_closure1 extends Closure implements GeneratedClosure {  
            public _run_closure1(Object _outerInstance, Object _thisObject) {  
                CallSite[] var3 = $getCallSiteArray();  
                super(_outerInstance, _thisObject);  
            }  
  
            public Object doCall(Object it) {  
                CallSite[] var2 = $getCallSiteArray();  
                return var2[0].callCurrent(this, "test");  
            }  
  
            @Generated  
            public Object doCall() {  
                CallSite[] var1 = $getCallSiteArray();  
                return this.doCall((Object)null);  
            }  
        }  
  
        return var1[1].callCurrent(this, new _run_closure1(this, this));  
    }  
}

Compilation and Execution of Gradle Scripts

The compilation of Gradle scripts goes through two rounds, referred to as Passes

Pass 1: CLASSPATH — Only the pluginsManagement and plugins sections from the buildscript/initscript (depending on the script) are compiled and executed

Pass 2: BODY — The remaining sections, excluding those from Pass 1, are compiled and executed

The reason for dividing the compilation into two Passes is that the top-level blocks like buildscript and plugins need to be executed first. The results of their execution serve as the prelude for the remaining sections. In reality, Gradle provides only a limited set of capabilities. It defines fundamental concepts such as tasks, the overall lifecycle, repositories, configurations, and dependencies.

The construction of the build is handled internally by plugins, which leverage these foundational capabilities. For example, the Java plugin adds configurations like implementation and api, binds action tasks like compileJava to lifecycle tasks like build, and orchestrates the entire Java compilation process.

The core entry point for loading Groovy scripts in Gradle is the FileCacheBackedScriptClassCompiler.compile method. It primarily involves the compilation and loading the compiled class files. The compilation process is handled in the CompileToCrossBuildCacheAction, which ultimately delegates the task to DefaultScriptCompilationHandler.compileToDir.

The compileToDir method is a critical part of the process, and understanding this will give you a good grasp of Groovy script compilation in Gradle.

compileToDir

go through its parameters

source: ScriptSource
classLoader: ClassLoader
configuration: CompilerConfiguration
classesDir: File
metadataDir: File
extractingTransformer: CompileOperation
scriptBaseClass: Class<? extends Script>
verifier: Action<? super ClassNode>

ScriptSource

Script source refers to the source file of the script, which is the actual script file itself. It can also be referred to as the script’s source code.

Script content string refers to the string representation of the script’s content. It represents the actual code present in the script.

In the context of the compileToDir method, these parameters are used to analyze and process the script’s source and content during the compilation process.

1. The conventionally established default files in Gradle. the build.gradle, settings.gradle, and init.gradle.
2. When using apply from, script files can be imported either from local sources or remote sources.

classesDir

The files are saved in the directory .gradle/caches/8.0 (gradle version)/scripts folder. When creating a new file from the cache, the GroovyClassLoader supports specifying the target directory where the file should be saved. The compiled classes are then serialized and stored in the specified directory.

It’s important to note that the specific directory where a script is saved may vary based on the script itself. Each script is assigned a unique identifier, and its corresponding directory is determined based on that identifier within the Gradle cache structure.

In summary, the cache directory .gradle/caches/8.0/scripts is used to store and retrieve compiled script files, with the ability to specify the target directory for saving the compiled classes.

settings.gradle -> settings
build.gradle -> proj

• Pass2

The Pass 2 phase involves more transformers, but the logic is simpler and clearer:

1. BuildScriptTransformer serves as the entry point, where the following transformers are consolidated.
2. FilteringScriptTransformer filters out the block sections that were already compiled in Pass 1, as they don’t need to be processed again.
3. TaskDefinitionScriptTransformer validates the task method calls.
4. FixMainScriptTransformer removes the default main function that is generated after compiling a Groovy script. If there is a main { } closure inside, it will default to calling the static main function, so this step removes it.
5. StatementLabelsScriptTransformer removes the labeled_statements , one of the Groovy language feature, which is not applicable in Gradle scripts. This step removes any labeled statements.
6. ModelBlockTransformer deals with the deprecated model block, which can be ignored.
7. ImperativeStatementDetectingTransformer determines whether the script contains imperative statements, which affects whether the script can be deferred for execution.

These transformers collectively handle the processing and transformation of the Groovy script during the Pass 2 phase.

Besides, there are two detectors that perform validation:

1. packageDetector checks whether the script contains the package keyword declaration. Gradle scripts are not allowed to have a package declaration.
2. emptyScriptDetector checks whether the script is empty. In this context, “empty” here not only mean that the script has no content at all. It refers to cases where the script consists solely of comments or statements that have no effect. In such cases, the script is considered empty. For all empty scripts, Gradle uses the same file cache to avoid generating excessive files.

These detectors contribute to the validation process, ensuring that certain rules are followed in Gradle scripts.

scriptBaseClass

The main functionality is actually implemented in the DefaultScript class, while other scripts are mostly overrides of specific methods.

Scripts such as build.gradle and settings.gradle are compiled using Groovy and need to extend the groovy.lang.Script class. The buildscript, plugins, apply and other instructs actually invoke methods defined in the DefaultScript class.

For build.gradle, a corresponding Project object is generated and associated with the script. Methods like task, dependencies, files etc., are accessed through this object. The actual method dispatch is handled by Groovy’s method dispatch mechanism, taking into account the priority of method resolution.

The outer blocks, such as apply and buildscript, are essentially method calls.

This code is wrapped inside the run method of the compiled Groovy Script class. The creation of the Project object can be seen in the ProjectFactory.createProject method, where reflection is used to instantiate the Project object.

The most important method is evaluate, which is an abstract method that is eventually delegated to BuildScriptProcessor.execute. It creates a ScriptPlugin, and the loading of build.gradle is handled by the DefaultScriptPluginFactory.ScriptPluginImpl implementation. The apply method of this implementation compiles the script and executes it through Pass 1 and Pass 2.

verifier

In the Pass 2 stage, the closure for Rules is processed. During the code generation phase of Groovy script compilation, there is a hook for Rules. Rules, in conjunction with the model, used to be an early way of writing plugins but is now considered deprecated. Therefore, we won’t delve into it further here.

The Dynamic capabilities of a Script

Apply Plugin

There are two ways to introduce plugins:

1. apply plugin: This was the earlier way of including plugins in a script. It allows you to apply a plugin directly using the apply keyword followed by the plugin identifier. For example: apply plugin: 'java'. This method still works and is supported by Gradle.
2. plugins block: This is the newer and recommended way of configuring plugins in Gradle. It provides more capabilities and flexibility. Inside the plugins block, you can specify the plugins you want to apply, their versions, and any additional configuration. For example:

plugins {
 id 'java'
 id 'org.springframework.boot' version '2.5.2'
}

The plugins block allows you to include plugins from external build sources using the includeBuild directive. This ensures that the plugin is available by building the included build before applying the plugin. This capability is not available in the apply plugin method.

Both methods serve the purpose of including plugins, but the plugins block offers more advanced features and is the recommended approach by Gradle.

In the apply method style, which is more commonly used in projects, you typically see code like this:

apply plugin: 'java'

This is actually a syntactic sugar provided by Groovy. In this case, plugin: 'java' creates a map with a key of 'plugin' and a value of 'java', and then passes this map as an argument to the apply method.

Internally, this will eventually call DefaultPluginManager.doApply. It initializes the plugin corresponding to the given id and then calls the apply method of the plugin. If you have experience with custom plugin development, you may recognize that implementing a plugin requires implementing the apply method, which is what gets invoked here.

Within the apply method, you can register tasks, add custom extensions to the project, and perform other necessary plugin setup. For example, the java plugin registers the java extension during this phase.

DefaultProject

In the Gradle script loading process, it is known that build.gradle itself is compiled as a Script, and for each build.gradle file, a corresponding Project object is created. Within the Script, we can use properties and methods like tasks to interact with the Project object. But how is this achieved?

This is made possible by leveraging a feature of the Groovy language called “Runtime and compile-time metaprogramming”. You can refer to the documentation for more details.

The compiled build.gradle script becomes a GroovyObject, and Groovy’s dynamic nature is based on this. When a method is invoked on a Groovy object, it can be intercepted using the invokeMethod(String name, Object args) method. This means that before the method call is executed, it first goes through the invokeMethod method.

There are also methods available for accessing and modifying properties, such as setProperty(String property, Object newValue) and getProperty(String property). Additionally, the methodMissing method works in conjunction with invokeMethod to provide fallback behavior. By utilizing these mechanisms, the Script object can dynamically operate on the properties and methods of the Project object.

DefaultScript inherits from BasicScript, and it is BasicScript that implements these methods. Within BasicScript, there is a dynamicObject object, which is responsible for the association between the Script and Project objects. When a method is called in the script and not found within the Script itself, the dynamicObject is used to locate the corresponding method in the Project object.

However, there is another requirement: the Project object also needs to be a Groovy object in order to have dynamic capabilities and serve as a “dynamicObject”. Therefore, the DefaultProject object also needs to inherit from groovy.lang.GroovyObject. This is achieved through a special classloader. In the actual source code, Project is initialized using reflection. During runtime, this special classloader modifies the parent class of Project using ASM technology. The details of this process can be found in the AsmBackedClassGenerator class. The Project class has mixInDsl set to true, which adds the GROOVY_OBJECT_TYPE mixin, enabling dynamic method dispatch through invokeMethod and dynamic property access capabilities through property.

if (mixInDsl) {
  interfaceTypes.add(DYNAMIC_OBJECT_AWARE_TYPE.getInternalName());
  interfaceTypes.add(GROOVY_OBJECT_TYPE.getInternalName());
}

Reference Links

Integrating Groovy into applications
Closures
Runtime and compile-time metaprogramming