What is programming?; Simply programming means reading some data on input, operating on that data, and giving some output. Metaprogramming is a programming technique in which we can read codes as input, operate on those codes, and provide generated codes or change the running programs’ behavior. Assume we want to manage dependencies between our classes and inject themes in our code-base(dependency injection). Here we must write boilerplate codes to inject our dependencies into our code-base. How can we solve this problem? Do we have tools that generate these boilerplate codes based on metadata? With metaprogramming, we can fix these kinds of problems.
All programming languages that developers can use for meta-programming purposes and have meta-features called metalanguage, like Kotlin, C, Java, and Python.
Compilers, Assemblers, Interpreters, Linkers, Debuggers, Code Generators are writing with metaprogramming concepts and tools. In this article, I want to discuss some base concepts of metaprogramming and how to write meta programs that act on runtime or compile-time with kotlin meta-features.
Meta-Programming Concepts in Kotlin
Meta-Programming lets us write programs that can write code, modify the program’s behavior in runtime, generate codes based on metadata, etc. This type of programming with kotlin has some concepts that I want to describe here.
Annotations
In Java and Kotlin Annotations, it is a form of metadata that provides data about a program without direct effect on the operation of the code they annotate. We use annotations for giving some information to the compiler annotation processing in compile-time and runtime. Annotations added to Java, based on Java Specification Request 175.
| // Declare Annotation in Kotlin | |
| @Target(AnnotationTarget.CLASS, AnnotationTarget.FUNCTION) | |
| @Retention(AnnotationRetention.SOURCE) | |
| @Repeatable | |
| annotation class HelloWorldAnnotation |
HelloWorldAnnotation.kt
The Target defines where HelloWorldAnnotation can be used. In the above example, HelloWorldAnnotation can only be used in the above classes and methods.
Retention determines whether an annotation is sored in binary or source code. Kotlin has three retention for annotations, Source(isn’t stored in binary output), Binary(stored in binary output but not visible for reflation), and Runtime(stored in binary output and visible for reflection).
Repeatable allows using HelloWorldAnnotation on a single element multiple times.
One of the essential annotation features in kotlin is use-site targets. Imagine annotating the kotlin data class field with annotation. When this code is compiled to JVM, there are multiple possible locations for annotation in the generated byte code. Here you can see the complete list of annotation use-site targets in kotlin.
DSL(Domain Specific Language)
DSL is a specialized language used for a specific purpose. While Kotlin can be leveraged to write any number of programs, a DSL focuses on a particular purpose like DSLs for building UI’s.
Metadata Annotation
Metadata annotation is extra information stored in annotations in Java class files produced by the Kotlin JVM compiler. You will see the metadata annotation if you decompiled the java version of the kotlin class source code.
| @Retention(AnnotationRetention.RUNTIME) | |
| @Target(AnnotationTarget.CLASS) | |
| @SinceKotlin("1.3") | |
| annotation class Metadata( | |
| @get:JvmName("k") val kind: Int = 1, | |
| @get:JvmName("mv") val metadataVersion: IntArray = [], | |
| @get:JvmName("bv") val bytecodeVersion: IntArray = [1, 0, 3], | |
| @get:JvmName("di") val data1: Array<String> = [], | |
| @get:JvmName("d2") val data2: Array<String> = [], | |
| @get:JvmName("xs") val extraString: String = "", | |
| @SinceKotlin("1.2") @get:JvmName("pn") val packageName: String = "", | |
| @SinceKotlin("1.1") @get:JvmName("xi") val extraInt: Int = 0 | |
| ) |
MetaData.kt
This annotation is available in runtime by reflection and compile-time by annotation processing api. Also, all parameters have shorted names because kotlin wants to reduce the final file size. Here you can read more about the metadata annotation parameters.
Run-Time Meta-Programming(RTMP)
With RTMP, programs can operate on themselves at introspecting and modifying their own structure and behavior. Kotlin publishes artifacts for working with reflection(kotlin-reflect.jar). The reason for this artifact’s existence is to reduce the size of programs that do not use reflections at all. For using reflection in kotlin, we must add kotlin-reflect artifacts in our dependencies based on the build-system.
| dependencies { | |
| implementation "org.jetbrains.kotlin:kotlin-reflect:{latest_version}" | |
| } |
I desire to talk about some concepts of reflection. Still, if you want complete tutorials, There are so many blogs for reflection in kotlin, for example, kotlin official documents, medium blog posts, etc.
Kotlin References
We can reference classes, properties, functions in kotlin with the below types.
We use KClass to reference classes at runtime, and with this reference, we can read the constructors, get supertypes, check the class visibility, and many other things. Also, KFunction and KProperty are used to reference functions and properties, and we can call these references at runtime with KCallable reference.
Reflection Disadvantages
Introspecting structure and behavior of programs at runtime have many use cases like serializing and deserializing XML and JSON, Security testing, etc. But why this opinion exists that we must prevent this fantastic feature of meta-programming?
Unexpected Side Effects
Imagine we have a singleton class with double check pattern. This singleton can be violated by reflection at runtime, and we can make any number of objects from the singleton class.
| fun main() { | |
| val singletonKClass = Singleton::class | |
| singletonKClass.constructors.firstOrNull { it.visibility == KVisibility.PRIVATE }?.also { privateConstructor -> | |
| /** | |
| * We give access to private constructor and we can create instance | |
| * from [Singleton] class [privateConstructor.call] method | |
| */ | |
| privateConstructor.isAccessible = true | |
| val singletonNewObject = privateConstructor.call() | |
| singletonKClass.declaredMemberProperties.firstOrNull { it.visibility == KVisibility.PRIVATE }?.also { dataProperty -> | |
| /** | |
| * We give access to private property and we can get value with [dataProperty.get] method. | |
| */ | |
| dataProperty.isAccessible = true | |
| dataProperty.get(singletonNewObject).toString().also(::println) | |
| } | |
| } | |
| } |
Give Access to Private Constructors and Properties with Reflection
This example is just one of the examples of the side-effects problem in reflection.
Peformance Overhead
Without any benchmark, we can guess the reflection has lower performance than direct access. But how much slower? To find out these, we must have a benchmark test with tools like JMH or kotlinx-benchmark, which I want to discuss in another article. Below you can read oracle’s explanation about the performance overhead in reflection.
Oracle
Because reflection involves types that are dynamically resolved, certain Java virtual machine optimizations can not be performed. Consequently, reflective operations have slower performance than their non-reflective counterparts, and should be avoided in sections of code which are called frequently in performance-sensitive applications.
Secutrity Restrictions
We can completely disable reflection in our programs based on security issues, etc. This item is not directly disadvantage, but it can cause many concerns. There are a couple of ways to disable reflection, which you can see one of them below.
| /** | |
| * Simply security manager that check the packages and if equal | |
| * to "java.lang.reflect" or "kotlin.reflect" throws [SecurityException] | |
| */ | |
| class AppSecurityManager : SecurityManager() { | |
| override fun checkPackageAccess(pkg: String?) { | |
| if (pkg == "java.lang.reflect" || pkg == "kotlin.reflect") { | |
| throw SecurityException("Reflection is not allowed in this program") | |
| } | |
| super.checkPackageAccess(pkg) | |
| } | |
| } | |
| fun main() { | |
| System.setSecurityManager(AppSecurityManager()) | |
| // Using reflection cause [SecurityException] | |
| Person::class.createInstance().name.also(::println) | |
| } |
Disable Refelection With SecurityManager
Here in the RTMP-Kotlin Github repository, you can find more examples and codes about run-time-meta-programming.
Job Offers
OUR VIDEO RECOMMENDATION
Jobs
Compile-Time Meta-Programming(CTMP)
With CTMP, we can process metadata inside the codebase for generating codes, analyzing, etc. There are many problems that CTMP can solve. For example, every android engineer remembers we have many issues with SQLite in android, and CTMP allows room to solve that problem. Also, dagger2, View-Binding, Moshi, Retrofit, and other tools use CTMP to solve their problems. Below you can see some essential concepts on CTMP in kotlin.
Annotation Processing
Annotation processing is a technique in which javac scans the code-base to find and return annotations and annotated elements based on standard APIs for java language and annotation processors. With annotation processing api, we can ask the compiler what elements are annotated with XAnnotation? The compiler returns these elements with a bunch of other useful information.
Famous libraries widely use this technique for generating code purposes like dagger2, room, moshi, etc.
Java Specification Request 269
We need Language Model APIs and tools for processing Annotations; these APIs are added to java based on JSR-269. This Java Specification Request includes two types of APIs: one API that models Java Langauge, and an API for writing annotation processors.
JSR-269 APIs contains three packages in “javax.annotation” package for annotation processors(processing package), modeling the language(lang package), and tools(tools package). Below you can see a simple annotation processor based on JSR-269 APIs that use kotlinpoet to generate kotlin source codes.
| /** | |
| * This is simple processor that use JSR-269 API and kotlinpoet library to generate | |
| * extension function for annotated classes with [HelloWorldAnnotation] | |
| */ | |
| class HelloWorldProcessor : AbstractProcessor() { | |
| override fun process(annotations: MutableSet<out TypeElement>?, roundEnv: RoundEnvironment?): Boolean { | |
| val typeElement = annotations?.firstOrNull() ?: return false | |
| for (annotatedElement in roundEnv?.getElementsAnnotatedWith(typeElement) ?: setOf()) { | |
| val elementPackageName = processingEnv.elementUtils.getPackageOf(annotatedElement).qualifiedName.toString() | |
| FileSpec.builder( | |
| packageName = elementPackageName, | |
| fileName = "${annotatedElement.simpleName}_Generated" | |
| ).also { currentClassFileSpec -> | |
| FunSpec.builder("printHelloWorld").receiver( | |
| ClassName( | |
| elementPackageName, | |
| annotatedElement.simpleName.toString() | |
| ) | |
| ).addStatement("println(\"Hello world!\")").build().also(currentClassFileSpec::addFunction) | |
| }.build().writeTo(processingEnv.filer) | |
| } | |
| return true | |
| } | |
| override fun getSupportedAnnotationTypes(): MutableSet<String> = mutableSetOf<String>().apply { | |
| add(HelloWorldAnnotation::class.java.canonicalName) | |
| } | |
| override fun getSupportedSourceVersion(): SourceVersion = SourceVersion.latestSupported() | |
| } |
Simple Processor Based on JSR-269 APIs
Annotation Processing Tool (apt)
apt is a command-line utility for annotation processing. It includes a set of reflective APIs and supporting infrastructure to process program annotations(JSR 175). These reflective APIs provide a build-time, source-based, read-only view of program structure. They are designed to cleanly model the JavaTM programming language’s type system after the addition of generics (JSR 14).
apt tool and mirror api have been deprecated since JDK7 and probably removed in the next JDK versions. Oracle proposes using JSR-269 API for annotation processing.
Kotlin Annotation Processing Tool (kapt)
JSR-269 works for java, and the JetBrains team could rewrite JSR-269 completely for kotlin, but this does not work for mixed projects that’s use kotlin and java. Another way that JetBrains could solve these problems is to generate java source code from kotlin code and feed them into the javac compiler, it’s work, but it needs two compiler runs and has some side effects. Another way is pretending the kotlin binaries are java sources because naturally, the kotlin compiler generates binaries that are compatible with javac; JetBrains uses this solution with the kapt name. Here is a complete guideline for using kapt with Gradle
The Kapt compiles Kotlin code into Java stubs. To generate stubs, kapt needs to resolve all meta-data in the Kotlin program. The stub generation costs about 1/3 of a complete kotlinc analysis and the same order of kotlinc code-generation. For many annotation processors, this is much longer than the time spent in the processors themselves.
JSR-269 API does not make enough for writing annotation processors for kotlin, and kotlin provides some libraries and API to solve the problems that we have for processing kotlin source codes like process internal modifiers, etc. One of these libraries is The Kotlin JVM metadata manipulation library, where you can get the kotlin meta-data annotation while processing the annotations.
Kotlin Symbol Processing (KSP)
Kotlin Symbol Processing is a compiler plugin with a more straightforward API to resolve problems of compiler plugins such as compiler changes effects. KSP designed and aligned with kotlin and fully understands kotlin specific features like extension functions, declaration-site variance, etc.
Easy to use API
KSP is spicifclly designed for kotlin and provides a simplified compiler plugin API that leverages the power of Kotlin while keeping the learning curve at a minimum.
Hide Compiler Changes
If we use compiler plugins, we must change our processor every time that compiler version is updated, and we must use the compiler complex API, but if we use KSP because the API and Implementation are separated, we don’t need to change the processor codes whenever the compiler is updated.
Support JVM/Android, JS, Native & Multiplatform
One of the essential features of KSP is that not bound to JVM, and from KSP version 1.0.1, KSP can be used on other platforms.
Incrementality
Incremental processing is an approach that lets us re-use processing output when we do not change the source code. For fast build time, Incremental processing is currently enabled by default in KSP.
No modificaion; Generation Only
In KSP, the source code modification is disabled because modifying the source code during the processing causes many side effects.
2x Faster Then KAPT
Based on the Kotlin benchmarks, the KSP is 2x faster than KAPT on some libraries like glide and room.
Libraries that support KSP
Currently, Room, Moshi, RxHttp, Kotshi, Lyricist, Lich SavedState, gRPC Dekorator, EasyAdater are supported KSP. Here you can see the full list.
KSP API
KSP Contains three packages wich you can find in KSP at GitHub.
- Processing
This package contains interfaces that we need for building processors like the CodeGenerator interface for generating source files, KSPLogger for logs during processing, and the base SymbolProcessor that we use as an interface for our processors. - Symbol
This package contains the classes and interfaces that we need to model the kotlin classes, functions, such as KSClassDeclaration, which models class declarations, including class, interface and object. - Visitor
This package contains the base type of visitors like KSEmptyVisitor, KSVisitor, etc.
Below you can see the simple Processor with KSP API that uses some classes and interfaces from the above packages that generate an extension function for annotated class with HelloWorldAnnotation.
| /** | |
| * This is simple processor that use KSP API to generate | |
| * extension function for annotated classes with [HelloWorldAnnotation] | |
| */ | |
| class HelloWorldProcessor( | |
| private val logger: KSPLogger, | |
| private val codeGenerator: CodeGenerator | |
| ) : SymbolProcessor { | |
| override fun process(resolver: Resolver): List<KSAnnotated> { | |
| val symbols = resolver.getSymbolsWithAnnotation(HelloWorldAnnotation::class.qualifiedName!!) | |
| val returnList = symbols.filter { !it.validate() }.toList() | |
| symbols.filter { it is KSClassDeclaration && it.validate() } | |
| .forEach { it.accept(HelloWorldAnnotatedClassVisitor(), Unit) } | |
| return returnList | |
| } | |
| inner class HelloWorldAnnotatedClassVisitor : KSVisitorVoid() { | |
| override fun visitClassDeclaration(classDeclaration: KSClassDeclaration, data: Unit) { | |
| super.visitClassDeclaration(classDeclaration, data) | |
| val packageName = classDeclaration.qualifiedName?.getQualifier() ?: kotlin.run { | |
| logger.error("packageName is empty") | |
| return | |
| } | |
| val className = classDeclaration.simpleName.getShortName() | |
| codeGenerator.createNewFile( | |
| Dependencies( | |
| true, | |
| classDeclaration.containingFile!! | |
| ), | |
| packageName, | |
| "${className}_Generated" | |
| ).use { file -> | |
| file.write(""" | |
| package $packageName\n | |
| fun ${packageName}.${className}.printHelloWorld() { | |
| println("Hello world!") | |
| } | |
| """.trimIndent().toByteArray()) | |
| } | |
| } | |
| } | |
| } |
HelloWorldProcessor.kt
Here in the CTMP-Kotlin Github repository, you can find more examples and codes about compile-time-meta-programming.
Conclusion
We review the meta-programming concepts, RTMP and CTMP in kotlin. As you know, tools are created to solve our problems. These days, many libraries and big companies use these two techniques to resolve their problems. Currently, reflection API in kotlin is used for RTMP, and KSP is the best choice for resolving problems with CTMP.
In the following articles, I will try to describe the KSP API and some other models that metaprogramming could help solve our problems.
Refrences
- Metaprogramming
- Java : Getting Started with the Annotation Processing Tool, apt
- Kapt: Annotation Processing for Kotlin
- Kotlin KSP Documents
- JSR-269
- JSR-269 Java Doc
- JSR 175: A Metadata Facility for the JavaTM Programming Language
- Java The Reflection API
- My Experience with RTMP, CTMP and other types of meta-programming in kotlin and java
This article was originally published on proandroiddev.com on April 25, 2022
