hello everyone hi my name is jing bo yan i'm from facebook i've been starting uh in facebook for like seven years ago and working at andrew for the past five years um and in the recent one and two years i've been working with kotlin stuff like dependency injection uh or like annotation processor kind of thing um today we'll be sharing my learnings about kotlin compiler plugin

okay okay yeah today we'll be talking about a kotlin tool called calling compiler plugin before we get deep into what it is let me tell you our story behind it it all starts when developers try to use android sustries in order to add your own sisters log a very common code pattern is wrapping your code with trees.begin section and end section and a try filing block however over time our developers start to want a very small and reasonable simplification can't we annotate or mess up with these annotations at trees messed up and let that automatically generate our code for us so oracle can be as clean as these um if you think can't we just do that what you mentioned with annotation processor congratulations you're wrong you fall into the same trap as i did um in case not everyone is familiar with annotation processor when you write an annotation like add get in retrofit or at inject in dagger to what actually happens is annotation processor or run at compile time to parse uh what and where you annotate and maybe generate new files accordingly so it's natural to consider annotation processor may solve or sister's example here however in java stack in fact annotation processor is only expected to verify files or generate new files it's not exactly to modify existing files so it cannot do the transformation we wanted for tracing if you're one of those naughty engineers that still want to hack it through unfortunately it becomes pretty much impossible now in kotlin in fact coupling files are converted to java stops before passing to java annotation processor so that's how it's able to reuse java annotation processor therefore all the annotation processor can see is messed up signatures like this it doesn't even have access to original source code not to say modify it that's why we turn our eye to this new kotlin tool called kotlin compiler plugin it's one level deeper than annotation processor in fact a kotlin run annotation processor in kapt which itself is a compiler plugin and because it's one level deeper coding compiler plugin has much more power and they do have access to the source codes therefore it can transform code and even if you just want to parse and not transform the code you do have more access than the message signature now i hope you can take a moment and think about your own project here is there any code you wish to transform

to inspire you but not to limit your thoughts here's a few examples of use cases that i have thought about the most intuitive motivation of transforming code is by no doubt simplifying api an example before we mentioned wrapping code with sisters similarly many may want to wrap their code with loggings for example start time and end time log that last but not least if you did cochin with annotation processor before you would probably notice calling the generated method can be a bit painful the auto-generated method name can be ugly calling it is sometimes read in the ide with compiler plugin you may just call a simple stat method instead and transform it into the auto generate method another potential use case is optimizing performance maybe you can latify your programs maybe you can replace string with index to reduce allocation or just compare with annotation processor who generate messages in the new class maybe you can just generally mess up inside the existing class or even modify the message inline this can feel like a macro optimization to you but think about what if you have 200 classes or in facebook you can have 2000 usages of this annotation then you would have saved 2000 new class which can improve the app size and class loads last but not least in your own project it's all up to you you can use your creative thought and come up with many more useful ideas for your own so why don't we start with it already um hold on for one second because although it's powerful making a calling compiler plugin is far from easy in fact there's no official documentation there's no public api for it yet um and at least for me this is the first kotlin compiler plugin to facebook um so also facebook use back instead of gradle so i'm also unsure it will work on the back in the beginning for you this is also a new tool so you might have similar concern before introducing to your project i would guess last but not least a few examples we found seems indicating we need to rewrite by code in order to transform the code which is a bit scary if you want more developer to write it but we figure out all these one by one if there's no documentation we learn from examples if we worry it may never work we make a prototype with simple transformation as a proof of concept the buy code line seems hard to resolve but we see a project called aromata which indicates the possibility of parsing the roll call instead of the bike home that's why today we're going to share our experience on how to write our own code transformer by compiler plugin without dealing with by code time to go with real code the workflow contains three sections griddle plugin which wraps your compiler plugin so that the other griddle project can just easily use it a cut incompiler plugin which registers your own step to the compiler notice the analysis handler extension is highlighted spoiler this is the very central step that achieved the code transformation you can look forward to it last is the tree visitor it makes parsing the kotlin file easier by breaking it down to a syntax tree we'll start with the griddle plugin it contains a plugin and a sap plugin you can declare a griddle plugin easily in the build script by using the java griddle plugin a few important points here are the group id the version and the griddle plugin id you also do need to write an implementation class that extend the plugin which does nothing right now now you already finished a simple griddle plugin upon publishing you can apply this as plug-in on other griddle projects or sub projects by adding this line to their build script the idea used here is again the griddle plugin id you declared on the right now griddle plug-in has a lot more than that like how to publish it online or to publish it locally but this talk is about kotlin compiler plug-in so we won't cover all those today you can try it out yourself by adding a griddle you need or you can check out all these useful links i found we did mention there is a plugin and a sap plugin so now is the kotlin griddle sap plugin to start with you need to add two more dependencies related to kotlin we'll add it on top of the almost empty griddle plugin we created before in fact we'll make the same class also extend the code ingredients plugin this will introduce four more method to implement i'll work through the important part here first of all you need to provide the group id the version and the artifact id here you need to match the one you declared before in the greater build script the artifact id if you haven't explicitly declared before will be the project name now here you need to also provide the kotlin compiler plugin id uh notice that's not the credo plugging id and you can also provide plug-in options to it which is basically input params here we provide the output directory as input param to indicate uh we want the transform file to be put in the build folder that's you already finished the sap plugin you do need to add this to this resource file so the compiler notice you have a step plugin here um if you're a hardcore developer like me yes i'm very hardcore uh you may wonder why we need both a plugin and a step plugin what are they right based on my investigation in short the brutal plugin is for allowing you to apply this plugin to other griddle projects uh well they calling riddle stuff again make sure when the plugin is applied an an x plugin param will be provided to the calling c command line which uh will be the one that adds this compiler plugin to the compiler uh plugin options by which i mean input programs again is also attached here to this program therefore if you are not using griddle then you don't need to write any of these two files but you do need to manually provide the x plugin param to call in c command line for example for us facebook use buck instead of griddle so uh back will have a kotlin compiler plugin per rom for you to add is this uh plum x plugin program to the command line that concludes the griddle plugin and it's zap again now let's talk about the kotlin compiler plugin it contains command line processor and component registrar and they will register an analysis handler extension which we mentioned will be the central step of this code transformation to start with we need another extra dependency to use code related to kotlin plugin the command line processor and component register is the easy part we mentioned the griddle plugin provide an x-packing param to the command line well these two just read the x valve around from the command line for the input program and use these to create and register the extension here's the code for the command line processor it looks identical i know but the key part is it declared a plugin id and a plugin option if you recall uh this is the ones we put in the griddle stuff plugin this is the the id and the input param we put there for each param it found in the command line it would save it in the configuration with a unique key and then when component register is called after the command line processor it will simply read the param from the configuration again you need to put this to class in the resource file now the most important step claim we create an extension with given params and register or extension to the analysis handler extension step now this is where the real magic begins everything before is more or less just to get the extension seen by the compiler they're more like procedure work basically you just have to roll those files in order to register to gradle and kotlin it's all for eventually register this one extension which can add your own callback at certain step of the compiler now we chose analysis handler extension uh to add it to the analysis step as it is early in the compilation process and they still have access to the role source code there are also many other extension types like um script evaluation class builder interceptor ir generation these runs in other step of the compiler the class builder interceptor extension is the one you might find in another talk about transforming the code by cloud income power plugin which have its own benefits um doing the buy code just we think it's simpler to start with transforming the roll code instead of the buy code there's no clear documentation on what you do at each extension point so if you're interested feel free to explore them and you may find your own favorite extension that can do your own magic now the real transformation begins by extending the analysis handler extension you get a callback by implementing this do analysis method the good thing about this method is you get to receive a files param which is a multiple collection of raw source codes so you can mutate this collection by replacing the old source file with a new source file that we achieved the code transformation one thing to notice is the new transform file need to be saved to an virtual file instead of just the file object in the memory uh here we save it in the output directory we pass in how to parse the old file text and generate a new file text is really up to you it's just a string processor so even a regular expression can do but you can see i use a tree wizard instead which we'll talk about in the next section to sum up calling compiler plugin contains a component a command line processor and a component register but the root work sees in the extension for which we chose another standard extension which can access mutable collection of files so we can achieve transformation next is the tree visitor i mentioned we can just use a regular expression for the string processing but parsing the entire file as one gigantic string isn't very sustainable for the long run kt tree visitor makes it much easier by breaking down your calling file into calling class calling function calling property so you can parse and transform them more intuitively to start the tree visitor you can file accept these then it will go through the file as a syntax tree which will call the visit name function at every name function in that and we override it with the name function here to do the transformation by the way there is many other visit functions you can overwrite like visits primary constructor with its property so you can do many more custom transformation as you like inside the wizard message we check whether the annotation contains uh at street method we hard code it here to make it simple in the real world you should probably pass it in then we wrap the original block body with trees begin section and trees.n section and the try finally block yes after 25 page we finally see this trace code again now there's still one last step which is saving the element and replacing string to a patch and apply all the patch together after visiting the entire file we have a snapshot of this helper function at the end thus we successfully transform to the left to the right you may notice we use fully qualified name here which can be a bit ugly but don't worry the transform code similar to annotation processors generated code is invisible to user unless they intentionally want to see it then they can go check out the build folder otherwise they won't even notice their source file is replaced in the middle of the build to ruin the whole process we have the griddle plugin which wraps your compiler plugin so that other griddle project can apply it we have the kotlin compiler plugin which register your own step to the calling compiler among these the analysis handler extension is the various steps that achieve the code transformation last is the tree visitor which is for helping the extension to parse the calling file easier i break it down to a syntax stream now congrats you almost graduated from jingbo's magic transformer school but before you leave the room there is one last thing i like to remind you which is don't abuse the magic if you use transformation without care you can easily confuse all the other developer imagine you read other people's calling code and see why this code is not even syntax correct and it can compile um we would have who would have saw they are actually transforming their code in a compiler planning so when you design your code transformer don't just think about making your api pretty but also think about whether other developers can understand it some other drawbacks are the debugging can be a bit harder breakpoint in the ide can be off by a few lines for now the only solution i found is keeping things in one line when you transform it's possible using later extensions that modified the bicode will solve it is one thing to consider also error messages can link you to the transform file in the build folder instead of the original code which can also be confusing so don't just think about don't just transform whenever you you can transform only when you really need if it can be achieved by a kotlin feature for example use that instead the rule of thumb is to keep your transformation simple minimal and understandable here's a few useful helper methods i used in my code and i've uploaded this tracer compiler plug into my github tool so you can also check that out that shall prove it really works and i'm not lying hopefully you guys find this talk helpful and are more interested in kotlin compiler plugin now here's a few useful links thank you guys for listening

um okay now let's wait for the slidell questions to come

actually is there any slight question oh i see how does this feed for a code base with both java and kotlin code

and no question but thanks for the great talk okay let's first answer how does it fit for the code base with both java and calling code so um for for for for back and gradle i i don't know how they feel they use they may split the module differently uh but for a module that use the kotlin uh code and they use the kotlin compiler then uh it will be able to have this coding compiler plugin so if it is a pure java module that doesn't have the calling compiler then it doesn't have the java the calling compiler plugin um uh yeah so yeah it depends on how you split the module

i think for us we mostly encourage like if a module just have calling code then we can use that if otherwise if it's a java module then we don't have that can you share the link to the repo um

like the link to the github uh yeah i mean in the speaker blog in the uh the other tab the speaker blog i have shared the slide i don't remember if i share the uh git github repo i'll see later if i don't then i'll paste it there

there's more question would this work with compiled library code like if we want sisters if you wanted to sister is a library i mean if this is your by compile library you mean like other people's code um no i don't think it works in other people's code like you have to be able to apply this plugin to this module i don't think you can apply the plugin to to their build script if it is other people's library but if this is your library then you can certainly do that

java parser can be used for processing the java code and then calling compiler plugin for kotlin

perhaps you have a parser um do you mean like the java annotation processor or is that another thing um

anyway um this is this is a question or an answer to the previous question

um all right um yeah so if um if that's about it let me just mention one thing about the um kotlin thing the the call in java module i mean for us we use the java annotation processor for the java modules we use the general annotation processor still works in the kotlin module where kapt and we also add on to that the um this uh compiler plugin as an addition the performance of kipt seems very poor compared to java does that make sense to just apply annotation processor for java files um and instead for cutting files you mean like maybe just use the compiler plugin so so from i mean i don't really know about i i know a little bit about the performance of the kpt uh i think if you configure it differently the performance can be a bit different but overall it is uh slower than java i can sort of understand because they need to um first turn the java file to calling files to a java stub and then apply it to the cutting compiler kpt and then to the calling compiler and everything um yeah but for us it's not reasonable to remake everything in the compiler plugin so mostly i think the annotation processor is still used for the java files and the annotation processor still works for the kotlin files and if some of it doesn't fulfill all the needs of you like you want some transformation or you want to read something in line that the message signature doesn't contain or you find it too slow then you can probably consider making a calling compiler plugin that's what we do sort of um yeah i think we are a bit over time uh if that's sort of all the question um then maybe we can and the talk here uh thank you guys for attending and hope you have a great day and enjoy the joycon