The Magic of Compiler Extensions

00:14

okay

00:15

welcome everybody to our droid con

00:16

online webinar this is the the final

00:18

version of our

00:19

of the final we have a three-part series

00:21

uh on advanced kotlin this is our final

00:23

uh webinar in the advanced kotlin series

00:26

and we're really excited

00:27

for our speaker today we're excited to

00:29

be with you for this final edition

00:31

uh and just wanted before we have andre

00:33

come on do his presentation wanted to do

00:35

the usual

00:36

a few housekeeping items with you uh so

00:39

i'm gonna get started because we've got

00:40

a great presentation we don't want to

00:41

take too much time away from that

00:44

as you know all of the droidcom online

00:47

webinars are available at droidcon.com

00:50

this webinar should be available

00:52

tomorrow uh androidcom.com it should be

00:55

indexed

00:55

so you can search searchable uh and you

00:58

should be able to find all those in case

00:60

you missed anything else in this series

01:01

or from our jetpack series or

01:04

or our multi-platform series or a

01:06

hands-on in-depth series

01:07

you should be able to find everything at

01:08

droidcom.com and we look forward to

01:11

seeing you over there

01:13

if you have not used a slido before

01:17

please note that uh slido is where we do

01:20

our q a

01:22

so if you would head over to slido.com

01:25

please put in this code also you can

01:27

check in the chat function in your

01:28

interface window

01:30

uh and only my colleague will be sending

01:31

that over as a direct link

01:33

so please use slido uh to to ask your

01:36

questions don't

01:37

use the the question function in the

01:39

interface here you slido

01:41

what that'll allow you to do is in case

01:43

you see a question that you wanted to

01:44

ask

01:45

that's already been asked you can vote

01:46

on it give it a thumbs up um

01:48

if you think it's a really important

01:49

question a really smart question vote on

01:51

it too

01:52

and that way when andre's doing his q a

01:54

all of the most important questions go

01:56

to the top

01:57

so we have time to answer them so make

01:59

sure you head over to slido.com right

02:00

now

02:01

open that up in your browser and get

02:03

ready to put your questions in there

02:05

just to give you a preview what's coming

02:06

up is you know we started

02:08

uh our our droid con online webinar

02:10

series in april

02:12

we've gone through as mentioned a few

02:14

different series on jetpack and

02:15

multi-platform

02:17

we're wrapping this up with with three

02:18

more time for questions too

03:17

so we're really excited to welcome andre

03:18

here um

03:20

i'm going to change the moderator over

03:22

to him

03:23

uh and so we can let him be the um

03:26

presenter now and andre feel free to

03:28

turn on your screen

03:29

or turn on your webcam and we'll be able

03:31

to uh

03:32

to see you as well there we go right

03:36

yes so so take it away it's all yours

03:41

all right all right so i hope you can

03:43

see me well

03:44

and the presentation as well and you can

03:48

hear me well as well because it's just

03:49

like really hard to

03:50

check it here so i'm really actually

03:53

excited to be on droidcon online i've

03:54

been following this whole thing like the

03:56

whole webinar series

03:57

for this time and it's really exciting

03:60

to be a part of it

04:02

uh myself and today i'm gonna be

04:05

focusing on one of my

04:06

like areas that was more interesting to

04:09

me personally for the last year i was

04:11

focusing on it really a lot it is a

04:14

compiler and a particular coating

04:16

compiler

04:17

and its extension or compiler plugins

04:20

you might have heard

04:20

them about that before but it's really

04:23

interesting

04:24

area which you can actually go deeper

04:27

and check how coding

04:29

works and why there's certain things the

04:31

way it does

04:32

so let's start and the first thing about

04:35

compiler is this

04:36

actually tool that we uh

04:39

that we use to take our source code and

04:41

compile to some kind of

04:43

form of binary and we don't really care

04:45

about how it works that much we have

04:47

like our build system like gradle or

04:48

something like that depending on what

04:50

you use or maybe using command line

04:51

interface that's also fine

04:53

but we don't really need to go deeper in

04:55

that

04:56

and check what's inside this magic box

04:59

and today's talks is right about this

05:02

magic box i'm going to be talking

05:03

how it works what kind of things he does

05:05

inside of our source code and how it

05:07

produces binaries

05:09

and with that i will keep the focus on

05:12

actually extending it for

05:13

our needs and doing some kind of meta

05:16

programming congenital things

05:19

but first allow me to introduce myself a

05:21

little so as you already saw i'm like

05:23

android engineer i'm doing android for a

05:25

while and yeah that's my photo before

05:27

before i started doing compilers

05:29

um and that's where i met kotlin

05:32

while doing android and that was the

05:34

perspective that i came into this

05:36

uh kind of whole cooking compiler

05:39

area with cotton i have really long

05:43

relationship like i've been using it

05:44

since

05:45

before the release and to be honest i

05:49

never look back

05:50

and as i started going into compiler i

05:52

realized that cotton actually has an

05:54

open one so you can just go on the

05:55

github check the code and do whatever

05:57

you like

05:58

and that's where i went i learned a lot

06:01

of stuff

06:02

and today today's talk is mostly result

06:05

of this research in the code base

06:07

and made the amazing things alone there

06:08

i don't work for jetbrains just to

06:10

clarify

06:11

and i mostly like talk to them on uh

06:14

like slack and other channels but didn't

06:17

really

06:17

participate in the creation of the

06:19

compiler itself

06:22

so this talk will be in the three parts

06:25

first i will try to clear the line

06:27

between annotation rotation

06:28

and this it will be like java annotation

06:30

procession and compiler plugins from

06:32

code inside

06:33

and when this line is cleared when we

06:35

get this understanding we can go into

06:37

the plugins themselves

06:38

and we'll go that first theoretical

06:41

things

06:41

like basically look at how the compiler

06:43

works and we'll do it step by step

06:46

and later we'll go to create the plugins

06:50

like extend these compiler structures we

06:52

already have and we'll go through

06:54

more practical things like some code and

06:58

examples of apis that we have at this

06:60

particular moment of time

07:02

all right so what's the difference

07:04

between annotation procession and those

07:06

plugins

07:06

they are both kind of playing the

07:08

similar role in the ecosystem

07:10

the kind of meta programming tool which

07:12

allows you to do some kind of code

07:13

generation and some kind of things

07:15

and on the surface they're quite similar

07:18

so

07:19

when you start using that you start with

07:20

the source file so they both take

07:23

source files of each respective platform

07:25

obviously but jetbrains actually created

07:27

tools

07:27

so you can use for example gem

07:30

annotation persistent we've gotten

07:32

when you generate these steps or uh

07:35

quoting compiler actually can understand

07:37

the java files when it compiles

07:39

you can just link to link to them but at

07:42

the same time

07:43

you won't be using quotes in the java

07:44

ecosystem because like what

07:46

doesn't compile uh java files for that

07:48

you have java compiler

07:50

and similar way annotation system kind

07:52

of work with coding without those steps

07:56

the first big distinction comes of how

07:59

we approach input

07:60

with annotation processing here we

08:03

uh get source files and we emit

08:06

new source files and then we can repeat

08:08

this over and over again

08:11

uh with different processor maybe the

08:13

same one but it just like multi-step

08:14

process can be repeated

08:16

there are some uh exceptions like lone

08:18

poke or stuff like that but they usually

08:20

using some kind of internal api majority

08:22

of the processor i've seen they actually

08:24

created the new files

08:25

and compiler is on the totally opposite

08:28

side of the scale so it doesn't emit

08:30

like new source files that you will be

08:32

later feeding and compiling it actually

08:33

creates in a binary

08:34

and it's really focusing on doing that

08:36

efficiently so the

08:38

plugin here is not like something that

08:40

manipulates with like source files or

08:41

not

08:42

area it actually uh some kind of

08:45

transform

08:46

like the extension on inside the

08:49

compiler that you can use

08:50

to adjust the generated code so the

08:53

result we will see only there

08:56

there is a way to do something like

08:59

annotation processing dance in kotlin

09:00

and ksp or code and symbol processing

09:03

you may be heard of that is like tool

09:05

from

09:05

google which went with alpha like in

09:07

june so there is a link below you can

09:10

check it out if you want

09:11

so essentially it's replicating like

09:13

this single pass annotation processing

09:15

using internal

09:16

kind of plug-in api that is there or was

09:19

created originally for kpt i think

09:21

uh and it's kind of doing the same but

09:24

majority of the plugins actually just

09:26

focusing on transforming the

09:29

source to the binaries

09:33

and the last thing i wanted to draw on

09:35

this distinction is actually

09:36

documentation because when you go into

09:39

annotation processing work it will have

09:40

been

09:41

quite a long time around and you just

09:44

have a lot of different materials you

09:46

have tutorial stocks and

09:48

the api itself is really stable so you

09:52

won't really expect the uh you won't

09:54

really expect it to break

09:56

like your annotation processing to break

09:58

when you're changing versions of like

09:59

that

09:59

with cotton block is totally opposite

10:01

you kind of

10:03

have really experimental api which was

10:04

mostly created for internal needs of jet

10:06

brains

10:08

originally and right now they slowly go

10:10

in into different directions like some

10:11

more stable thing

10:12

so it's some specifics they are changing

10:15

almost every release right now

10:16

and documentation is essentially

10:18

non-existent

10:20

so thankfully the code is available for

10:21

you to research in the first place and

10:23

community is quite helpful if you're

10:26

stuck and if you want to try something

10:28

out you usually point

10:29

out where to start where to search for

10:33

one thing i wanted to note here while we

10:35

are on uh annotation process in the

10:37

plugin

10:38

area is that why we don't really like

10:42

kpt in coding

10:43

why we'll compile an ad so the uh

10:47

most problematic part here is generating

10:49

this type so basically take

10:50

all of the quotient source source files

10:53

then you take all declaration compiler

10:55

to java so you create new java source

10:57

files and this process is quite slow

10:59

and it obviously depends on the module

11:01

size but usually takes like 20 30

11:03

percent

11:03

of the compilation itself and it cannot

11:07

be skipped

11:07

because as i said java annotation

11:09

processing cannot really understand what

11:10

happens inside the button so you need to

11:12

convert these files forward

11:13

with plugins you essentially go to

11:15

coding files and just compile

11:17

yes you may have some limited uh limited

11:20

kind of understanding of the java files

11:22

as well but if you focus in on coding

11:24

here in the first place like if you have

11:25

gotten

11:26

database or code base so it's

11:29

uh much faster to do so and just work

11:32

with the plugins directly

11:37

and plugins actually allows us to do

11:38

even more things so you can focus on a

11:40

multi-platform create your plugin which

11:42

uh does some kind of meta programming or

11:44

code generation on all the platforms

11:46

that codes and supports

11:47

you can change the code that user writes

11:49

yes you can do this with some kind of

11:52

part of the uh java coupon or java

11:54

annotation processing part but

11:56

it's still much better formulated here

11:59

inside the

12:00

uh kotlin capillary plugins and you have

12:03

way more possibilities of actually

12:05

detecting the things that you want to

12:06

change so you don't have to be focused

12:07

on rotations only

12:08

you can just go around change some

12:10

function calls or probably you have some

12:12

interesting class that extends some

12:13

marker interface you want to add some

12:15

things to that or generate some

12:17

uh some basic uh implementations of the

12:20

methods for interfacing for that you can

12:22

do it and you don't need any kind of

12:23

annotation on

12:24

any additional code to do so

12:29

but obviously there are some drawbacks

12:31

in here and first of all

12:32

is not a big on my from my point of view

12:35

but

12:36

still is important for many it's like

12:37

multiple passes so intelligent

12:39

protection you can go around and apply

12:41

different processors and

12:42

try to uh together as a multi-step

12:45

approach

12:46

with coding compiler you cannot really

12:47

do so and

12:49

to be honest compiler plugins frequently

12:52

even don't work with each other

12:54

i'm not sure if in serialization for

12:55

example right now work with compose well

12:57

and so on

12:58

and the second thing here is that you

13:01

don't really see the generated code that

13:02

much

13:03

you can check the by bytecode that is

13:05

emitted by jvm

13:06

for example you can dump some internals

13:08

of the compiler

13:10

but from user perspective it's much

13:12

easier to just go to generated java

13:14

files

13:14

like like you do with annotation

13:16

procession and check whatever whatever

13:18

generated there

13:19

you can do you cannot do this with the

13:20

compiler because they just focus on

13:22

emitting the final binary

13:24

so we have to leave that

13:27

all right so i hope we kind of

13:32

give you some understanding of what is

13:33

annotation procession what the plugins

13:34

where the line is landscape of the meta

13:36

programming that we have

13:40

and now we're moving completely word of

13:42

the compile

13:43

and as i said uh this part to be

13:46

like more theoretical we will go through

13:49

the parts of compiler how they work

13:51

together and what the structures they

13:53

pray so later we can utilize this

13:55

knowledge of actually using apis

13:57

that export the compound yeah

14:00

i establish already that there is uh the

14:03

source

14:04

files that getting transformed into the

14:07

binaries and the binders can do

14:09

different platforms because we have

14:11

kotlin

14:12

if look at this a bit deeper

14:16

we can see that this process kind of

14:19

happens in two big steps

14:20

first of them is called front-end and

14:22

second one is called back-end

14:24

so the front-end is where compiler tries

14:26

to understand what the hell have you

14:28

written in the source code that are

14:29

trying to fit in there

14:31

and here it validates errors both

14:34

syntaxical

14:35

and logical ones and this understanding

14:38

later packed into this structure the

14:40

structure is really

14:41

useful for water in the back end

14:44

what we do it with that data in the back

14:46

end we actually generate it for the

14:48

target platform

14:49

so that's where we already have complete

14:51

understanding what it happens inside

14:53

of the code that you provided to the

14:55

compiler and we can generate the

14:58

binaries or whatever the uh

15:01

targets target platform sources are

15:04

here before generating backend also goes

15:07

through the

15:08

optimization steps like for example in

15:10

line-in or

15:11

it could be some plasma specific stuff

15:14

like interrupt

15:14

the intel interrupt for jvm

15:18

for getter setters for properties or

15:19

something like that

15:21

so this is basically the front end we're

15:23

trying to understand what happens and

15:25

the back end we use an asset design to

15:27

generate

15:27

the code for the target classroom let's

15:29

go one layer deeper

15:31

and we'll start with front-end and trace

15:35

uh trace this snippet through all of the

15:38

structures that it gets converted into

15:40

so we'll understand how it goes

15:42

what steps are there on the way

15:45

and the first step is to convert this

15:47

code snippet to psi

15:49

actually it's based on this kind of

15:51

multi-step process but it's the first

15:52

one that we

15:53

exposed as a compiler plug-in writer so

15:55

we'll step in there

15:57

so psi is the program structure

15:59

interface

16:00

and is really close to its function on

16:02

an abstract syntax tree if we

16:04

ever if you know what that is and

16:08

while parsing the code like parsing

16:09

these three compiler already

16:11

checked for all the syntax errors it

16:13

knows that parentheses is all in place

16:15

there are no missing commas and so on

16:18

and

16:18

this three that it gives us it also is

16:20

immutable

16:21

so it's directly tied to the source code

16:24

that we've written

16:25

so we cannot really change it that much

16:29

uh this tree is also starts with the

16:31

file so basically we have this

16:32

collection of these trees for each of

16:34

the files that we

16:35

collect there and it really helps us to

16:38

analyze effectively

16:40

for example if we interested what is the

16:42

class that continuously selected

16:44

function

16:44

we just walk up the tree we find node of

16:46

the type class

16:48

and here we go or if we're interested in

16:51

a

16:51

like the connection other way like we

16:54

want to check what are the functions

16:56

that are inside the

16:57

class that we are purchasing we can do

16:59

the same by just going down the tree

17:02

in general psi is used by jbrains in

17:04

many places if most of the it is

17:07

and they have quite good to me quite

17:09

good documentation on that but not

17:11

unlike coding part of the website but

17:13

unlike

17:13

eddy part of the website so you can

17:16

check it out if you want more answers

17:17

with your house and

17:19

wise

17:22

so right now we have this uh tree parsed

17:26

but we still don't have the complete

17:28

information about what happens

17:30

inside these source files to demonstrate

17:32

that i added a simple call

17:34

so basically right now we have println

17:36

function that we call

17:38

inside of the example function and

17:41

this printer line is not really in the

17:42

current file and it's not really

17:45

even any current like uh set of files

17:48

that we are compiling it's in totally

17:50

different artifact

17:52

but we still need to validate the

17:53

parameters we need to check that we use

17:55

string that we pass is actually correct

17:57

and probably return type we're also

17:59

interested in that and

18:00

later we have to generate a bytecode for

18:03

that so basically

18:04

forwarding to correct instructions in

18:05

the bytecode so we all need to

18:07

link this information to the description

18:09

of that function

18:10

and for that uh coding compiler has this

18:14

structure name descriptors these cuters

18:17

can be seen as a metadata that is

18:18

associated with this psi

18:21

uh and it describes more information

18:23

about the class for example

18:25

uh you can check what is the

18:26

constructors of the class what is the

18:28

type

18:29

uh what are the type parameters that it

18:32

has

18:32

and function could be like parameter

18:34

infor and like name and obviously all

18:36

these things

18:37

so with the descriptors they can also

18:40

exist

18:40

outside of psi for example every

18:42

artifact is also part into these

18:44

descriptors

18:45

that are organized in the tree as well

18:48

ns3

18:49

also has two other elements that we

18:50

don't see in psi directly

18:52

first of them is like module the

18:54

compilation unit

18:55

uh basically all of the elements that

18:59

have during current compilation like our

19:01

current files that we compiling

19:03

some dependencies that we have all of

19:04

the classes functions can be available

19:06

from that module

19:07

and they are separated with different

19:09

package fragments usually like for

19:10

different arts in fact you have package

19:12

fragments something like that

19:13

and they are using like fully qualified

19:15

name the one that you usually write in

19:16

the import

19:17

you can actually get access to the class

19:20

like description

19:20

of the class of the function or whatever

19:23

is declaration you are looking for

19:26

so let's say that we checked the imports

19:27

and we found our println function

19:29

and now we wrap it into resolved call

19:32

which keeps the context of all of the

19:34

things that we're calling here so for

19:35

example let's say that uh

19:37

right now uh here we have the function

19:39

descriptor for println

19:40

and hello as a parameter so we just

19:42

remember that there is a constant as a

19:44

parameter here in

19:45

hello we can also kind of capture some

19:49

elements of the outer context for

19:51

example we inside the class and we want

19:52

to use some field inside this println

19:54

uh call we can also do that you it also

19:57

captured using this result called

19:58

mechanism

19:60

so these are two elements uh which

20:02

extensively use an

20:03

analysis and if you're writing compiler

20:05

plugins you probably

20:07

will try to manipulate and somehow

20:09

utilize it to

20:10

acquire some knowledge about the code

20:12

that you actually combine right now

20:15

and uh this connection between psi

20:18

and all of these elements is starting

20:21

inside the building context

20:22

you can look at it as like a hash map

20:25

which contains a different connection

20:28

between different comments

20:29

you can add your own connections there

20:32

for example if you're interested in some

20:34

metadata

20:35

specific for your plugin you can add it

20:37

there and

20:38

you can sometimes alter the existing

20:41

ones

20:42

but it's probably not the best way to

20:44

approach them

20:48

so we right now parsed all this data

20:50

into psi into descriptors and into these

20:53

result codes

20:54

and we can move into generating part

20:58

plug into the back end

20:59

and the back end is quite interesting

21:00

here because you have to support all the

21:02

different

21:03

platforms

21:09

and because of that because we have to

21:10

support these platforms

21:12

uh we have a back-end separated we at

21:15

least we had a security for most power

21:17

but front it was shared between

21:20

different platforms

21:22

so it was like that before the new

21:24

back-end came in so probably you heard

21:26

about this ir

21:27

or intermediate representation back and

21:29

then we have and right now it looks

21:31

something like that so it will be coming

21:33

somewhere in like one

21:34

for something unstable you can enable it

21:36

for a long time all right

21:38

and the interesting part here is that

21:40

despite behind all of these different

21:42

platforms with their custom tools for

21:44

generation so for jvm you for example

21:46

have object web asm

21:47

for native series lvm compilation that

21:49

takes place and for javascript there if

21:51

i remember there are some custom tools

21:53

for that

21:53

so right now with this intermediate

21:56

representation you can define some

21:58

common transforms and for us as a

21:60

compiler plugin writers

22:01

it's particularly important because we

22:04

can't

22:04

write our plugin like code generation

22:06

for a plugin only once

22:08

and apply it to all the platforms which

22:10

is quite cool for my opinion

22:11

um the api by itself is also improved a

22:15

lot with

22:16

when ir came so right now it's much

22:18

easier and much more accessible from my

22:20

point of view

22:22

and right now majority of the new

22:24

plugins that

22:26

are kind of support kind of are coming

22:28

right now on the scene they

22:29

support all this new backend as a

22:32

default so for example compose

22:33

it works by default on ir back and

22:36

that's why it's

22:38

kind of yeah

22:41

uh the exceptions from that probably the

22:43

first party from jetbrains

22:45

you have like you need to support all

22:47

the backhands in there and

22:48

if i remember correctly and wheel from

22:50

square also is written using old

22:52

backhand

22:52

because they want to use tool right now

22:54

before the ir is right

22:57

so ir by itself is also a tree

22:60

and it contains all the information that

23:02

you had in psi

23:04

and descriptors together so basically

23:06

this complete information about each of

23:08

the nodes

23:09

it's much less variables on psi so you

23:11

don't have

23:12

different nodes for identifiers and so

23:14

on entry point here is

23:16

module as we saw with the descriptors

23:17

but right now it contains

23:19

all of the modules that we compile right

23:22

now so basically files that we compile

23:23

right now

23:24

and the important part of this tree is

23:27

that right now it's mutable so you can

23:28

just go there

23:29

change whatever function you want the

23:31

body of the calls whatever you're

23:32

interested in and this is

23:37

extensively used in many plugins for

23:39

example compose modifies majority of the

23:41

calls with the composable functions and

23:42

so on

23:45

uh the extension that we have for

23:49

like adding new stuff to ir is

23:52

applied right after it is created so we

23:55

can focus on generated code

23:57

as close to original sources other

23:59

original source files as it could be

24:01

and later it goes through the process of

24:03

lowering or optimization so lowering is

24:05

basically this

24:06

as i understand the places kind of this

24:08

path for all of the

24:09

files that does one particular thing so

24:12

for example

24:13

there was uh this tweet from jay quarter

24:16

some time ago

24:16

said lowering for multi-line strings

24:19

which

24:20

uh does stream marching on the compile

24:21

time or

24:23

there is lower in the for example right

24:25

function perhaps function

24:28

uh with on jvm with the classes on top

24:30

so something like that

24:31

and they usually go for all of the files

24:34

and do only one particular thing and

24:36

later

24:36

this optimized code is uh generated to

24:39

the target ladder so like bytecode

24:41

what's already interesting

24:44

all right so this is for the compiler

24:46

first we have a front end

24:48

trying to understand what happens inside

24:50

the

24:51

source files and validate that this code

24:52

is actually correct and later we go to

24:55

the back end

24:55

and here we are focusing on optimizing

24:58

that

24:59

uh structures that we had and generating

25:02

the resulting

25:03

code for the target platform that we

25:04

have compiled right now

25:06

i hope it gives you at least some

25:08

understanding about it happens

25:09

or what is happening inside so let's

25:12

right now try to add something to it

25:16

and that's where we go into actual

25:17

creatine plugins

25:22

i will be jumping here through the code

25:25

and try to run the through process of

25:27

creating a plugin

25:28

with different pieces that can change

25:30

there

25:31

and the first note i wanted to mention

25:33

is that it's actually really

25:35

experimental all api that i mentioned

25:37

right now

25:38

so as i said you frequently have your

25:41

plugin break after like version of the

25:43

eighth of the coding

25:44

with minor one and i'm pretty sure it

25:47

will gonna be changed in the future

25:48

because

25:49

right now many kind of big plugins come

25:51

to release the compose base uh

25:53

by itself is worth money so it just

25:57

uh right now it is this experimental api

25:59

that we need to be aware of

26:00

but many things at least on conceptual

26:02

level

26:03

uh will stay the same basically now is

26:06

based

26:06

on the price we pay to get in compiler

26:09

plugin development basic early

26:12

all right so what is the compiler plugin

26:14

in the first place it is basically a jar

26:16

that you compile using the dependency of

26:18

the compiler

26:19

and later you can add this chart using

26:22

the

26:23

command line parameter x plugin so here

26:25

i did it in gradle

26:27

but in gradle you actually can do much

26:28

better you have the

26:31

transformation you can use like myelin

26:33

artifacts or mymac originates

26:35

or you can use like any kind of

26:36

artifacts that you got used to in new

26:38

ground

26:39

so you can notice here that there is

26:41

different

26:42

clasp for native and it's just because

26:44

how

26:46

shadow jar is set up for jvmgs compiler

26:49

and native one there is nothing beacon

26:51

there you can just mimic it

26:53

by yourself but it's something to be

26:54

aware of

26:57

so we connected our plugin and the first

26:60

thing that gets connected is a company's

27:02

register

27:03

as i said it's connected through the

27:04

service loader and as with anything

27:06

that uses service loader in a jvm we can

27:09

use auto server

27:11

annotation processor to connect that and

27:14

when compiler gets our component

27:15

register

27:16

it calls this method uh to register the

27:19

components

27:20

and the components or extensions that

27:23

are available there

27:24

they look like this so you have the

27:25

interface some extension slide

27:27

and it's companion object has a method

27:31

to register

27:32

like instance of it and there are many

27:34

extensions like that inside the compiler

27:36

so you for example have

27:38

uh one that allows you to modify iron

27:42

some particular phase

27:43

or you have one that allows you to check

27:45

for errors or the one

27:46

that allows you to kind of other

27:49

separate descriptors like custom things

27:51

to so compile will be aware of

27:53

during analysis on the front end so

27:57

basically each compiler appliance

27:59

usually multi implements multitude of

28:01

those checks

28:02

and their combined provide the whole

28:04

functionality of the plugin

28:07

just an example that's how you connect

28:09

that you have your extension

28:11

you have the method to register it and

28:13

your instance and that's

28:15

basically extension registered

28:18

that's why if you're looking through the

28:20

some compiler plugin implementation that

28:22

you're interested in

28:23

you can go to company registrar and see

28:25

all of the

28:27

extensions that are connected right now

28:29

and this can give you at least some

28:31

overview what happened it

28:32

what happens inside that because you may

28:34

be aware that this extension that this

28:35

on this particular face

28:36

or some another and later you can go

28:39

like face by face and trying to

28:40

understand

28:41

what the compiler plugin does in each of

28:43

those

28:46

and as an example of writing the plugin

28:48

i will

28:49

look at two really small subsets of the

28:52

quote and actualization

28:54

uh code link serialization by itself is

28:56

a really good example of the plugin that

28:58

you made right

28:59

i totally recommend you go there check

29:01

it out and see

29:02

how it works inside like the one that

29:04

you actually could use in production

29:06

but here i will be taking like small

29:07

subsidy for those who do know

29:10

what that is it's basically the magic

29:12

annotation that you cannot take your

29:13

classes with

29:14

and laser regenerates some bulletproof

29:17

so you can

29:18

integrate it with a serialization

29:20

framework

29:21

library that it has what exactly does it

29:24

generate

29:25

basically two parts first is this custom

29:26

serializer that explains how you will be

29:29

serializing code it has better to

29:30

serialize deserialize and this

29:32

child serializer should actually explain

29:34

uh how the structure will be looking

29:36

and uh this

29:39

custom serializer is available through

29:42

the method

29:43

in a companion object that you can

29:46

basically use

29:47

to specify to the library how you want

29:50

the object to be serialized

29:52

so that will be focusing today so it

29:53

will be like a method and then some

29:55

implementation of the serial license

29:58

and we'll be doing that in two parts so

29:60

first of all first of them

30:01

is to add serialize because uh during

30:04

analysis because compiler doesn't really

30:06

know that we have this new method so if

30:07

you look at that

30:08

in the companion object we have a new

30:10

method we can call outside of this class

30:12

and compile is not really aware that

30:14

this method exists right now so we need

30:15

to tell it like an analysis phase there

30:17

is method that exists

30:19

and later we will actually generate

30:20

implementation of the serializer

30:22

and the functions let's start with the

30:26

first one

30:27

so here we'll be adding serialization

30:29

serializer descriptor

30:30

during analysis stage and to do that we

30:33

will use the

30:35

extension that is called synthetic

30:36

resolve extension it allows you to add

30:38

properties

30:39

methods and classes i think that's it uh

30:43

two different like different classes

30:45

that you have inside content

30:48

and you can do you do it in like two

30:50

stages

30:51

first you specify that there is this new

30:52

identifier that you are interested in so

30:55

basically saying that there is new

30:56

identify serializer

30:57

inside this companion object of the

30:59

serializable class and later you specify

31:01

the metadata associated with this

31:03

identifier

31:05

so let's go through adding actually that

31:07

method yeah that's the method

31:10

so first we specify the identifier and

31:13

here we say

31:14

okay if this companion object is

31:17

actually serializable

31:19

oh if this component is actually inside

31:22

a serializable class

31:23

which we check using the annotation we

31:26

will add like return this new identifier

31:29

that we are interested in

31:30

otherwise we just add anything

31:34

then this name like then we get called

31:37

with another method

31:38

and here it's very important to check

31:41

that

31:42

we are called with is the one that we

31:44

are

31:45

ourselves because compiler will be just

31:47

calling this with all the classrooms you

31:49

encounter with all the names

31:50

but it encounters in sizes like tool

31:52

string equals some

31:54

custom implementations and so on

31:59

right so

32:03

here we check that all of these things

32:05

that we've been called is correct is

32:07

actually the method that we are

32:08

interested in

32:09

and later we create a descriptor

32:12

and add it to this result basically it

32:15

will collect

32:16

that coding expects from us

32:20

and creating the descriptor is actually

32:22

quite simple so

32:23

you just take the class that is is

32:26

inside

32:26

you add some annotations right now we

32:28

don't have any sheets mc

32:30

e and the name and he also specified the

32:32

synthesized so cotton won't be looking

32:34

for the psi

32:35

source of this uh serial

32:38

for this descriptor there

32:41

and installation basically continuation

32:43

of creation is just

32:45

uh kind of the specifics of how compile

32:48

applied uh

32:49

how compiler parses the functions

32:51

internally

32:52

so when we initialize an app we need to

32:54

specify what are the type and

32:56

usual parameters of the function return

32:59

type of that

32:60

and we're also saying that there is some

33:02

flux that you're interested in it's like

33:04

final

33:04

and public so we added this function and

33:08

compiling our knowledge we can call

33:10

it it will generate the correct links

33:12

link in white code or something

33:14

but we will fail with the abstract

33:17

syntax of after abstract method

33:18

or something it's basically saying that

33:21

there is no implementation

33:22

in actual by that's what we need to do

33:24

on the back end

33:26

and that will be using ir generation

33:28

extension

33:30

uh it will give us the motion fragment

33:33

so basically this top element

33:34

that we are in the then and later we can

33:36

show you how to hold three base on that

33:38

and we also have the environment of this

33:41

inside

33:42

in form of the plug-in context

33:45

so this is an implementation of the

33:48

actual generation i

33:50

created this machine part is the

33:52

lowering

33:53

so lowering as i said it passed through

33:55

the whole

33:57

whole like all of the compile

33:60

uh files being compiled right now that

34:04

we want to something with and right now

34:06

i'm interested in classes because that's

34:07

what i'm marked as serializable

34:09

so i'm using class lowering past and it

34:11

will be called on each of the

34:12

instances of classes that uh

34:16

that the plugin will encounter inside

34:18

the module fragment and for each of

34:21

these

34:22

uh for these three classes i just run

34:24

generate and later i check if i need to

34:25

generate something

34:28

so the first thing when you start

34:30

generating are actually taking it is

34:32

like

34:33

which what exactly doing it's right like

34:35

which comments exactly should be there

34:36

or what instructions

34:37

and for that usually when you're writing

34:39

bytecode you just go

34:41

and check what the resulting plus files

34:43

but with ir is a bit harder because you

34:46

like on a level

34:46

higher than that so here you have

34:50

basically some compiler arcs that allow

34:52

you to dump this internal state with ir

34:54

in the component

34:55

so you specify the directory where you

34:57

want all these dots to be

34:59

and the phases that you want to download

35:01

so the phase is basically each of those

35:02

optimizations

35:03

here i've chosen the first one i found

35:05

in a compiler for jvm

35:08

but you can just write all in all tabs

35:11

and you'll just enumerate them you know

35:13

and just will be

35:14

there like after all optimizations you

35:17

will see what the

35:18

ir how it's changed if mostly you're

35:20

interested only in the first one because

35:22

that's this

35:22

that you're supposed to be generating

35:26

um and these ir is really huge so

35:30

there's just one data class in final

35:31

annotation

35:32

but it's really a lot of things that

35:34

code engineering raised there and iris

35:36

was really verbose but it's much easier

35:38

to read

35:39

uh because it just feel like structure

35:43

and you can just find the identifiers

35:46

you are

35:46

interested in and later just go for that

35:49

through the

35:50

tree that is connected so here for

35:52

example i have this object

35:54

uh definition that is there and we have

35:57

the ir that is

35:58

uh kind of associated to that and now we

36:01

can just use the

36:04

use the builders inside the compiler to

36:05

go one by one

36:07

and just generate these things for

36:09

example here i'm using builder called

36:10

build class

36:11

and add in this class

36:14

inside the parent companion object that

36:16

they have

36:17

and the super type as well which also

36:20

mentioned

36:20

like mentioned in these super types

36:24

field inside the ir the same thing you

36:27

do

36:27

for the constructor you just have this

36:29

builder and you can

36:31

just basically write all the fields that

36:33

you see inside them

36:36

and for the body it's like two commas

36:39

that you have in there

36:40

and for that you can use the builders

36:42

for the budgets that they have inside

36:43

color again and it's basically the same

36:45

thing the interesting part here is this

36:47

dsl with the only plus

36:48

which basically allows you to add your

36:51

statement to a list of statements inside

36:52

the

36:53

blog block which is the body of the

36:55

function

36:57

all right i won't go through the whole

36:58

generation of ir for that particular

37:00

case

37:02

but you can find the complete example

37:03

before or also some

37:05

examples of how to check for errors and

37:08

some other things

37:10

for using these links i just like

37:12

repository which has this compiler

37:14

plugin with some readme which you can

37:16

check what it actually does

37:19

uh just to sum up what we did with the

37:22

plugin

37:23

is that probably most plugins which are

37:24

not like generating source code

37:26

as the case be that's what they do they

37:28

usually modify a little bit on an

37:30

analysis stage

37:31

they add some checks that their things

37:33

are correct and they also add the

37:35

uh some discrete or some metadata to say

37:37

that they added some new

37:39

classes or some methods or something

37:41

like that and

37:42

when they uh go to the

37:46

backend stage they just do like pure

37:49

code changes basically altering whatever

37:51

is there on the ir level

37:52

as you saw with these builders and this

37:54

builder is really helpful because that's

37:56

how they do all of these lowerings

37:57

inside the compiler so there is a really

37:59

huge functionality in there

38:00

and majority of those is more or less

38:03

stable so

38:04

it doesn't really break that much in

38:06

that layer right now

38:08

all right uh

38:12

that's basically it for me right now

38:15

so let's sum up first of all

38:18

i wanted to say that because i guess

38:20

that's really experimental apigee brains

38:22

doesn't give us any guarantees

38:23

and uh i wouldn't just put them directly

38:27

to the production

38:28

unless you are really you really kind of

38:32

are free to

38:33

uh take this new release of courtly and

38:35

make sure that everything works right a

38:37

lot of tests for that and so on

38:38

so it's like a little bit harder to do

38:41

it with the usual the more stable things

38:43

but at the same time you have a lot of

38:45

possibilities that this compiler open

38:47

like you could not really do annotations

38:50

and you have a multi-platform thing and

38:53

also the impact on the compilation speed

38:56

is much lower because you don't have to

38:58

make java tools to understand because

39:02

and also one thing is that many

39:04

compilers that have done like exactly

39:06

the way i described they actually

39:07

support

39:08

incremental compilation out of the box

39:10

which is also good

39:11

so here i want to say that compiler

39:13

plugins is pretty

39:14

new thing but they're for sure here to

39:17

stay because

39:17

many of the companies are invested in

39:20

also for example google with their

39:21

compose

39:22

uh we have jet brands with multitude of

39:24

applicants they have written from the

39:25

conception of

39:26

cotton straight away like personalized

39:28

and all of the android things that we

39:30

did

39:31

so this is something that i've been

39:33

looking into and

39:34

was really excited about i hope i give

39:37

you some understanding what it is

39:39

and maybe you're excited about this as

39:41

well

39:43

um and also before i finish

39:46

i want to mention aroma method because

39:48

you cannot really talk about compiler

39:49

plugins and communicative without

39:50

mentioning aromatics

39:52

and quite a big community that focusing

39:54

Global CSS

Follow us

Team droidcon

Get in touch with us

Quicklinks