Tech Showcases,
Developer Resources &
Partners
droidcon News
The Magic of Compiler Extensions
Speaker: Andrei Shikov
Source: droidcon Online 2020
Advanced Kotlin Series Webinar (Part 3 of 3)
Transcript
English
00:14
okay
00:15
welcome everybody to our droid con
00:16
online webinar this is the the final
00:18
version of our
00:19
of the final we have a three-part series
00:21
uh on advanced kotlin this is our final
00:23
uh webinar in the advanced kotlin series
00:26
and we're really excited
00:27
for our speaker today we're excited to
00:29
be with you for this final edition
00:31
uh and just wanted before we have andre
00:33
come on do his presentation wanted to do
00:35
the usual
00:36
a few housekeeping items with you uh so
00:39
i'm gonna get started because we've got
00:40
a great presentation we don't want to
00:41
take too much time away from that
00:44
as you know all of the droidcom online
00:47
webinars are available at droidcon.com
00:50
this webinar should be available
00:52
tomorrow uh androidcom.com it should be
00:55
indexed
00:55
so you can search searchable uh and you
00:58
should be able to find all those in case
00:60
you missed anything else in this series
01:01
or from our jetpack series or
01:04
or our multi-platform series or a
01:06
hands-on in-depth series
01:07
you should be able to find everything at
01:08
droidcom.com and we look forward to
01:11
seeing you over there
01:13
if you have not used a slido before
01:17
please note that uh slido is where we do
01:20
our q a
01:22
so if you would head over to slido.com
01:25
please put in this code also you can
01:27
check in the chat function in your
01:28
interface window
01:30
uh and only my colleague will be sending
01:31
that over as a direct link
01:33
so please use slido uh to to ask your
01:36
questions don't
01:37
use the the question function in the
01:39
interface here you slido
01:41
what that'll allow you to do is in case
01:43
you see a question that you wanted to
01:44
ask
01:45
that's already been asked you can vote
01:46
on it give it a thumbs up um
01:48
if you think it's a really important
01:49
question a really smart question vote on
01:51
it too
01:52
and that way when andre's doing his q a
01:54
all of the most important questions go
01:56
to the top
01:57
so we have time to answer them so make
01:59
sure you head over to slido.com right
02:00
now
02:01
open that up in your browser and get
02:03
ready to put your questions in there
02:05
just to give you a preview what's coming
02:06
up is you know we started
02:08
uh our our droid con online webinar
02:10
series in april
02:12
we've gone through as mentioned a few
02:14
different series on jetpack and
02:15
multi-platform
02:17
we're wrapping this up with with three
02:18
more on our hands-on and in-depth um
02:22
items here so we have on september 10th
02:25
that's something it's all about testing
02:27
ui testing other testing it's going to
02:28
be a great webinar
02:30
uh on september 17th we're going to be
02:32
looking at new form factors
02:34
particularly things like dual screen
02:35
form factors uh for microsoft duo that's
02:38
going to be a lot of fun
02:39
uh and then our final one on september
02:41
24th speeding up builds
02:43
building a devops pipeline so that's
02:45
what we've got coming up for you right
02:46
now
02:47
uh we have today uh andrei shiko if
02:50
you're an android engineer for magic lab
02:52
if you had registered and looked at the
02:54
program originally you had would have
02:56
expected maybe a second speaker
02:58
we apologize our second speaker had to
03:00
drop out last minute there were some
03:01
medical complications in his family
03:04
and so we wish he and his family well
03:06
but we're excited to have andre here and
03:07
also
03:08
it'll allow him in fact to do this in a
03:10
little more relaxed way
03:12
uh and take a little bit more time than
03:14
we normally would maybe we'll have some
03:15
more time for questions too
03:17
so we're really excited to welcome andre
03:18
here um
03:20
i'm going to change the moderator over
03:22
to him
03:23
uh and so we can let him be the um
03:26
presenter now and andre feel free to
03:28
turn on your screen
03:29
or turn on your webcam and we'll be able
03:31
to uh
03:32
to see you as well there we go right
03:36
yes so so take it away it's all yours
03:41
all right all right so i hope you can
03:43
see me well
03:44
and the presentation as well and you can
03:48
hear me well as well because it's just
03:49
like really hard to
03:50
check it here so i'm really actually
03:53
excited to be on droidcon online i've
03:54
been following this whole thing like the
03:56
whole webinar series
03:57
for this time and it's really exciting
03:60
to be a part of it
04:02
uh myself and today i'm gonna be
04:05
focusing on one of my
04:06
like areas that was more interesting to
04:09
me personally for the last year i was
04:11
focusing on it really a lot it is a
04:14
compiler and a particular coating
04:16
compiler
04:17
and its extension or compiler plugins
04:20
you might have heard
04:20
them about that before but it's really
04:23
interesting
04:24
area which you can actually go deeper
04:27
and check how coding
04:29
works and why there's certain things the
04:31
way it does
04:32
so let's start and the first thing about
04:35
compiler is this
04:36
actually tool that we uh
04:39
that we use to take our source code and
04:41
compile to some kind of
04:43
form of binary and we don't really care
04:45
about how it works that much we have
04:47
like our build system like gradle or
04:48
something like that depending on what
04:50
you use or maybe using command line
04:51
interface that's also fine
04:53
but we don't really need to go deeper in
04:55
that
04:56
and check what's inside this magic box
04:59
and today's talks is right about this
05:02
magic box i'm going to be talking
05:03
how it works what kind of things he does
05:05
inside of our source code and how it
05:07
produces binaries
05:09
and with that i will keep the focus on
05:12
actually extending it for
05:13
our needs and doing some kind of meta
05:16
programming congenital things
05:19
but first allow me to introduce myself a
05:21
little so as you already saw i'm like
05:23
android engineer i'm doing android for a
05:25
while and yeah that's my photo before
05:27
before i started doing compilers
05:29
um and that's where i met kotlin
05:32
while doing android and that was the
05:34
perspective that i came into this
05:36
uh kind of whole cooking compiler
05:39
area with cotton i have really long
05:43
relationship like i've been using it
05:44
since
05:45
before the release and to be honest i
05:49
never look back
05:50
and as i started going into compiler i
05:52
realized that cotton actually has an
05:54
open one so you can just go on the
05:55
github check the code and do whatever
05:57
you like
05:58
and that's where i went i learned a lot
06:01
of stuff
06:02
and today today's talk is mostly result
06:05
of this research in the code base
06:07
and made the amazing things alone there
06:08
i don't work for jetbrains just to
06:10
clarify
06:11
and i mostly like talk to them on uh
06:14
like slack and other channels but didn't
06:17
really
06:17
participate in the creation of the
06:19
compiler itself
06:22
so this talk will be in the three parts
06:25
first i will try to clear the line
06:27
between annotation rotation
06:28
and this it will be like java annotation
06:30
procession and compiler plugins from
06:32
code inside
06:33
and when this line is cleared when we
06:35
get this understanding we can go into
06:37
the plugins themselves
06:38
and we'll go that first theoretical
06:41
things
06:41
like basically look at how the compiler
06:43
works and we'll do it step by step
06:46
and later we'll go to create the plugins
06:50
like extend these compiler structures we
06:52
already have and we'll go through
06:54
more practical things like some code and
06:58
examples of apis that we have at this
06:60
particular moment of time
07:02
all right so what's the difference
07:04
between annotation procession and those
07:06
plugins
07:06
they are both kind of playing the
07:08
similar role in the ecosystem
07:10
the kind of meta programming tool which
07:12
allows you to do some kind of code
07:13
generation and some kind of things
07:15
and on the surface they're quite similar
07:18
so
07:19
when you start using that you start with
07:20
the source file so they both take
07:23
source files of each respective platform
07:25
obviously but jetbrains actually created
07:27
tools
07:27
so you can use for example gem
07:30
annotation persistent we've gotten
07:32
when you generate these steps or uh
07:35
quoting compiler actually can understand
07:37
the java files when it compiles
07:39
you can just link to link to them but at
07:42
the same time
07:43
you won't be using quotes in the java
07:44
ecosystem because like what
07:46
doesn't compile uh java files for that
07:48
you have java compiler
07:50
and similar way annotation system kind
07:52
of work with coding without those steps
07:56
the first big distinction comes of how
07:59
we approach input
07:60
with annotation processing here we
08:03
uh get source files and we emit
08:06
new source files and then we can repeat
08:08
this over and over again
08:11
uh with different processor maybe the
08:13
same one but it just like multi-step
08:14
process can be repeated
08:16
there are some uh exceptions like lone
08:18
poke or stuff like that but they usually
08:20
using some kind of internal api majority
08:22
of the processor i've seen they actually
08:24
created the new files
08:25
and compiler is on the totally opposite
08:28
side of the scale so it doesn't emit
08:30
like new source files that you will be
08:32
later feeding and compiling it actually
08:33
creates in a binary
08:34
and it's really focusing on doing that
08:36
efficiently so the
08:38
plugin here is not like something that
08:40
manipulates with like source files or
08:41
not
08:42
area it actually uh some kind of
08:45
transform
08:46
like the extension on inside the
08:49
compiler that you can use
08:50
to adjust the generated code so the
08:53
result we will see only there
08:56
there is a way to do something like
08:59
annotation processing dance in kotlin
09:00
and ksp or code and symbol processing
09:03
you may be heard of that is like tool
09:05
from
09:05
google which went with alpha like in
09:07
june so there is a link below you can
09:10
check it out if you want
09:11
so essentially it's replicating like
09:13
this single pass annotation processing
09:15
using internal
09:16
kind of plug-in api that is there or was
09:19
created originally for kpt i think
09:21
uh and it's kind of doing the same but
09:24
majority of the plugins actually just
09:26
focusing on transforming the
09:29
source to the binaries
09:33
and the last thing i wanted to draw on
09:35
this distinction is actually
09:36
documentation because when you go into
09:39
annotation processing work it will have
09:40
been
09:41
quite a long time around and you just
09:44
have a lot of different materials you
09:46
have tutorial stocks and
09:48
the api itself is really stable so you
09:52
won't really expect the uh you won't
09:54
really expect it to break
09:56
like your annotation processing to break
09:58
when you're changing versions of like
09:59
that
09:59
with cotton block is totally opposite
10:01
you kind of
10:03
have really experimental api which was
10:04
mostly created for internal needs of jet
10:06
brains
10:08
originally and right now they slowly go
10:10
in into different directions like some
10:11
more stable thing
10:12
so it's some specifics they are changing
10:15
almost every release right now
10:16
and documentation is essentially
10:18
non-existent
10:20
so thankfully the code is available for
10:21
you to research in the first place and
10:23
community is quite helpful if you're
10:26
stuck and if you want to try something
10:28
out you usually point
10:29
out where to start where to search for
10:33
one thing i wanted to note here while we
10:35
are on uh annotation process in the
10:37
plugin
10:38
area is that why we don't really like
10:42
kpt in coding
10:43
why we'll compile an ad so the uh
10:47
most problematic part here is generating
10:49
this type so basically take
10:50
all of the quotient source source files
10:53
then you take all declaration compiler
10:55
to java so you create new java source
10:57
files and this process is quite slow
10:59
and it obviously depends on the module
11:01
size but usually takes like 20 30
11:03
percent
11:03
of the compilation itself and it cannot
11:07
be skipped
11:07
because as i said java annotation
11:09
processing cannot really understand what
11:10
happens inside the button so you need to
11:12
convert these files forward
11:13
with plugins you essentially go to
11:15
coding files and just compile
11:17
yes you may have some limited uh limited
11:20
kind of understanding of the java files
11:22
as well but if you focus in on coding
11:24
here in the first place like if you have
11:25
gotten
11:26
database or code base so it's
11:29
uh much faster to do so and just work
11:32
with the plugins directly
11:37
and plugins actually allows us to do
11:38
even more things so you can focus on a
11:40
multi-platform create your plugin which
11:42
uh does some kind of meta programming or
11:44
code generation on all the platforms
11:46
that codes and supports
11:47
you can change the code that user writes
11:49
yes you can do this with some kind of
11:52
part of the uh java coupon or java
11:54
annotation processing part but
11:56
it's still much better formulated here
11:59
inside the
12:00
uh kotlin capillary plugins and you have
12:03
way more possibilities of actually
12:05
detecting the things that you want to
12:06
change so you don't have to be focused
12:07
on rotations only
12:08
you can just go around change some
12:10
function calls or probably you have some
12:12
interesting class that extends some
12:13
marker interface you want to add some
12:15
things to that or generate some
12:17
uh some basic uh implementations of the
12:20
methods for interfacing for that you can
12:22
do it and you don't need any kind of
12:23
annotation on
12:24
any additional code to do so
12:29
but obviously there are some drawbacks
12:31
in here and first of all
12:32
is not a big on my from my point of view
12:35
but
12:36
still is important for many it's like
12:37
multiple passes so intelligent
12:39
protection you can go around and apply
12:41
different processors and
12:42
try to uh together as a multi-step
12:45
approach
12:46
with coding compiler you cannot really
12:47
do so and
12:49
to be honest compiler plugins frequently
12:52
even don't work with each other
12:54
i'm not sure if in serialization for
12:55
example right now work with compose well
12:57
and so on
12:58
and the second thing here is that you
13:01
don't really see the generated code that
13:02
much
13:03
you can check the by bytecode that is
13:05
emitted by jvm
13:06
for example you can dump some internals
13:08
of the compiler
13:10
but from user perspective it's much
13:12
easier to just go to generated java
13:14
files
13:14
like like you do with annotation
13:16
procession and check whatever whatever
13:18
generated there
13:19
you can do you cannot do this with the
13:20
compiler because they just focus on
13:22
emitting the final binary
13:24
so we have to leave that
13:27
all right so i hope we kind of
13:32
give you some understanding of what is
13:33
annotation procession what the plugins
13:34
where the line is landscape of the meta
13:36
programming that we have
13:40
and now we're moving completely word of
13:42
the compile
13:43
and as i said uh this part to be
13:46
like more theoretical we will go through
13:49
the parts of compiler how they work
13:51
together and what the structures they
13:53
pray so later we can utilize this
13:55
knowledge of actually using apis
13:57
that export the compound yeah
14:00
i establish already that there is uh the
14:03
source
14:04
files that getting transformed into the
14:07
binaries and the binders can do
14:09
different platforms because we have
14:11
kotlin
14:12
if look at this a bit deeper
14:16
we can see that this process kind of
14:19
happens in two big steps
14:20
first of them is called front-end and
14:22
second one is called back-end
14:24
so the front-end is where compiler tries
14:26
to understand what the hell have you
14:28
written in the source code that are
14:29
trying to fit in there
14:31
and here it validates errors both
14:34
syntaxical
14:35
and logical ones and this understanding
14:38
later packed into this structure the
14:40
structure is really
14:41
useful for water in the back end
14:44
what we do it with that data in the back
14:46
end we actually generate it for the
14:48
target platform
14:49
so that's where we already have complete
14:51
understanding what it happens inside
14:53
of the code that you provided to the
14:55
compiler and we can generate the
14:58
binaries or whatever the uh
15:01
targets target platform sources are
15:04
here before generating backend also goes
15:07
through the
15:08
optimization steps like for example in
15:10
line-in or
15:11
it could be some plasma specific stuff
15:14
like interrupt
15:14
the intel interrupt for jvm
15:18
for getter setters for properties or
15:19
something like that
15:21
so this is basically the front end we're
15:23
trying to understand what happens and
15:25
the back end we use an asset design to
15:27
generate
15:27
the code for the target classroom let's
15:29
go one layer deeper
15:31
and we'll start with front-end and trace
15:35
uh trace this snippet through all of the
15:38
structures that it gets converted into
15:40
so we'll understand how it goes
15:42
what steps are there on the way
15:45
and the first step is to convert this
15:47
code snippet to psi
15:49
actually it's based on this kind of
15:51
multi-step process but it's the first
15:52
one that we
15:53
exposed as a compiler plug-in writer so
15:55
we'll step in there
15:57
so psi is the program structure
15:59
interface
16:00
and is really close to its function on
16:02
an abstract syntax tree if we
16:04
ever if you know what that is and
16:08
while parsing the code like parsing
16:09
these three compiler already
16:11
checked for all the syntax errors it
16:13
knows that parentheses is all in place
16:15
there are no missing commas and so on
16:18
and
16:18
this three that it gives us it also is
16:20
immutable
16:21
so it's directly tied to the source code
16:24
that we've written
16:25
so we cannot really change it that much
16:29
uh this tree is also starts with the
16:31
file so basically we have this
16:32
collection of these trees for each of
16:34
the files that we
16:35
collect there and it really helps us to
16:38
analyze effectively
16:40
for example if we interested what is the
16:42
class that continuously selected
16:44
function
16:44
we just walk up the tree we find node of
16:46
the type class
16:48
and here we go or if we're interested in
16:51
a
16:51
like the connection other way like we
16:54
want to check what are the functions
16:56
that are inside the
16:57
class that we are purchasing we can do
16:59
the same by just going down the tree
17:02
in general psi is used by jbrains in
17:04
many places if most of the it is
17:07
and they have quite good to me quite
17:09
good documentation on that but not
17:11
unlike coding part of the website but
17:13
unlike
17:13
eddy part of the website so you can
17:16
check it out if you want more answers
17:17
with your house and
17:19
wise
17:22
so right now we have this uh tree parsed
17:26
but we still don't have the complete
17:28
information about what happens
17:30
inside these source files to demonstrate
17:32
that i added a simple call
17:34
so basically right now we have println
17:36
function that we call
17:38
inside of the example function and
17:41
this printer line is not really in the
17:42
current file and it's not really
17:45
even any current like uh set of files
17:48
that we are compiling it's in totally
17:50
different artifact
17:52
but we still need to validate the
17:53
parameters we need to check that we use
17:55
string that we pass is actually correct
17:57
and probably return type we're also
17:59
interested in that and
18:00
later we have to generate a bytecode for
18:03
that so basically
18:04
forwarding to correct instructions in
18:05
the bytecode so we all need to
18:07
link this information to the description
18:09
of that function
18:10
and for that uh coding compiler has this
18:14
structure name descriptors these cuters
18:17
can be seen as a metadata that is
18:18
associated with this psi
18:21
uh and it describes more information
18:23
about the class for example
18:25
uh you can check what is the
18:26
constructors of the class what is the
18:28
type
18:29
uh what are the type parameters that it
18:32
has
18:32
and function could be like parameter
18:34
infor and like name and obviously all
18:36
these things
18:37
so with the descriptors they can also
18:40
exist
18:40
outside of psi for example every
18:42
artifact is also part into these
18:44
descriptors
18:45
that are organized in the tree as well
18:48
ns3
18:49
also has two other elements that we
18:50
don't see in psi directly
18:52
first of them is like module the
18:54
compilation unit
18:55
uh basically all of the elements that
18:59
have during current compilation like our
19:01
current files that we compiling
19:03
some dependencies that we have all of
19:04
the classes functions can be available
19:06
from that module
19:07
and they are separated with different
19:09
package fragments usually like for
19:10
different arts in fact you have package
19:12
fragments something like that
19:13
and they are using like fully qualified
19:15
name the one that you usually write in
19:16
the import
19:17
you can actually get access to the class
19:20
like description
19:20
of the class of the function or whatever
19:23
is declaration you are looking for
19:26
so let's say that we checked the imports
19:27
and we found our println function
19:29
and now we wrap it into resolved call
19:32
which keeps the context of all of the
19:34
things that we're calling here so for
19:35
example let's say that uh
19:37
right now uh here we have the function
19:39
descriptor for println
19:40
and hello as a parameter so we just
19:42
remember that there is a constant as a
19:44
parameter here in
19:45
hello we can also kind of capture some
19:49
elements of the outer context for
19:51
example we inside the class and we want
19:52
to use some field inside this println
19:54
uh call we can also do that you it also
19:57
captured using this result called
19:58
mechanism
19:60
so these are two elements uh which
20:02
extensively use an
20:03
analysis and if you're writing compiler
20:05
plugins you probably
20:07
will try to manipulate and somehow
20:09
utilize it to
20:10
acquire some knowledge about the code
20:12
that you actually combine right now
20:15
and uh this connection between psi
20:18
and all of these elements is starting
20:21
inside the building context
20:22
you can look at it as like a hash map
20:25
which contains a different connection
20:28
between different comments
20:29
you can add your own connections there
20:32
for example if you're interested in some
20:34
metadata
20:35
specific for your plugin you can add it
20:37
there and
20:38
you can sometimes alter the existing
20:41
ones
20:42
but it's probably not the best way to
20:44
approach them
20:48
so we right now parsed all this data
20:50
into psi into descriptors and into these
20:53
result codes
20:54
and we can move into generating part
20:58
plug into the back end
20:59
and the back end is quite interesting
21:00
here because you have to support all the
21:02
different
21:03
platforms
21:09
and because of that because we have to
21:10
support these platforms
21:12
uh we have a back-end separated we at
21:15
least we had a security for most power
21:17
but front it was shared between
21:20
different platforms
21:22
so it was like that before the new
21:24
back-end came in so probably you heard
21:26
about this ir
21:27
or intermediate representation back and
21:29
then we have and right now it looks
21:31
something like that so it will be coming
21:33
somewhere in like one
21:34
for something unstable you can enable it
21:36
for a long time all right
21:38
and the interesting part here is that
21:40
despite behind all of these different
21:42
platforms with their custom tools for
21:44
generation so for jvm you for example
21:46
have object web asm
21:47
for native series lvm compilation that
21:49
takes place and for javascript there if
21:51
i remember there are some custom tools
21:53
for that
21:53
so right now with this intermediate
21:56
representation you can define some
21:58
common transforms and for us as a
21:60
compiler plugin writers
22:01
it's particularly important because we
22:04
can't
22:04
write our plugin like code generation
22:06
for a plugin only once
22:08
and apply it to all the platforms which
22:10
is quite cool for my opinion
22:11
um the api by itself is also improved a
22:15
lot with
22:16
when ir came so right now it's much
22:18
easier and much more accessible from my
22:20
point of view
22:22
and right now majority of the new
22:24
plugins that
22:26
are kind of support kind of are coming
22:28
right now on the scene they
22:29
support all this new backend as a
22:32
default so for example compose
22:33
it works by default on ir back and
22:36
that's why it's
22:38
kind of yeah
22:41
uh the exceptions from that probably the
22:43
first party from jetbrains
22:45
you have like you need to support all
22:47
the backhands in there and
22:48
if i remember correctly and wheel from
22:50
square also is written using old
22:52
backhand
22:52
because they want to use tool right now
22:54
before the ir is right
22:57
so ir by itself is also a tree
22:60
and it contains all the information that
23:02
you had in psi
23:04
and descriptors together so basically
23:06
this complete information about each of
23:08
the nodes
23:09
it's much less variables on psi so you
23:11
don't have
23:12
different nodes for identifiers and so
23:14
on entry point here is
23:16
module as we saw with the descriptors
23:17
but right now it contains
23:19
all of the modules that we compile right
23:22
now so basically files that we compile
23:23
right now
23:24
and the important part of this tree is
23:27
that right now it's mutable so you can
23:28
just go there
23:29
change whatever function you want the
23:31
body of the calls whatever you're
23:32
interested in and this is
23:37
extensively used in many plugins for
23:39
example compose modifies majority of the
23:41
calls with the composable functions and
23:42
so on
23:45
uh the extension that we have for
23:49
like adding new stuff to ir is
23:52
applied right after it is created so we
23:55
can focus on generated code
23:57
as close to original sources other
23:59
original source files as it could be
24:01
and later it goes through the process of
24:03
lowering or optimization so lowering is
24:05
basically this
24:06
as i understand the places kind of this
24:08
path for all of the
24:09
files that does one particular thing so
24:12
for example
24:13
there was uh this tweet from jay quarter
24:16
some time ago
24:16
said lowering for multi-line strings
24:19
which
24:20
uh does stream marching on the compile
24:21
time or
24:23
there is lower in the for example right
24:25
function perhaps function
24:28
uh with on jvm with the classes on top
24:30
so something like that
24:31
and they usually go for all of the files
24:34
and do only one particular thing and
24:36
later
24:36
this optimized code is uh generated to
24:39
the target ladder so like bytecode
24:41
what's already interesting
24:44
all right so this is for the compiler
24:46
first we have a front end
24:48
trying to understand what happens inside
24:50
the
24:51
source files and validate that this code
24:52
is actually correct and later we go to
24:55
the back end
24:55
and here we are focusing on optimizing
24:58
that
24:59
uh structures that we had and generating
25:02
the resulting
25:03
code for the target platform that we
25:04
have compiled right now
25:06
i hope it gives you at least some
25:08
understanding about it happens
25:09
or what is happening inside so let's
25:12
right now try to add something to it
25:16
and that's where we go into actual
25:17
creatine plugins
25:22
i will be jumping here through the code
25:25
and try to run the through process of
25:27
creating a plugin
25:28
with different pieces that can change
25:30
there
25:31
and the first note i wanted to mention
25:33
is that it's actually really
25:35
experimental all api that i mentioned
25:37
right now
25:38
so as i said you frequently have your
25:41
plugin break after like version of the
25:43
eighth of the coding
25:44
with minor one and i'm pretty sure it
25:47
will gonna be changed in the future
25:48
because
25:49
right now many kind of big plugins come
25:51
to release the compose base uh
25:53
by itself is worth money so it just
25:57
uh right now it is this experimental api
25:59
that we need to be aware of
26:00
but many things at least on conceptual
26:02
level
26:03
uh will stay the same basically now is
26:06
based
26:06
on the price we pay to get in compiler
26:09
plugin development basic early
26:12
all right so what is the compiler plugin
26:14
in the first place it is basically a jar
26:16
that you compile using the dependency of
26:18
the compiler
26:19
and later you can add this chart using
26:22
the
26:23
command line parameter x plugin so here
26:25
i did it in gradle
26:27
but in gradle you actually can do much
26:28
better you have the
26:31
transformation you can use like myelin
26:33
artifacts or mymac originates
26:35
or you can use like any kind of
26:36
artifacts that you got used to in new
26:38
ground
26:39
so you can notice here that there is
26:41
different
26:42
clasp for native and it's just because
26:44
how
26:46
shadow jar is set up for jvmgs compiler
26:49
and native one there is nothing beacon
26:51
there you can just mimic it
26:53
by yourself but it's something to be
26:54
aware of
26:57
so we connected our plugin and the first
26:60
thing that gets connected is a company's
27:02
register
27:03
as i said it's connected through the
27:04
service loader and as with anything
27:06
that uses service loader in a jvm we can
27:09
use auto server
27:11
annotation processor to connect that and
27:14
when compiler gets our component
27:15
register
27:16
it calls this method uh to register the
27:19
components
27:20
and the components or extensions that
27:23
are available there
27:24
they look like this so you have the
27:25
interface some extension slide
27:27
and it's companion object has a method
27:31
to register
27:32
like instance of it and there are many
27:34
extensions like that inside the compiler
27:36
so you for example have
27:38
uh one that allows you to modify iron
27:42
some particular phase
27:43
or you have one that allows you to check
27:45
for errors or the one
27:46
that allows you to kind of other
27:49
separate descriptors like custom things
27:51
to so compile will be aware of
27:53
during analysis on the front end so
27:57
basically each compiler appliance
27:59
usually multi implements multitude of
28:01
those checks
28:02
and their combined provide the whole
28:04
functionality of the plugin
28:07
just an example that's how you connect
28:09
that you have your extension
28:11
you have the method to register it and
28:13
your instance and that's
28:15
basically extension registered
28:18
that's why if you're looking through the
28:20
some compiler plugin implementation that
28:22
you're interested in
28:23
you can go to company registrar and see
28:25
all of the
28:27
extensions that are connected right now
28:29
and this can give you at least some
28:31
overview what happened it
28:32
what happens inside that because you may
28:34
be aware that this extension that this
28:35
on this particular face
28:36
or some another and later you can go
28:39
like face by face and trying to
28:40
understand
28:41
what the compiler plugin does in each of
28:43
those
28:46
and as an example of writing the plugin
28:48
i will
28:49
look at two really small subsets of the
28:52
quote and actualization
28:54
uh code link serialization by itself is
28:56
a really good example of the plugin that
28:58
you made right
28:59
i totally recommend you go there check
29:01
it out and see
29:02
how it works inside like the one that
29:04
you actually could use in production
29:06
but here i will be taking like small
29:07
subsidy for those who do know
29:10
what that is it's basically the magic
29:12
annotation that you cannot take your
29:13
classes with
29:14
and laser regenerates some bulletproof
29:17
so you can
29:18
integrate it with a serialization
29:20
framework
29:21
library that it has what exactly does it
29:24
generate
29:25
basically two parts first is this custom
29:26
serializer that explains how you will be
29:29
serializing code it has better to
29:30
serialize deserialize and this
29:32
child serializer should actually explain
29:34
uh how the structure will be looking
29:36
and uh this
29:39
custom serializer is available through
29:42
the method
29:43
in a companion object that you can
29:46
basically use
29:47
to specify to the library how you want
29:50
the object to be serialized
29:52
so that will be focusing today so it
29:53
will be like a method and then some
29:55
implementation of the serial license
29:58
and we'll be doing that in two parts so
29:60
first of all first of them
30:01
is to add serialize because uh during
30:04
analysis because compiler doesn't really
30:06
know that we have this new method so if
30:07
you look at that
30:08
in the companion object we have a new
30:10
method we can call outside of this class
30:12
and compile is not really aware that
30:14
this method exists right now so we need
30:15
to tell it like an analysis phase there
30:17
is method that exists
30:19
and later we will actually generate
30:20
implementation of the serializer
30:22
and the functions let's start with the
30:26
first one
30:27
so here we'll be adding serialization
30:29
serializer descriptor
30:30
during analysis stage and to do that we
30:33
will use the
30:35
extension that is called synthetic
30:36
resolve extension it allows you to add
30:38
properties
30:39
methods and classes i think that's it uh
30:43
two different like different classes
30:45
that you have inside content
30:48
and you can do you do it in like two
30:50
stages
30:51
first you specify that there is this new
30:52
identifier that you are interested in so
30:55
basically saying that there is new
30:56
identify serializer
30:57
inside this companion object of the
30:59
serializable class and later you specify
31:01
the metadata associated with this
31:03
identifier
31:05
so let's go through adding actually that
31:07
method yeah that's the method
31:10
so first we specify the identifier and
31:13
here we say
31:14
okay if this companion object is
31:17
actually serializable
31:19
oh if this component is actually inside
31:22
a serializable class
31:23
which we check using the annotation we
31:26
will add like return this new identifier
31:29
that we are interested in
31:30
otherwise we just add anything
31:34
then this name like then we get called
31:37
with another method
31:38
and here it's very important to check
31:41
that
31:42
we are called with is the one that we
31:44
are
31:45
ourselves because compiler will be just
31:47
calling this with all the classrooms you
31:49
encounter with all the names
31:50
but it encounters in sizes like tool
31:52
string equals some
31:54
custom implementations and so on
31:59
right so
32:03
here we check that all of these things
32:05
that we've been called is correct is
32:07
actually the method that we are
32:08
interested in
32:09
and later we create a descriptor
32:12
and add it to this result basically it
32:15
will collect
32:16
that coding expects from us
32:20
and creating the descriptor is actually
32:22
quite simple so
32:23
you just take the class that is is
32:26
inside
32:26
you add some annotations right now we
32:28
don't have any sheets mc
32:30
e and the name and he also specified the
32:32
synthesized so cotton won't be looking
32:34
for the psi
32:35
source of this uh serial
32:38
for this descriptor there
32:41
and installation basically continuation
32:43
of creation is just
32:45
uh kind of the specifics of how compile
32:48
applied uh
32:49
how compiler parses the functions
32:51
internally
32:52
so when we initialize an app we need to
32:54
specify what are the type and
32:56
usual parameters of the function return
32:59
type of that
32:60
and we're also saying that there is some
33:02
flux that you're interested in it's like
33:04
final
33:04
and public so we added this function and
33:08
compiling our knowledge we can call
33:10
it it will generate the correct links
33:12
link in white code or something
33:14
but we will fail with the abstract
33:17
syntax of after abstract method
33:18
or something it's basically saying that
33:21
there is no implementation
33:22
in actual by that's what we need to do
33:24
on the back end
33:26
and that will be using ir generation
33:28
extension
33:30
uh it will give us the motion fragment
33:33
so basically this top element
33:34
that we are in the then and later we can
33:36
show you how to hold three base on that
33:38
and we also have the environment of this
33:41
inside
33:42
in form of the plug-in context
33:45
so this is an implementation of the
33:48
actual generation i
33:50
created this machine part is the
33:52
lowering
33:53
so lowering as i said it passed through
33:55
the whole
33:57
whole like all of the compile
33:60
uh files being compiled right now that
34:04
we want to something with and right now
34:06
i'm interested in classes because that's
34:07
what i'm marked as serializable
34:09
so i'm using class lowering past and it
34:11
will be called on each of the
34:12
instances of classes that uh
34:16
that the plugin will encounter inside
34:18
the module fragment and for each of
34:21
these
34:22
uh for these three classes i just run
34:24
generate and later i check if i need to
34:25
generate something
34:28
so the first thing when you start
34:30
generating are actually taking it is
34:32
like
34:33
which what exactly doing it's right like
34:35
which comments exactly should be there
34:36
or what instructions
34:37
and for that usually when you're writing
34:39
bytecode you just go
34:41
and check what the resulting plus files
34:43
but with ir is a bit harder because you
34:46
like on a level
34:46
higher than that so here you have
34:50
basically some compiler arcs that allow
34:52
you to dump this internal state with ir
34:54
in the component
34:55
so you specify the directory where you
34:57
want all these dots to be
34:59
and the phases that you want to download
35:01
so the phase is basically each of those
35:02
optimizations
35:03
here i've chosen the first one i found
35:05
in a compiler for jvm
35:08
but you can just write all in all tabs
35:11
and you'll just enumerate them you know
35:13
and just will be
35:14
there like after all optimizations you
35:17
will see what the
35:18
ir how it's changed if mostly you're
35:20
interested only in the first one because
35:22
that's this
35:22
that you're supposed to be generating
35:26
um and these ir is really huge so
35:30
there's just one data class in final
35:31
annotation
35:32
but it's really a lot of things that
35:34
code engineering raised there and iris
35:36
was really verbose but it's much easier
35:38
to read
35:39
uh because it just feel like structure
35:43
and you can just find the identifiers
35:46
you are
35:46
interested in and later just go for that
35:49
through the
35:50
tree that is connected so here for
35:52
example i have this object
35:54
uh definition that is there and we have
35:57
the ir that is
35:58
uh kind of associated to that and now we
36:01
can just use the
36:04
use the builders inside the compiler to
36:05
go one by one
36:07
and just generate these things for
36:09
example here i'm using builder called
36:10
build class
36:11
and add in this class
36:14
inside the parent companion object that
36:16
they have
36:17
and the super type as well which also
36:20
mentioned
36:20
like mentioned in these super types
36:24
field inside the ir the same thing you
36:27
do
36:27
for the constructor you just have this
36:29
builder and you can
36:31
just basically write all the fields that
36:33
you see inside them
36:36
and for the body it's like two commas
36:39
that you have in there
36:40
and for that you can use the builders
36:42
for the budgets that they have inside
36:43
color again and it's basically the same
36:45
thing the interesting part here is this
36:47
dsl with the only plus
36:48
which basically allows you to add your
36:51
statement to a list of statements inside
36:52
the
36:53
blog block which is the body of the
36:55
function
36:57
all right i won't go through the whole
36:58
generation of ir for that particular
37:00
case
37:02
but you can find the complete example
37:03
before or also some
37:05
examples of how to check for errors and
37:08
some other things
37:10
for using these links i just like
37:12
repository which has this compiler
37:14
plugin with some readme which you can
37:16
check what it actually does
37:19
uh just to sum up what we did with the
37:22
plugin
37:23
is that probably most plugins which are
37:24
not like generating source code
37:26
as the case be that's what they do they
37:28
usually modify a little bit on an
37:30
analysis stage
37:31
they add some checks that their things
37:33
are correct and they also add the
37:35
uh some discrete or some metadata to say
37:37
that they added some new
37:39
classes or some methods or something
37:41
like that and
37:42
when they uh go to the
37:46
backend stage they just do like pure
37:49
code changes basically altering whatever
37:51
is there on the ir level
37:52
as you saw with these builders and this
37:54
builder is really helpful because that's
37:56
how they do all of these lowerings
37:57
inside the compiler so there is a really
37:59
huge functionality in there
38:00
and majority of those is more or less
38:03
stable so
38:04
it doesn't really break that much in
38:06
that layer right now
38:08
all right uh
38:12
that's basically it for me right now
38:15
so let's sum up first of all
38:18
i wanted to say that because i guess
38:20
that's really experimental apigee brains
38:22
doesn't give us any guarantees
38:23
and uh i wouldn't just put them directly
38:27
to the production
38:28
unless you are really you really kind of
38:32
are free to
38:33
uh take this new release of courtly and
38:35
make sure that everything works right a
38:37
lot of tests for that and so on
38:38
so it's like a little bit harder to do
38:41
it with the usual the more stable things
38:43
but at the same time you have a lot of
38:45
possibilities that this compiler open
38:47
like you could not really do annotations
38:50
and you have a multi-platform thing and
38:53
also the impact on the compilation speed
38:56
is much lower because you don't have to
38:58
make java tools to understand because
39:02
and also one thing is that many
39:04
compilers that have done like exactly
39:06
the way i described they actually
39:07
support
39:08
incremental compilation out of the box
39:10
which is also good
39:11
so here i want to say that compiler
39:13
plugins is pretty
39:14
new thing but they're for sure here to
39:17
stay because
39:17
many of the companies are invested in
39:20
also for example google with their
39:21
compose
39:22
uh we have jet brands with multitude of
39:24
applicants they have written from the
39:25
conception of
39:26
cotton straight away like personalized
39:28
and all of the android things that we
39:30
did
39:31
so this is something that i've been
39:33
looking into and
39:34
was really excited about i hope i give
39:37
you some understanding what it is
39:39
and maybe you're excited about this as
39:41
well
39:43
um and also before i finish
39:46
i want to mention aroma method because
39:48
you cannot really talk about compiler
39:49
plugins and communicative without
39:50
mentioning aromatics
39:52
and quite a big community that focusing
39:54
more on the functional thing so like
39:56
we're doing a lot of interesting
39:57
stuff on uh type level extensions and
40:00
content
40:02
but more importantly they're writing a
40:03
framework for
40:05
us like developers of those plugins
40:09
and uh those
40:12
though this framework allows you to have
40:14
like more or less competitive experience
40:16
with the idea and like compile a part of
40:20
that so basically
40:21
joining those together it's more or less
40:23
in the alpha stage but the
40:25
uh opportunities that it gives to you
40:26
now is already quite huge so i recommend
40:28
to check out and there is
40:30
a hash from the question like slack if
40:32
you're interested to join the channel
40:33
ask them how that works and what other
40:35
things
40:37
and yeah thank you for listening i hope
40:39
you enjoyed this
40:41
you can find me on twitter or telegram
40:42
by using these handles on top
40:44
you can find this example with the
40:46
plugin that
40:47
went through in the bottom and if you're
40:50
interested in compiler things so you
40:52
started writing your own you beat stack
40:54
you can go to the
40:55
link channel compiler and ask questions
40:58
there are there are people from
40:58
jetbrains they
40:59
really kind of recently started to
41:01
finally answer the questions about the
41:03
plugins
41:04
and sometimes something as well so
41:05
probably even i can help you if
41:07
something
41:07
goes wrong so right that's it for me
41:10
thank you very much
41:14
well honor thank you again so much for
41:16
for doing this i know
41:18
when you and i talked yesterday we
41:20
talked a little about some
41:21
you know you've given this presentation
41:23
before in russian
41:24
uh and get up and then internally and um
41:27
it's always whether there's questions
41:30
getting astronauts always depends
41:31
because it's kind of a
41:33
a new and esoteric topic so right now
41:36
i'm seeing in slido that we don't have
41:38
any
41:38
questions currently about the
41:40
presentation but i did want to give
41:42
some people a second to think about it
41:44
um but i appreciate the fact that you've
41:46
also given your contact information
41:49
if anybody does have questions uh after
41:51
they've watched the video
41:53
or whatnot they can reach you on
41:55
telegram or on
41:56
on twitter to ask questions and i
41:58
believe um uli will also be sending out
42:01
we have a copy of the of the slides
42:03
correct
42:05
oh here you know what we do have one
42:08
question i'll have really um show this
42:11
one up on the screen then
42:13
it looks like we did finally get a
42:14
question really earlier
42:18
that's actually quite cool
42:22
okay so all right yeah
42:26
let me just move it over
42:30
right so yeah that's probably the
42:33
hardest question that i could answer
42:35
uh so what's the best way of um
42:39
get familiar with all those extensions
42:41
of the
42:42
plugins of the compiler um right now
42:45
there is
42:46
not really much so from me the best
42:48
experience was to go through the
42:50
actual plugins that other people written
42:53
because they give you clarification
42:54
experience they have a goal that you
42:55
want to achieve
42:56
and have the things how they did it for
42:58
example cod linux serialization was like
43:02
one of more like harder ones to maybe
43:04
procedure but at the same time it was
43:07
really interesting to go through that
43:08
like elements of how they add the method
43:11
here
43:11
or how they modify the
43:14
coding like what do they do in the code
43:16
gene station so on
43:17
uh there are some simple examples out of
43:20
there and unfortunately i don't remember
43:21
right now there was
43:23
uh one from it was one uh doing some
43:26
kind of
43:26
assertion like smart asserts it was
43:28
really small just doing some kind of i
43:30
really like back and forth called gene
43:32
things
43:32
and i probably that's the nice way to
43:34
start so i recommend you to like search
43:37
for
43:37
i think it's also like power resort or
43:39
something like that if i
43:41
remember correctly so maybe you can
43:43
check those
43:44
and start from there right now because
43:46
documentation is really lacking
43:48
um one of the other things maybe i'm not
43:51
sure if
43:52
people from aerometa actually given any
43:55
uh given any like books about a
43:58
particular framework that would be like
43:60
well they actually did that mostly
44:01
but you're gonna go there and ask how
44:04
these things works as well but it's
44:05
probably like
44:06
usually you go to this like channel to
44:08
ask if you don't have something working
44:10
right it's just
44:11
really hard to ask people to explain to
44:13
you what is there
44:23
well all right i think that you you
44:25
answer the hardest question
44:27
uh the one and only so anyway
44:30
well good very good well we're really
44:32
again excited that we had you with us to
44:34
wrap up our
44:34
our advanced kotlin series he certainly
44:37
did a great job and we really appreciate
44:38
you being here
44:39
and talking to us about a topic that i
44:41
think not a lot of people have a lot of
44:42
experience with
44:43
but it definitely is important opens uh
44:45
i was looking into it after you and i
44:47
talked yesterday definitely
44:49
using the compilers extensions opens a
44:51
lot of doors to extending functions and
44:53
things i think it's really
44:54
a useful tool so i'm excited that you
44:56
were able to teach us about that
44:58
uh as it anyway this video will be
44:60
available online at droidcom.com
45:02
tomorrow
45:02
feel free to ping andre on telegram or
45:05
in or
45:06
twitter for any questions and again we
45:08
uh andre thank you and we look forward
45:10
to seeing everybody
45:11
soon uh for the last three of our online
45:13
uh joycon webinar series
45:15
and we look forward to seeing you all at
45:16
droid conor of droidcon america's an
45:18
apac
45:19
coming up soon that's it we're signing
45:22
off oh
45:23
we did one final question sneak in go
45:25
ahead
45:27
right so others are available only on
45:29
intelligence so this is interesting part
45:31
of it goes like
45:32
uh if you're talking about compiler
45:34
plugins they're available on the
45:36
compile itself so it's like coding see
45:38
things like just runs through
45:39
item yeah so psi is baked into the
45:42
compiler
45:43
itself so it's not really
45:47
something limited to only of the
45:48
intelligent platform
45:50
uh i think the uh why they chosen phi in
45:53
the first place
45:54
is because they like had the in mind
45:58
this integration of the tooling and
46:00
you have to agree like in tooling for
46:02
kotlin it's truly amazing
46:04
so a big part of that is how big the
46:07
plate is how the compile is structured
46:09
on the front and where it understands
46:10
the code because they shared a lot of
46:12
common parts
46:13
with the idea and that's why it repeats
46:15
i chose
46:16
but you know it's totally it's totally
46:19
unbounded from the utility you can use
46:21
quotient without intelligent
46:22
and probably some people do for sure
46:30
all right okay
46:33
i think we're good we're going to sign
46:35
off but again thanks for being here we
46:36
look forward to seeing
46:37
you in another event take care everyone
46:39
have a good day
46:41
bye
droidcon News
Tech Showcases, Developer Resources & Partners
EmployerBrandingHeader
jobs.droidcon.com
![]() Latest Android Jobs
Kotlin Weekly
![]() Your weekly dose of Kotlin
ProAndroidDev
![]() Android Tech Blogs, Case Studies and Step-by-Step Coding
Zalando
![]() Meet one of Berlin's top employers
Academy for App Success
![]() Google Play resources tailored for the global droidcon community |
Droidcon is a registered trademark of Mobile Seasons GmbH Copyright © 2020. All rights reserved.