hello everyone welcome to our panel on uh gradle build and testing at scale um i'm your moderator zach uh i guess is the part where i do i pick people to tell them to introduce themselves or john who are you this is the first question of the panel hi uh i'm john rodriguez uh i'm a software engineer working at square on the cash app product and yeah i dabble in the gradle things a lot for android nelson my name's nelson i'm a solutions engineer at gradle so that means i help customers optimize their build through trials as well as our big customers using gradle enterprise so i see a lot of different gradle builds and maven builds

cookie hello my name is chu ki chan i am an android developer and i don't touch much of gradle but uh testing is kind of an area that i'm interested in well yeah we we didn't get very many testing questions on twitter so i assume that that means just testing is a solved problem and uh it should be easy awesome um ivan uh hi everyone i'm ivan i'm a software engineer uh working on the android pedal plugin and also content build tooling mostly trying to make all the android developers happy and have faster builds and more correct builds oh and israel hey my name is israel and i work at dropbox in the mobile foundation team and i'm mainly doing android and a lot of riddle for the team

cool um it looks like we're still waiting for some questions to appear in the slido so who are you oh i i introduced myself at the beginning hi i'm zach i'm i'm here to ask the questions um so uh the first question while we're waiting for something to slide out from twitter ralph asks uh so gradle has a large and like wide api surface area that has a lot of legacy apis in it i'm paraphrasing a bit here certain apis can easily break performance improvements especially when it comes to configuration caching should gradle stop supporting backwards compatibility for certain apis or do it more aggressively nelson i'm going to throw this one to you first under the gun here yeah um okay so i should say i'm not on the build tool team on the gradle enterprise side that being said this question comes up a lot we're constantly in discussions about the best way to handle this we are thinking of deprecating a lot of apis for gradle 7.0 um the the reason we don't deprecate everything is of course backward compatibility we have a lot of large customers using legacy apis but this is definitely a thing that's starting to hurt us a lot um as you know as gradle builds get older and older and these older non-performant features are being used and we're not only in discussions with our large partners we're also in discussions with google about what should we deprecate what which we get rid of uh to make sure that gradle builds are as performant as possible so yes it's definitely at the top of our mind and it's also good that you ask for this because that means that this is something the community wants not just hey we're going to make it faster by killing this api i'd like to add to that one thing that i've seen some plug-in authors do is add test kit tests to test a matrix of different gradle versions for their plugin um enabling features like caching and configuration caching that's a well desired feature coming up um that can be onerous because now every plugin author needs to copy each other's test suites in order to guarantee that their plug-ins meet it it'd be nice if gradle could maybe team up with some of the work that i think that the android gradle plug-in team did when um uh gotham on uber came up with that large-scale project to test agp against and maybe do some cross-section of testing uh plug-ins over large projects across gradle versions i think would be a really nice way to try to maybe catch some of those things earlier because i agree it's a it is a pain point especially for many plug-in authors

there's one thing uh yeah i just wanted to add uh one thing for all the plug-in authors who are interested in you know losing using the latest and the greatest apis um there are some flags in gradle that you can you can pass there is one uh that we are using in atp warning mode equals fail dash dash warning mode equals fail that one is a great way just to catch user dissolve any of the deprecated apis it is still not as strict as we'd like it to be but at the same time what we're trying to do on the atp side is sort of like to design apis that are also moving the ecosystem in the right direction like we're working on the new variant api that's only using providers and properties we're trying to make it hard to do things in the wrong way and that's also very important like that you know that i think the the plugins authors that that see that they have like a big usage in the android community i'd love like for all of them you know to start switching to these new apis i mean we say new but they've been around for almost like six months to a year and a half and some of them two years so uh yeah it would be nice to start moving a bit faster there

cool um so anonymous asks do you uh keep a org.gradle.cachingtrue enabled for your local builds as well i see a lot of my builds fail when i have it turned on especially when the change is dagger related

i'm going to toss this one to j-rod who uh i know has worked a lot in dagger and large gradle projects with it uh yeah i'm actually checking my local gradle files right now uh to see if i have disabled i tend to toggle these flags around a lot so um i think the answer to that is yes we do yes we do toggle it on cache anyway and it has been helpful there are occasional cases with dagger and assisted inject where um incremental compilation will fail um and i believe a teammate of ours igor just found a certain use case for that but i think it um our solution there has been just to rmrf the build directory they don't happen as much as they used to but more often than not the caching equals true flag it has been proven beneficial for us yeah i'd like also to add one thing there like there are some there were some caching issues in the copying gradle plugin um and some of them were fixed but if you have like a you know reproduction please do file bugs like these are sometimes we hit these edge cases and having actual reproductions or more data to compare like comparing different gradle build scans or something could be super useful in you know fixing these ultimately caching should just work so we just treat these as bugs so please you know please follow them and provide a possible more information on that i wanted to add that with these flags usually they are beneficial and sometimes there are some such cases that they break for some reason and in those cases feel back if that's persistent and it can be reproduced easily but in general they are beneficial so it's better to keep them on otherwise you will not get any caching which is pretty bad depending on how easy the app is

cool so uh next we have uh we had one question on twitter related to this as well um so anonymous asks how do you suggest managing multiple build gradle files in an app with feature modules and i think a sort of related one that we got on twitter from p was uh do you have any advice basically around organization structure for your repo to try to maximize like build performance

um for the for the first part of that question organization um if if you have many modules you notice there's a lot of like copy pasta like which plugins you'll apply um what configurations you'll have for your lint options a lot of those common configurations can be pulled out into your root build gradle um and there's there's plenty of sample code to to brock over github for this but you can do something like you know if you are an app module then you can apply these settings if you're a library module you can apply these settings and so that can kind of reduce some of the boilerplate um i think for many of our featured modules at this point we just supply the dependencies block because those will differ from module to module and just reduces a lot of the copy pasta i think the next natural step would be if you can generate a lot of those build build-out gradles uh yourself i haven't seen a common solution to that across orgs i think if anything every work has kind of their own way to to generate their uh build gradle files appropriately but that that first step of pulling common config out into the route i think will drastically clean up a lot of the boilerplate you'll see yeah and if i can add to that pulling out common configuration is in general a good idea because then you can if you using build scans you can see in the build scans where what part of what script contributed to the build time whereas if it's all in one giant file you don't know what part of that file contributed so that's another benefit that being said there's also there's many different practices around here there's included builds there's plugins there's build source

and yeah it really depends on what exactly the logic is going forward we're going to recommend having using conventions plugins so these are plugins that you use basically as part of a composite build you declare it in the composite build and then you apply that to your build scripts your build source and then you apply that plug-in where needed instead of having these uh build scripts everywhere that you apply to each project so that's that's one suggestion that we're gonna have going forward but of course there are different needs that require different solutions so there's no one size best fits all

cool so our top question right now is another one from anonymous um what's that number of tests that you run in your apps we're running about 5k unit and 1k integration uh and end-to-end tests and ci of course hates us chookie do you want to take a shot at this yeah well i think everybody should give a number as well so we have a more variety i think um so the app that i'm working on is about 400 i would say espresso instrumentation test and yes like you do need to run it on ci and it's a extra step of uh making it so that it's less flaky it takes less time so yeah um so far i still haven't seen like an easy way to have a big test suite you just have to go through the paint but maybe others have a different experience i'm gonna add a twist onto this can you mention in your answer if you use emulators or if you use like a device lab like ftl

i'm just like okay so just to make sure people know ftl is a firebase test lab and i think you can actually ask for either real device or emulator so it's not uh either or situation so you could run through emulators with uh firebase tests well we use firebase tesla so like i say answer to that yeah on cache we recently migrated from our own proprietary emulator solution to firebase test lab shout out to nelson's flagler plug-in for making that process very seamless it uses flank um and yeah we have only well we had only two ui tests i think we've grown to five and that is um there's we we are growing the number of those integration tests however we um people use integration tests i would posit for many different reasons uh why we have done it in the past has not only been to make sure we have end-to-end coverage of certain things but also using screenshot tests we've since moved to a different project that we developed in-house at cash app called paparazzi that allows us to run a lot of those screenshot tests on the jvm and minimizing that dependence uh on using espresso for that because as truekey pointed out uh making an espresso test to cover many parts of your app is difficult you have race conditions flakiness etc and so um yeah having an idling resource for the many types of android animations i know unless you disable animations et cetera are always just common things you have to worry about with espresso so we try to minimize that and keep that at the very top of the testing pyramid um at dropbox i want to commend a project that uh mike uh chris and will and my team did wrestling where we had this 75 minutes time to merge because of those 6000 tests that we have to run every time before we merge and the way the way they got it down to 25 minutes was by doing selective testing and only running the test of the class of the test modified by that uh quarterback uh by that change right and um that's open source by the way so you can check it out in a dropbox uh github and then what we what we'll do is chart all the unit tests um per module except in the we have a big one still a monolith that has 25 of the code and that one we compile one we run unit test and ui test at the same time so we avoid uh adding extra extra processing on the cache on on the build time for for the big monolith and yeah selective testing have definitely helped us bring that down and improving everyone's ci life i guess i wanna yeah oh god i was also going to add to that um but i was going to go on for a bit so maybe you should go first if you just have a quick comment uh i was going to add on um on top of this how do you manage flaky tests and do your strategies like differ at all as you scale up from a few hundred to maybe a few thousand hmm that's okay that's a good question maybe we should get to that one next let me finish this one um yeah there's of course a bunch of strategies to scaling unit tests and and ui tests um like john mentioned there's uh fladle where you can scale and parallelize all your tests on firebase test lab with the gradle plug-in um and then as israel mentioned you can use uh this cradle plug-in affected module detector gradle itself of course has build caching so if the runtime class path of the module of the tests that are running don't change gradle will not rerun those if you're using the gradle build cache another feature you can use of gradle is max parallel forks if you enable max problem forks in a module you can run your tests in parallel that also helps speed up builds for unit tests gradle also recently announced a new feature called test distribution which will allow you to run your unit tests on multiple test distribution agents whether locally or on ci and that will also significantly help speed up your test run times both locally and on ci so if your laptop locks up when you start running unit tests that helps with that and then i guess also to plug another cradle enterprise feature and answer the flaky test question uh gradle enterprise also has flaky test management tools so there's a flaky test plug-in that we that we have as part of gradle enterprise and what it does is if any unit test fails it'll re-run it once and if that test fails and then succeeds as part of the same build then it'll be marked as flaky and you can see all your flaky tests you can see which tests are the most flaky using the gradle enterprise dashboard uh we just launched a new feature that lets you see the actual error messages side by side of all the different flaky test failures and compare that to the tests that failed so to see if the failures are different when they fail versus when they're flaky um as for strategies to handle this i know right now at cradle we're having a once a week we're going through our flaky tests and uh just fixing them so we just spend a morning we're just fixing them we're doing this until the end of the year uh just because we have a lot of flaky tests right now but that's i guess the way i've handled it at companies before is just have a day go through flaky tests and yeah i don't know what are other strategies that you guys have

i think the strategy is if it's really all and the person that brought it doesn't work there anymore we just ignore it and then at some point somebody's gonna let us know that that shouldn't be ignored and we fix it but no but it's true that uh accountability is really important um and we definitely take uh care of those that are failing uh we ignore them to make sure that we don't block anyone from merging i think that's the most important for us but we fix them as soon as possible yeah so so far um i have worked with teams that are not that big so it flakey test is definitely something like super annoying um this may be controversial but i am actually a believer that if you have a flaky test and you know your adult strategy is like trying to run it multiple times and if it passed once then it counts as passing because sometimes the flanking has come from the fact that oh when you click on the drop down it's not rendered yet before you uh try to click on a an option for example so it's not necessarily telling you that your code is broken so if you have a flaky test that you cannot make it past like i am actually a believer to just get rid of the test to be honest like it's better to make sure that all your tests when they fail they actually indicate a problem then getting tired of seeing tests failing but not actually showing you bugs because well first even if a big team right when people lose confidence in the value of tests then it's really hard for people to continue investing time in the test so i know some people are really obsessed about test coverage but to me it's more important to have good tests than have a lot of tests that don't tell you anything i like it i heard a rumor once about a company where if uh someone maintained a service and then they left then they turned off the service until someone else complained about the service being off and then they were the new owners i think it was a joke but i do wonder sometimes that that would work well it's the same theory as if you have boxes that you moved and you haven't opened them in a year that means you don't need those things in that box right so exactly um so moving on through the questions here so we have one from rodrigo uh how do you guys see jetpack compose um x probe electric tests um guessing the needs in combination with um do you think that there's hope for a faster instrumentation test with them uh i'm going to send this one over to yvonne

i think like probably verbal lecture is going to be able to handle some of them i'm not sure but i can imagine that in in the same way with the current uh layout solution you still want to run on the device in order to have like proper testing of the behavior of the platform so it would assume like you would still need to run instrumentation tests like on the actual devices so i don't think those are going going away fully however like we're trying to address some of the pain points especially about caching and up-to-dateness of tests by trying to model the state on the device a bit better and we're working on that with the with the new nitrogen plug-in that should like allow us to know what the state of the device uh and thus be able to actually cache integration tests another one from uh anonymous asking what are the top three things that you would recommend to try if your build time is uh if your build time in ci is really long i'm gonna send this one to nelson and jrod yeah um this is a a good question and obviously it depends on your setup i'm a little bit biased here i'm going to say run a build scan and then take a look for red flags so a big red flag would be if garbage collection time is high increase your heap size um if if one specific task is dominating the build time that could mean split up that module into different builds so that you know not the up-to-date checking mechanisms and incremental compilation can work better um of course yeah there's a bunch of things to test make sure there's no plug-ins so check your configuration time make sure there's no plugins that are adding a significant amount of time to the configuration phase um i guess yeah those are three things so i'll let j rot or someone else answer yeah i think those are good starting points as well just speaking from experience on the app that i work on um try to see if like the local how different the local build and ci builds are different for example we use amazon workers and so they may have beefier ram than our local machines also but at the same time because of compute cost they may have fewer cores and that can drastically affect the parallelism of your builds and which that just means longer wall clock time um what else nelson brings a good point about the plug-ins you use i mean it's it's very easy and handy to use a plug-in from open source but it's also sometimes one of those plug-ins may inadvertently drag your build down and so it's build scans are helpful for being able to profile that information quickly parallelizing and modularizing again if you have more cores that means you can run more things in parallel also using plugins that have adopted the worker api to achieve even better parallelism um are some things to maybe kind of start off with but usually cpu cores and ram are the bottlenecks for a lot of these things yeah with we've done a bunch of things at dropbox so first uh uh great get a bigger machine as you all said that always helps but uh then taking a look at the build scan and checking the configuration phase and the execution phase in the configuration phase we had this one instance where we added a plugin and a plugin double the time of configuration and nelson knows about this that's why he's laughing about it and then and then so it was difficult for us because at that time we didn't have any telemetry to tell us when that happened so i had to go down in time to figure out what was it and thanks to build scandal we will we had a really good signal of what was happening and uh and so that's one the second one is execution especially try to clean the bill from clean the cash the local cache build one clean build two and see in the second one the second bill the cash ability of those cashable tasks and check if there has been any change and they should be cached or not and if you can even compare between those two by capturing the input then you will be able to have a lot of information on what's wrong in your in your tasks um a nice thing that's pretty much uh eat cool one other thing that uh i'll i'll put myself on the panel for a second here that i want to mention because we just solved a lot of our slowdowns with this is uh gradle build scans will tell you also how much time uh your built spends garbage collecting and we found out that lint spent 20 of its time garbage collecting so we just gave a little bit more memory and we got back like 10 minutes of ci time so there can be easy wins to be found if you look close enough moving on to the next question omar asks we currently use the dash dash build cash flag on ci bills to benefit from our remote build cash would it be a good practice to use that flag also on developer local build

yes all right cool

one little caveat to that be aware of the dash dash no build cache flags and the dash dash rerun tasks flasks uh sorry task options um just in case something corrupts your build cache doesn't happen too often but it's good to know that they exist yeah to add to that usually uh what what you can also do is you can add org dot gradle dot caching equals true this came up in an earlier question and then just enable the build cache for all the builds and that's actually what we recommend at gradle that's what we use in the gradle build tool and that's what most people use they have ci jobs that seed the remote build cache and then local jobs local builds will not push to the remote build cache will only pull from the remote build cache and that's yeah that's the recommended setup and i think that's also in the docs as well oh one caveat though be aware of negative avoidance savings tasks if that happens to you you can use the gradle doctor plugin it'll tell you about that but that's a whole other topic i don't want to get into right now you beat my curiosity i'll ask you later

standard disclaimer that nelson is the author of that plugin um uh yvonne i think this one is gonna be for you uh chow song asks when agp aligns with the versioning uh of gradle uh which is for background context a reference to the next major version of agp will be i guess seven to match the gradle version that's built with yes yep um would it be possible to fix the lint versioning as well so that we don't have to keep adding 23 to the current agp value for lint tools that's a great question something actually we're discussing internally there's a the whole like versioning issue is uh like is lens should it be version with adp or does it more belong to studio because it's also like it's a in ide thing as well so at the moment like we're trying to figure out um sort of like where it belongs naturally and based on that we're gonna figure out uh what is the worsening schema that it should follow so yeah no definite answer yet

um anonymous asks on ci it's easy to have memory issues when running tests what's the best strategy to manage memory between tests or between tasks that spawn java processes and the gradle daemon

it's hard to know the exact dimensions of the ci and test infrastructure you're working with but two suggestions i tend to find are make sure you have a minimum of four gigs of ram on your ci workers and try to minimize the robo-electric dependencies you have because it loads a lot of that the android platform into ram um again those are just very general ideas that i know has affected us in the past yeah i want to say this is also a really tricky issue because with workers the more cpu cores you have the more memory you consume but you don't necessarily have more memory if you have more cpu cores and if you're running on different machines those will scale in different directions so this is this is definitely uh at the top of our minds how to make gradle uh scale to different machines workers cpu cores and memory and those types of things but i will say it's not very easy just tweak experiment and measure is is the best solution right now

there could be a memory late too so look for those

cool um the current top question we actually uh cherry picked out earlier so we've already gone through that one um a good one that we have in here is where do you see the kotlin dsl going in android builds yeah um so yeah where do you see it going i think it's going to be used more and more and i think it helps people understand uh gradle because of the ide autocompletion uh one of the biggest downsides of using the kotlin dsl for build scripts was the compilation time so when you compile your build scripts it slows them down when you need to compile or build scripts the kotlin build scripts are about 100 times slower to compile than the groovy ones the difference is about about 10 milliseconds for groovy scripts versus one second for kotlin dsl scripts but in gradle 6.8 this is significantly significantly significantly improved both in terms of speed and memory consumption and so uh i see the kotlin dsl script being used more and more and then the second part to this is there's also compile avoidance for uh build source changes for your build script so that'll mean that you won't have to recompile those as often even if they are faster to compile so yeah in terms of speed it's definitely getting a lot better i don't know about other other features or anything if anyone else wants to comment go ahead yeah it's great to hear the performance improvement and the compilation with build source that's great too and yeah definitely we are using more calling because the ide helps so much to learn the apis or to you know not so well known abs it gives the ui autocomplaining easily um moving on here uh this one's going to be for yvonne any updates on project nitrogen

so the updates are that you're working on it actively um we are doing a bunch of work in the android gradle plug-in to support uh nitrogen as the test executor so that should allow you to plug in you know device farms whatever you have for running the actual tests um so yeah not not too much info uh you know but like basically working on it actively working on it so hopefully something should happen during 7.0

cool um i'm gonna take one from uh twitter here um

shoot where was it uh i lost the question that i was going to ask uh oh what's the current state of configuration caching and what can i get by using it or i guess investing and making sure that your project supports it where i can i'd love to take this one um so the current state is that gradle 6.7 has pretty good support for configuration caching but 6.8 fixes even more things so one of the things that you can do is you can make sure that you're using the latest adp content gradle plugin and cradle so this means uh integrated plugin 4.2 alpha 16. i think that's the last one we released you can use content gradle plugin 1 420 rc is currently out i think 1420 final should be out soon-ish and gradle 6.8 uh currently milestone 3 is published but i think the final should be out in a couple of weeks and um you should just update and try it out just um pass that configuration cache to your build and just check uh you get like a very nice html report saying like if there are any violations uh you get like a pretty neat output that you can use to debug other things and also you can just reach out to other plug-in authors uh and tell them you know just file bugs on github or reach out to them directly asking if they can make their plugins compatible with configuration caching because we see pretty good results on some of the projects like configuration times that were like minute and a half going to only a couple of seconds also task execution time simply because you're able to run all the tasks in parallel also within the project goes down the dependency resolution is cached so you don't pay cost there when it comes to incremental build there are like numerous benefits so i'd really love to see community uh adopting this yeah i believe through the help of stefan nicolas once at the square we've been able to have sql delight and wire gradle plug-ins already able to support them again if we missed anything please file issues as yvonne pointed out but i think those should be ready for the gradle configuration caching feature

yeah one thing i wanted to add there is configuration cache is not necessarily an all or nothing thing for your project or your build you can try it out and there might be like a specific hot path that it works for you and your project if for example uh certain tasks are not run so i would recommend even trying it out in the current stable versions if you can uh and then you know file bugs give feedback uh and then since there was a question on twitter about this about how the teams collaborate i want to say configuration cache was a huge collaboration between gradle google jetbrains and even square actually square helped a lot in testing the feature in in their real world projects so yeah it's been it's been pretty cool to see all the teams come together collaborate and and get it across the finish line and obviously it's not just those teams but also the entire ecosystem of gradle all the plug-ins all the plug-in authors have really put in the extra effort to sometimes rewrite their entire plugins to get it working

we've got just a few minutes left here so i want to try to run through some of these next question we have is generate build config takes more than 20 seconds for one of my modules what can i do to figure out why it's taking that long um yvonne i'm guessing what they're trying to ask is should i file a bug report and where you should tracker.google.com please do file bugs it should not take that long we'll try to get to the bottom of it one thing i did want to add there is it could be that your generate build config calls a function in your project like let's say calls get and there's lots of files on your file system touching git or this function makes a network request that could also be a reason for that taking long so just make sure that all your values are constant or not taking long in there

um do you have or anonymous as do you have a recommendation about gradle module size in an android app let's say when it's bigger than x lines of code um or maybe like build time it's better to split it or vice versa do you have any sort of patterns that you enforce in your own repos it can be as small as possible if it says one concern and only one concern that's totally fine to be its own module um and then on the flip side of things usually you know when you create a new app everything widens up in the app module we've been pulling large chunks at least on cache app we've moved into a tab based interface in the last i guess year and so there are five tabs and we've been modularizing out of app to have each tab in its own module and then that tab itself could be broken down even further so um you can come at it from both ends either start a new module no matter how small or break down really really big modules into smaller ones to just get the benefit of paralyzed compilation yeah we've been doing that uh distract distracting the monolith and creating feature modules and like even so feature modules and that every everybody's been really happy about it the only caveat here is that uh we had a we have a huge many many apps and uh modules uh monorepo and then when we open we have a single settings radio and when we open that in android studio the gradle thing uh takes forever so watch out for that we are working on a focus mode at some point on 2021 because this is not going to scale much from now i think we're at time here um i want to actually just run through and let each panelist if people want to reach out to you with other questions or um anything what's the best way to do that uh for me i'm on twitter and jrodbx jrodbx

uh for me you can contact me on twitter i also have my website osaki.com osa cky and then there you'll find a bunch of ways to contact me yeah feel free to reach out to me on twitter as well i think the twitter handle is you know on the on the website with the talk yeah same for me on twitter r-a-l-l-a-t rather yeah i'm chucky on twitter

cool uh thank you everyone for attending thank you to our panel for uh taking the time to sit through these and uh cool have a good day everyone thanks so much thanks zach for moderating and everyone else thank you for joining take care thank you take care bye