background
At the beginning of 2018, the Zhihu Android client was in the middle stage of componentization, and the splitting and building of components was in full swing. Thanks to componentization, Java files can be compiled into class files in advance, and the overall compilation time of the app has also been improved to a certain extent. However, one day, the compilation time of the main project suddenly increased from 4 minutes to 10 minutes, which seriously affected the development efficiency of the students who were still working on the main project, and at the same time almost doubled the consumption of CI resources, so it was necessary to investigate what the problem was.
Locate the problem
First of all, when I added the --profile parameter at compile time, I found that the growth part was in the javac stage, and the corresponding gradle task was compile{flavor}{buildType}JavaWithJavac, while the time taken in the other compilation stages was basically unchanged, and it was obvious that there was a problem when compiling the java file to class.
Since the problem was not found immediately, and there were many MRs submitted, I didn't know what the change caused the problem immediately, so I sacrificed a binary search method, and finally found the commit that compiled slowly, and then saw that there was nothing suspicious about the code, but simply split a component into three components - the account component was the first component to be split when the componentization was split, and it was relatively rough to disassemble, with a lot of other business code and basic code, so this commit The account component is split into three components: the page frame, the account, and a common component. Could it be that some change in this split triggered a compilation bug?
At the same time, we compare the compilation time of other business components, and find that the time of JAVAC of each business component also increases to varying degrees, among which the more dependent layers of components, the greater the increase in compilation time, and the growth of time is basically exponential with the dependency level.
For example, if there are four components ABCD, A depends on B, B depends on C, and C depends on D, then if the compilation time of component D increases by 1 minute, the compilation time of component C increases by about 2 minutes, component B increases by about 4 minutes, and A grows by about 8 minutes.
Comparing the code before and after the component split, most of the time it is just a simple movement of the file, and the other changes are very simple, it is hard to imagine what compiler problems will be triggered in this situation, after all, almost 100% of the time, suspecting that the compiler will only slap itself in the face.
So what could be the problem? We know that Android compilation does not only have a javac stage, but also other compilation processes, and there are a bunch of other Android-related products in the aar file in addition to the jar file.
Find a top-level business component and compare the compiled product aar file:
As shown in the picture above, it is clear that there is a setter_stores.bin that the size of the file has increased dramatically, and if nothing else, the packaging time should have something to do with this file.
Then we analyzed the AAR of other components, and found that the increase in the volume of this file was roughly the same as the increase in the compilation time of JAVAC, and the increase in volume and compilation time was roughly exponential with the number of databindings in the branch with the largest number of databinding components in all branches of the dependency tree. Other words:
Originally, it only takes 2 minutes to compile Java, and there is an extra layer of normal libraries in the dependency layer, and the compilation time will not change, but if there is an additional layer of DataBinding-enabled libraries, it may increase by 1 minute, and if there are two more layers of DataBinding, it will increase by 4 minutes.
With this guess in mind, the solution is clear: remove the DataBinding property of some components. Since the page framework and the Common component do not have much DataBinding code, the DataBinding of these two components is directly removed, and the packaging time is restored.
Although the immediate problem is solved, it does not really solve the problem, and maybe one day the DataBinding dependency layer will increase again, and the problem will reappear. Sure enough, after the last split of componentization, compilation times skyrocketed again. At that time, all the last community business was dismantled from the main project, and due to the complexity of the main project business, it became a number of business components after being dismantled, and these business components were coupled due to historical reasons, so there were finally several more DataBinding dependency layers, and these components used a lot of DataBinding code, which made it a little unrealistic to remove DataBinding dependencies: first of all, the development cost of transforming these codes, and more importantly, QA The cost of regression is high, and there is no real solution to the problem, and no one can be sure if there will be any more dependency tiers in the future, unless we prohibit the use of DataBinding. It's time to dig into the root cause of slow compilation caused by DataBinding.
Where exactly is the slowness
To see where the bottom is slow, we need to know what it is doing during the period of packaging, let's use the tool visual VM to take a look (jstack can also be):
1 ) 打开 Visual VM
2) Execute the main project compilation command to locate the compiled process ID (7689 in the example)
3) When compiling and executing to the javac stage (e.g. compileDebugJavaWithJavac), right-click on the process name → Thread Dump to dump the thread down and execute it several more times
4 ) Since we have already guessed that DataBinding is causing the problem, we directly searched for databinding in the dumped message, and found that it was stuck in the same place every time:
"Task worker for ':'" #522 prio=5 os_prio=0 tid=0x00007f4af4447800 nid=0x30a0 runnable [0x00007f4af8a70000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap$TreeNode.find(HashMap.java:1864)
at java.util.HashMap$TreeNode.find(HashMap.java:1874)
at java.util.HashMap$TreeNode.find(HashMap.java:1874)
at java.util.HashMap$TreeNode.find(HashMap.java:1874)
at java.util.HashMap$TreeNode.find(HashMap.java:1874)
at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:1994)
at java.util.HashMap.putVal(HashMap.java:638)
at java.util.HashMap.put(HashMap.java:612)
at android.databinding.tool.store.SetterStore.merge(SetterStore.java:1173)
at android.databinding.tool.store.SetterStore.merge(SetterStore.java:1153)
at android.databinding.tool.store.SetterStore.load(SetterStore.java:185)
at android.databinding.tool.store.SetterStore.create(SetterStore.java:176)
at android.databinding.tool.Context.init(Context.kt:49)
at android.databinding.annotationprocessor.ProcessDataBinding.doProcess(ProcessDataBinding.java:95)
at android.databinding.annotationprocessor.ProcessDataBinding.process(ProcessDataBinding.java:73)
所有 dump 的信息都卡在了 android.databinding.tool.store.SetterStore.merge(SetterStore.java:1173) 这一行上,很明显,直接原因就是这里了:SetterStore#merge 方法有鬼。
SetterStore 是做什么的
In Android DataBinding, you can bind data to the property of the view in xml, for example, if you want to bind a String to the text property of the TextView, you only need to declare it in the xml
<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="@{string_content}"/> Then give the text property of the TextView how to bind the data
@BindingAdapter("android:text")
public static void setText(TextView view, CharSequence text) {
view.setText(text);
} At compile time, databinding-compiler will find the BindingAdapter corresponding to the xml attribute and convert the binding syntax in the xml into a method annotated by BindingAdapter, so that the data binding is implemented.
However, databinding-compiler generates code by processing the source code through the annotationProcessor, generally speaking, the annotationProcessor will not process annotations outside of this module, such as annotations in other aar files, so if the BindingAdapter is defined in the Base library, how to make the projects that depend on the Base library also use the Base library What about BindingAdapter?
After studying the source code of databinding-compiler, I found that the answer is in setter_stores.bin: databinding-compiler will store the resulting BindingAdapter and some other elements in an instance of the SetterStore.Intermediate class, and setter_stores.bin is the result of serialization of this object. It is eventually packaged into an aar for use by the referrer. The referencer will deserialize the setter_store.bin of all the libraries it depends on at compile time to get several instances of the Intermediate class, and then generate a merged SetterStore for the annotationProcessor to use, so that the annotationProcessor can use the BindingAdapter defined by other projects. The merged data is eventually serialized into a setter_stores.bin file.
So what SetterStore does is serialize, deserialize, and merge other setter_stores.bin files.
Why is it slowing down
Let's take a look at the process of merge:
The first step is to take all the setter_store.bin files and deserialize them and merge them, each loop is merging the setter_store of a library into a new store, which looks fine
private static SetterStore load(ModelAnalyzer modelAnalyzer,
GenerationalClassUtil generationalClassUtil) {
IntermediateV3 store = new IntermediateV3();
List<Intermediate> previousStores = generationalClassUtil
.loadObjects(GenerationalClassUtil.ExtensionFilter.SETTER_STORE);
for (Intermediate intermediate : previousStores) {
merge(store, intermediate);
}
return new SetterStore(modelAnalyzer, store);
} If you look at the method of merging individual setter_store, it looks normal, merging each type of data in turn
private static void merge(IntermediateV3 store, Intermediate dumpStore) {
IntermediateV3 intermediateV3 = (IntermediateV3) dumpStore.upgrade();
merge(store.adapterMethods, intermediateV3.adapterMethods); // 堆栈信息表示卡在这一行
merge(store.renamedMethods, intermediateV3.renamedMethods);
merge(store.conversionMethods, intermediateV3.conversionMethods);
store.multiValueAdapters.putAll(intermediateV3.multiValueAdapters);
store.untaggableTypes.putAll(intermediateV3.untaggableTypes);
merge(store.inverseAdapters, intermediateV3.inverseAdapters);
merge(store.inverseMethods, intermediateV3.inverseMethods);
store.twoWayMethods.putAll(intermediateV3.twoWayMethods);
} Looking at the merge method of merging individual items, there seems to be no problem with Map merging
private static <K, V, D> void merge(HashMap<K, HashMap<V, D>> first,
HashMap<K, HashMap<V, D>> second) {
for (K key : second.keySet()) {
HashMap<V, D> firstVals = first.get(key);
HashMap<V, D> secondVals = second.get(key);
if (firstVals == null) {
first.put(key, secondVals);
} else {
for (V key2 : secondVals.keySet()) {
if (!firstVals.containsKey(key2)) {
firstVals.put(key2, secondVals.get(key2)); // 堆栈信息表示卡在这一行
}
}
}
}
} The inside of the setter_store is a map (see SetterStore.java#IntermediateV1), so if nothing else, you'll end up with a small deduplicated setter_store. But when we open these generated setter_store.bin files, we will find that there are a huge number of duplicates in it, the same BindindAdapter appears multiple times in the same map, and BindingAdapter is stored in the IntermediateV1#adapterMethods field, the type is HashMap<String, HashMap<AccessorKey, MethodDescription>>, is there something wrong with this map?
Guess a lot of repetition should be related to the poor design of the key's hashcode and equals, adapterMethods is a double Map, the first layer of the key is String, obviously no problem, pass, the second layer of the key is a class AccessorKey, take a look at the source code of this class:
private static class AccessorKey implements Serializable {
private static final long serialVersionUID = 1;
public final String viewType;
public final String valueType;
public AccessorKey(String viewType, String valueType) {
this.viewType = viewType;
this.valueType = valueType;
}
@Override
public int hashCode() {
return mergedHashCode(viewType, valueType);
}
@Override
public boolean equals(Object obj) {
if (obj instanceof AccessorKey) {
AccessorKey that = (AccessorKey) obj;
return viewType.equals(that.valueType) && valueType.equals(that.valueType);
} else {
return false;
}
}
@Override
public String toString() {
return "AK(" + viewType + ", " + valueType + ")";
}
} Double-check the equals line
return viewType.equals(that.valueType) && valueType.equals(that.valueType); viewType.equals(that.valueType) 肯定是恒为 false。
According to the above analysis, the original work of merge was to deduplicate and merge the setter_store of all dependent libraries, but now because of the wrong writing of equals, each key must be different, and the effect of deduplication is not achieved at all
Let's assume that in the simplest case, we have library D dependent on C, C dependent on B, and B dependent on A, all of which have databinding enabled and none of the adapterMethods are defined, and assuming that the databinding library itself already contains 50 adapterMethods, then:
A relies on the databinding library, and A has 50 adapterMethods in the final setter_store
B relies on A and the databinding library, and B ends up with 50 + 50 = 100 adapterMethods in the final setter_store
C 依赖了 A、B 和 databinding 库，C 最终的 setter_store 中包含了 100 + 50 + 50 = 200个 adapterMethods
D 依赖了 A、B、C 和 databinding 库，D 最终的 setter_store 中包含了 200 + 100 + 50 + 50 = 400个 adapterMethods
This is consistent with the conclusion above that the increase in volume is roughly exponential to the length of the longest chain in the dependency tree containing the databinding component
The dependency level of our main project has reached 8 layers, so the repetition rate is 1/2^7, in the actual use environment, the dependency is not only a single-chain dependency, each dependency level may have multiple libraries, so in fact, the final dependency will be doubled several times, and finally the hash collision rate of our custom BindingMethod is close to 99.9% (862/863), and the BindingMethod that comes with databinding has quadrupled. The hash collision rate reached 99.97% (3497/3498), and the entire setter_store.bin file has reached 33M, when in fact there are only about 100 BindingMethods after deduplication.
solution
一、修改 databinding-compiler。
Once you know where the problem is, it's easy to modify the line where the problem is
return viewType.equals(that.viewType) && valueType.equals(that.valueType); 重新打包 databinding-compiler,使用后速度果然快了
2. Prohibition of unreasonable dependence
Although the problem of slow packaging caused by DataBinding has been solved, there are too many dependency layers in the project, and there are many cases that cause dependency problems: first, when a large business is split, it is split into several interdependent components at a time, resulting in more layers, and second, some students are greedy for convenience when developing, and when the interaction between components occurs, they directly reference other components (instead of referencing component interfaces), resulting in complex dependencies. So our solution is to grade the components:
- Clarify the division of business components, middleware, and basic components
- Direct dependencies between business components are prohibited, lower-level components are prohibited from relying on upper-level components, and cyclic dependencies are prohibited
- The existing unreasonable dependencies should be reorganized and resolved according to the rating
- Rating components is implemented into a centralized configuration file, and the gradle plugin is used to prohibit erroneous dependencies
Subsequent
This bug has existed for a year, and I encountered it when the Android Gradle Plugin was still version 3.0 at the beginning of 2018, and I only found out that it was a DataBinding problem, but I didn't think it might be a bug, so I simply removed the DataBinding code of a few components, and then I explored this problem again when it was version 3.2.1, and it has not been fixed. A few days before writing this article, the official version 3.3 was released, and this issue has been fixed :SetterStore.java#1357
Not only that, but also quietly added a compareTo method:
public int compareTo(@NonNull AccessorKey other) {
int viewTypeCmp = nullableCompare(viewType, other.viewType);
if (viewTypeCmp == 0) {
return nullableCompare(valueType, other.valueType);
} else {
return viewTypeCmp;
}
} If you don't know compareTo, you can see this Liao Xuefeng # Java Map correct use
作者:Stomach pork
Source: https://zhuanlan.zhihu.com/p/58010428