Buck: buck autodeps

buck autodeps

THIS COMMAND IS DEPRECATED AND NOT ACTIVELY MAINTAINED!

tl;dr

Deprecated feature to auto-generate the deps and exported_deps for the following rules:

The Problem

Maintaining your deps by hand in a BUCK file can be tedious. In practice, most programming languages have an existing mechanism for expressing dependencies in the source code, so what appears in your deps often duplicates that information. For example, in Java, there is often a straightforward mapping between your import statements and your deps.

To make matters worse, the tools that you use to write in your programming language often help keep the dependencies in your code up to date, but do not help with your BUCK files. Let's consider Java, which has many great IDEs that support all sorts of far-reaching refactoring operations. Although the IDE may make it simple to do something like extract an interface for a class, it will not help you split the new interface into its own java_library() rule in Buck and update all of your deps, as appropriate.

This is bad. Particularly because developers frequently work around this in one of two ways:

  • They let small modules grow into large ones out of concern that the cost of having to update so many BUCK files outweights the benefit of having smaller modules.
  • They split up the srcs of a large library, but copy-paste the deps. The result is that the resulting libraries end up with superfluous items in the deps, which can slow down build times.
Developers should not have to choose between using the full power of their IDE and fast build times.

Our Experimental Solution

The idea is that you can opt-in to letting Buck auto-generate the deps of individual java_library() and java_test() rules by adding the argument autodeps = True to the rule. In practice, a rule with autodeps = True should almost always have empty arguments for deps and exported_deps.

After you have added autodeps = True and you have deleted your deps, the next step is to run buck autodeps. For every BUCK file with an autodeps = True, a BUCK.autodeps file will be generated alongside of it. The BUCK.autodeps file contains the list of autogenerated dependencies, as well as a SHA-1 signature of its contents. (Signature checking is currently disabled, but we plan to turn it on by default once we believe buck autodeps is reliable.)

Now when Buck parses a BUCK file for building, it looks alongside it for the presence of a BUCK.autodeps file. If present, it uses it to amend the deps and exported_deps of the rules declared therein. As things work today, we recommend checking in your BUCK.autodeps files. Admittedly, checking in generated code is gross, causes merge conflicts, etc. However, until we are sure that buck autodeps is performant, checking in the files ensures that someone who checks out the code can build right away without waiting for buck autodeps to run. Also, in the early stages of buck autodeps, it makes it easier to identify when buck autodeps has decided to change what it generates. (Such changes should be sanity-checked against changes in the .java code that accompany the changes to BUCK.autodeps files.)

As you can see from the Troubleshooting section, there are some ways that buck autodeps can end up doing the wrong thing. Our goal is to make sure we provide you with an escape hatch if you run into trouble.

How Required and Provided Symbols Are Extracted

When buck autodeps is run, Buck only looks at non-generated srcs for the rule and parses (but does not compile) the .java files. The goal of parsing is to extract symbols and put them in the following buckets:

  • required symbols are symbols that must be on the $CLASSPATH when the srcs are compiled.
  • exported symbols are also symbols that must be on the $CLASSPATH when the srcs are compiled, but have the additional constraint that they need to be on the $CLASSPATH when someone else depends on this rule. (Roughly, they correspond to types that are visible via the non-private API of the file.)
  • provided symbols are the types of the .class files when the rule is built (i.e., the types that the rule provides).

Here is a simple example:

package com.example;

import com.example.foo.Bar;

class Foo extends Bar {
  private final IBazFactory factory = new BazFactory();

  public Baz makeBaz() throws BazFactoryException {
    return factory.newBaz();
  }
}
In this example:
  • The required symbols are com.example.BazFactory and com.example.IBazFactory. They are needed for compilation, but do not appear in the non-private API of Foo.
  • The exported symbols are com.example.foo.Bar, com.example.Baz, and com.example.BazFactoryException.
  • The lone provided symbol is com.example.Foo.

In the absence of compilation, some expressions are ambiguous when it comes to extracting symbols. Buck assumes that you are following traditional Java naming conventions when deciding how to resolve an ambiguity. For example, if you have the following code:

Object obj = com.foo.Bar.baz();

Buck assumes that this is a fully qualified reference to the class com.foo.Bar with a static method named baz(). It is also possible for this to refer to a class whose fully-qualified name is com.foo with a field named Bar with a static method named baz(). However, the latter would violate standard Java naming conventions because foo should start with an uppercase letter if it identifies a class, so Buck will assume the former, instead. As a result, Buck adds com.foo.Bar to the set of required symbols for the build rule that contains this Java code.

Similarly, generics can also introduce an ambiguity when parsing. For example, consider the following code:

package com.example;

class Foo<T> {}

Without knowing all types that are available in com.example, it is impossible for buck autodeps to know whether T is a generic type or a reference to an existing type, com.example.T.

Buck's heuristic for resolving this ambiguity is: if the name of the type is in all caps, it is assumed to refer to a generic type. Admittedly, there are some types in the JDK itself that violate this heuristic, such as java.net.URL. If this is an issue, the rule that provides or uses the type will have to be explicitly added to deps or exported_deps, as appropriate.

The class that is responsible for extracting these symbols from the .java files is JavaFileParser.java, and the corresponding unit test is JavaFileParserTest.java. It is quite possible that there are edge cases that this code is missing, so feel free to send out a pull request with a unit test to improve the existing heuristics.

Finally, there are some types that may be provided by a java_library rule, but cannot be determined from parsing the srcs because they are provided dynamically. (For example, using a genrule to generate one of the srcs). In this case, you must specify the names of the generated types via the generated_symbols argument to java_library.

How Autodeps Are Assigned

The process of auto-generating dependencies is pretty straightforward. After the parse phase, Buck has a set of provided, exported, and required symbols for all rules of the following types in your project (regardless of the rule's autodeps argument):

(Note that, at the time of this writing, android_resource is not in this list.)

For each required and exported symbol for a rule with autodeps = True, an entry in a BUCK.autodeps file is generated as follows (the build target is added to deps for required symbols and to exported_deps for exported symbols):

  • If there is a prefix match in .buckconfig under the java-package-mappings option in the [autodeps] section, the corresponding build target is used.
  • If there is exactly one visible provider of the symbol, its build target is used.
  • If there are multiple visible providers, then a warning is printed. The developer is responsible for deciding how to resolve the warning. (Often, this is done by restricting the visibility of all but one provider.)
  • If there are zero [visible] providers, then a check is done for providers that appear in the exported_deps of other visible rules.
    • If there is exactly one, its build target is used.
    • If there is more than one, a warning is printed.
    • If there are zero, nothing is printed and Buck assumes that the user will take action if/when compilation fails, as a result. It's possible that this is an edge case where the user has to hardcode something in the rule's deps or exported_deps.
The results are cleaned up such that deps and exported_deps are disjoint sets for each build rule.

Signing

By default, BUCK.autodeps files contain a signature line at the top like the following:

#@# GENERATED FILE: DO NOT MODIFY c9a06044ccac3d1356249ab06a01f82874510b5d #@#

The signature in this line must match the SHA-1 of the contents of the rest of the file. If not, an error will be raised when parsing the corresponding BUCK file.

The purpose of the signature is to help ensure that the BUCK.autodeps is produced by running buck autodeps rather than someone mangling it by hand. The problem with manually fixing a BUCK.autodeps is that those changes will be blown away by the next person who runs BUCK.autodeps, which may introduce an issue that is difficult to debug.

Nevertheless, some teams dislike the signatures because the friction they introduce [with regards to merge conflicts when using source control] may not outweigh the benefits of the barrier they present to hand-editing the files. Therefore, it is possible to disable this behavior in your .buckconfig file:

[autodeps]
    include_signature = false

When this option is set, the signature line will not be inserted when buck autodeps is run and the build file parser will not check for the signature line when reading BUCK.autodeps files.

Troubleshooting

When buck autodeps does not generate the correct dependencies, there are several things you can do to fix it, depending on the problem. First you must determine whether the dependencies are underspecified or overspecified.

Fixing Underspecified Deps

There are several ways to solve this problem.

The quickest and dirtiest solution is to add the missing build target manually to the deps argument of your build rule. This is the least desirable solution because if your library evolves to the point where it no longer needs that dependency, you will probably never discover the change and you will be over-depping, slowing down your build times. This may be a reasonable thing to do locally while you are developing, but you should clean this up before you commit your change. (At least add a TODO or a FIXME so you have some chance of coming back to the problem in the future and fixing it.)

Note that we managed to add support for autodeps = True to all of the java_library and java_test rules in Buck without resorting to adding hardcoded deps. There were several cases where we decided to use the default of autodeps = False where we had sensitive code where it would have been an error for it to take on any dependencies. We would generally include a comment inside the empty deps array with a reason explaining why we do not want to auto-generate dependencies for that rule.

We don't load classes via reflection in Buck, but if we did, that would also be a legitimate case for manually adding something to your deps. Another case was a dependency on a java_library that contained only resources, so there was no way for Buck to statically determine that dependency.

Although buck autodeps is working well for Buck, your code may exercise a valid case that Buck does not cover. See the note above about verifying Buck's logic in JavaFileParser.java to ensure it is doing the right thing.

Fixing Overspecified Deps

If buck autodeps introduces dependencies that you did not want, then you have two options:

  • Modify the visibility of the unwanted rule such that it is not visible to the rule that is ending up with it in its deps. The buck autodeps command will not include rules that it knows are not visible to the target rule. (The one exception is when there is an explicit mapping in your .buckconfig file. Buck does not check the visibility when applying such mappings.)
  • Add the build target of the unwanted dependency to the provided_deps of your rule that is ending up with the unwanted dependency.