A bit of history
Historically, Buck relied on a Python DSL to describe build files and macros. This enabled Buck users to implement many features without having to modify Buck's core. Although Python worked fine for local builds and small repositories, when used at scale, the ability to access the host environment and perform arbitrary actions without Buck's knowledge led to non-deterministic builds, hard-to-debug issues, and slow parsing.
To address some of these issues, we introduced features such as
allow_unsafe_import(), but ultimately we were unable to provide proper sandboxing for deterministic parsing, and a new solution had to be put in place.
In order to tackle the limitations of the Python DSL parser, we added multiple-language support and a built-in parser for the Skylark language. The new parser provides an alternative to the Python DSL parser.
Enabling the Skylark parser
In order to enable Skylark support for your project, add the following section to your
[parser] polyglot_parsing_enabled = true default_build_file_syntax = SKYLARK
We recommend Skylark for new projects and it will become the default in the future. However, if most of your build files or macros rely on Python DSL features and you're not ready to invest in migrating to Skylark, replace
default_build_file_syntax = SKYLARK
default_build_file_syntax = PYTHON_DSL
to use the Python DSL parser by default.
Because Skylark will eventually become the default, we recommended that you migrate as soon as possible. To make that easier, Buck gives you control over which parser it uses for individual build files. If you add
# BUILD FILE SYNTAX: SKYLARK
as the first line of a build file, Buck uses the Skylark parser. If instead, you add
# BUILD FILE SYNTAX: PYTHON_DSL
Buck uses the Python DSL parser. If neither of these lines is present, Buck uses the parser specified in the
[parser] section of
It's best to enable the Skylark parser globally in
.buckconfig and add
# BUILD FILE SYNTAX: PYTHON_DSL
to any build files that continue to rely on Python DSL features.
Note: All of the options above require you to enable support for multi-language (polyglot) parsing:
[parser] polyglot_parsing_enabled = true
Migrating from Python to Skylark
The Skylark language was specifically created to address the issues mentioned previously—as well as other issues—which is why Skylark will eventually replace the Python DSL as the language for build files and extension files. Unfortunately, migration cannot be fully automated, so here we describe some ways to resolve common issues when migrating from the Python DSL to Skylark.
include_defs() function is not supported in Skylark because it can contaminate the symbol table of the execution environment and make automated refactoring more difficult.
To replace an invocation such as
- find all symbols defined in the
my_macro.bzlfile that are actually used by the including file, say, for example,
- replace the
include_defsinvocation with an equivalent
load()invocation that explicitly imports the needed symbols:
load("//tools:my_macro.bzl", "foo", "bar")
export_file(name="my_macro.bzl")were defined in a
toolspackage build file. This means that instead of using the
//package/extension.bzlsyntax expected by
include_defs(), you should use the
//package:extension.bzlsyntax expected by
For Skylark, replace environment variables with equivalent configuration variables. The implicit nature of environment variables frequently results in non-reproducible builds because of differences in the values of environment variables across machines.
For example, in your build file or extension file, instead of
my_var = py_sdk.os.env.get('MY_VAR', 'foo')
my_var = read_config('my_project', 'my_var', 'foo')
Then, when calling Buck, instead of passing
export MY_VAR='some_value' buck <args>
pass a configuration flag
buck <args> --config my_project.my_var=foo
or better yet, define these configuration values in your
Note: When using the Python DSL parser it's possible to invoke the
read_config() function directly during extension-file evaluation or indirectly through other function invocations. Indirect invocation of
read_config() is not supported with the Skylark parser in order to track the use of configuration options more precisely. Because of this, a top-level invocation of
read_config() such as:
bar = read_config(<args>)
either has to be performed in a build file directly or, preferably, moved into a descriptively-named function within an extension file. In the case where configuration options are used to instantiate expensive objects which should be created only once, consider replacing a top-level invocation such as
FOO = expensive1() if read_config(<args>) else expensive2()
with something like
_EXPENSIVE1 = expensive1() _EXPENSIVE2 = expensive2() def foo(): return _EXPENSIVE1 if read_config(<args>) else _EXPENSIVE2
While it can result in the instantiation of an unnecessary and expensive object, it might still be more efficient than instantiating one of the expensive objects during each
foo invocation. Having said that, we recommend that you start simply and optimize only if you notice performance issues.
isinstance() function is not available in Skylark because Skylark does not support inheritance. However, some usages of
isinstance() can be replaced with the
type function. For example,
can be replaced with
type(foo) == type('str')
In Skylark, we've replaced the
get_base_path()function with the equivalent—but more appropriately named—package_name() function. Note that in build files, it's invoked as
package_name(), but in extension files, it's invoked as
native.package_name(). Using the
native prefix is consistent with the rest of the built-in functions provided by Buck. If there is a strong desire to use the old name instead, you can assign the new function to the legacy function name:
get_base_path = native.package_name get_base_path()
del arr and
del dictionary['key'] are not supported. Instead, use
arr_val = arr.pop(1) dict_val = dictionary.pop('key')
Classes are not supported. Replace classes with a combination of structs and functions. In addition to being simpler, structs are more memory efficient. For example, a class such as
class Foo: def __init__(self, foo, bar=None): ... def some_method(self, param): ... ... foo = Foo('foo', bar='yo') res = foo.some_method(some_param)
can be replaced with something such as
def some_function(foo_instance, param): ... foo = struct(foo='foo', bar='yo') res = some_function(foo, some_param)
You can also track state in variables defined in the same extension file, but you cannot expose any mutators, since all variables are immutable once the extension file is evaluated. This is intentional and prevents race conditions because build files as well as extension files must support efficient parallel evaluation.
Regular expressions (import re)
Regular expressions are not supported in Skylark due to unbounded runtime and resource usage, but often regular expressions can be replaced with string functions.
Example: Match characters at the end of a string
Example: Match characters at the beginning of a string
Example: Match characters at both the beginning and end of a string
"some_text" in foo
Raising and catching exceptions is not supported. Instead, use the
fail function to stop the evaluation of a build or extension file, and report an error.
For example, instead of
raise Exception("attribute_name: foo")
Since usage of
fail triggers non-recoverable errors and halts parsing, it cannot be used for control flow.
While loops are not supported due to unbounded runtime. Instead, use a
for loop with a bounded range. Usage of
while True: ...
should be replaced with
for _ in range(<reasonable limit>):
followed by an extra check after the loop to make sure the loop terminated before all the iterations were exhausted.
Python modules cannot be imported in Skylark. However, you can replace many safe Python functions with analogous functions from Skylib. For example, you can replace
paths.basename, and you can replace
In order to use Skylib, clone its repository from GitHub into a local directory. Then, configure that repository as a Buck cell by adding
[repositories] bazel_skylib = path/to/skylib_checkout
Load the functions that you need, using
load(). For example,
Here is an example from the Skylib website:
load("@bazel_skylib//:lib.bzl", "paths", "shell") p = paths.basename("foo.bar") s = shell.quote(p)
Consider running the Skylint linting tool on your extension (
.bzl) files. Skylint can catch many common issues and suggest fixes. Unfortunately, some constructs in Python can cause Skylint to crash. Some examples are:
- Nested functions. Nested functions should be moved to top-level scope.
- Usage of
not foo in. You should instead use
foo not in. Note that
foo not inis recommended by the flake8 style-enforcement tool.
When debugging an issue, an effective strategy is to bisect your code by commenting out parts of the file and rerunning Skylint.
Testing your changes
The easiest way to verify that your changes have not affected your build rules is by checking if the corresponding rule keys have changed. Before making your changes, capture rule keys with the following command:
buck targets --show-rulekey //path/to/targets/... > before
After making your changes, run the command again, redirecting to a different file.
buck targets --show-rulekey //path/to/targets/... > after
Now that you have captured the before and after rule keys, use the following command to compare them:
diff before after
There should be no differences unless your changes affected the semantics of some of the build definitions or macros. In order to get more insight into what exactly has changed, you can use the command
buck audit rules path/to/BUCK command
on individual build files to see how Buck expanded the macros in them.