a fast, scalable, multi-language and extensible build system

Bazel

Last update: Jan 4, 2023

Overview

Bazel

{Fast, Correct} - Choose two

Build and test software of any size, quickly and reliably.

Speed up your builds and tests: Bazel rebuilds only what is necessary. With advanced local and distributed caching, optimized dependency analysis and parallel execution, you get fast and incremental builds.
One tool, multiple languages: Build and test Java, C++, Android, iOS, Go, and a wide variety of other language platforms. Bazel runs on Windows, macOS, and Linux.
Scalable: Bazel helps you scale your organization, codebase, and continuous integration solution. It handles codebases of any size, in multiple repositories or a huge monorepo.
Extensible to your needs: Easily add support for new languages and platforms with Bazel's familiar extension language. Share and re-use language rules written by the growing Bazel community.

Getting Started

Install Bazel
Get started with Bazel
Follow our tutorials:
- Build C++
- Build Java
- Android
- iOS

Documentation

Contributing to Bazel

See CONTRIBUTING.md

Comments

Initial attempt at direct s3 remote cache
This is an initial attempt to support #4844

It directly supports S3 as a simple remote cache. After spending some time looking at how AWS credentials are done it looks like its easier to use the SDK, instead of hand rolling the auth directly.

Since the SDK provides the backbone for auth support, it makes sense to include the S3 parts of the sdk as well.

In this PR we use the S3 SDK directly to provide S3 remote cache support. This support should be on par with the google datastore support, and has similar options in its usage.

Thoughts?

TODO

[x] Remove requirement on S3 SDK

[x] Remove external jar requirements

[x] Chase down SSL failures (not related to S3 I think) (was #4944)

[x] Chase down connection failures (not related to S3 I think) (was #4944)

[x] Make work correctly with S3-Accelerator

[x] Make Region crash more user friendly

[x] Real world test

[x] Documentation

[x] Allow for credential profiles to be configured directly

[x] I want to take a bit more work at AwsHttpAuthAdapter as its constructor is messy for testing

[x] Unit tests for changes

[x] EC2 instance credentials test

[x] Figure out how to support security tokens

[x] Tidy up command line options (could use some help here as to what the bazel team want the options to look like)

[x] Replace the ThreadLocal date formatters with javax.time ones

[x] Stare at, and maybe profile the Aws auth code for inefficiencies

[x] Make sure the remote documentation makes sense for the option

[x] Solve the terrible, slow, horrible and unstable downloads ~~~this patch~~~ (was #4944) causes

cla: yes
opened by GregBowyer 189
Release 4.2 - June 2021
Status of Bazel 4.2

This release will use Bazel 4.1.0 as its baseline and we will apply selected cherry-picks and backports on top of it. Please request cherry-picks that you'd like to get into Bazel 4.2.0 here via a comment.

Expected release date: June 2021

List of release blockers

To report a release-blocking bug, please file a bug using the Release blocker label, and cc me.

Task list:

[x] Pick release baseline: https://github.com/bazelbuild/bazel/commit/c4975efdd7ede7b46637bf353209d9ac371181a5

[x] Create release candidate: https://releases.bazel.build/4.2.0/rc1/

[x] Check downstream projects: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2105

[x] Create draft release announcement

[x] Send for review the release announcement PR: https://github.com/bazelbuild/bazel-blog/pull/258

[x] Push the release, notify package maintainers: ...

[x] Update the documentation

[x] Push the blog post

[x] Update the release page

P1 type: process release team-OSS
opened by philwo 124

failed to install bazel on Red Hat 6.7

I would like to install bazel from source, and use bazel to compile tensorflow on a cluster running redhat 6.7. When I try to install bazel, the glibc version (2.12) is too old. I do not have root access to the cluster. Is it possible to install tensorflow in this case?

My system information:

-bash-4.1$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.7 (Santiago)
-bash-4.1$ which gcc
/usr/bin/gcc
-bash-4.1$ gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) 
-bash-4.1$ ldd --version
ldd (GNU libc) 2.12

The system has newer gcc installed as well. I tried using it, bazel still won't compile.

-bash-4.1$ /usr/local/gcc/4.8.4/bin/gcc -v
Using built-in specs.
COLLECT_GCC=/usr/local/gcc/4.8.4/bin/gcc
COLLECT_LTO_WRAPPER=/usr/local/gcc/4.8.4/libexec/gcc/x86_64-unknown-linux-gnu/4.8.4/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ../configure --prefix=/usr/local/gcc/4.8.4
Thread model: posix
gcc version 4.8.4 (GCC)

When I was compiling bazel using gcc 4.8.4, I got the following error:

bazel-0.1.1/_bin/build-runfiles: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found

How can I install the missing dependency locally, and have bazel pick up the right version of glib? What part of tools/cpp/CROSSTOOL should I modify? What environment variables should I set before ./compile.sh

I am trying to install bazel-0.1.1 instead of bazel-0.1.2 because the latter does not compile tensorflow. https://github.com/tensorflow/tensorflow/issues/469

type: bug P2 platform: other team-Configurability

opened by digitalsword 110

Release 4.1 - May 2021
Status of Bazel 4.1

This release will use Bazel 4.0.0 as its baseline and we will apply selected cherry-picks and backports on top of it. Please request cherry-picks that you'd like to get into Bazel 4.1.0 here via a comment.

Expected release date: March 2021

List of release blockers

To report a release-blocking bug, please file a bug using the Release blocker label, and cc me.

Task list:

[x] Pick release baseline: https://github.com/bazelbuild/bazel/commit/6b33bdb1e22514304c0e35ce8e067f2175685245

[x] Create release candidate: https://releases.bazel.build/4.1.0/rc1/

[x] Check downstream projects: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2040

[x] Create draft release announcement

[x] Send for review the release announcement PR: https://github.com/bazelbuild/bazel-blog/pull/256

[x] Push the release, notify package maintainers: @vbatts @petemounce @excitoon

[x] Update the documentation

[ ] Push the blog post

[ ] Update the release page

P1 type: process release team-OSS
opened by philwo 107
java rules should be able to depend on Skylark rules

See:

https://github.com/bazelbuild/rules_scala/blob/master/test/BUILD#L15

The java libraries should probe to see if there are jars, transitive_runtime deps, etc...
type: feature request P1

opened by johnynek 105
Release 4.0 - December 2020
Status of Bazel 4.0

Target baseline: 2020-11-12

Expected release date: 2021-01-18

List of release blockers

To report a release-blocking bug, please file a bug using the Release blocker label, and cc me.

Task list:

[x] Update GitHub issues for incompatible changes

[x] Pick release baseline: 37a429ad12b4c9e6a62dbae4881a1ff03b81ab40

[x] Create release candidate: https://releases.bazel.build/4.0.0/rc10/

[x] Check downstream projects:

[x] Create draft release announcement

[x] Send for review the release announcement PR:

[x] Push the release, notify package maintainers:

[x] Update the documentation

[x] Push the blog post

[x] Update the release page

P1 type: process release team-OSS
opened by philwo 98
kotlin_rules and Ijars: Update the Ijar tooling to do the right thing.
I have been working on a new set of kotlin rules and they are currently being migrated to bazelbuild/rules_kotlin.

One goal with the design of the Kotlin JVM ruleset is to stay as close to the native java rules and infrastructure as possible. The automatic Ijars generation is problematic as the java rules auto create ijars. Ijars generated for Kotlin work in a lot of cases, but inlined methods break (might be other cases).

Proposal: Adapt Ijar to do the right thing for Kotlin Jars.

I propose making a change to the behaviour of the ijar tool so that it does the right thing when it comes to Kotlin metadata. Adding this change to Bazel core simplifies the Kotlin rules and allows more reuse out of java common infrastructure. I propose the ijar tool be updated to:

Initially to detect a Kotlin jar and ignore it for processing -- perhaps just symlink the ijar to the full jar ? This should be straight forward a Kotlin jar has some markers that can be identified without processing bytecode.

Adapt the tool to generate valid Kotlin Ijars. Afaik this means retaining certain bytecode elements and marker files.

Motivating example 1: dep management tooling / java_import.

I have just added a kotlin_import rule to:

Get around the automatic ijar generation by java_import, and

Ensure the Jar has a KotlinInfo provider so that full_compile_jars are always selected when compiling Kotlin.

Any fancy java dependency management tooling that we develop will likely work via orchestrations of java_import, java_library and java_import_external it's a shame to have to add a layer for Kotlin when it's barely needed.
type: feature request P3 team-Rules-Java
opened by hsyed 93
$(location) should use runtime path when used in jvm_flags attribute
We have a binary which requires the javac.jar be present in its bootclasspath. Prior to bazel 0.4.4, the following invocation worked (but was obviously less than ideal):

jvm_flags = [ "-Xbootclasspath/p:$$JAVA_RUNFILES/bazel_tools/third_party/java/jdk/langtools/javac.jar", ],

With bazel 0.4.4 the file name specific path has broken, as the actual file name of the target is now javac-9-dev-r3297-1.jar.

Ideally, we'd be able to use $(location @bazel_tools//third_party/java/jdk/langtools:javac_jar), however that expands to external/bazel_tools/third_party/java/jdk/langtools/javac-9-dev-r3297-1.jar which is the build path and not the runtime path for that file. Needless to say, I'd like to avoid hard-coding another path which will break or fragile inline path-munging hacks. It seems like the correct answer is for $(location) to use the runtime path when specified in jvm_flags.
type: feature request P1
opened by shahms 88
tensorflow building issue on ubuntu 14.04: Accessing remote repository through a proxy

As (https://github.com/tensorflow/tensorflow/issues/67) is declared as Bazel issue I'm writing here. So I'm having the same issue with the original post. As mentioned in the original post, by manually wget and placing the jpeg library the error message about jpeg has gone. However there are still many error messages, and by trying multiple times the error message seems to change but always fails with downloading something. These are the variations.

Variation 1: ERROR: /home/name/work/tensorflow/tensorflow/core/platform/default/build_config/BUILD:36:1: no such package '@re2//': https://github.com/google/re2.git: cannot open git-upload-pack and referenced by '//tensorflow/core/platform/default/build_config:platformlib'.

Variation 2: ERROR: /home/name/work/tensorflow/tensorflow/core/platform/default/build_config/BUILD:36:1: no such package '@png_archive//': Error downloading from https://storage.googleapis.com/libpng-public-archive/libpng-1.2.53.tar.gz to /home/name/.cache/bazel/_bazel_name/894ee6ce07893c5c665fe600d592867f/external/png_archive: Error downloading https://storage.googleapis.com/libpng-public-archive/libpng-1.2.53.tar.gz to /home/name/.cache/bazel/_bazel_name/894ee6ce07893c5c665fe600d592867f/external/png_archive/libpng-1.2.53.tar.gz: storage.googleapis.com and referenced by '//tensorflow/core/platform/default/build_config:platformlib'.

Variation 3: ERROR: /home/name/work/tensorflow/tensorflow/tensorflow.bzl:224:3: no such package '@gemmlowp//': https://github.com/google/gemmlowp.git: cannot open git-upload-pack and referenced by '//tensorflow/core:kernels'.
type: feature request P2

opened by lacartelacarte 87
undocumented usage of perl for cc_test
Description of the problem / feature request:

cc_test uses a inline perl script for failed tests. https://github.com/bazelbuild/bazel/blob/eb067ea88749a5635cc8ee8954cde2b767f1eb61/tools/test/test-setup.sh#L153

Feature requests: what underlying problem are you trying to solve with this feature?

The usage of perl is not documented. Windows does not have Perl installed by default.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

execute a failing test on windows will print:

==================== Test output for //Test/Unit:Unit (shard 1 of 8): [...] external/bazel_tools/tools/test/test-setup.sh: line 152: perl: command not found ================================================================================

What operating system are you running Bazel on?

Windows 10

What's the output of bazel info release?

0.10.1

The perl script replaces invalid XML characters and invalid sequence in CDATA. To get rid of perl bash or python could be used.
type: bug P1 platform: windows
opened by ahippler 86
Support coverage

We've gotten a number of requests to support coverage. Here's an email I wrote in response to one of those requests, which I'm reposting here as a feature request:

Ideally, we'd want blaze coverage to exist and work in Bazel, just as it does in Blaze. The code is mostly there, but not quite. The coverage command itself doesn't actually do a lot of work; it sets --collect_code_coverage and a default --instrumentation_filter, and it optionally runs a tool to generate an HTML report (which is almost never used internally, where we push to a service instead). Otherwise it's identical to the test command, and the two flags can be manually set.

Internally, we have a wrapper script (collect_coverage.sh) that wraps an individual test invocation and uses a tool (lcov_merger.par) to merge the coverage data from that invocation, which both aren't in Bazel, and Bazel also doesn't setup the environment correctly to tell the wrapper script about output locations (StandaloneTestStrategy.java).

If you actually tried, you'd get an error like this one: ERROR: /usr/local/google/home/ulfjack/Projects/os-bazel/src/test/cpp/BUILD:5:1: output 'src/test/cpp/blaze_util_test/coverage.dat' was not created.

It doesn't seem very hard to make the necessary Bazel changes, but we can't export the scripts as-is.

From my reading of the lcov documentation, lcov --capture looks a lot like what lcov_merger.par does, though I can't be completely certain (the docs are a bit scarce). It may make sense for us to integrate with lcov rather than export lcov_merger.par (or rewrite from scratch).
type: feature request P1 coverage

opened by ulfjack 77
[bazel.build] Problem with /docs/user-manual: would be nice to document how the output of --workspace_status_command can be used

It would be nice to have a self-contained example of --workspace_status_command use.

While the page describes how to make a script, it does not even link to a place that explains how the resulting output can be used.

opened by filmil 0
ActionMetadataHandler.constructTreeArtifactValueFromFilesystem() causes files to be considered modified
Description of the bug:

We are using --experimental_guard_against_concurrent_changes and producing a TreeArtifact from a directory where files have 0755 permissions. When Bazel builds this target, it executes our code (producing files with 0755 permissions), but then https://source.corp.google.com/piper///depot/google3/third_party/bazel/src/main/java/com/google/devtools/build/lib/skyframe/ActionMetadataHandler.java;l=347 iterates through the TreeArtifact and chmods each file to 0555 permissions. The permission change doesn't cause us problems, but this updates the ctime for those files, so when RemoteSpawnCache.checkForConcurrentModifications() confirms that no concurrent modifications were made, it flags those ctime changes and refuses to upload the artifacts to the remote cache.

My observations of this behavior came from using the --host_jvm_debug flag when invoking Bazel, and setting breakpoints within the code from the 6.0.0 tag.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

gLinux

What is the output of bazel info release?

release 6.0.0-pre.20221012.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

https://github.com/bazelbuild/bazel.git f2cdf9e65eadfb5365127eabcc72dedce7781405 06deebfb5b73f848de5a0ea0e00fcfaa26788d1f

Have you found anything relevant by searching the web?

https://github.com/bazelbuild/bazel/issues/5566 describes other instances of fragility with this flag, including an example from djmarcin for symlinks in tree artifact output, but nothing about file permissions for tree artifacts.

Any other information, logs, or outputs that you want to share?

No response
opened by tbaing 0
Release X.Y.Z - $MONTH $YEAR
Status of Bazel X.Y.Z

Target baseline: [date]

Expected release date: [date]

List of release blockers

To report a release-blocking bug, please add a comment with the text @bazel-io flag to the issue. A release manager will triage it and add it to the milestone.

To cherry-pick a mainline commit into X.Y.Z, simply send a PR against the release-X.Y.Z branch.

Task list:

[ ] Pick release baseline: [link to base commit]

[ ] Create release candidate: X.Y.Zrc1

[ ] Check downstream projects

[ ] Create draft release announcement

[ ] Send the release announcement PR for review: [link to bazel-blog PR]

[ ] Push the release and notify package maintainers: [link to comment notifying package maintainers]

[ ] Update the documentation

[ ] Push the blog post: [link to blog post]

[ ] Update the release page

P1 type: process release team-OSS
opened by Pandabar007 0
Do not count tests as failed that have not started

Before this commit, when the user interrupted a test run, tests that had not started until that point were counted as failed, whereas tests that were already running were counted as skipped.

With this commmit, both types of tests are counted as skipped.
team-Performance awaiting-review

opened by fmeum 1
[docs] Fix link to rules_apple tutorial

Looks like a search and replace went wrong in https://github.com/bazelbuild/bazel/commit/0c4cf501c83df845a1c9bf70fbaca955b0f51a4c. This is an external link, so it shouldn't have been changed, since it was a 404 now.
awaiting-PR-merge

opened by BalestraPatrick 0