User-level statically defined tracing probes have been placed in various applications and runtimes, including Java, Node.js, MySQL, and PostgreSQL. These allow API-stable scripts to be written, that do not depend on tracing raw user-level functions (uprobes).

As an example of hacking in USDT tracing using ftrace, see: http://www.brendangregg.com/blog/2015-07-03/hacking-linux-usdt-ftrace.html . The unpublished script I referred to is: https://gist.github.com/brendangregg/f1b3d09c14088522065b

For a simple example to trace:

1. Create tick-dtrace.d:

provider tick {
        probe loop(int);
}

#pragma D attributes Evolving/Evolving/ISA provider node provider
#pragma D attributes Private/Private/Unknown provider node module
#pragma D attributes Private/Private/Unknown provider node function
#pragma D attributes Private/Private/ISA provider node name
#pragma D attributes Evolving/Evolving/ISA provider node args

2. Then create an object file:

# apt-get install -y systemtap-sdt-dev        # adds "dtrace"
# dtrace -G -s tick-dtrace.d -o tick-dtrace.o

3. Create the target program, tick-main.c:

#include <stdio.h>
#include <unistd.h>
/* from systemtap-sdt */
#include <sys/sdt.h>

int
main(int argc, char *argv[])
{
        int i;

        for (i = 0; i < 5; i++) {
                DTRACE_PROBE1(tick, loop, i);
                printf("hi: %d\n", i);
                sleep(1);
        }

        return (0);
}

4. Compile tick-main:

gcc -c tick-main.c
gcc -o tick tick-main.o tick-dtrace.o

5. Check it has USDT probes:

readelf -n tick

Notes at offset 0x0000021c with length 0x00000020:
  Owner                 Data size       Description
  GNU                  0x00000010       NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 2.6.35

Notes at offset 0x0000023c with length 0x00000024:
  Owner                 Data size       Description
  GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 2b94c0e7c684a001a34d685c862b33aa51ff7672

Notes at offset 0x00000a08 with length 0x00000044:
  Owner                 Data size       Description
  stapsdt              0x0000002e       NT_STAPSDT (SystemTap probe descriptors)
    Provider: tick
    Name: loop
    Location: 0x0000000000400558, Base: 0x0000000000400628, Semaphore: 0x0000000000000000
    Arguments: -4@-4(%rbp)

See NT_STAPSDT etc.

This is a basic probe. There is another type, isenabled, which I discussed in the blog post, and requires a semaphore to activate.

Overview

These patches add features that allow BCC to support cross-development workflows where the development machine and the target machine running the BPF program are different.

This is achieved by integrating the BPFd (BPF Daemon) project into BCC. BPFd is a standalone executable that can be loaded onto a remote target device, and which then can act as a proxy for BCC for whenever BCC wishes to perform an operation on the system (e.g. load BPF programs, read /proc/kallsyms, attach kprobes, etc.).

Through this arrangement, BCC can be used to profile a remote target device while it mostly runs on a separate host machine. The advantage of this arrangement is that it removes the need to have kernel sources and the LLVM stack on the target machine. These can, instead, be kept on the host. This arrangement therefore reduces the space required on a target for BCC tools to run, which is a key benefit for devices that have a more limited disk space (e.g. embedded devices).

In addition, the above set-up also allows embedded developers who use a cross-compiler in their workflow to run clang on a different architecture than the target's architecture, thus facilitating cross-compilation development.

For more information, please check out this LWN article which explains the purpose of BPFd and how it works in more detail.

Integration of BPFd sources into BCC

These patches add the sources for the BPFd executable into the BCC source tree. This is done to ensure that BCC and BPFd remain compatible with each other.

BPFd depends on existing BCC components such as libbpf.c and bcc_syms.cc to function. However, the converse is also true: BCC makes calls to BPFd via BCC's Python interface. The Python interface is the main way by which communication happens between BCC and BPFd, and so any changes there could break interoperability. As a result, it is not sufficient for BPFd to just use libbcc and remain independent like other projects (e.g. bpftrace).

Therefore, to ensure that BCC and BPFd are always compatible with each other, it is more feasible to keep them in the same tree instead of keeping them separate. This is also why these patches come with smoke tests which ensure that the interoperability between BCC and BPFd isn't broken silently when changes are made to either.

Tools

The tools that currently work for remote devices with these patches are as follows:

• Biolatency
• Biosnoop
• Biotop
• Cachestat
• Cachetop
• Filetop
• Hardirqs
• Offcputime
• Opensnoop
• Profile
• Runqlen
• Stackcount
• Syscount
• Trace

Reviewed-by: Joel Fernandes ([email protected])

I understand that adding kernel tracepoint support in BPF is going to take a while. In the meantime, would it be possible to put together a tool that relies on kprobes? What I had in mind was to enable the tracepoint using ftrace and then place a kprobe in the trace_event_raw_event_* function or in trace_event_buffer_commit, or some similar location. The kprobe would have access to the entry structure defined by TRACE_EVENT.

Is that a direction worth pursuing? Would it be possible (for efficiency) to instruct the ftrace infrastructure to discard the event and not even put it in the ftrace buffer?

I recently ran into a problem when trying to attach to a USDT in a shared library. Specifically, the shared library was loaded into the address space of a chrooted process. If I just provide the pid of this process, initializing USDT object fails because all paths in /proc/[pid]/maps for the target are relative to its chroot, while the process I'm trying to attach from is not in a chroot and therefore the paths don't make sense and USDT initialization can't look through shared lib ELFs to find the right probe.

Similarly, if I try to initialize the USDT object with pid and bin_path to the shared lib, initialization succeeds but actually attaching fails: address lookup for the usdt's semaphore in pid's address space fails because /proc/[pid]/maps shared lib path is expected to match provided bin_path exactly, but maps paths are chroot-relative so this doesn't work.

Implementation evolved significantly since this PR was created, look at this comment for details about impl.

This isn't necessarily a bug in bcc per se, more a write-up of what doesn't work and why - hopefully it will at least help others from going down the rabbit hole I did.

There may be a way for bcc to fix this by finding another way to increment the semaphore, but I don't really see how.

In the chromium kernel source code, here is code in fs/proc/base.c that prevents processes from writing to their own memory maps for security reasons:

static ssize_t mem_write(struct file *file, const char __user *buf,
       size_t count, loff_t *ppos)
{
#ifdef CONFIG_SECURITY_CHROMIUMOS_READONLY_PROC_SELF_MEM
  return -EACCES;
#else
  return mem_rw(file, (char __user*)buf, count, ppos, 1);
#endif
}

This variable is enabled by default in the kernel used by container OS, such as in google's GKE offering: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master/overlay-lakitu/sys-kernel/lakitu-kernel-4_14/files/base.config#3016

This means that anyone trying to use bcc on a chromium derived OS, and especially anyone trying to use bcc on GKE, will probably also hit this if they try to use usdt probes.

During the process of enabling a usdt probe, some probes need to be enabled by writing to a semaphore - this isn't true of all usdt probes, but is probably true of many (I ran into this with ruby's usdt probes). As the dtrace docs indicate, this is a means to avoid expensive processing around the probe, only adding this extra info/processing if the probe is enabled.

The code that handles this in bcc is here: https://github.com/iovisor/bcc/blob/c2e2a26b8624492018a14d5eebd4a50b869c911f/src/cc/usdt/usdt.cc#L109-L113

And it is essentially the same as the approach described here.

However, this leads to probes silently failing to be enabled if run against a kernel with the above hardening. Using strace, it is obvious why it fails:

openat(AT_FDCWD, "/proc/726288/mem", O_RDWR) = 72
lseek(72, 94200854600568, SEEK_SET)     = 94200854600568
read(72, "\0\0", 2)                     = 2
lseek(72, 94200854600568, SEEK_SET)     = 94200854600568
write(72, "\1\0", 2)                    = -1 EACCES (Permission denied)
close(72)                               = 0

Note that this only will happen for probes where readelf --notes indicates a value for the sempahore:

  stapsdt              0x00000059       NT_STAPSDT (SystemTap probe descriptors)
    Provider: ruby
    Name: cmethod__entry
    Location: 0x000000000019999d, Base: 0x00000000002d8ec0, Semaphore: 0x000000000052bb54
    Arguments: 8@32(%rsp) 8@40(%rsp) 8@48(%rsp) -4@56(%rsp)

As there actually is a sempahore indicated here, this USDT probe would be affected. A similar probe in libc would not be affected and can be attached to as the semaphore is not required for the "enable" mechanism:

  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: libc
    Name: memory_heap_free
    Location: 0x000000000019bfd0, Base: 0x00000000001bdd48, Semaphore: 0x0000000000000000
    Arguments: 8@%r11 8@%rax

 # ./tools/trace.py t:syscalls:sys_enter_newfstat
Ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
Failed to attach BPF to tracepoint

Here's newfstat.py:

#!/usr/bin/python

from __future__ import print_function
from bcc import BPF

# load BPF program
b = BPF(text="""
TRACEPOINT_PROBE(syscalls, sys_enter_newfstat) {
    bpf_trace_printk("%d\\n", args->fd);
    return 0;
};
""", debug=0)

# header
print("%-18s %-16s %-6s %s" % ("TIME(s)", "COMM", "PID", "FD"))

# format output
while 1:
    try:
        (task, pid, cpu, flags, ts, msg) = b.trace_fields()
    except ValueError:
        continue
    print("%-18.9f %-16s %-6d %s" % (ts, task, pid, msg))

output:

# ./newfstat.py 
ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
Traceback (most recent call last):
  File "./newfstat.py", line 12, in <module>
    """, debug=0)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 212, in __init__
    self._trace_autoload()
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 767, in _trace_autoload
    self.attach_tracepoint(tp=tp, fn_name=fn.name)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 585, in attach_tracepoint
    raise Exception("Failed to attach BPF to tracepoint")
Exception: Failed to attach BPF to tracepoint

Testing on 4.8-rc4. Same problem with other syscall tracepoints.

I can use kprobes as a workaround in the meantime, but this should be a nice example of using tracepoints...

Scripts should never assume what codetable an external user has and try to decode. Doing that has huge potential to cause pain and misery for users (a common python3 sickness nowadays). A better option is for python3-users to live with b'' syntax or provide an option --decode=xxx to each script.

Revert "Python 3 compatibility fixes around string handling (#986)" This reverts commit 78948e4aae6aa0d06806d452d193320936d59dc7.

The offending patch produces the following test failure on Jessie with backports 4.9 kernel:

Traceback (most recent call last): File "../../tools/slabratetop.py", line 123, in print("%-32s %6d %10d" % (k.name.decode(), v.count, v.size)) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc1 in position 2: ordinal not in range(128)

I am trying to run bcc on openSUSE Linux and there is a problem with the kernel header path it seems:

> ./hello_world.py
In file included from <built-in>:316:
<command line>:2:10: fatal error: './include/linux/kconfig.h' file not found
#include "./include/linux/kconfig.h"
         ^
1 error generated.
Traceback (most recent call last):
  File "./hello_world.py", line 11, in <module>
    BPF(text='void kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); }').trace_print()
  File "/usr/local/lib/python2.7/site-packages/bcc/__init__.py", line 126, in __init__
    raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module

The file exists at /usr/src/linux/include/linux/kconfig.h , I am wondering if there is a special setup I should be doing before using bcc.

gentrace.py attaches to functions and traces their parameter values into a histogram or a raw data collection. A simple command-line syntax specifies the probe point, the parameter values to capture, and whether a histogram or raw data is desired.

From the examples -- print a histogram of buffer sizes passed to write() across all processes, where the file descriptor was 1 (STDOUT):

# ./gentrace.py "hist:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1"
[11:25:28]
hist:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1
     count               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 1        |****************************************|
        16 -> 31         : 0        |                                        |
        32 -> 63         : 0        |                                        |
        64 -> 127        : 1        |****************************************|
[11:25:29]
hist:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1
     count               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 2        |********                                |
        16 -> 31         : 0        |                                        |
        32 -> 63         : 1        |****                                    |
        64 -> 127        : 9        |****************************************|
[11:25:30]
hist:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1
     count               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 3        |*******                                 |
        16 -> 31         : 0        |                                        |
        32 -> 63         : 2        |****                                    |
        64 -> 127        : 17       |****************************************|

When running tools/funccount.py inside a container, we discovered that no func counting actually happened.

Further investigation shows the reason is due to the pid filtering like below:

    u32 pid = bpf_get_current_pid_tgid() >> 32;
    if (pid != <pid_arg_passed_in>) { return 0; }

The <pid_arg_passed_in> is typically the pid of a process to be traced inside the container. On the other hand, bpf_get_current_pid_tgid() in the kernel will get the pid outside the container. Such a mismatch will cause pid != <pid_arg_passed_in> evaluated to be true, and bpf program returns 0 prematurely.

Most of our bcc tools have pid based filtering. That means most of them will not run properly inside the container.

How to resolve the issue? I can think of two options: (1). Have one more option in each of the tools which have pid as its arguments. This option will specify host pid, the program will take this pid in the bpf pid filtering if it is available. Users can get the host pid by using ps command on the host. (2). Invent new kernel helpers to: . helper 1: get namespace id of the current process . helper 2: pass in the process id and namespace id and get back true if the the current process is the one having the same <ns_id, pid_inside_ns>.

Any comments or suggestions? cc @brendangregg @goldshtn @4ast

Run BCC test suite with github actions

With this commit, pushes to branches can trigger tests directly on
Github Actions.

Here it will be able to test against kernel 4.14 and 5.0.0, which correspond
to the latest Ubuntu LTS kernel releases.

Tests are run for both Debug and Release modes.

Tests run inside docker, in an ubuntu 19.04 container base.

For the github workflow:

- The test_libbcc suite is run first, to potentially fail fast on these faster
unit tests.
- The Python test suite is run

Some of these tests are allowed to fail, but the failure is still reported:

- In catch2 tests, using the [!mayfail] tag, where this will be displayed in
the test summary
- In Python unittests, using the `@mayFail("Reason")` decorator, which is
introduce for consistency with catch2. These will report the reason for the
failure, and log this to an artifact of the test.

test_libbcc

Currently this appears to actually be running more test cases than on jenkins:

test cases:   37 |   35 passed | 2 failed
assertions: 2414 | 2412 passed | 2 failed

But on jenkins for libbcc I see:

3: test cases:  37 |  36 passed | 1 failed
3: assertions: 626 | 624 passed | 2 failed

Python tests

There are a few python tests that needed to be skipped for now with @mayFail. Fixing these I think is outside of the scope of adding github actions, and these failures seem to be legitimate.

These failures are summarized as part of the action run, and the logs are explicitly added as an artifact. An issue should be cut to fix these up once this is merged.

Since kernel commit fcb14cb1bdac("new iov_iter flavour - ITER_UBUF"), tty_write() will use ITER_UBUF. If do not add support for ITER_UBUF, ttysnoop will not work

I have linking problem if I want to build the tree after installing the described dependencies. I have ubuntu 20.04 installed and some dependencies failed ;-( E: Unable to locate package libllvm14 E: Unable to locate package llvm-14-dev E: Unable to locate package libclang-14-dev ... E: Unable to locate package libllvm3.7 E: Couldn't find any package by glob 'libllvm3.7' E: Couldn't find any package by regex 'libllvm3.7' E: Unable to locate package llvm-3.7-dev E: Couldn't find any package by glob 'llvm-3.7-dev' E: Couldn't find any package by regex 'llvm-3.7-dev' E: Unable to locate package libclang-3.7-dev E: Couldn't find any package by glob 'libclang-3.7-dev' E: Couldn't find any package by regex 'libclang-3.7-dev'

When I try to build the tools I get the following error: [ 36%] Built target bcc_py_python3 [ 36%] Built target bcc-lua [ 36%] Built target bps [ 36%] Linking CXX executable CGroupTest /usr/bin/ld: /usr/lib/llvm-10/lib/libclangCodeGen.a(BackendUtil.cpp.o): in function (anonymous namespace)::EmitAssemblyHelper::EmitAssemblyWithNewPassManager(clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >)': (.text._ZN12_GLOBAL__N_118EmitAssemblyHelper30EmitAssemblyWithNewPassManagerEN5clang13BackendActionESt10unique_ptrIN4llvm17raw_pwrite_streamESt14default_deleteIS5_EE+0x1f15): undefined reference togetPollyPluginInfo()' collect2: error: ld returned 1 exit status make[2]: *** [examples/cpp/CMakeFiles/CGroupTest.dir/build.make:158: examples/cpp/CGroupTest] Error 1 make[1]: *** [CMakeFiles/Makefile2:1087: examples/cpp/CMakeFiles/CGroupTest.dir/all] Error 2 make: *** [Makefile:141: all] Error 2

Any Idea how I can fix that? I saw that those packages are not available for ubuntu 20.04.

Thanks in advance. BR, Erwin

Currently softirq only report the time but not the event counts. If we have both time and counts it will be better understand the frequency and duration of events.

Signed-off-by: Hailong Liu [email protected]

Hi，when i made and installed these tools in my aarch-64 OS,I got these problem: BIOTOP FILETOP TCPSTATUS ... I try to take some measures but not useful,please help me,thanks! PS1: my OS info: Linux e04-g4 5.10.110-yocto-standard #1 SMP PREEMPT Tue Jan 3 09:00:28 UTC 2023 aarch64 GNU/Linux PS2: my kernel config: `CONFIG_TASKS_TRACE_RCU=y CONFIG_BPF_SYSCALL=y CONFIG_TRACEPOINTS=y CONFIG_UPROBES=y CONFIG_BINARY_PRINTF=y

CONFIG_DEBUG_INFO_BTF=y CONFIG_STACKTRACE=y CONFIG_NOP_TRACER=y CONFIG_RING_BUFFER=y CONFIG_EVENT_TRACING=y CONFIG_CONTEXT_SWITCH_TRACER=y CONFIG_TRACING=y CONFIG_FTRACE=y CONFIG_FTRACE_SYSCALLS=y CONFIG_BRANCH_PROFILE_NONE=y CONFIG_UPROBE_EVENTS=y CONFIG_BPF_EVENTS=y CONFIG_DYNAMIC_EVENTS=y CONFIG_PROBE_EVENTS=y CONFIG_CGROUP_BPF=y CONFIG_SOCK_CGROUP_DATA=y CONFIG_BPF_LSM=y CONFIG_BPF_JIT_ALWAYS_ON=y CONFIG_USERMODE_DRIVER=y CONFIG_KPROBES=y CONFIG_KRETPROBES=y CONFIG_BPF_STREAM_PARSER=y CONFIG_STREAM_PARSER=y CONFIG_NET_SOCK_MSG=y CONFIG_KPROBE_EVENTS=y CONFIG_FUNCTION_ERROR_INJECTION=y CONFIG_DEBUG_FS=y CONFIG_KPROBES_ON_FTRACE=y CONFIG_HAVE_KPROBES_ON_FTRACE=y CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS=y CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y CONFIG_FTRACE_MCOUNT_USE_CC=y CONFIG_SECURITY=y CONFIG_SECURITY_NETWORK=y

#third CONFIG_NET_CLS_BPF=m CONFIG_NET_ACT_BPF=m CONFIG_LWTUNNEL=y CONFIG_LWTUNNEL_BPF=y

#fourth CONFIG_IKHEADERS=y CONFIG_HAVE_BPF_JIT=y CONFIG_BPF_JIT=y CONFIG_HAVE_EBPF_JIT=y CONFIG_DEBUG_INFO=y

CONFIG_NETFILTER_XT_MATCH_BPF=m CONFIG_BPF_KPROBE_OVERRIDE=y

For PIE binaries, BCC fails to attach uprobes by virtual address. To demonstrate the issue, we provide the following self contained repo: https://github.com/etep/pie-bcc-issue

Note that uprobe attach by binary address, i.e. the address as found in the ELF file by nm e.g. works fine.

We would like feedback on the following:

1. should uprobe attach support virtual address for PIE binaries?
2. if yes, should there be a separate interface, e.g.: attach_uprobe_by_virtual_address and attach_uprobe_by_binary_address?

We would be open to authoring a PR to resolve the issue, pursuant to decisions on the previous two questions. Also, would work with the BCC authors to ensure the PR is consistent with BCC coding practices and best practices.

A recent change caused arrays of multi-word types to not be detected properly leading to errors such as:

Type: 'unsigned char[256]' not recognized. Please define the data with ctypes manually

Fix the regular expression to allow multi-word types in arrays.

Fixes: ca1d3fd07a84 ("bcc: Fix array type handling due to llvm changes") Cc: [email protected] Signed-off-by: Adrian Moreno [email protected]