Capturing and timing all the method invocations is a great way of obtaining a relatively precise application performance profile. Unless you need to do that for short and frequently called methods, that is.

Profile Collection Overhead

It is obvious that capturing the performance profile would incur some additional overhead - getting and processing the timestamps, correlating the events, building up the profile, they all come at a price which must be paid in order to get some meaningful information.

Short And Frequent Methods

This overhead is usually projected to a time constant. With clever data structures and algorithms it can become a really small constant. It may sound good at first but what it really means is that the effective overhead for shorter methods is higher - eg. a short method usually running for 30ns would suddenly run for 100ns with profiling turned on (and this would be a pretty fast profiling runtime, too) meaning that the execution speed has just increased by more than 200%.

Adding insult to injury such short methods have the bad habit of being run really frequently and they performance profile is not that interesting at all - they are already as fast as they can be.

Sampling

To summarize - short, frequently repeating methods are causing the biggest relative overhead while contributing almost no additional insight into the application performance profile. In fact, we might get the same information by capturing just each n-th invocation. Grabbing just a a sample of all the method invocations.

Adaptive Sampling

The trickier part of setting up the sampilng is coming up with the appropriate sampling rate. Too high and the overhead reduction might not even be noticeable, too low and you might miss valuable information.

Adaptive sampling tries to address this problem with accepting an overhead target and then dynamically adjusting the sampling rate to meet the target. This approach guarantees that the maximum of information will be collected without exceeding te overhead target. As a bonus the system will react to changing method invocation times due to changing execution environment.

Sampling In BTrace

In the upcoming version of BTrace (1.3) it is possible to define sampling for Kind.CALL and Kind.ENTRY locations.

Fig.1 Adaptive Sampling

@BTrace public class AllCalls1Sampled {
  @OnMethod(clazz="javax.swing.JTextField", method="/.*/",
            location=@Location(value=Kind.CALL, clazz="/.*/", method="/.*/"))
  @Sampled(kind = Sampled.Sampler.Adaptive)
  public static void m(@Self Object self, @TargetMethodOrField String method, @ProbeMethodName   String probeMethod) { // all calls to the methods with signature "()"
    println(method + " in " + probeMethod);
  }
}

Fig.2 Standard Sampling

@BTrace public class AllMethodsSampled {
  @OnMethod(
      clazz="/javax\\.swing\\..*/",
      method="/.*/"
  )
  // standard sampling; capture each 20-th invocation
  @Sampled(kind = Sampled.Sampler.Const, mean = 20)
  public static void m(@Self Object o, @ProbeClassName String probeClass, @ProbeMethodName String probeMethod) {
      println("this = " + o);
      print("entered " + probeClass);
      println("." + probeMethod);
  }
}

Wrapup

Sampling seems to be a good way to achieve a balance between the amount of information a profiler is able to collect and the overhead it incurs to the profiled application. The drawback is the necessity to specify the sampling rate but it can very well be mitigated by using the adaptive variety.

StatsD

StatsD is becoming certainly an ubiquituous standard for delivering monitoring metrics. It defines a very simple, yet extremly extensible textual format for transfering metrics of various kinds. The data is usually transferred in the form of UDP packets (low latency) but other transports might be used as well.

Metrics

Metric is defined by its name and type. There are no restrictions on the names.

StatsD defines basic metrics types

• counters - can be incremented or decremented by certain delta
• gauges - reflect the last set value; no increments/decrements
• timers - used for timing data

More details here.

Servers

There are more than a few server side implementations, varying in the throughput or the amount of extensions they are adding. In general the server side takes care of collecting the metrics from one or more clients (producers) and aggregating the data to provide eg. averages, variations or smoothing out samples. All of this is happening asynchronously in relation to the metrics producers.

Clients

There might be even more StatsD client implementations than there is the server ones. In its simplest form a StatsD client will provide convenience methods to manipulate the metrics and take care of preparing the UDP packets to be sent to collectors.

BTrace Integration

The only way to collect any meaningful data in BTrace is to print it either to stdout or a file. While this is straight-forward and does not require any additional infrastructure it brings a lot of pain when trying to integrate BTrace with monitoring tools.

BTrace StatsD Client

Good news! Thanks to the nature of the StatsD metrics transfer protocol adding the StatsD client to BTrace is fairly simple and does not require adding any 3rd party libraries. It is just a matter of adding one extension class.

com.sun.btrace.services.impl.Statsd

This client also supports the extensions from DogStatsD while keeping the compatibility with the original StatsD

Using StatsD Client

@BTrace class StatsdExample {
  // declare the variable to be injected by Statsd service
  @Injected(factoryMethod = "getInstance") private static Statsd sd;

  @OnMethod(...)
  public static void m(...) {
    sd.increment("my.metric.a"); // simple metric value
    sd.increment("my.metric.b", "regular,distribution:gaussian"); // tagged metric value
  }
}

Configuring StatsD Collector

By default the BTrace StatsD client will forward the metrics UDP packets to localhost and port 8125 (the defaults for many of the StatsD servers).

In order to configure a different collector one can use the new BTrace launcher argument -statsd

btrace -statsd host:port ...

The BTrace agent also accepts statsd argument directly

java -javaagent:btrace-agent.jar=statsd=host:port,...

These settings are not dynamic, however. Once set and used they will remain the same. The subsequent attaches using the btrace launcher are not able to modify the settings. This might be addressed in the future if it, in fact, turns out to be a problem.

Conclusion

Including StatsD in the BTrace tool box will enable very easy integration with various monitoring suites like graphite or Datadog.

In my previous blog post I was sketching up a possible solution to BTrace extensibility.

A few experiments with AST and bytecode verifiers later I came up with a relatively easy way to provide the extensibility of the core BTrace functionality by the means of external plugins. Very importantly, introducing external plugins doesn’t incur any additional overhead on top of the one caused by the extension functionality itself.

Creating a New Service

Creating a new service is just a matter of extending one of the following two classes

• com.sun.btrace.services.spi.SimpleService
• com.sun.btrace.services.spi.RuntimeService

SimpleService

A stateless service which can even be a singleton. It has no access to the BTrace runtime and can rely on the provided values.

RuntimeService

For each BTraceRuntime a new service instance will be created. The service can use BTrace features exposed via BTraceRuntime - eg. send messages over the wire channel.

Using Services

Services become available by adding the jars containing them to the bootclasspath. (eg. via agent options)

The recommended way to use services in the script is to create fields of the appropriate service type and annotate them by the @Injected annotation.

@BTrace public class MyTrace {
    // a simple service instantiated via the default noarg constructor
    @Injected private static MyService svc;

    // a simple service instantiated via the given static method
    @Injected(factoryMethod = "singleton") private static MyService svc1;

    // a runtime-aware service instantiated via the default single arg (*BTraceRuntime*) constructor
    @Injected(ServiceType.RUNTIME) private static RtService svc2;

    @OnMethod(....)
    public static void interceptMethod(...) {
        svc.process(...);
        svc1.process(...);
        svc2.process(...);
    }
}

There is an alternative approach valid only for locally accessible services.

@OnMethod(...)
public void handler(...) {
    // a simple service instantiated via the default noarg constructor
    MyService svc = Service.simple(MyService.class);
    
    // a simple service instantiated via the given static method
    MyService svc1 = Service.simple("singleton", MyService.class)

    // a runtime-aware service instantiated via the default single arg (*BTraceRuntime*) constructor
    RtService svc2 = Service.runtime(RtService.classs);

    svc.process(...);
    svc1.process(...);
    svc2.process(...);
}

These two approaches are functionally equivalent - it really depends on the script author which one she should choose.

Planning

This functionality is planned for the upcoming version 1.3 of BTrace.

Integration is tracked by this pull request