Testing

Halide uses CTest as its primary test platform and runner.

Organization

Halide's tests are organized beneath the top-level test/ directory. These folders are described below:

Folder	Description
autoschedulers/$AS	Test for the $AS (e.g. adams2019) autoscheduler
common	Code that may be shared across multiple tests
correctness	Tests that check correctness of various compiler properties
error	Tests that expect an exception to be thrown (or abort() to be called)
failing_with_issue	Correctness tests that are associated with a particular issue on GitHub
fuzz	Fuzz tests. Read more at FuzzTesting.md
generator	Tests of Halide's AOT compilation infrastructure.
integration	Tests of Halide's CMake package for downstream use, including cross compilation.
performance	Tests that check that certain schedules indeed improve performance.
runtime	Unit tests for the Halide runtime library
warning	Tests that expected warnings are indeed issued.

The tests in each of these directories are given CTest labels corresponding to the directory name. Thus, one can use ctest -L generator to run only the generator tests. The performance tests configure CTest to not run them concurrently with other tests (including each other).

The vast majority of our tests are simple C++ executables that link to Halide, perform some checks, and print the special line Success! upon successful completion. There are three main exceptions to this:

First, the warning tests are expected to print a line that reads Warning: and do not look for Success!.

Second, some tests cannot run in all scenarios; for example, a test that measures CUDA performance requires a CUDA-capable GPU. In these cases, tests are expected to print [SKIP] and exit and not print Success! or Warning:.

Finally, the error tests are expected to throw an (uncaught) exception that is not a Halide::InternalError (i.e. from a failing internal_assert). The logic for translating uncaught exceptions into successful tests is in test/common/expect_abort.cpp.

Debugging with LLDB

We provide helpers for pretty-printing Halide's IR types in LLDB. The .lldbinit file at the repository root will load automatically if you launch lldb from this directory and your ~/.lldbinit file contains the line,

settings set target.load-cwd-lldbinit true

If you prefer to avoid such global configuration, you can directly load the helpers with the LLDB command,

command script import ./tools/lldbhalide.py

again assuming that the repository root is your current working directory.

To see the benefit of using these helpers, let us debug correctness_bounds:

$ lldb ./build/test/correctness/correctness_bounds
(lldb) breakpoint set --file bounds.cpp --line 18
Breakpoint 1: where = correctness_bounds`main + 864 at bounds.cpp:18:12, address = 0x0000000100002054
(lldb) run
Process 29325 launched: '/Users/areinking/dev/Halide/build/test/correctness/correctness_bounds' (arm64)
Defining function...
Process 29325 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x0000000100002054 correctness_bounds`main(argc=1, argv=0x000000016fdff160) at bounds.cpp:18:12
   15       g(x, y) = min(x, y);
   16       h(x, y) = clamp(x + y, 20, 100);
   17   
-> 18       Var xo("xo"), yo("yo"), xi("xi"), yi("yi");
   19   
   20       Target target = get_jit_target_from_environment();
   21       if (target.has_gpu_feature()) {
Target 0: (correctness_bounds) stopped.
(lldb) 

Now we can try to inspect the Func h. Without the helpers, we see:

(lldb) v h
(Halide::Func) {
  func = {
    contents = {
      strong = (ptr = 0x0000600002486a20)
      weak = nullptr
      idx = 0
    }
  }
  pipeline_ = {
    contents = (ptr = 0x0000000000000000)
  }
}

But if we load the helpers and try again, we get a much more useful output:

(lldb) command script import ./tools/lldbhalide.py
(lldb) v h
... lots of output ...

The amount of output here is maybe a bit too much, but we gain the ability to more narrowly inspect data about the func:

(lldb) v h.func.init_def.values
...
(std::vector<Halide::Expr>) h.func.init_def.values = size=1 {
  [0] = max(min(x + y, 100), 20)
}

These helpers are particularly useful when using graphical debuggers, such as the one found in CLion.