做毕设的时候，我曾经遇到一个多线程的BUG。这个BUG表现得较为诡异，会导致数据随机出错。由于找不出什么规律，一开始我还是挺头疼的。查了半天后我发现，相关的日志有多线程下共享数据访问问题的迹象（即所谓的data race），所以很快确诊是多线程部分代码存在逻辑错误。这个问题的解决办法很简单，就是把相关的代码review下，找出data race的部分并加以修正。虽然BUG是搞定了，不过我还是想找到一个自动化工具，能够检测出代码中潜在的线程安全问题。这样就能把BUG消灭在萌芽之中，而不是等到事后才睁大眼睛揪它出来。
搜索了下，发现了两个适合做这个的工具，Valgrind和ThreadSanitizer。今天就来介绍下这两个工具。
Valgrind
Valgrind一般用做内存泄露和访存越界检测，除此之外，其实它也支持对data race及一些简单的多线程问题的检查。Valgrind工具集里面，helgrind和drd都能用来完成这种检测。你可以用valgrind --tool=helgrind或valgrind --tool=drd来启用它。只要应用使用的线程模型是POSIX thread（pthread），这两个工具就能进行检测。这两个工具间差别不大，下面我就基于helgrind来介绍下用法：
先上一段有问题的示例代码：
// raceCondition.cpp
#include <pthread.h>void *write_buffer(void *args)
{pthread_t *buffer = static_cast<pthread_t *>(args);*buffer = pthread_self();pthread_exit(0);return NULL;
}int main()
{pthread_t *buffer = new pthread_t[2];pthread_t a, b;pthread_create(&a, NULL, write_buffer, buffer);pthread_create(&b, NULL, write_buffer, buffer);pthread_join(a, NULL);pthread_join(b, NULL);delete []buffer;return 0;
}这段代码有一个刻意为之的问题，线程a和线程b写入了同一个缓冲区。
用Valgrind可以检测出问题：
==5697== ---Thread-Announcement------------------------------------------
==5697==
==5697== Thread #3 was created
==5697== at 0x545943E: clone (clone.S:74)
==5697== by 0x5148199: do_clone.constprop.3 (createthread.c:75)
==5697== by 0x51498BA: create_thread (createthread.c:245)
==5697== by 0x51498BA: pthread_create@@GLIBC_2.2.5 (pthread_create.c:611)
==5697== by 0x4C30E0D: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==5697== by 0x400928: main (in /home/lzx/C/thread_error/a.out)
==5697==
==5697== ---Thread-Announcement------------------------------------------
==5697==
==5697== Thread #2 was created
==5697== at 0x545943E: clone (clone.S:74)
==5697== by 0x5148199: do_clone.constprop.3 (createthread.c:75)
==5697== by 0x51498BA: create_thread (createthread.c:245)
==5697== by 0x51498BA: pthread_create@@GLIBC_2.2.5 (pthread_create.c:611)
==5697== by 0x4C30E0D: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==5697== by 0x40090B: main (in /home/lzx/C/thread_error/a.out)
==5697==
==5697== ---Thread-Announcement------------------------------------------
==5697==
==5697== Thread #1 is the program's root thread
==5697==
==5697== ----------------------------------------------------------------
==5697==
==5697== Possible data race during write of size 8 at 0x5C40040 by thread #3
==5697== Locks held: none
==5697== at 0x4008BB: write_buffer(void*) (in /home/lzx/C/thread_error/a.out)
==5697== by 0x4C30FA6: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==5697== by 0x5149181: start_thread (pthread_create.c:312)
==5697== by 0x545947C: clone (clone.S:111)
==5697==
==5697== This conflicts with a previous write of size 8 by thread #2
==5697== Locks held: none
==5697== at 0x4008BB: write_buffer(void*) (in /home/lzx/C/thread_error/a.out)
==5697== by 0x4C30FA6: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==5697== by 0x5149181: start_thread (pthread_create.c:312)
==5697== by 0x545947C: clone (clone.S:111)
==5697== Address 0x5c40040 is 0 bytes inside a block of size 16 alloc'd
==5697== at 0x4C2CC20: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==5697== by 0x4008EA: main (in /home/lzx/C/thread_error/a.out)
==5697== Block was alloc'd by thread #1输出结果包括data race的内存位置、内存区域大小和涉及的线程，以及调用栈。
如果编译程序时加了-g选项，那么输出的调用栈中会有具体的位置：
==7993== This conflicts with a previous write of size 8 by thread #2
==7993== Locks held: none
==7993== at 0x4008BB: write_buffer(void*) (raceCondition.cpp:8)
==7993== by 0x4C30FA6: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==7993== by 0x5149181: start_thread (pthread_create.c:312)
==7993== by 0x545947C: clone (clone.S:111)
==7993== Address 0x5c40040 is 0 bytes inside a block of size 16 alloc'd
==7993== at 0x4C2CC20: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==7993== by 0x4008EA: main (raceCondition.cpp:17)
==7993== Block was alloc'd by thread #1Valgrind记录了每个线程的内存访问情况，如果多个线程对同一个内存地址的访问没有限定次序（诸如happen before这样的memory model细则），就会被判为“Possible data race”。
Valgrind的检测同样对C++11提供的thread库生效（只要底层用的还是pthread）：
#include <thread>using namespace std;void write_buffer(thread::id *buffer)
{*buffer = this_thread::get_id();
}int main()
{thread::id *buffer = new thread::id[2];thread a(write_buffer, buffer);thread b(write_buffer, buffer);a.join();b.join();return 0;
}输出报告跟pthread版本的差不多。由于输出太长，这里就不贴了。
除了data race，Valgrind也能检测出一些简单的多线程问题，比如线程结束时没有释放锁：
#include <pthread.h>pthread_mutex_t mutex;void *still_locked(void *args)
{(void)args;pthread_mutex_lock(&mutex);pthread_exit(0);return NULL;
}int main()
{pthread_mutex_init(&mutex, NULL);pthread_t a;pthread_create(&a, NULL, still_locked, NULL);pthread_join(a, NULL);return 0;
}==6316== Thread #2: Exiting thread still holds 1 lock
==6316== at 0x4E4521F: start_thread (pthread_create.c:457)即使线程detach了也能检测出来。
#include <pthread.h>pthread_mutex_t mutex;void *still_locked(void *args)
{(void)args;pthread_detach(pthread_self());pthread_mutex_lock(&mutex);pthread_exit(0);return NULL;
}int main()
{pthread_mutex_init(&mutex, NULL);pthread_t a;pthread_create(&a, NULL, still_locked, NULL);return 0;
}==6574== Thread #2: Exiting thread still holds 1 lock
==6574== at 0x4E4521F: start_thread (pthread_create.c:457)ThreadSanitizer
ThreadSanitizer是另外一个检测多线程问题的工具，集成于gcc 4.8和clang 3.2以上的版本。
换句话说，只要你的编译器版本不太旧，那么你就可以立刻启用它。
对于clang，需要使用下列的编译/链接选项：
clang -fsanitize=thread -fPIE -pie -g对于gcc，可能还要加上-ltsan
gcc -fsanitize=thread -fPIE -pie -g -ltsan如果出现了链接错误，检查下是否有libtsan这个库。
以上节展示的第一段代码为例：
$ g++ raceCondition.cpp -fsanitize=thread -fPIE -pie -g -ltsan
$ ./a.out
==================
WARNING: ThreadSanitizer: data race (pid=8425)Write of size 8 at 0x7d020000eff0 by thread T2:#0 write_buffer(void*) /home/lzx/C/thread_error/raceCondition.cpp:8 (exe+0x000000000c1b)#1 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9)Previous write of size 8 at 0x7d020000eff0 by thread T1:#0 write_buffer(void*) /home/lzx/C/thread_error/raceCondition.cpp:8 (exe+0x000000000c1b)#1 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9)Location is heap block of size 16 at 0x7d020000eff0 allocated by main thread:#0 operator new[](unsigned long) ??:0 (libtsan.so.0+0x00000001cfe2)#1 main /home/lzx/C/thread_error/raceCondition.cpp:17 (exe+0x000000000c5b)Thread T2 (tid=8427, running) created by main thread at:#0 pthread_create ??:0 (libtsan.so.0+0x00000001eccb)#1 main /home/lzx/C/thread_error/raceCondition.cpp:21 (exe+0x000000000c9d)Thread T1 (tid=8426, finished) created by main thread at:#0 pthread_create ??:0 (libtsan.so.0+0x00000001eccb)#1 main /home/lzx/C/thread_error/raceCondition.cpp:20 (exe+0x000000000c7e)SUMMARY: ThreadSanitizer: data race /home/lzx/C/thread_error/raceCondition.cpp:8 write_buffer(void*)
==================
ThreadSanitizer: reported 1 warnings输出结果跟Valgrind的大同小异。ThreadSanitizer的检测机制跟Valgrind相似，也是检测各线程对内存的访问是否有序。不同的是，ThreadSanitizer会在编译时给特定的访存操作注入监控指令，而不是在运行时监控全部的访存操作。这么一来，ThreadSanitizer的内存占用和性能损耗会比Valgrind的少很多，这也是它的主打优点。
ThreadSanitizer的另一个主打优点是，它支持的data race检测要比Valgrind的更多。
不过，在某些方面（比如上文提到的线程结束时没有释放锁）的检测，ThreadSanitizer却又不如Valgrind。
结论
这两个工具之间，我偏好Valgrind。在资源占用方面，除非你的项目已经达到Chrome级别，否则不用太在意运行测试的用时；在功能方面，两者间差异不大；而ThreadSanitizer用起来相对麻烦一些。它需要特定的编译指令，一旦跟现有的编译方式冲突就很蛋疼了。
事实上，如果要我给这两个工具打分，满分100我只能给70.
这两个工具的输出都很含糊。Possible data race？Previous write by thread X？在现实应用中使用时，出问题之处要比上述的示例代码难理解多了。而且，Valgrind的多线程问题检测有一定可能出现误报。（之前在毕设的应用中就遇到过）
另外，只有进行了内存访问才会触发data race的检测。对于一类小概率触发的data race问题，这两个工具不一定能检测出来。
写一个randomRaceCondition.cpp作为例子：
#include <cstdlib>
#include <ctime>
#include <pthread.h>void *write_buffer(void *args) {pthread_t *buffer = static_cast<pthread_t *>(args);if (rand() % 2 == 0) { // 现在线程a和b都进行访存操作的概率为1/4*buffer = pthread_self();}pthread_exit(0);return NULL;
}int main()
{srand(time(0));pthread_t *buffer = new pthread_t[2];pthread_t a, b;pthread_create(&a, NULL, write_buffer, buffer);pthread_create(&b, NULL, write_buffer, buffer);pthread_join(a, NULL);pthread_join(b, NULL);delete []buffer;return 0;
}无论是Valgrind还是ThreadSanitizer，在单次运行内检测出data race都是个随机事件了。
最后，Valgrind/ThreadSanitizer所能检测出的线程问题只占了一小部分。对于许多棘手的多线程问题，它们也无能为力。工具报告没问题并不确保代码没问题，要写出线程安全的代码，还是得多花点心思。