Ret was uint32_t truncating the uint64_t __readlink return, and was
compared against the unrelated getdents64 BufSize (1024) instead of
sizeof(TargetPath) (NameMax, 4096). A truncated readlink of exactly
NameMax bytes also wrote one byte past TargetPath.
This commit enables compatibility of instrumentation-file-append-pid and
instrumentation-sleep-time options. It also requires keeping the
counters mapping between the watcher process and the instrumented binary
process in shared mode. This is useful when we instrument a shared
library that is used by several tasks running on the target system. In
case when we cannot wait for every task to complete, we must use the
sleep-time option. Without append-pid option, we would overwrite the
profile at the same path but collected from different tasks, leading to
unexpected or suboptimal optimization effects.
Co-authored-by: Vasily Leonenko <vasily.leonenko@huawei.com>
Disable all stderr diagnostic output on Android since there is typically
no terminal to read diagnostic message. The `noinline`annotation is to
keep same inline decision before and after this change. On AArch64
the `.text` section in instr runtime library is now ~4.8 KB smaller.
Use `GlobalWriteProfileMutex` to synchronize between data reset and
dump. Between static counter reset and increment, we use atomic store
in counter reset - the counter increment sequence inserted within user
code already takes care of thread safety, so we just need to make sure
the counter reset code is also thread safe (no torn write to counter).
While the current max memory size is sufficient for most binaries, a few
binaries may encouter insufficient allocated memory space.
Allow specify the max memory size of the instrumentation bump allocator.
The BOLT runtime currently does not close the FD pointing to
/proc/self/map_files. This does not actually hurt anything but was at
least a bit of a red herring for me when looking through strace on a
BOLT instrumented binary.
`__bolt_instr_data_dump()` will find instrumented binary name by
iterating through entries under directory `/proc/self/map_files`,
and then open the binary and memory map it onto heap in order
to locate `.bolt.instr.tables` section to read the descriptions.
If binary name is already known and/or binary is already opened
as memory mapped, we can pass binary name and/or memory
buffer directly to `__bolt_instr_data_dump()` to save some work.
This patch adds code generation for RISCV64 instrumentation.The work
involved includes the following three points:
a) Implements support for instrumenting direct function call and jump
on RISC-V which relies on , Atomic instructions
(used to increment counters) are only available on RISC-V when the A
extension is used.
b) Implements support for instrumenting direct function inderect call
by implementing the createInstrumentedIndCallHandlerEntryBB and
createInstrumentedIndCallHandlerExitBB interfaces. In this process, we
need to accurately record the target address and IndCallID to ensure
the correct recording of the indirect call counters.
c)Implemented the RISCV64 Bolt runtime library, implemented some system
call interfaces through embedded assembly. Get the difference between
runtime addrress of .text section andstatic address in section header
table, which in turn can be used to search for indirect call
description.
However, the community code currently has problems with relocation in
some scenarios, but this has nothing to do with instrumentation. We
may continue to submit patches to fix the related bugs.
Closes https://github.com/llvm/llvm-project/issues/63097
Before merging please make sure the change to
bolt/include/bolt/Passes/StokeInfo.h is correct.
bolt/include/bolt/Passes/StokeInfo.h
```diff
// This Pass solves the two major problems to use the Stoke program without
- // proting its code:
+ // probing its code:
```
I'm still not happy about the awkward wording in this comment.
bolt/include/bolt/Passes/FixRelaxationPass.h
```
$ ed -s bolt/include/bolt/Passes/FixRelaxationPass.h <<<'9,12p'
// This file declares the FixRelaxations class, which locates instructions with
// wrong targets and fixes them. Such problems usually occures when linker
// relaxes (changes) instructions, but doesn't fix relocations types properly
// for them.
$
```
bolt/docs/doxygen.cfg.in
bolt/include/bolt/Core/BinaryContext.h
bolt/include/bolt/Core/BinaryFunction.h
bolt/include/bolt/Core/BinarySection.h
bolt/include/bolt/Core/DebugData.h
bolt/include/bolt/Core/DynoStats.h
bolt/include/bolt/Core/Exceptions.h
bolt/include/bolt/Core/MCPlusBuilder.h
bolt/include/bolt/Core/Relocation.h
bolt/include/bolt/Passes/FixRelaxationPass.h
bolt/include/bolt/Passes/InstrumentationSummary.h
bolt/include/bolt/Passes/ReorderAlgorithm.h
bolt/include/bolt/Passes/StackReachingUses.h
bolt/include/bolt/Passes/StokeInfo.h
bolt/include/bolt/Passes/TailDuplication.h
bolt/include/bolt/Profile/DataAggregator.h
bolt/include/bolt/Profile/DataReader.h
bolt/lib/Core/BinaryContext.cpp
bolt/lib/Core/BinarySection.cpp
bolt/lib/Core/DebugData.cpp
bolt/lib/Core/DynoStats.cpp
bolt/lib/Core/Relocation.cpp
bolt/lib/Passes/Instrumentation.cpp
bolt/lib/Passes/JTFootprintReduction.cpp
bolt/lib/Passes/ReorderData.cpp
bolt/lib/Passes/RetpolineInsertion.cpp
bolt/lib/Passes/ShrinkWrapping.cpp
bolt/lib/Passes/TailDuplication.cpp
bolt/lib/Rewrite/BoltDiff.cpp
bolt/lib/Rewrite/DWARFRewriter.cpp
bolt/lib/Rewrite/RewriteInstance.cpp
bolt/lib/Utils/CommandLineOpts.cpp
bolt/runtime/instr.cpp
bolt/test/AArch64/got-ld64-relaxation.test
bolt/test/AArch64/unmarked-data.test
bolt/test/X86/Inputs/dwarf5-cu-no-debug-addr-helper.s
bolt/test/X86/Inputs/linenumber.cpp
bolt/test/X86/double-jump.test
bolt/test/X86/dwarf5-call-pc-function-null-check.test
bolt/test/X86/dwarf5-split-dwarf4-monolithic.test
bolt/test/X86/dynrelocs.s
bolt/test/X86/fallthrough-to-noop.test
bolt/test/X86/tail-duplication-cache.s
bolt/test/runtime/X86/instrumentation-ind-calls.s
This commit adds support for AArch64 in instrumentation runtime library,
including AArch64 system calls.
Also this commit divides syscalls into target-specific files.
Reviewed By: rafauler, yota9
Differential Revision: https://reviews.llvm.org/D151942
Because indirect call tables use static addresses for call sites, but pc
values recorded by runtime may be subject to ASLR in PIE, we couldn't
find indirect call descriptions by their runtime address in PIE. It
resulted in [unknown] entries in profile for all indirect calls. We need
to substract base address of .text from runtime addresses to get the
corresponding static addresses. Here we create a getter for base address
of .text and substract it's return value from recorded PC values. It
converts them to static addresses, which then may be used to find the
corresponding indirect call descriptions.
Reviewed By: rafauler
Differential Revision: https://reviews.llvm.org/D154121
When a binary is instrumented with --instrumentation-sleep-time and
instrumentation-wait-forks options and lauched, the profile is
periodically written until all the forks die. The problem is that we
cannot wait for the whole process tree, and we have no way to tell when
it's safe to read the profile. Hovewer, if we keep profile open
throughout the life of the process tree, we can use fuser to determine
when writing is finished.
Reviewed By: rafauler
Differential Revision: https://reviews.llvm.org/D154436
The issue was caused by the absence of placement new definition. It
worked for clang and thus passed Phabricator checks, but broke when
compiled with GCC on buildbot.
Full problem description: https://reviews.llvm.org/D153771#4468239
Original patch description:
In absence of instrumentation-file-append-pid option,
global allocator uses shared pages for allocation. However, since it is a
global variable, it gets COW'd after fork if instrumentation-sleep-time
is used, or if a process forks by itself. This means it handles the same
pages to every process which causes hash table corruption. Thus, if we
want shared pages, we need to put the allocator itself in a shared page,
which we do in this commit in __bolt_instr_setup.
I also added a couple of assertions to sanity-check the hash table.
Reviewed By: rafauler, Amir
Differential Revision: https://reviews.llvm.org/D153771
In a very rare case that mmap call fails, we'll at least get a message
instead of segfault.
Reviewed By: rafauler, Amir
Differential Revision: https://reviews.llvm.org/D154056
Since there are no other means to debug the instrumentation library
other than using stdout, having a function to print hash table entries
is very useful.
Reviewed By: rafauler, Amir
Differential Revision: https://reviews.llvm.org/D153771
In absence of instrumentation-file-append-pid option,
global allocator uses shared pages for allocation. However, since it is a
global variable, it gets COW'd after fork if instrumentation-sleep-time
is used, or if a process forks by itself. This means it handles the same
pages to every process which causes hash table corruption. Thus, if we
want shared pages, we need to put the allocator itself in a shared page,
which we do in this commit in __bolt_instr_setup.
I also added a couple of assertions to sanity-check the hash table.
Reviewed By: rafauler, Amir
Differential Revision: https://reviews.llvm.org/D153771
The point of append-pid option is to record separate profiles for
separate forks, which is impossible when counters are the same for
every process. It leads to a sum of all profiles in every file, plus
GlobalWriteProfileMutex located in a shared memory prevents some
processes from dumping their data at all.
Reviewed By: rafauler, Amir
Differential Revision: https://reviews.llvm.org/D153771
In a very rare case that mmap call fails, we'll at least get a message
instead of segfault.
Reviewed By: rafauler, Amir
Differential Revision: https://reviews.llvm.org/D154056
The BOLT instrumentation runtime uses a bump allocator that has a fixed amount of maximum memory. In some cases, this memory limit is not large enough (https://github.com/llvm/llvm-project/issues/59174). We are hitting this limit when instrumenting the Rust compiler with BOLT.
This change increases the memory of the bump allocator used for writing the resulting BOLT profile.
Reviewed By: rafauler
Differential Revision: https://reviews.llvm.org/D151891
`__bolt_instr_data_dump()` does not lock the hash tables when iterating
over them, so the iteration can happen concurrently with a modification
done in another thread, when the table is in an inconsistent state. This
also has been observed in practice, when it caused a segmentation fault.
We fix this by locking hash tables during iteration. This is done by taking
the lock in `forEachElement()`.
The only other site of iteration, `resetCounters()`, has been correctly
locking the table even before this patch. This patch removes its `Lock`
because the lock is now taken in the inner `forEachElement()`.
Reviewed By: maksfb, yota9
Differential Revision: https://reviews.llvm.org/D129089
Compiler can generate calls to some functions implicitly, even under
constraints of freestanding environment. Make sure these functions are
available in our runtime objects.
Fixes test failures on some systems after https://reviews.llvm.org/D128960.
Reviewed By: yota9
Differential Revision: https://reviews.llvm.org/D129168
Summary:
Refactor remaining bolt sources to follow the braces rule for if/else/loop from
[LLVM Coding Standards](https://llvm.org/docs/CodingStandards.html).
(cherry picked from FBD33345885)
Summary: In some cases __open() is returning 0 for me. The open
syscall will return a negative number (-1) on error so 0 should be
valid.
This happens when one compiles a BOLT instrumented executable of
pyston (python implementation) and afterwards pip installs a package
which needs to be compiled and sets CFLAGS=-pipe. I could not reduce
it down to a small testcase but I guess it one can trigger when
manually closing std{in, out, err}. Everything seems to work
normally when disabling the assert for 0 in getBinaryPath() - I
decided to also modify the second case in readDescriptions() even
though I did not run into that one yet.
(cherry picked from FBD32409548)
Summary:
Increase the hard limit from 256 to 4096.
This fixes the 'Assertion failed: failed to open binary path' error I'm seeing.
(cherry picked from FBD31911946)
Summary:
This patch temporarily disables instrumentation and higufy build not for
x86 platforms to be able to build llvm-bolt tool on aarch64.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD31738306)
Summary:
Sync the file with storage device on data dump to stabilize
instrumentation testing
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD31738021)
Summary:
This commit introduces TryLock usage for SimpleHashTable getter to
avoid deadlock and relax syscalls usage which causes significant
overhead in runtime.
The old behavior left under -conservative-instrumentation option passed
to instrumentation library.
Also, this commit includes a corresponding test case: instrumentation of
executable which performs indirect calls from common code and signal
handler.
Note: in case if TryLock was failed to acquire the lock - this indirect
call will not be accounted in the resulting profile.
Vasily Leonenko,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30821949)
Summary:
To avoid RELATIVE relocations avoid using of GOT table
by using hidden visibility for all symbols in library.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092712)
Summary:
The trampolines are no loger pointers to the functions. For
propper name resolving by bolt use extern "C" for all external symbols
in instr.cpp
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092698)
Summary:
This commit introduces -instrumentation-binpath argument used
to point instuqmented binary in runtime in case if /proc/self/map_files
path is not accessible due to access restriction issues.
Vasily Leonenko
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092681)
Summary:
This commit adds support for opening libs based on links
/proc/self/map_files. For this we're getting current virtual address
and searching the lib in the directory with such address range. After
that, we're getting full path to the binary by using readlink
function. Direct read from link in /proc/self/map_files entries is not
possible because of lack of permissions.
Elvina Yakubova,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092422)
