1249259Sdim//===-- MemorySanitizer.cpp - detector of uninitialized reads -------------===// 2249259Sdim// 3249259Sdim// The LLVM Compiler Infrastructure 4249259Sdim// 5249259Sdim// This file is distributed under the University of Illinois Open Source 6249259Sdim// License. See LICENSE.TXT for details. 7249259Sdim// 8249259Sdim//===----------------------------------------------------------------------===// 9249259Sdim/// \file 10249259Sdim/// This file is a part of MemorySanitizer, a detector of uninitialized 11249259Sdim/// reads. 12249259Sdim/// 13249259Sdim/// The algorithm of the tool is similar to Memcheck 14249259Sdim/// (http://goo.gl/QKbem). We associate a few shadow bits with every 15249259Sdim/// byte of the application memory, poison the shadow of the malloc-ed 16249259Sdim/// or alloca-ed memory, load the shadow bits on every memory read, 17249259Sdim/// propagate the shadow bits through some of the arithmetic 18249259Sdim/// instruction (including MOV), store the shadow bits on every memory 19249259Sdim/// write, report a bug on some other instructions (e.g. JMP) if the 20249259Sdim/// associated shadow is poisoned. 21249259Sdim/// 22249259Sdim/// But there are differences too. The first and the major one: 23249259Sdim/// compiler instrumentation instead of binary instrumentation. This 24249259Sdim/// gives us much better register allocation, possible compiler 25249259Sdim/// optimizations and a fast start-up. But this brings the major issue 26249259Sdim/// as well: msan needs to see all program events, including system 27249259Sdim/// calls and reads/writes in system libraries, so we either need to 28249259Sdim/// compile *everything* with msan or use a binary translation 29249259Sdim/// component (e.g. DynamoRIO) to instrument pre-built libraries. 30249259Sdim/// Another difference from Memcheck is that we use 8 shadow bits per 31249259Sdim/// byte of application memory and use a direct shadow mapping. This 32249259Sdim/// greatly simplifies the instrumentation code and avoids races on 33249259Sdim/// shadow updates (Memcheck is single-threaded so races are not a 34249259Sdim/// concern there. Memcheck uses 2 shadow bits per byte with a slow 35249259Sdim/// path storage that uses 8 bits per byte). 36249259Sdim/// 37249259Sdim/// The default value of shadow is 0, which means "clean" (not poisoned). 38249259Sdim/// 39249259Sdim/// Every module initializer should call __msan_init to ensure that the 40249259Sdim/// shadow memory is ready. On error, __msan_warning is called. Since 41249259Sdim/// parameters and return values may be passed via registers, we have a 42249259Sdim/// specialized thread-local shadow for return values 43249259Sdim/// (__msan_retval_tls) and parameters (__msan_param_tls). 44249259Sdim/// 45249259Sdim/// Origin tracking. 46249259Sdim/// 47249259Sdim/// MemorySanitizer can track origins (allocation points) of all uninitialized 48249259Sdim/// values. This behavior is controlled with a flag (msan-track-origins) and is 49249259Sdim/// disabled by default. 50249259Sdim/// 51249259Sdim/// Origins are 4-byte values created and interpreted by the runtime library. 52249259Sdim/// They are stored in a second shadow mapping, one 4-byte value for 4 bytes 53249259Sdim/// of application memory. Propagation of origins is basically a bunch of 54249259Sdim/// "select" instructions that pick the origin of a dirty argument, if an 55249259Sdim/// instruction has one. 56249259Sdim/// 57249259Sdim/// Every 4 aligned, consecutive bytes of application memory have one origin 58249259Sdim/// value associated with them. If these bytes contain uninitialized data 59249259Sdim/// coming from 2 different allocations, the last store wins. Because of this, 60249259Sdim/// MemorySanitizer reports can show unrelated origins, but this is unlikely in 61249259Sdim/// practice. 62249259Sdim/// 63249259Sdim/// Origins are meaningless for fully initialized values, so MemorySanitizer 64249259Sdim/// avoids storing origin to memory when a fully initialized value is stored. 65249259Sdim/// This way it avoids needless overwritting origin of the 4-byte region on 66249259Sdim/// a short (i.e. 1 byte) clean store, and it is also good for performance. 67261991Sdim/// 68261991Sdim/// Atomic handling. 69261991Sdim/// 70261991Sdim/// Ideally, every atomic store of application value should update the 71261991Sdim/// corresponding shadow location in an atomic way. Unfortunately, atomic store 72261991Sdim/// of two disjoint locations can not be done without severe slowdown. 73261991Sdim/// 74261991Sdim/// Therefore, we implement an approximation that may err on the safe side. 75261991Sdim/// In this implementation, every atomically accessed location in the program 76261991Sdim/// may only change from (partially) uninitialized to fully initialized, but 77261991Sdim/// not the other way around. We load the shadow _after_ the application load, 78261991Sdim/// and we store the shadow _before_ the app store. Also, we always store clean 79261991Sdim/// shadow (if the application store is atomic). This way, if the store-load 80261991Sdim/// pair constitutes a happens-before arc, shadow store and load are correctly 81261991Sdim/// ordered such that the load will get either the value that was stored, or 82261991Sdim/// some later value (which is always clean). 83261991Sdim/// 84261991Sdim/// This does not work very well with Compare-And-Swap (CAS) and 85261991Sdim/// Read-Modify-Write (RMW) operations. To follow the above logic, CAS and RMW 86261991Sdim/// must store the new shadow before the app operation, and load the shadow 87261991Sdim/// after the app operation. Computers don't work this way. Current 88261991Sdim/// implementation ignores the load aspect of CAS/RMW, always returning a clean 89261991Sdim/// value. It implements the store part as a simple atomic store by storing a 90261991Sdim/// clean shadow. 91261991Sdim 92249259Sdim//===----------------------------------------------------------------------===// 93249259Sdim 94249259Sdim#include "llvm/Transforms/Instrumentation.h" 95249259Sdim#include "llvm/ADT/DepthFirstIterator.h" 96249259Sdim#include "llvm/ADT/SmallString.h" 97249259Sdim#include "llvm/ADT/SmallVector.h" 98276479Sdim#include "llvm/ADT/StringExtras.h" 99261991Sdim#include "llvm/ADT/Triple.h" 100249259Sdim#include "llvm/IR/DataLayout.h" 101249259Sdim#include "llvm/IR/Function.h" 102249259Sdim#include "llvm/IR/IRBuilder.h" 103249259Sdim#include "llvm/IR/InlineAsm.h" 104276479Sdim#include "llvm/IR/InstVisitor.h" 105249259Sdim#include "llvm/IR/IntrinsicInst.h" 106249259Sdim#include "llvm/IR/LLVMContext.h" 107249259Sdim#include "llvm/IR/MDBuilder.h" 108249259Sdim#include "llvm/IR/Module.h" 109249259Sdim#include "llvm/IR/Type.h" 110276479Sdim#include "llvm/IR/ValueMap.h" 111249259Sdim#include "llvm/Support/CommandLine.h" 112249259Sdim#include "llvm/Support/Compiler.h" 113249259Sdim#include "llvm/Support/Debug.h" 114249259Sdim#include "llvm/Support/raw_ostream.h" 115249259Sdim#include "llvm/Transforms/Utils/BasicBlockUtils.h" 116249259Sdim#include "llvm/Transforms/Utils/Local.h" 117249259Sdim#include "llvm/Transforms/Utils/ModuleUtils.h" 118249259Sdim 119249259Sdimusing namespace llvm; 120249259Sdim 121276479Sdim#define DEBUG_TYPE "msan" 122276479Sdim 123288943Sdimstatic const unsigned kOriginSize = 4; 124249259Sdimstatic const unsigned kMinOriginAlignment = 4; 125249259Sdimstatic const unsigned kShadowTLSAlignment = 8; 126249259Sdim 127280031Sdim// These constants must be kept in sync with the ones in msan.h. 128280031Sdimstatic const unsigned kParamTLSSize = 800; 129280031Sdimstatic const unsigned kRetvalTLSSize = 800; 130280031Sdim 131276479Sdim// Accesses sizes are powers of two: 1, 2, 4, 8. 132276479Sdimstatic const size_t kNumberOfAccessSizes = 4; 133276479Sdim 134249259Sdim/// \brief Track origins of uninitialized values. 135249259Sdim/// 136249259Sdim/// Adds a section to MemorySanitizer report that points to the allocation 137249259Sdim/// (stack or heap) the uninitialized bits came from originally. 138276479Sdimstatic cl::opt<int> ClTrackOrigins("msan-track-origins", 139249259Sdim cl::desc("Track origins (allocation sites) of poisoned memory"), 140276479Sdim cl::Hidden, cl::init(0)); 141249259Sdimstatic cl::opt<bool> ClKeepGoing("msan-keep-going", 142249259Sdim cl::desc("keep going after reporting a UMR"), 143249259Sdim cl::Hidden, cl::init(false)); 144249259Sdimstatic cl::opt<bool> ClPoisonStack("msan-poison-stack", 145249259Sdim cl::desc("poison uninitialized stack variables"), 146249259Sdim cl::Hidden, cl::init(true)); 147249259Sdimstatic cl::opt<bool> ClPoisonStackWithCall("msan-poison-stack-with-call", 148249259Sdim cl::desc("poison uninitialized stack variables with a call"), 149249259Sdim cl::Hidden, cl::init(false)); 150249259Sdimstatic cl::opt<int> ClPoisonStackPattern("msan-poison-stack-pattern", 151296417Sdim cl::desc("poison uninitialized stack variables with the given pattern"), 152249259Sdim cl::Hidden, cl::init(0xff)); 153249259Sdimstatic cl::opt<bool> ClPoisonUndef("msan-poison-undef", 154249259Sdim cl::desc("poison undef temps"), 155249259Sdim cl::Hidden, cl::init(true)); 156249259Sdim 157249259Sdimstatic cl::opt<bool> ClHandleICmp("msan-handle-icmp", 158249259Sdim cl::desc("propagate shadow through ICmpEQ and ICmpNE"), 159249259Sdim cl::Hidden, cl::init(true)); 160249259Sdim 161249259Sdimstatic cl::opt<bool> ClHandleICmpExact("msan-handle-icmp-exact", 162249259Sdim cl::desc("exact handling of relational integer ICmp"), 163249259Sdim cl::Hidden, cl::init(false)); 164249259Sdim 165249259Sdim// This flag controls whether we check the shadow of the address 166249259Sdim// operand of load or store. Such bugs are very rare, since load from 167249259Sdim// a garbage address typically results in SEGV, but still happen 168249259Sdim// (e.g. only lower bits of address are garbage, or the access happens 169249259Sdim// early at program startup where malloc-ed memory is more likely to 170249259Sdim// be zeroed. As of 2012-08-28 this flag adds 20% slowdown. 171249259Sdimstatic cl::opt<bool> ClCheckAccessAddress("msan-check-access-address", 172249259Sdim cl::desc("report accesses through a pointer which has poisoned shadow"), 173249259Sdim cl::Hidden, cl::init(true)); 174249259Sdim 175249259Sdimstatic cl::opt<bool> ClDumpStrictInstructions("msan-dump-strict-instructions", 176249259Sdim cl::desc("print out instructions with default strict semantics"), 177249259Sdim cl::Hidden, cl::init(false)); 178249259Sdim 179276479Sdimstatic cl::opt<int> ClInstrumentationWithCallThreshold( 180276479Sdim "msan-instrumentation-with-call-threshold", 181276479Sdim cl::desc( 182276479Sdim "If the function being instrumented requires more than " 183276479Sdim "this number of checks and origin stores, use callbacks instead of " 184276479Sdim "inline checks (-1 means never use callbacks)."), 185276479Sdim cl::Hidden, cl::init(3500)); 186249259Sdim 187280031Sdim// This is an experiment to enable handling of cases where shadow is a non-zero 188280031Sdim// compile-time constant. For some unexplainable reason they were silently 189280031Sdim// ignored in the instrumentation. 190280031Sdimstatic cl::opt<bool> ClCheckConstantShadow("msan-check-constant-shadow", 191280031Sdim cl::desc("Insert checks for constant shadow values"), 192280031Sdim cl::Hidden, cl::init(false)); 193261991Sdim 194288943Sdimstatic const char *const kMsanModuleCtorName = "msan.module_ctor"; 195288943Sdimstatic const char *const kMsanInitName = "__msan_init"; 196288943Sdim 197249259Sdimnamespace { 198249259Sdim 199280031Sdim// Memory map parameters used in application-to-shadow address calculation. 200280031Sdim// Offset = (Addr & ~AndMask) ^ XorMask 201280031Sdim// Shadow = ShadowBase + Offset 202280031Sdim// Origin = OriginBase + Offset 203280031Sdimstruct MemoryMapParams { 204280031Sdim uint64_t AndMask; 205280031Sdim uint64_t XorMask; 206280031Sdim uint64_t ShadowBase; 207280031Sdim uint64_t OriginBase; 208280031Sdim}; 209280031Sdim 210280031Sdimstruct PlatformMemoryMapParams { 211280031Sdim const MemoryMapParams *bits32; 212280031Sdim const MemoryMapParams *bits64; 213280031Sdim}; 214280031Sdim 215280031Sdim// i386 Linux 216288943Sdimstatic const MemoryMapParams Linux_I386_MemoryMapParams = { 217280031Sdim 0x000080000000, // AndMask 218280031Sdim 0, // XorMask (not used) 219280031Sdim 0, // ShadowBase (not used) 220280031Sdim 0x000040000000, // OriginBase 221280031Sdim}; 222280031Sdim 223280031Sdim// x86_64 Linux 224288943Sdimstatic const MemoryMapParams Linux_X86_64_MemoryMapParams = { 225296417Sdim#ifdef MSAN_LINUX_X86_64_OLD_MAPPING 226280031Sdim 0x400000000000, // AndMask 227280031Sdim 0, // XorMask (not used) 228280031Sdim 0, // ShadowBase (not used) 229280031Sdim 0x200000000000, // OriginBase 230296417Sdim#else 231296417Sdim 0, // AndMask (not used) 232296417Sdim 0x500000000000, // XorMask 233296417Sdim 0, // ShadowBase (not used) 234296417Sdim 0x100000000000, // OriginBase 235296417Sdim#endif 236280031Sdim}; 237280031Sdim 238288943Sdim// mips64 Linux 239288943Sdimstatic const MemoryMapParams Linux_MIPS64_MemoryMapParams = { 240288943Sdim 0x004000000000, // AndMask 241288943Sdim 0, // XorMask (not used) 242288943Sdim 0, // ShadowBase (not used) 243288943Sdim 0x002000000000, // OriginBase 244288943Sdim}; 245288943Sdim 246288943Sdim// ppc64 Linux 247288943Sdimstatic const MemoryMapParams Linux_PowerPC64_MemoryMapParams = { 248288943Sdim 0x200000000000, // AndMask 249288943Sdim 0x100000000000, // XorMask 250288943Sdim 0x080000000000, // ShadowBase 251288943Sdim 0x1C0000000000, // OriginBase 252288943Sdim}; 253288943Sdim 254296417Sdim// aarch64 Linux 255296417Sdimstatic const MemoryMapParams Linux_AArch64_MemoryMapParams = { 256296417Sdim 0, // AndMask (not used) 257296417Sdim 0x06000000000, // XorMask 258296417Sdim 0, // ShadowBase (not used) 259296417Sdim 0x01000000000, // OriginBase 260296417Sdim}; 261296417Sdim 262280031Sdim// i386 FreeBSD 263288943Sdimstatic const MemoryMapParams FreeBSD_I386_MemoryMapParams = { 264280031Sdim 0x000180000000, // AndMask 265280031Sdim 0x000040000000, // XorMask 266280031Sdim 0x000020000000, // ShadowBase 267280031Sdim 0x000700000000, // OriginBase 268280031Sdim}; 269280031Sdim 270280031Sdim// x86_64 FreeBSD 271288943Sdimstatic const MemoryMapParams FreeBSD_X86_64_MemoryMapParams = { 272280031Sdim 0xc00000000000, // AndMask 273280031Sdim 0x200000000000, // XorMask 274280031Sdim 0x100000000000, // ShadowBase 275280031Sdim 0x380000000000, // OriginBase 276280031Sdim}; 277280031Sdim 278288943Sdimstatic const PlatformMemoryMapParams Linux_X86_MemoryMapParams = { 279288943Sdim &Linux_I386_MemoryMapParams, 280288943Sdim &Linux_X86_64_MemoryMapParams, 281280031Sdim}; 282280031Sdim 283288943Sdimstatic const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams = { 284296417Sdim nullptr, 285288943Sdim &Linux_MIPS64_MemoryMapParams, 286280031Sdim}; 287280031Sdim 288288943Sdimstatic const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams = { 289296417Sdim nullptr, 290288943Sdim &Linux_PowerPC64_MemoryMapParams, 291288943Sdim}; 292288943Sdim 293296417Sdimstatic const PlatformMemoryMapParams Linux_ARM_MemoryMapParams = { 294296417Sdim nullptr, 295296417Sdim &Linux_AArch64_MemoryMapParams, 296296417Sdim}; 297296417Sdim 298288943Sdimstatic const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams = { 299288943Sdim &FreeBSD_I386_MemoryMapParams, 300288943Sdim &FreeBSD_X86_64_MemoryMapParams, 301288943Sdim}; 302288943Sdim 303249259Sdim/// \brief An instrumentation pass implementing detection of uninitialized 304249259Sdim/// reads. 305249259Sdim/// 306249259Sdim/// MemorySanitizer: instrument the code in module to find 307249259Sdim/// uninitialized reads. 308249259Sdimclass MemorySanitizer : public FunctionPass { 309249259Sdim public: 310276479Sdim MemorySanitizer(int TrackOrigins = 0) 311261991Sdim : FunctionPass(ID), 312276479Sdim TrackOrigins(std::max(TrackOrigins, (int)ClTrackOrigins)), 313280031Sdim WarningFn(nullptr) {} 314276479Sdim const char *getPassName() const override { return "MemorySanitizer"; } 315276479Sdim bool runOnFunction(Function &F) override; 316276479Sdim bool doInitialization(Module &M) override; 317249259Sdim static char ID; // Pass identification, replacement for typeid. 318249259Sdim 319249259Sdim private: 320249259Sdim void initializeCallbacks(Module &M); 321249259Sdim 322249259Sdim /// \brief Track origins (allocation points) of uninitialized values. 323276479Sdim int TrackOrigins; 324249259Sdim 325249259Sdim LLVMContext *C; 326249259Sdim Type *IntptrTy; 327249259Sdim Type *OriginTy; 328249259Sdim /// \brief Thread-local shadow storage for function parameters. 329249259Sdim GlobalVariable *ParamTLS; 330249259Sdim /// \brief Thread-local origin storage for function parameters. 331249259Sdim GlobalVariable *ParamOriginTLS; 332249259Sdim /// \brief Thread-local shadow storage for function return value. 333249259Sdim GlobalVariable *RetvalTLS; 334249259Sdim /// \brief Thread-local origin storage for function return value. 335249259Sdim GlobalVariable *RetvalOriginTLS; 336249259Sdim /// \brief Thread-local shadow storage for in-register va_arg function 337249259Sdim /// parameters (x86_64-specific). 338249259Sdim GlobalVariable *VAArgTLS; 339249259Sdim /// \brief Thread-local shadow storage for va_arg overflow area 340249259Sdim /// (x86_64-specific). 341249259Sdim GlobalVariable *VAArgOverflowSizeTLS; 342249259Sdim /// \brief Thread-local space used to pass origin value to the UMR reporting 343249259Sdim /// function. 344249259Sdim GlobalVariable *OriginTLS; 345249259Sdim 346249259Sdim /// \brief The run-time callback to print a warning. 347249259Sdim Value *WarningFn; 348276479Sdim // These arrays are indexed by log2(AccessSize). 349276479Sdim Value *MaybeWarningFn[kNumberOfAccessSizes]; 350276479Sdim Value *MaybeStoreOriginFn[kNumberOfAccessSizes]; 351276479Sdim 352249259Sdim /// \brief Run-time helper that generates a new origin value for a stack 353249259Sdim /// allocation. 354261991Sdim Value *MsanSetAllocaOrigin4Fn; 355249259Sdim /// \brief Run-time helper that poisons stack on function entry. 356249259Sdim Value *MsanPoisonStackFn; 357276479Sdim /// \brief Run-time helper that records a store (or any event) of an 358276479Sdim /// uninitialized value and returns an updated origin id encoding this info. 359276479Sdim Value *MsanChainOriginFn; 360249259Sdim /// \brief MSan runtime replacements for memmove, memcpy and memset. 361249259Sdim Value *MemmoveFn, *MemcpyFn, *MemsetFn; 362249259Sdim 363280031Sdim /// \brief Memory map parameters used in application-to-shadow calculation. 364280031Sdim const MemoryMapParams *MapParams; 365280031Sdim 366249259Sdim MDNode *ColdCallWeights; 367249259Sdim /// \brief Branch weights for origin store. 368249259Sdim MDNode *OriginStoreWeights; 369249259Sdim /// \brief An empty volatile inline asm that prevents callback merge. 370249259Sdim InlineAsm *EmptyAsm; 371288943Sdim Function *MsanCtorFunction; 372249259Sdim 373249259Sdim friend struct MemorySanitizerVisitor; 374249259Sdim friend struct VarArgAMD64Helper; 375288943Sdim friend struct VarArgMIPS64Helper; 376296417Sdim friend struct VarArgAArch64Helper; 377249259Sdim}; 378296417Sdim} // anonymous namespace 379249259Sdim 380249259Sdimchar MemorySanitizer::ID = 0; 381249259SdimINITIALIZE_PASS(MemorySanitizer, "msan", 382249259Sdim "MemorySanitizer: detects uninitialized reads.", 383249259Sdim false, false) 384249259Sdim 385276479SdimFunctionPass *llvm::createMemorySanitizerPass(int TrackOrigins) { 386276479Sdim return new MemorySanitizer(TrackOrigins); 387249259Sdim} 388249259Sdim 389249259Sdim/// \brief Create a non-const global initialized with the given string. 390249259Sdim/// 391249259Sdim/// Creates a writable global for Str so that we can pass it to the 392249259Sdim/// run-time lib. Runtime uses first 4 bytes of the string to store the 393249259Sdim/// frame ID, so the string needs to be mutable. 394249259Sdimstatic GlobalVariable *createPrivateNonConstGlobalForString(Module &M, 395249259Sdim StringRef Str) { 396249259Sdim Constant *StrConst = ConstantDataArray::getString(M.getContext(), Str); 397249259Sdim return new GlobalVariable(M, StrConst->getType(), /*isConstant=*/false, 398249259Sdim GlobalValue::PrivateLinkage, StrConst, ""); 399249259Sdim} 400249259Sdim 401249259Sdim/// \brief Insert extern declaration of runtime-provided functions and globals. 402249259Sdimvoid MemorySanitizer::initializeCallbacks(Module &M) { 403249259Sdim // Only do this once. 404249259Sdim if (WarningFn) 405249259Sdim return; 406249259Sdim 407249259Sdim IRBuilder<> IRB(*C); 408249259Sdim // Create the callback. 409249259Sdim // FIXME: this function should have "Cold" calling conv, 410249259Sdim // which is not yet implemented. 411249259Sdim StringRef WarningFnName = ClKeepGoing ? "__msan_warning" 412249259Sdim : "__msan_warning_noreturn"; 413280031Sdim WarningFn = M.getOrInsertFunction(WarningFnName, IRB.getVoidTy(), nullptr); 414249259Sdim 415276479Sdim for (size_t AccessSizeIndex = 0; AccessSizeIndex < kNumberOfAccessSizes; 416276479Sdim AccessSizeIndex++) { 417276479Sdim unsigned AccessSize = 1 << AccessSizeIndex; 418276479Sdim std::string FunctionName = "__msan_maybe_warning_" + itostr(AccessSize); 419276479Sdim MaybeWarningFn[AccessSizeIndex] = M.getOrInsertFunction( 420276479Sdim FunctionName, IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), 421280031Sdim IRB.getInt32Ty(), nullptr); 422276479Sdim 423276479Sdim FunctionName = "__msan_maybe_store_origin_" + itostr(AccessSize); 424276479Sdim MaybeStoreOriginFn[AccessSizeIndex] = M.getOrInsertFunction( 425276479Sdim FunctionName, IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), 426280031Sdim IRB.getInt8PtrTy(), IRB.getInt32Ty(), nullptr); 427276479Sdim } 428276479Sdim 429261991Sdim MsanSetAllocaOrigin4Fn = M.getOrInsertFunction( 430261991Sdim "__msan_set_alloca_origin4", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy, 431280031Sdim IRB.getInt8PtrTy(), IntptrTy, nullptr); 432280031Sdim MsanPoisonStackFn = 433280031Sdim M.getOrInsertFunction("__msan_poison_stack", IRB.getVoidTy(), 434280031Sdim IRB.getInt8PtrTy(), IntptrTy, nullptr); 435276479Sdim MsanChainOriginFn = M.getOrInsertFunction( 436280031Sdim "__msan_chain_origin", IRB.getInt32Ty(), IRB.getInt32Ty(), nullptr); 437249259Sdim MemmoveFn = M.getOrInsertFunction( 438249259Sdim "__msan_memmove", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), 439280031Sdim IRB.getInt8PtrTy(), IntptrTy, nullptr); 440249259Sdim MemcpyFn = M.getOrInsertFunction( 441249259Sdim "__msan_memcpy", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), 442280031Sdim IntptrTy, nullptr); 443249259Sdim MemsetFn = M.getOrInsertFunction( 444249259Sdim "__msan_memset", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt32Ty(), 445280031Sdim IntptrTy, nullptr); 446249259Sdim 447249259Sdim // Create globals. 448249259Sdim RetvalTLS = new GlobalVariable( 449280031Sdim M, ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8), false, 450276479Sdim GlobalVariable::ExternalLinkage, nullptr, "__msan_retval_tls", nullptr, 451261991Sdim GlobalVariable::InitialExecTLSModel); 452249259Sdim RetvalOriginTLS = new GlobalVariable( 453276479Sdim M, OriginTy, false, GlobalVariable::ExternalLinkage, nullptr, 454276479Sdim "__msan_retval_origin_tls", nullptr, GlobalVariable::InitialExecTLSModel); 455249259Sdim 456249259Sdim ParamTLS = new GlobalVariable( 457280031Sdim M, ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), false, 458276479Sdim GlobalVariable::ExternalLinkage, nullptr, "__msan_param_tls", nullptr, 459261991Sdim GlobalVariable::InitialExecTLSModel); 460249259Sdim ParamOriginTLS = new GlobalVariable( 461280031Sdim M, ArrayType::get(OriginTy, kParamTLSSize / 4), false, 462280031Sdim GlobalVariable::ExternalLinkage, nullptr, "__msan_param_origin_tls", 463280031Sdim nullptr, GlobalVariable::InitialExecTLSModel); 464249259Sdim 465249259Sdim VAArgTLS = new GlobalVariable( 466280031Sdim M, ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), false, 467276479Sdim GlobalVariable::ExternalLinkage, nullptr, "__msan_va_arg_tls", nullptr, 468261991Sdim GlobalVariable::InitialExecTLSModel); 469249259Sdim VAArgOverflowSizeTLS = new GlobalVariable( 470276479Sdim M, IRB.getInt64Ty(), false, GlobalVariable::ExternalLinkage, nullptr, 471276479Sdim "__msan_va_arg_overflow_size_tls", nullptr, 472261991Sdim GlobalVariable::InitialExecTLSModel); 473249259Sdim OriginTLS = new GlobalVariable( 474276479Sdim M, IRB.getInt32Ty(), false, GlobalVariable::ExternalLinkage, nullptr, 475276479Sdim "__msan_origin_tls", nullptr, GlobalVariable::InitialExecTLSModel); 476249259Sdim 477249259Sdim // We insert an empty inline asm after __msan_report* to avoid callback merge. 478249259Sdim EmptyAsm = InlineAsm::get(FunctionType::get(IRB.getVoidTy(), false), 479249259Sdim StringRef(""), StringRef(""), 480249259Sdim /*hasSideEffects=*/true); 481249259Sdim} 482249259Sdim 483249259Sdim/// \brief Module-level initialization. 484249259Sdim/// 485249259Sdim/// inserts a call to __msan_init to the module's constructor list. 486249259Sdimbool MemorySanitizer::doInitialization(Module &M) { 487288943Sdim auto &DL = M.getDataLayout(); 488276479Sdim 489280031Sdim Triple TargetTriple(M.getTargetTriple()); 490288943Sdim switch (TargetTriple.getOS()) { 491288943Sdim case Triple::FreeBSD: 492288943Sdim switch (TargetTriple.getArch()) { 493288943Sdim case Triple::x86_64: 494288943Sdim MapParams = FreeBSD_X86_MemoryMapParams.bits64; 495288943Sdim break; 496288943Sdim case Triple::x86: 497288943Sdim MapParams = FreeBSD_X86_MemoryMapParams.bits32; 498288943Sdim break; 499288943Sdim default: 500288943Sdim report_fatal_error("unsupported architecture"); 501288943Sdim } 502249259Sdim break; 503288943Sdim case Triple::Linux: 504288943Sdim switch (TargetTriple.getArch()) { 505288943Sdim case Triple::x86_64: 506288943Sdim MapParams = Linux_X86_MemoryMapParams.bits64; 507288943Sdim break; 508288943Sdim case Triple::x86: 509288943Sdim MapParams = Linux_X86_MemoryMapParams.bits32; 510288943Sdim break; 511288943Sdim case Triple::mips64: 512288943Sdim case Triple::mips64el: 513288943Sdim MapParams = Linux_MIPS_MemoryMapParams.bits64; 514288943Sdim break; 515288943Sdim case Triple::ppc64: 516288943Sdim case Triple::ppc64le: 517288943Sdim MapParams = Linux_PowerPC_MemoryMapParams.bits64; 518288943Sdim break; 519296417Sdim case Triple::aarch64: 520296417Sdim case Triple::aarch64_be: 521296417Sdim MapParams = Linux_ARM_MemoryMapParams.bits64; 522296417Sdim break; 523288943Sdim default: 524288943Sdim report_fatal_error("unsupported architecture"); 525288943Sdim } 526249259Sdim break; 527249259Sdim default: 528288943Sdim report_fatal_error("unsupported operating system"); 529249259Sdim } 530249259Sdim 531288943Sdim C = &(M.getContext()); 532249259Sdim IRBuilder<> IRB(*C); 533276479Sdim IntptrTy = IRB.getIntPtrTy(DL); 534249259Sdim OriginTy = IRB.getInt32Ty(); 535249259Sdim 536249259Sdim ColdCallWeights = MDBuilder(*C).createBranchWeights(1, 1000); 537249259Sdim OriginStoreWeights = MDBuilder(*C).createBranchWeights(1, 1000); 538249259Sdim 539288943Sdim std::tie(MsanCtorFunction, std::ignore) = 540288943Sdim createSanitizerCtorAndInitFunctions(M, kMsanModuleCtorName, kMsanInitName, 541288943Sdim /*InitArgTypes=*/{}, 542288943Sdim /*InitArgs=*/{}); 543249259Sdim 544288943Sdim appendToGlobalCtors(M, MsanCtorFunction, 0); 545288943Sdim 546261991Sdim if (TrackOrigins) 547261991Sdim new GlobalVariable(M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage, 548261991Sdim IRB.getInt32(TrackOrigins), "__msan_track_origins"); 549249259Sdim 550261991Sdim if (ClKeepGoing) 551261991Sdim new GlobalVariable(M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage, 552261991Sdim IRB.getInt32(ClKeepGoing), "__msan_keep_going"); 553249259Sdim 554249259Sdim return true; 555249259Sdim} 556249259Sdim 557249259Sdimnamespace { 558249259Sdim 559249259Sdim/// \brief A helper class that handles instrumentation of VarArg 560249259Sdim/// functions on a particular platform. 561249259Sdim/// 562249259Sdim/// Implementations are expected to insert the instrumentation 563249259Sdim/// necessary to propagate argument shadow through VarArg function 564249259Sdim/// calls. Visit* methods are called during an InstVisitor pass over 565249259Sdim/// the function, and should avoid creating new basic blocks. A new 566249259Sdim/// instance of this class is created for each instrumented function. 567249259Sdimstruct VarArgHelper { 568249259Sdim /// \brief Visit a CallSite. 569249259Sdim virtual void visitCallSite(CallSite &CS, IRBuilder<> &IRB) = 0; 570249259Sdim 571249259Sdim /// \brief Visit a va_start call. 572249259Sdim virtual void visitVAStartInst(VAStartInst &I) = 0; 573249259Sdim 574249259Sdim /// \brief Visit a va_copy call. 575249259Sdim virtual void visitVACopyInst(VACopyInst &I) = 0; 576249259Sdim 577249259Sdim /// \brief Finalize function instrumentation. 578249259Sdim /// 579249259Sdim /// This method is called after visiting all interesting (see above) 580249259Sdim /// instructions in a function. 581249259Sdim virtual void finalizeInstrumentation() = 0; 582249259Sdim 583249259Sdim virtual ~VarArgHelper() {} 584249259Sdim}; 585249259Sdim 586249259Sdimstruct MemorySanitizerVisitor; 587249259Sdim 588249259SdimVarArgHelper* 589249259SdimCreateVarArgHelper(Function &Func, MemorySanitizer &Msan, 590249259Sdim MemorySanitizerVisitor &Visitor); 591249259Sdim 592276479Sdimunsigned TypeSizeToSizeIndex(unsigned TypeSize) { 593276479Sdim if (TypeSize <= 8) return 0; 594276479Sdim return Log2_32_Ceil(TypeSize / 8); 595276479Sdim} 596276479Sdim 597249259Sdim/// This class does all the work for a given function. Store and Load 598249259Sdim/// instructions store and load corresponding shadow and origin 599249259Sdim/// values. Most instructions propagate shadow from arguments to their 600249259Sdim/// return values. Certain instructions (most importantly, BranchInst) 601249259Sdim/// test their argument shadow and print reports (with a runtime call) if it's 602249259Sdim/// non-zero. 603249259Sdimstruct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> { 604249259Sdim Function &F; 605249259Sdim MemorySanitizer &MS; 606249259Sdim SmallVector<PHINode *, 16> ShadowPHINodes, OriginPHINodes; 607249259Sdim ValueMap<Value*, Value*> ShadowMap, OriginMap; 608276479Sdim std::unique_ptr<VarArgHelper> VAHelper; 609261991Sdim 610261991Sdim // The following flags disable parts of MSan instrumentation based on 611261991Sdim // blacklist contents and command-line options. 612249259Sdim bool InsertChecks; 613276479Sdim bool PropagateShadow; 614261991Sdim bool PoisonStack; 615261991Sdim bool PoisonUndef; 616261991Sdim bool CheckReturnValue; 617249259Sdim 618249259Sdim struct ShadowOriginAndInsertPoint { 619261991Sdim Value *Shadow; 620261991Sdim Value *Origin; 621249259Sdim Instruction *OrigIns; 622261991Sdim ShadowOriginAndInsertPoint(Value *S, Value *O, Instruction *I) 623249259Sdim : Shadow(S), Origin(O), OrigIns(I) { } 624249259Sdim }; 625249259Sdim SmallVector<ShadowOriginAndInsertPoint, 16> InstrumentationList; 626249259Sdim SmallVector<Instruction*, 16> StoreList; 627249259Sdim 628249259Sdim MemorySanitizerVisitor(Function &F, MemorySanitizer &MS) 629249259Sdim : F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)) { 630288943Sdim bool SanitizeFunction = F.hasFnAttribute(Attribute::SanitizeMemory); 631261991Sdim InsertChecks = SanitizeFunction; 632276479Sdim PropagateShadow = SanitizeFunction; 633261991Sdim PoisonStack = SanitizeFunction && ClPoisonStack; 634261991Sdim PoisonUndef = SanitizeFunction && ClPoisonUndef; 635261991Sdim // FIXME: Consider using SpecialCaseList to specify a list of functions that 636261991Sdim // must always return fully initialized values. For now, we hardcode "main". 637261991Sdim CheckReturnValue = SanitizeFunction && (F.getName() == "main"); 638249259Sdim 639249259Sdim DEBUG(if (!InsertChecks) 640249259Sdim dbgs() << "MemorySanitizer is not inserting checks into '" 641249259Sdim << F.getName() << "'\n"); 642249259Sdim } 643249259Sdim 644276479Sdim Value *updateOrigin(Value *V, IRBuilder<> &IRB) { 645276479Sdim if (MS.TrackOrigins <= 1) return V; 646276479Sdim return IRB.CreateCall(MS.MsanChainOriginFn, V); 647276479Sdim } 648249259Sdim 649288943Sdim Value *originToIntptr(IRBuilder<> &IRB, Value *Origin) { 650288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 651288943Sdim unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy); 652288943Sdim if (IntptrSize == kOriginSize) return Origin; 653288943Sdim assert(IntptrSize == kOriginSize * 2); 654288943Sdim Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false); 655288943Sdim return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8)); 656288943Sdim } 657288943Sdim 658288943Sdim /// \brief Fill memory range with the given origin value. 659288943Sdim void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr, 660288943Sdim unsigned Size, unsigned Alignment) { 661288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 662288943Sdim unsigned IntptrAlignment = DL.getABITypeAlignment(MS.IntptrTy); 663288943Sdim unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy); 664288943Sdim assert(IntptrAlignment >= kMinOriginAlignment); 665288943Sdim assert(IntptrSize >= kOriginSize); 666288943Sdim 667288943Sdim unsigned Ofs = 0; 668288943Sdim unsigned CurrentAlignment = Alignment; 669288943Sdim if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) { 670288943Sdim Value *IntptrOrigin = originToIntptr(IRB, Origin); 671288943Sdim Value *IntptrOriginPtr = 672288943Sdim IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0)); 673288943Sdim for (unsigned i = 0; i < Size / IntptrSize; ++i) { 674288943Sdim Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i) 675288943Sdim : IntptrOriginPtr; 676288943Sdim IRB.CreateAlignedStore(IntptrOrigin, Ptr, CurrentAlignment); 677288943Sdim Ofs += IntptrSize / kOriginSize; 678288943Sdim CurrentAlignment = IntptrAlignment; 679288943Sdim } 680288943Sdim } 681288943Sdim 682288943Sdim for (unsigned i = Ofs; i < (Size + kOriginSize - 1) / kOriginSize; ++i) { 683288943Sdim Value *GEP = 684288943Sdim i ? IRB.CreateConstGEP1_32(nullptr, OriginPtr, i) : OriginPtr; 685288943Sdim IRB.CreateAlignedStore(Origin, GEP, CurrentAlignment); 686288943Sdim CurrentAlignment = kMinOriginAlignment; 687288943Sdim } 688288943Sdim } 689288943Sdim 690276479Sdim void storeOrigin(IRBuilder<> &IRB, Value *Addr, Value *Shadow, Value *Origin, 691276479Sdim unsigned Alignment, bool AsCall) { 692288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 693280031Sdim unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment); 694288943Sdim unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType()); 695296417Sdim if (Shadow->getType()->isAggregateType()) { 696288943Sdim paintOrigin(IRB, updateOrigin(Origin, IRB), 697288943Sdim getOriginPtr(Addr, IRB, Alignment), StoreSize, 698288943Sdim OriginAlignment); 699276479Sdim } else { 700276479Sdim Value *ConvertedShadow = convertToShadowTyNoVec(Shadow, IRB); 701288943Sdim Constant *ConstantShadow = dyn_cast_or_null<Constant>(ConvertedShadow); 702288943Sdim if (ConstantShadow) { 703288943Sdim if (ClCheckConstantShadow && !ConstantShadow->isZeroValue()) 704288943Sdim paintOrigin(IRB, updateOrigin(Origin, IRB), 705288943Sdim getOriginPtr(Addr, IRB, Alignment), StoreSize, 706288943Sdim OriginAlignment); 707288943Sdim return; 708288943Sdim } 709288943Sdim 710276479Sdim unsigned TypeSizeInBits = 711288943Sdim DL.getTypeSizeInBits(ConvertedShadow->getType()); 712276479Sdim unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits); 713276479Sdim if (AsCall && SizeIndex < kNumberOfAccessSizes) { 714276479Sdim Value *Fn = MS.MaybeStoreOriginFn[SizeIndex]; 715276479Sdim Value *ConvertedShadow2 = IRB.CreateZExt( 716276479Sdim ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex))); 717288943Sdim IRB.CreateCall(Fn, {ConvertedShadow2, 718288943Sdim IRB.CreatePointerCast(Addr, IRB.getInt8PtrTy()), 719288943Sdim Origin}); 720276479Sdim } else { 721276479Sdim Value *Cmp = IRB.CreateICmpNE( 722276479Sdim ConvertedShadow, getCleanShadow(ConvertedShadow), "_mscmp"); 723276479Sdim Instruction *CheckTerm = SplitBlockAndInsertIfThen( 724296417Sdim Cmp, &*IRB.GetInsertPoint(), false, MS.OriginStoreWeights); 725276479Sdim IRBuilder<> IRBNew(CheckTerm); 726288943Sdim paintOrigin(IRBNew, updateOrigin(Origin, IRBNew), 727288943Sdim getOriginPtr(Addr, IRBNew, Alignment), StoreSize, 728288943Sdim OriginAlignment); 729276479Sdim } 730276479Sdim } 731276479Sdim } 732276479Sdim 733276479Sdim void materializeStores(bool InstrumentWithCalls) { 734276479Sdim for (auto Inst : StoreList) { 735276479Sdim StoreInst &SI = *dyn_cast<StoreInst>(Inst); 736276479Sdim 737276479Sdim IRBuilder<> IRB(&SI); 738276479Sdim Value *Val = SI.getValueOperand(); 739276479Sdim Value *Addr = SI.getPointerOperand(); 740276479Sdim Value *Shadow = SI.isAtomic() ? getCleanShadow(Val) : getShadow(Val); 741249259Sdim Value *ShadowPtr = getShadowPtr(Addr, Shadow->getType(), IRB); 742249259Sdim 743249259Sdim StoreInst *NewSI = 744276479Sdim IRB.CreateAlignedStore(Shadow, ShadowPtr, SI.getAlignment()); 745249259Sdim DEBUG(dbgs() << " STORE: " << *NewSI << "\n"); 746249259Sdim (void)NewSI; 747249259Sdim 748276479Sdim if (ClCheckAccessAddress) insertShadowCheck(Addr, &SI); 749249259Sdim 750276479Sdim if (SI.isAtomic()) SI.setOrdering(addReleaseOrdering(SI.getOrdering())); 751261991Sdim 752280031Sdim if (MS.TrackOrigins && !SI.isAtomic()) 753280031Sdim storeOrigin(IRB, Addr, Shadow, getOrigin(Val), SI.getAlignment(), 754276479Sdim InstrumentWithCalls); 755249259Sdim } 756249259Sdim } 757249259Sdim 758276479Sdim void materializeOneCheck(Instruction *OrigIns, Value *Shadow, Value *Origin, 759276479Sdim bool AsCall) { 760276479Sdim IRBuilder<> IRB(OrigIns); 761276479Sdim DEBUG(dbgs() << " SHAD0 : " << *Shadow << "\n"); 762276479Sdim Value *ConvertedShadow = convertToShadowTyNoVec(Shadow, IRB); 763276479Sdim DEBUG(dbgs() << " SHAD1 : " << *ConvertedShadow << "\n"); 764288943Sdim 765288943Sdim Constant *ConstantShadow = dyn_cast_or_null<Constant>(ConvertedShadow); 766288943Sdim if (ConstantShadow) { 767288943Sdim if (ClCheckConstantShadow && !ConstantShadow->isZeroValue()) { 768288943Sdim if (MS.TrackOrigins) { 769288943Sdim IRB.CreateStore(Origin ? (Value *)Origin : (Value *)IRB.getInt32(0), 770288943Sdim MS.OriginTLS); 771288943Sdim } 772288943Sdim IRB.CreateCall(MS.WarningFn, {}); 773288943Sdim IRB.CreateCall(MS.EmptyAsm, {}); 774288943Sdim // FIXME: Insert UnreachableInst if !ClKeepGoing? 775288943Sdim // This may invalidate some of the following checks and needs to be done 776288943Sdim // at the very end. 777288943Sdim } 778288943Sdim return; 779288943Sdim } 780288943Sdim 781288943Sdim const DataLayout &DL = OrigIns->getModule()->getDataLayout(); 782288943Sdim 783288943Sdim unsigned TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType()); 784276479Sdim unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits); 785276479Sdim if (AsCall && SizeIndex < kNumberOfAccessSizes) { 786276479Sdim Value *Fn = MS.MaybeWarningFn[SizeIndex]; 787276479Sdim Value *ConvertedShadow2 = 788276479Sdim IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex))); 789288943Sdim IRB.CreateCall(Fn, {ConvertedShadow2, MS.TrackOrigins && Origin 790276479Sdim ? Origin 791288943Sdim : (Value *)IRB.getInt32(0)}); 792276479Sdim } else { 793249259Sdim Value *Cmp = IRB.CreateICmpNE(ConvertedShadow, 794249259Sdim getCleanShadow(ConvertedShadow), "_mscmp"); 795276479Sdim Instruction *CheckTerm = SplitBlockAndInsertIfThen( 796276479Sdim Cmp, OrigIns, 797276479Sdim /* Unreachable */ !ClKeepGoing, MS.ColdCallWeights); 798249259Sdim 799249259Sdim IRB.SetInsertPoint(CheckTerm); 800249259Sdim if (MS.TrackOrigins) { 801276479Sdim IRB.CreateStore(Origin ? (Value *)Origin : (Value *)IRB.getInt32(0), 802249259Sdim MS.OriginTLS); 803249259Sdim } 804288943Sdim IRB.CreateCall(MS.WarningFn, {}); 805288943Sdim IRB.CreateCall(MS.EmptyAsm, {}); 806249259Sdim DEBUG(dbgs() << " CHECK: " << *Cmp << "\n"); 807249259Sdim } 808276479Sdim } 809276479Sdim 810276479Sdim void materializeChecks(bool InstrumentWithCalls) { 811276479Sdim for (const auto &ShadowData : InstrumentationList) { 812276479Sdim Instruction *OrigIns = ShadowData.OrigIns; 813276479Sdim Value *Shadow = ShadowData.Shadow; 814276479Sdim Value *Origin = ShadowData.Origin; 815276479Sdim materializeOneCheck(OrigIns, Shadow, Origin, InstrumentWithCalls); 816276479Sdim } 817249259Sdim DEBUG(dbgs() << "DONE:\n" << F); 818249259Sdim } 819249259Sdim 820249259Sdim /// \brief Add MemorySanitizer instrumentation to a function. 821249259Sdim bool runOnFunction() { 822249259Sdim MS.initializeCallbacks(*F.getParent()); 823249259Sdim 824249259Sdim // In the presence of unreachable blocks, we may see Phi nodes with 825249259Sdim // incoming nodes from such blocks. Since InstVisitor skips unreachable 826249259Sdim // blocks, such nodes will not have any shadow value associated with them. 827249259Sdim // It's easier to remove unreachable blocks than deal with missing shadow. 828249259Sdim removeUnreachableBlocks(F); 829249259Sdim 830249259Sdim // Iterate all BBs in depth-first order and create shadow instructions 831249259Sdim // for all instructions (where applicable). 832249259Sdim // For PHI nodes we create dummy shadow PHIs which will be finalized later. 833276479Sdim for (BasicBlock *BB : depth_first(&F.getEntryBlock())) 834249259Sdim visit(*BB); 835249259Sdim 836276479Sdim 837249259Sdim // Finalize PHI nodes. 838276479Sdim for (PHINode *PN : ShadowPHINodes) { 839249259Sdim PHINode *PNS = cast<PHINode>(getShadow(PN)); 840276479Sdim PHINode *PNO = MS.TrackOrigins ? cast<PHINode>(getOrigin(PN)) : nullptr; 841249259Sdim size_t NumValues = PN->getNumIncomingValues(); 842249259Sdim for (size_t v = 0; v < NumValues; v++) { 843249259Sdim PNS->addIncoming(getShadow(PN, v), PN->getIncomingBlock(v)); 844276479Sdim if (PNO) PNO->addIncoming(getOrigin(PN, v), PN->getIncomingBlock(v)); 845249259Sdim } 846249259Sdim } 847249259Sdim 848249259Sdim VAHelper->finalizeInstrumentation(); 849249259Sdim 850276479Sdim bool InstrumentWithCalls = ClInstrumentationWithCallThreshold >= 0 && 851276479Sdim InstrumentationList.size() + StoreList.size() > 852276479Sdim (unsigned)ClInstrumentationWithCallThreshold; 853276479Sdim 854249259Sdim // Delayed instrumentation of StoreInst. 855249259Sdim // This may add new checks to be inserted later. 856276479Sdim materializeStores(InstrumentWithCalls); 857249259Sdim 858249259Sdim // Insert shadow value checks. 859276479Sdim materializeChecks(InstrumentWithCalls); 860249259Sdim 861249259Sdim return true; 862249259Sdim } 863249259Sdim 864249259Sdim /// \brief Compute the shadow type that corresponds to a given Value. 865249259Sdim Type *getShadowTy(Value *V) { 866249259Sdim return getShadowTy(V->getType()); 867249259Sdim } 868249259Sdim 869249259Sdim /// \brief Compute the shadow type that corresponds to a given Type. 870249259Sdim Type *getShadowTy(Type *OrigTy) { 871249259Sdim if (!OrigTy->isSized()) { 872276479Sdim return nullptr; 873249259Sdim } 874249259Sdim // For integer type, shadow is the same as the original type. 875249259Sdim // This may return weird-sized types like i1. 876249259Sdim if (IntegerType *IT = dyn_cast<IntegerType>(OrigTy)) 877249259Sdim return IT; 878288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 879249259Sdim if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) { 880288943Sdim uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType()); 881249259Sdim return VectorType::get(IntegerType::get(*MS.C, EltSize), 882249259Sdim VT->getNumElements()); 883249259Sdim } 884280031Sdim if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) { 885280031Sdim return ArrayType::get(getShadowTy(AT->getElementType()), 886280031Sdim AT->getNumElements()); 887280031Sdim } 888249259Sdim if (StructType *ST = dyn_cast<StructType>(OrigTy)) { 889249259Sdim SmallVector<Type*, 4> Elements; 890249259Sdim for (unsigned i = 0, n = ST->getNumElements(); i < n; i++) 891249259Sdim Elements.push_back(getShadowTy(ST->getElementType(i))); 892249259Sdim StructType *Res = StructType::get(*MS.C, Elements, ST->isPacked()); 893249259Sdim DEBUG(dbgs() << "getShadowTy: " << *ST << " ===> " << *Res << "\n"); 894249259Sdim return Res; 895249259Sdim } 896288943Sdim uint32_t TypeSize = DL.getTypeSizeInBits(OrigTy); 897249259Sdim return IntegerType::get(*MS.C, TypeSize); 898249259Sdim } 899249259Sdim 900249259Sdim /// \brief Flatten a vector type. 901249259Sdim Type *getShadowTyNoVec(Type *ty) { 902249259Sdim if (VectorType *vt = dyn_cast<VectorType>(ty)) 903249259Sdim return IntegerType::get(*MS.C, vt->getBitWidth()); 904249259Sdim return ty; 905249259Sdim } 906249259Sdim 907249259Sdim /// \brief Convert a shadow value to it's flattened variant. 908249259Sdim Value *convertToShadowTyNoVec(Value *V, IRBuilder<> &IRB) { 909249259Sdim Type *Ty = V->getType(); 910249259Sdim Type *NoVecTy = getShadowTyNoVec(Ty); 911249259Sdim if (Ty == NoVecTy) return V; 912249259Sdim return IRB.CreateBitCast(V, NoVecTy); 913249259Sdim } 914249259Sdim 915280031Sdim /// \brief Compute the integer shadow offset that corresponds to a given 916280031Sdim /// application address. 917280031Sdim /// 918280031Sdim /// Offset = (Addr & ~AndMask) ^ XorMask 919280031Sdim Value *getShadowPtrOffset(Value *Addr, IRBuilder<> &IRB) { 920296417Sdim Value *OffsetLong = IRB.CreatePointerCast(Addr, MS.IntptrTy); 921296417Sdim 922280031Sdim uint64_t AndMask = MS.MapParams->AndMask; 923296417Sdim if (AndMask) 924296417Sdim OffsetLong = 925296417Sdim IRB.CreateAnd(OffsetLong, ConstantInt::get(MS.IntptrTy, ~AndMask)); 926280031Sdim 927280031Sdim uint64_t XorMask = MS.MapParams->XorMask; 928296417Sdim if (XorMask) 929296417Sdim OffsetLong = 930296417Sdim IRB.CreateXor(OffsetLong, ConstantInt::get(MS.IntptrTy, XorMask)); 931280031Sdim return OffsetLong; 932280031Sdim } 933280031Sdim 934249259Sdim /// \brief Compute the shadow address that corresponds to a given application 935249259Sdim /// address. 936249259Sdim /// 937280031Sdim /// Shadow = ShadowBase + Offset 938249259Sdim Value *getShadowPtr(Value *Addr, Type *ShadowTy, 939249259Sdim IRBuilder<> &IRB) { 940280031Sdim Value *ShadowLong = getShadowPtrOffset(Addr, IRB); 941280031Sdim uint64_t ShadowBase = MS.MapParams->ShadowBase; 942280031Sdim if (ShadowBase != 0) 943280031Sdim ShadowLong = 944280031Sdim IRB.CreateAdd(ShadowLong, 945280031Sdim ConstantInt::get(MS.IntptrTy, ShadowBase)); 946249259Sdim return IRB.CreateIntToPtr(ShadowLong, PointerType::get(ShadowTy, 0)); 947249259Sdim } 948249259Sdim 949249259Sdim /// \brief Compute the origin address that corresponds to a given application 950249259Sdim /// address. 951249259Sdim /// 952280031Sdim /// OriginAddr = (OriginBase + Offset) & ~3ULL 953280031Sdim Value *getOriginPtr(Value *Addr, IRBuilder<> &IRB, unsigned Alignment) { 954280031Sdim Value *OriginLong = getShadowPtrOffset(Addr, IRB); 955280031Sdim uint64_t OriginBase = MS.MapParams->OriginBase; 956280031Sdim if (OriginBase != 0) 957280031Sdim OriginLong = 958280031Sdim IRB.CreateAdd(OriginLong, 959280031Sdim ConstantInt::get(MS.IntptrTy, OriginBase)); 960280031Sdim if (Alignment < kMinOriginAlignment) { 961280031Sdim uint64_t Mask = kMinOriginAlignment - 1; 962280031Sdim OriginLong = IRB.CreateAnd(OriginLong, 963280031Sdim ConstantInt::get(MS.IntptrTy, ~Mask)); 964280031Sdim } 965280031Sdim return IRB.CreateIntToPtr(OriginLong, 966280031Sdim PointerType::get(IRB.getInt32Ty(), 0)); 967249259Sdim } 968249259Sdim 969249259Sdim /// \brief Compute the shadow address for a given function argument. 970249259Sdim /// 971249259Sdim /// Shadow = ParamTLS+ArgOffset. 972249259Sdim Value *getShadowPtrForArgument(Value *A, IRBuilder<> &IRB, 973249259Sdim int ArgOffset) { 974249259Sdim Value *Base = IRB.CreatePointerCast(MS.ParamTLS, MS.IntptrTy); 975249259Sdim Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset)); 976249259Sdim return IRB.CreateIntToPtr(Base, PointerType::get(getShadowTy(A), 0), 977249259Sdim "_msarg"); 978249259Sdim } 979249259Sdim 980249259Sdim /// \brief Compute the origin address for a given function argument. 981249259Sdim Value *getOriginPtrForArgument(Value *A, IRBuilder<> &IRB, 982249259Sdim int ArgOffset) { 983276479Sdim if (!MS.TrackOrigins) return nullptr; 984249259Sdim Value *Base = IRB.CreatePointerCast(MS.ParamOriginTLS, MS.IntptrTy); 985249259Sdim Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset)); 986249259Sdim return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0), 987249259Sdim "_msarg_o"); 988249259Sdim } 989249259Sdim 990249259Sdim /// \brief Compute the shadow address for a retval. 991249259Sdim Value *getShadowPtrForRetval(Value *A, IRBuilder<> &IRB) { 992249259Sdim Value *Base = IRB.CreatePointerCast(MS.RetvalTLS, MS.IntptrTy); 993249259Sdim return IRB.CreateIntToPtr(Base, PointerType::get(getShadowTy(A), 0), 994249259Sdim "_msret"); 995249259Sdim } 996249259Sdim 997249259Sdim /// \brief Compute the origin address for a retval. 998249259Sdim Value *getOriginPtrForRetval(IRBuilder<> &IRB) { 999249259Sdim // We keep a single origin for the entire retval. Might be too optimistic. 1000249259Sdim return MS.RetvalOriginTLS; 1001249259Sdim } 1002249259Sdim 1003249259Sdim /// \brief Set SV to be the shadow value for V. 1004249259Sdim void setShadow(Value *V, Value *SV) { 1005249259Sdim assert(!ShadowMap.count(V) && "Values may only have one shadow"); 1006276479Sdim ShadowMap[V] = PropagateShadow ? SV : getCleanShadow(V); 1007249259Sdim } 1008249259Sdim 1009249259Sdim /// \brief Set Origin to be the origin value for V. 1010249259Sdim void setOrigin(Value *V, Value *Origin) { 1011249259Sdim if (!MS.TrackOrigins) return; 1012249259Sdim assert(!OriginMap.count(V) && "Values may only have one origin"); 1013249259Sdim DEBUG(dbgs() << "ORIGIN: " << *V << " ==> " << *Origin << "\n"); 1014249259Sdim OriginMap[V] = Origin; 1015249259Sdim } 1016249259Sdim 1017249259Sdim /// \brief Create a clean shadow value for a given value. 1018249259Sdim /// 1019249259Sdim /// Clean shadow (all zeroes) means all bits of the value are defined 1020249259Sdim /// (initialized). 1021249259Sdim Constant *getCleanShadow(Value *V) { 1022249259Sdim Type *ShadowTy = getShadowTy(V); 1023249259Sdim if (!ShadowTy) 1024276479Sdim return nullptr; 1025249259Sdim return Constant::getNullValue(ShadowTy); 1026249259Sdim } 1027249259Sdim 1028249259Sdim /// \brief Create a dirty shadow of a given shadow type. 1029249259Sdim Constant *getPoisonedShadow(Type *ShadowTy) { 1030249259Sdim assert(ShadowTy); 1031249259Sdim if (isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy)) 1032249259Sdim return Constant::getAllOnesValue(ShadowTy); 1033280031Sdim if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy)) { 1034280031Sdim SmallVector<Constant *, 4> Vals(AT->getNumElements(), 1035280031Sdim getPoisonedShadow(AT->getElementType())); 1036280031Sdim return ConstantArray::get(AT, Vals); 1037280031Sdim } 1038280031Sdim if (StructType *ST = dyn_cast<StructType>(ShadowTy)) { 1039280031Sdim SmallVector<Constant *, 4> Vals; 1040280031Sdim for (unsigned i = 0, n = ST->getNumElements(); i < n; i++) 1041280031Sdim Vals.push_back(getPoisonedShadow(ST->getElementType(i))); 1042280031Sdim return ConstantStruct::get(ST, Vals); 1043280031Sdim } 1044280031Sdim llvm_unreachable("Unexpected shadow type"); 1045249259Sdim } 1046249259Sdim 1047249259Sdim /// \brief Create a dirty shadow for a given value. 1048249259Sdim Constant *getPoisonedShadow(Value *V) { 1049249259Sdim Type *ShadowTy = getShadowTy(V); 1050249259Sdim if (!ShadowTy) 1051276479Sdim return nullptr; 1052249259Sdim return getPoisonedShadow(ShadowTy); 1053249259Sdim } 1054249259Sdim 1055249259Sdim /// \brief Create a clean (zero) origin. 1056249259Sdim Value *getCleanOrigin() { 1057249259Sdim return Constant::getNullValue(MS.OriginTy); 1058249259Sdim } 1059249259Sdim 1060249259Sdim /// \brief Get the shadow value for a given Value. 1061249259Sdim /// 1062249259Sdim /// This function either returns the value set earlier with setShadow, 1063249259Sdim /// or extracts if from ParamTLS (for function arguments). 1064249259Sdim Value *getShadow(Value *V) { 1065276479Sdim if (!PropagateShadow) return getCleanShadow(V); 1066249259Sdim if (Instruction *I = dyn_cast<Instruction>(V)) { 1067249259Sdim // For instructions the shadow is already stored in the map. 1068249259Sdim Value *Shadow = ShadowMap[V]; 1069249259Sdim if (!Shadow) { 1070249259Sdim DEBUG(dbgs() << "No shadow: " << *V << "\n" << *(I->getParent())); 1071249259Sdim (void)I; 1072249259Sdim assert(Shadow && "No shadow for a value"); 1073249259Sdim } 1074249259Sdim return Shadow; 1075249259Sdim } 1076249259Sdim if (UndefValue *U = dyn_cast<UndefValue>(V)) { 1077261991Sdim Value *AllOnes = PoisonUndef ? getPoisonedShadow(V) : getCleanShadow(V); 1078249259Sdim DEBUG(dbgs() << "Undef: " << *U << " ==> " << *AllOnes << "\n"); 1079249259Sdim (void)U; 1080249259Sdim return AllOnes; 1081249259Sdim } 1082249259Sdim if (Argument *A = dyn_cast<Argument>(V)) { 1083249259Sdim // For arguments we compute the shadow on demand and store it in the map. 1084249259Sdim Value **ShadowPtr = &ShadowMap[V]; 1085249259Sdim if (*ShadowPtr) 1086249259Sdim return *ShadowPtr; 1087249259Sdim Function *F = A->getParent(); 1088249259Sdim IRBuilder<> EntryIRB(F->getEntryBlock().getFirstNonPHI()); 1089249259Sdim unsigned ArgOffset = 0; 1090288943Sdim const DataLayout &DL = F->getParent()->getDataLayout(); 1091276479Sdim for (auto &FArg : F->args()) { 1092276479Sdim if (!FArg.getType()->isSized()) { 1093249259Sdim DEBUG(dbgs() << "Arg is not sized\n"); 1094249259Sdim continue; 1095249259Sdim } 1096288943Sdim unsigned Size = 1097288943Sdim FArg.hasByValAttr() 1098288943Sdim ? DL.getTypeAllocSize(FArg.getType()->getPointerElementType()) 1099288943Sdim : DL.getTypeAllocSize(FArg.getType()); 1100276479Sdim if (A == &FArg) { 1101280031Sdim bool Overflow = ArgOffset + Size > kParamTLSSize; 1102276479Sdim Value *Base = getShadowPtrForArgument(&FArg, EntryIRB, ArgOffset); 1103276479Sdim if (FArg.hasByValAttr()) { 1104249259Sdim // ByVal pointer itself has clean shadow. We copy the actual 1105249259Sdim // argument shadow to the underlying memory. 1106261991Sdim // Figure out maximal valid memcpy alignment. 1107276479Sdim unsigned ArgAlign = FArg.getParamAlignment(); 1108261991Sdim if (ArgAlign == 0) { 1109261991Sdim Type *EltType = A->getType()->getPointerElementType(); 1110288943Sdim ArgAlign = DL.getABITypeAlignment(EltType); 1111261991Sdim } 1112280031Sdim if (Overflow) { 1113280031Sdim // ParamTLS overflow. 1114280031Sdim EntryIRB.CreateMemSet( 1115280031Sdim getShadowPtr(V, EntryIRB.getInt8Ty(), EntryIRB), 1116280031Sdim Constant::getNullValue(EntryIRB.getInt8Ty()), Size, ArgAlign); 1117280031Sdim } else { 1118280031Sdim unsigned CopyAlign = std::min(ArgAlign, kShadowTLSAlignment); 1119280031Sdim Value *Cpy = EntryIRB.CreateMemCpy( 1120280031Sdim getShadowPtr(V, EntryIRB.getInt8Ty(), EntryIRB), Base, Size, 1121280031Sdim CopyAlign); 1122280031Sdim DEBUG(dbgs() << " ByValCpy: " << *Cpy << "\n"); 1123280031Sdim (void)Cpy; 1124280031Sdim } 1125249259Sdim *ShadowPtr = getCleanShadow(V); 1126249259Sdim } else { 1127280031Sdim if (Overflow) { 1128280031Sdim // ParamTLS overflow. 1129280031Sdim *ShadowPtr = getCleanShadow(V); 1130280031Sdim } else { 1131280031Sdim *ShadowPtr = 1132280031Sdim EntryIRB.CreateAlignedLoad(Base, kShadowTLSAlignment); 1133280031Sdim } 1134249259Sdim } 1135276479Sdim DEBUG(dbgs() << " ARG: " << FArg << " ==> " << 1136249259Sdim **ShadowPtr << "\n"); 1137280031Sdim if (MS.TrackOrigins && !Overflow) { 1138276479Sdim Value *OriginPtr = 1139276479Sdim getOriginPtrForArgument(&FArg, EntryIRB, ArgOffset); 1140249259Sdim setOrigin(A, EntryIRB.CreateLoad(OriginPtr)); 1141280031Sdim } else { 1142280031Sdim setOrigin(A, getCleanOrigin()); 1143249259Sdim } 1144249259Sdim } 1145280031Sdim ArgOffset += RoundUpToAlignment(Size, kShadowTLSAlignment); 1146249259Sdim } 1147249259Sdim assert(*ShadowPtr && "Could not find shadow for an argument"); 1148249259Sdim return *ShadowPtr; 1149249259Sdim } 1150249259Sdim // For everything else the shadow is zero. 1151249259Sdim return getCleanShadow(V); 1152249259Sdim } 1153249259Sdim 1154249259Sdim /// \brief Get the shadow for i-th argument of the instruction I. 1155249259Sdim Value *getShadow(Instruction *I, int i) { 1156249259Sdim return getShadow(I->getOperand(i)); 1157249259Sdim } 1158249259Sdim 1159249259Sdim /// \brief Get the origin for a value. 1160249259Sdim Value *getOrigin(Value *V) { 1161276479Sdim if (!MS.TrackOrigins) return nullptr; 1162280031Sdim if (!PropagateShadow) return getCleanOrigin(); 1163280031Sdim if (isa<Constant>(V)) return getCleanOrigin(); 1164280031Sdim assert((isa<Instruction>(V) || isa<Argument>(V)) && 1165280031Sdim "Unexpected value type in getOrigin()"); 1166280031Sdim Value *Origin = OriginMap[V]; 1167280031Sdim assert(Origin && "Missing origin"); 1168280031Sdim return Origin; 1169249259Sdim } 1170249259Sdim 1171249259Sdim /// \brief Get the origin for i-th argument of the instruction I. 1172249259Sdim Value *getOrigin(Instruction *I, int i) { 1173249259Sdim return getOrigin(I->getOperand(i)); 1174249259Sdim } 1175249259Sdim 1176249259Sdim /// \brief Remember the place where a shadow check should be inserted. 1177249259Sdim /// 1178249259Sdim /// This location will be later instrumented with a check that will print a 1179261991Sdim /// UMR warning in runtime if the shadow value is not 0. 1180261991Sdim void insertShadowCheck(Value *Shadow, Value *Origin, Instruction *OrigIns) { 1181261991Sdim assert(Shadow); 1182249259Sdim if (!InsertChecks) return; 1183249259Sdim#ifndef NDEBUG 1184249259Sdim Type *ShadowTy = Shadow->getType(); 1185249259Sdim assert((isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy)) && 1186249259Sdim "Can only insert checks for integer and vector shadow types"); 1187249259Sdim#endif 1188249259Sdim InstrumentationList.push_back( 1189261991Sdim ShadowOriginAndInsertPoint(Shadow, Origin, OrigIns)); 1190249259Sdim } 1191249259Sdim 1192261991Sdim /// \brief Remember the place where a shadow check should be inserted. 1193261991Sdim /// 1194261991Sdim /// This location will be later instrumented with a check that will print a 1195261991Sdim /// UMR warning in runtime if the value is not fully defined. 1196261991Sdim void insertShadowCheck(Value *Val, Instruction *OrigIns) { 1197261991Sdim assert(Val); 1198280031Sdim Value *Shadow, *Origin; 1199280031Sdim if (ClCheckConstantShadow) { 1200280031Sdim Shadow = getShadow(Val); 1201280031Sdim if (!Shadow) return; 1202280031Sdim Origin = getOrigin(Val); 1203280031Sdim } else { 1204280031Sdim Shadow = dyn_cast_or_null<Instruction>(getShadow(Val)); 1205280031Sdim if (!Shadow) return; 1206280031Sdim Origin = dyn_cast_or_null<Instruction>(getOrigin(Val)); 1207280031Sdim } 1208261991Sdim insertShadowCheck(Shadow, Origin, OrigIns); 1209261991Sdim } 1210261991Sdim 1211261991Sdim AtomicOrdering addReleaseOrdering(AtomicOrdering a) { 1212261991Sdim switch (a) { 1213261991Sdim case NotAtomic: 1214261991Sdim return NotAtomic; 1215261991Sdim case Unordered: 1216261991Sdim case Monotonic: 1217261991Sdim case Release: 1218261991Sdim return Release; 1219261991Sdim case Acquire: 1220261991Sdim case AcquireRelease: 1221261991Sdim return AcquireRelease; 1222261991Sdim case SequentiallyConsistent: 1223261991Sdim return SequentiallyConsistent; 1224261991Sdim } 1225261991Sdim llvm_unreachable("Unknown ordering"); 1226261991Sdim } 1227261991Sdim 1228261991Sdim AtomicOrdering addAcquireOrdering(AtomicOrdering a) { 1229261991Sdim switch (a) { 1230261991Sdim case NotAtomic: 1231261991Sdim return NotAtomic; 1232261991Sdim case Unordered: 1233261991Sdim case Monotonic: 1234261991Sdim case Acquire: 1235261991Sdim return Acquire; 1236261991Sdim case Release: 1237261991Sdim case AcquireRelease: 1238261991Sdim return AcquireRelease; 1239261991Sdim case SequentiallyConsistent: 1240261991Sdim return SequentiallyConsistent; 1241261991Sdim } 1242261991Sdim llvm_unreachable("Unknown ordering"); 1243261991Sdim } 1244261991Sdim 1245249259Sdim // ------------------- Visitors. 1246249259Sdim 1247249259Sdim /// \brief Instrument LoadInst 1248249259Sdim /// 1249249259Sdim /// Loads the corresponding shadow and (optionally) origin. 1250249259Sdim /// Optionally, checks that the load address is fully defined. 1251249259Sdim void visitLoadInst(LoadInst &I) { 1252249259Sdim assert(I.getType()->isSized() && "Load type must have size"); 1253261991Sdim IRBuilder<> IRB(I.getNextNode()); 1254249259Sdim Type *ShadowTy = getShadowTy(&I); 1255249259Sdim Value *Addr = I.getPointerOperand(); 1256280031Sdim if (PropagateShadow && !I.getMetadata("nosanitize")) { 1257249259Sdim Value *ShadowPtr = getShadowPtr(Addr, ShadowTy, IRB); 1258249259Sdim setShadow(&I, 1259249259Sdim IRB.CreateAlignedLoad(ShadowPtr, I.getAlignment(), "_msld")); 1260249259Sdim } else { 1261249259Sdim setShadow(&I, getCleanShadow(&I)); 1262249259Sdim } 1263249259Sdim 1264249259Sdim if (ClCheckAccessAddress) 1265261991Sdim insertShadowCheck(I.getPointerOperand(), &I); 1266249259Sdim 1267261991Sdim if (I.isAtomic()) 1268261991Sdim I.setOrdering(addAcquireOrdering(I.getOrdering())); 1269261991Sdim 1270249259Sdim if (MS.TrackOrigins) { 1271276479Sdim if (PropagateShadow) { 1272280031Sdim unsigned Alignment = I.getAlignment(); 1273280031Sdim unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment); 1274280031Sdim setOrigin(&I, IRB.CreateAlignedLoad(getOriginPtr(Addr, IRB, Alignment), 1275280031Sdim OriginAlignment)); 1276249259Sdim } else { 1277249259Sdim setOrigin(&I, getCleanOrigin()); 1278249259Sdim } 1279249259Sdim } 1280249259Sdim } 1281249259Sdim 1282249259Sdim /// \brief Instrument StoreInst 1283249259Sdim /// 1284249259Sdim /// Stores the corresponding shadow and (optionally) origin. 1285249259Sdim /// Optionally, checks that the store address is fully defined. 1286249259Sdim void visitStoreInst(StoreInst &I) { 1287249259Sdim StoreList.push_back(&I); 1288249259Sdim } 1289249259Sdim 1290261991Sdim void handleCASOrRMW(Instruction &I) { 1291261991Sdim assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I)); 1292261991Sdim 1293261991Sdim IRBuilder<> IRB(&I); 1294261991Sdim Value *Addr = I.getOperand(0); 1295261991Sdim Value *ShadowPtr = getShadowPtr(Addr, I.getType(), IRB); 1296261991Sdim 1297261991Sdim if (ClCheckAccessAddress) 1298261991Sdim insertShadowCheck(Addr, &I); 1299261991Sdim 1300261991Sdim // Only test the conditional argument of cmpxchg instruction. 1301261991Sdim // The other argument can potentially be uninitialized, but we can not 1302261991Sdim // detect this situation reliably without possible false positives. 1303261991Sdim if (isa<AtomicCmpXchgInst>(I)) 1304261991Sdim insertShadowCheck(I.getOperand(1), &I); 1305261991Sdim 1306261991Sdim IRB.CreateStore(getCleanShadow(&I), ShadowPtr); 1307261991Sdim 1308261991Sdim setShadow(&I, getCleanShadow(&I)); 1309280031Sdim setOrigin(&I, getCleanOrigin()); 1310261991Sdim } 1311261991Sdim 1312261991Sdim void visitAtomicRMWInst(AtomicRMWInst &I) { 1313261991Sdim handleCASOrRMW(I); 1314261991Sdim I.setOrdering(addReleaseOrdering(I.getOrdering())); 1315261991Sdim } 1316261991Sdim 1317261991Sdim void visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) { 1318261991Sdim handleCASOrRMW(I); 1319276479Sdim I.setSuccessOrdering(addReleaseOrdering(I.getSuccessOrdering())); 1320261991Sdim } 1321261991Sdim 1322249259Sdim // Vector manipulation. 1323249259Sdim void visitExtractElementInst(ExtractElementInst &I) { 1324261991Sdim insertShadowCheck(I.getOperand(1), &I); 1325249259Sdim IRBuilder<> IRB(&I); 1326249259Sdim setShadow(&I, IRB.CreateExtractElement(getShadow(&I, 0), I.getOperand(1), 1327249259Sdim "_msprop")); 1328249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1329249259Sdim } 1330249259Sdim 1331249259Sdim void visitInsertElementInst(InsertElementInst &I) { 1332261991Sdim insertShadowCheck(I.getOperand(2), &I); 1333249259Sdim IRBuilder<> IRB(&I); 1334249259Sdim setShadow(&I, IRB.CreateInsertElement(getShadow(&I, 0), getShadow(&I, 1), 1335249259Sdim I.getOperand(2), "_msprop")); 1336249259Sdim setOriginForNaryOp(I); 1337249259Sdim } 1338249259Sdim 1339249259Sdim void visitShuffleVectorInst(ShuffleVectorInst &I) { 1340261991Sdim insertShadowCheck(I.getOperand(2), &I); 1341249259Sdim IRBuilder<> IRB(&I); 1342249259Sdim setShadow(&I, IRB.CreateShuffleVector(getShadow(&I, 0), getShadow(&I, 1), 1343249259Sdim I.getOperand(2), "_msprop")); 1344249259Sdim setOriginForNaryOp(I); 1345249259Sdim } 1346249259Sdim 1347249259Sdim // Casts. 1348249259Sdim void visitSExtInst(SExtInst &I) { 1349249259Sdim IRBuilder<> IRB(&I); 1350249259Sdim setShadow(&I, IRB.CreateSExt(getShadow(&I, 0), I.getType(), "_msprop")); 1351249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1352249259Sdim } 1353249259Sdim 1354249259Sdim void visitZExtInst(ZExtInst &I) { 1355249259Sdim IRBuilder<> IRB(&I); 1356249259Sdim setShadow(&I, IRB.CreateZExt(getShadow(&I, 0), I.getType(), "_msprop")); 1357249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1358249259Sdim } 1359249259Sdim 1360249259Sdim void visitTruncInst(TruncInst &I) { 1361249259Sdim IRBuilder<> IRB(&I); 1362249259Sdim setShadow(&I, IRB.CreateTrunc(getShadow(&I, 0), I.getType(), "_msprop")); 1363249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1364249259Sdim } 1365249259Sdim 1366249259Sdim void visitBitCastInst(BitCastInst &I) { 1367296417Sdim // Special case: if this is the bitcast (there is exactly 1 allowed) between 1368296417Sdim // a musttail call and a ret, don't instrument. New instructions are not 1369296417Sdim // allowed after a musttail call. 1370296417Sdim if (auto *CI = dyn_cast<CallInst>(I.getOperand(0))) 1371296417Sdim if (CI->isMustTailCall()) 1372296417Sdim return; 1373249259Sdim IRBuilder<> IRB(&I); 1374249259Sdim setShadow(&I, IRB.CreateBitCast(getShadow(&I, 0), getShadowTy(&I))); 1375249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1376249259Sdim } 1377249259Sdim 1378249259Sdim void visitPtrToIntInst(PtrToIntInst &I) { 1379249259Sdim IRBuilder<> IRB(&I); 1380249259Sdim setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false, 1381249259Sdim "_msprop_ptrtoint")); 1382249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1383249259Sdim } 1384249259Sdim 1385249259Sdim void visitIntToPtrInst(IntToPtrInst &I) { 1386249259Sdim IRBuilder<> IRB(&I); 1387249259Sdim setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false, 1388249259Sdim "_msprop_inttoptr")); 1389249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1390249259Sdim } 1391249259Sdim 1392249259Sdim void visitFPToSIInst(CastInst& I) { handleShadowOr(I); } 1393249259Sdim void visitFPToUIInst(CastInst& I) { handleShadowOr(I); } 1394249259Sdim void visitSIToFPInst(CastInst& I) { handleShadowOr(I); } 1395249259Sdim void visitUIToFPInst(CastInst& I) { handleShadowOr(I); } 1396249259Sdim void visitFPExtInst(CastInst& I) { handleShadowOr(I); } 1397249259Sdim void visitFPTruncInst(CastInst& I) { handleShadowOr(I); } 1398249259Sdim 1399249259Sdim /// \brief Propagate shadow for bitwise AND. 1400249259Sdim /// 1401249259Sdim /// This code is exact, i.e. if, for example, a bit in the left argument 1402249259Sdim /// is defined and 0, then neither the value not definedness of the 1403249259Sdim /// corresponding bit in B don't affect the resulting shadow. 1404249259Sdim void visitAnd(BinaryOperator &I) { 1405249259Sdim IRBuilder<> IRB(&I); 1406249259Sdim // "And" of 0 and a poisoned value results in unpoisoned value. 1407249259Sdim // 1&1 => 1; 0&1 => 0; p&1 => p; 1408249259Sdim // 1&0 => 0; 0&0 => 0; p&0 => 0; 1409249259Sdim // 1&p => p; 0&p => 0; p&p => p; 1410249259Sdim // S = (S1 & S2) | (V1 & S2) | (S1 & V2) 1411249259Sdim Value *S1 = getShadow(&I, 0); 1412249259Sdim Value *S2 = getShadow(&I, 1); 1413249259Sdim Value *V1 = I.getOperand(0); 1414249259Sdim Value *V2 = I.getOperand(1); 1415249259Sdim if (V1->getType() != S1->getType()) { 1416249259Sdim V1 = IRB.CreateIntCast(V1, S1->getType(), false); 1417249259Sdim V2 = IRB.CreateIntCast(V2, S2->getType(), false); 1418249259Sdim } 1419249259Sdim Value *S1S2 = IRB.CreateAnd(S1, S2); 1420249259Sdim Value *V1S2 = IRB.CreateAnd(V1, S2); 1421249259Sdim Value *S1V2 = IRB.CreateAnd(S1, V2); 1422249259Sdim setShadow(&I, IRB.CreateOr(S1S2, IRB.CreateOr(V1S2, S1V2))); 1423249259Sdim setOriginForNaryOp(I); 1424249259Sdim } 1425249259Sdim 1426249259Sdim void visitOr(BinaryOperator &I) { 1427249259Sdim IRBuilder<> IRB(&I); 1428249259Sdim // "Or" of 1 and a poisoned value results in unpoisoned value. 1429249259Sdim // 1|1 => 1; 0|1 => 1; p|1 => 1; 1430249259Sdim // 1|0 => 1; 0|0 => 0; p|0 => p; 1431249259Sdim // 1|p => 1; 0|p => p; p|p => p; 1432249259Sdim // S = (S1 & S2) | (~V1 & S2) | (S1 & ~V2) 1433249259Sdim Value *S1 = getShadow(&I, 0); 1434249259Sdim Value *S2 = getShadow(&I, 1); 1435249259Sdim Value *V1 = IRB.CreateNot(I.getOperand(0)); 1436249259Sdim Value *V2 = IRB.CreateNot(I.getOperand(1)); 1437249259Sdim if (V1->getType() != S1->getType()) { 1438249259Sdim V1 = IRB.CreateIntCast(V1, S1->getType(), false); 1439249259Sdim V2 = IRB.CreateIntCast(V2, S2->getType(), false); 1440249259Sdim } 1441249259Sdim Value *S1S2 = IRB.CreateAnd(S1, S2); 1442249259Sdim Value *V1S2 = IRB.CreateAnd(V1, S2); 1443249259Sdim Value *S1V2 = IRB.CreateAnd(S1, V2); 1444249259Sdim setShadow(&I, IRB.CreateOr(S1S2, IRB.CreateOr(V1S2, S1V2))); 1445249259Sdim setOriginForNaryOp(I); 1446249259Sdim } 1447249259Sdim 1448249259Sdim /// \brief Default propagation of shadow and/or origin. 1449249259Sdim /// 1450249259Sdim /// This class implements the general case of shadow propagation, used in all 1451249259Sdim /// cases where we don't know and/or don't care about what the operation 1452249259Sdim /// actually does. It converts all input shadow values to a common type 1453249259Sdim /// (extending or truncating as necessary), and bitwise OR's them. 1454249259Sdim /// 1455249259Sdim /// This is much cheaper than inserting checks (i.e. requiring inputs to be 1456249259Sdim /// fully initialized), and less prone to false positives. 1457249259Sdim /// 1458249259Sdim /// This class also implements the general case of origin propagation. For a 1459249259Sdim /// Nary operation, result origin is set to the origin of an argument that is 1460249259Sdim /// not entirely initialized. If there is more than one such arguments, the 1461249259Sdim /// rightmost of them is picked. It does not matter which one is picked if all 1462249259Sdim /// arguments are initialized. 1463249259Sdim template <bool CombineShadow> 1464249259Sdim class Combiner { 1465249259Sdim Value *Shadow; 1466249259Sdim Value *Origin; 1467249259Sdim IRBuilder<> &IRB; 1468249259Sdim MemorySanitizerVisitor *MSV; 1469249259Sdim 1470249259Sdim public: 1471249259Sdim Combiner(MemorySanitizerVisitor *MSV, IRBuilder<> &IRB) : 1472276479Sdim Shadow(nullptr), Origin(nullptr), IRB(IRB), MSV(MSV) {} 1473249259Sdim 1474249259Sdim /// \brief Add a pair of shadow and origin values to the mix. 1475249259Sdim Combiner &Add(Value *OpShadow, Value *OpOrigin) { 1476249259Sdim if (CombineShadow) { 1477249259Sdim assert(OpShadow); 1478249259Sdim if (!Shadow) 1479249259Sdim Shadow = OpShadow; 1480249259Sdim else { 1481249259Sdim OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType()); 1482249259Sdim Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop"); 1483249259Sdim } 1484249259Sdim } 1485249259Sdim 1486249259Sdim if (MSV->MS.TrackOrigins) { 1487249259Sdim assert(OpOrigin); 1488249259Sdim if (!Origin) { 1489249259Sdim Origin = OpOrigin; 1490249259Sdim } else { 1491276479Sdim Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin); 1492276479Sdim // No point in adding something that might result in 0 origin value. 1493276479Sdim if (!ConstOrigin || !ConstOrigin->isNullValue()) { 1494276479Sdim Value *FlatShadow = MSV->convertToShadowTyNoVec(OpShadow, IRB); 1495276479Sdim Value *Cond = 1496276479Sdim IRB.CreateICmpNE(FlatShadow, MSV->getCleanShadow(FlatShadow)); 1497276479Sdim Origin = IRB.CreateSelect(Cond, OpOrigin, Origin); 1498276479Sdim } 1499249259Sdim } 1500249259Sdim } 1501249259Sdim return *this; 1502249259Sdim } 1503249259Sdim 1504249259Sdim /// \brief Add an application value to the mix. 1505249259Sdim Combiner &Add(Value *V) { 1506249259Sdim Value *OpShadow = MSV->getShadow(V); 1507276479Sdim Value *OpOrigin = MSV->MS.TrackOrigins ? MSV->getOrigin(V) : nullptr; 1508249259Sdim return Add(OpShadow, OpOrigin); 1509249259Sdim } 1510249259Sdim 1511249259Sdim /// \brief Set the current combined values as the given instruction's shadow 1512249259Sdim /// and origin. 1513249259Sdim void Done(Instruction *I) { 1514249259Sdim if (CombineShadow) { 1515249259Sdim assert(Shadow); 1516249259Sdim Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I)); 1517249259Sdim MSV->setShadow(I, Shadow); 1518249259Sdim } 1519249259Sdim if (MSV->MS.TrackOrigins) { 1520249259Sdim assert(Origin); 1521249259Sdim MSV->setOrigin(I, Origin); 1522249259Sdim } 1523249259Sdim } 1524249259Sdim }; 1525249259Sdim 1526249259Sdim typedef Combiner<true> ShadowAndOriginCombiner; 1527249259Sdim typedef Combiner<false> OriginCombiner; 1528249259Sdim 1529249259Sdim /// \brief Propagate origin for arbitrary operation. 1530249259Sdim void setOriginForNaryOp(Instruction &I) { 1531249259Sdim if (!MS.TrackOrigins) return; 1532249259Sdim IRBuilder<> IRB(&I); 1533249259Sdim OriginCombiner OC(this, IRB); 1534249259Sdim for (Instruction::op_iterator OI = I.op_begin(); OI != I.op_end(); ++OI) 1535249259Sdim OC.Add(OI->get()); 1536249259Sdim OC.Done(&I); 1537249259Sdim } 1538249259Sdim 1539249259Sdim size_t VectorOrPrimitiveTypeSizeInBits(Type *Ty) { 1540249259Sdim assert(!(Ty->isVectorTy() && Ty->getScalarType()->isPointerTy()) && 1541249259Sdim "Vector of pointers is not a valid shadow type"); 1542249259Sdim return Ty->isVectorTy() ? 1543249259Sdim Ty->getVectorNumElements() * Ty->getScalarSizeInBits() : 1544249259Sdim Ty->getPrimitiveSizeInBits(); 1545249259Sdim } 1546249259Sdim 1547249259Sdim /// \brief Cast between two shadow types, extending or truncating as 1548249259Sdim /// necessary. 1549261991Sdim Value *CreateShadowCast(IRBuilder<> &IRB, Value *V, Type *dstTy, 1550261991Sdim bool Signed = false) { 1551249259Sdim Type *srcTy = V->getType(); 1552249259Sdim if (dstTy->isIntegerTy() && srcTy->isIntegerTy()) 1553261991Sdim return IRB.CreateIntCast(V, dstTy, Signed); 1554249259Sdim if (dstTy->isVectorTy() && srcTy->isVectorTy() && 1555249259Sdim dstTy->getVectorNumElements() == srcTy->getVectorNumElements()) 1556261991Sdim return IRB.CreateIntCast(V, dstTy, Signed); 1557249259Sdim size_t srcSizeInBits = VectorOrPrimitiveTypeSizeInBits(srcTy); 1558249259Sdim size_t dstSizeInBits = VectorOrPrimitiveTypeSizeInBits(dstTy); 1559249259Sdim Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits)); 1560249259Sdim Value *V2 = 1561261991Sdim IRB.CreateIntCast(V1, Type::getIntNTy(*MS.C, dstSizeInBits), Signed); 1562249259Sdim return IRB.CreateBitCast(V2, dstTy); 1563249259Sdim // TODO: handle struct types. 1564249259Sdim } 1565249259Sdim 1566276479Sdim /// \brief Cast an application value to the type of its own shadow. 1567276479Sdim Value *CreateAppToShadowCast(IRBuilder<> &IRB, Value *V) { 1568276479Sdim Type *ShadowTy = getShadowTy(V); 1569276479Sdim if (V->getType() == ShadowTy) 1570276479Sdim return V; 1571276479Sdim if (V->getType()->isPtrOrPtrVectorTy()) 1572276479Sdim return IRB.CreatePtrToInt(V, ShadowTy); 1573276479Sdim else 1574276479Sdim return IRB.CreateBitCast(V, ShadowTy); 1575276479Sdim } 1576276479Sdim 1577249259Sdim /// \brief Propagate shadow for arbitrary operation. 1578249259Sdim void handleShadowOr(Instruction &I) { 1579249259Sdim IRBuilder<> IRB(&I); 1580249259Sdim ShadowAndOriginCombiner SC(this, IRB); 1581249259Sdim for (Instruction::op_iterator OI = I.op_begin(); OI != I.op_end(); ++OI) 1582249259Sdim SC.Add(OI->get()); 1583249259Sdim SC.Done(&I); 1584249259Sdim } 1585249259Sdim 1586276479Sdim // \brief Handle multiplication by constant. 1587276479Sdim // 1588276479Sdim // Handle a special case of multiplication by constant that may have one or 1589276479Sdim // more zeros in the lower bits. This makes corresponding number of lower bits 1590276479Sdim // of the result zero as well. We model it by shifting the other operand 1591276479Sdim // shadow left by the required number of bits. Effectively, we transform 1592276479Sdim // (X * (A * 2**B)) to ((X << B) * A) and instrument (X << B) as (Sx << B). 1593276479Sdim // We use multiplication by 2**N instead of shift to cover the case of 1594276479Sdim // multiplication by 0, which may occur in some elements of a vector operand. 1595276479Sdim void handleMulByConstant(BinaryOperator &I, Constant *ConstArg, 1596276479Sdim Value *OtherArg) { 1597276479Sdim Constant *ShadowMul; 1598276479Sdim Type *Ty = ConstArg->getType(); 1599276479Sdim if (Ty->isVectorTy()) { 1600276479Sdim unsigned NumElements = Ty->getVectorNumElements(); 1601276479Sdim Type *EltTy = Ty->getSequentialElementType(); 1602276479Sdim SmallVector<Constant *, 16> Elements; 1603276479Sdim for (unsigned Idx = 0; Idx < NumElements; ++Idx) { 1604296417Sdim if (ConstantInt *Elt = 1605296417Sdim dyn_cast<ConstantInt>(ConstArg->getAggregateElement(Idx))) { 1606296417Sdim APInt V = Elt->getValue(); 1607296417Sdim APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros(); 1608296417Sdim Elements.push_back(ConstantInt::get(EltTy, V2)); 1609296417Sdim } else { 1610296417Sdim Elements.push_back(ConstantInt::get(EltTy, 1)); 1611296417Sdim } 1612296417Sdim } 1613296417Sdim ShadowMul = ConstantVector::get(Elements); 1614296417Sdim } else { 1615296417Sdim if (ConstantInt *Elt = dyn_cast<ConstantInt>(ConstArg)) { 1616276479Sdim APInt V = Elt->getValue(); 1617276479Sdim APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros(); 1618296417Sdim ShadowMul = ConstantInt::get(Ty, V2); 1619296417Sdim } else { 1620296417Sdim ShadowMul = ConstantInt::get(Ty, 1); 1621276479Sdim } 1622276479Sdim } 1623276479Sdim 1624276479Sdim IRBuilder<> IRB(&I); 1625276479Sdim setShadow(&I, 1626276479Sdim IRB.CreateMul(getShadow(OtherArg), ShadowMul, "msprop_mul_cst")); 1627276479Sdim setOrigin(&I, getOrigin(OtherArg)); 1628276479Sdim } 1629276479Sdim 1630276479Sdim void visitMul(BinaryOperator &I) { 1631276479Sdim Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0)); 1632276479Sdim Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1)); 1633276479Sdim if (constOp0 && !constOp1) 1634276479Sdim handleMulByConstant(I, constOp0, I.getOperand(1)); 1635276479Sdim else if (constOp1 && !constOp0) 1636276479Sdim handleMulByConstant(I, constOp1, I.getOperand(0)); 1637276479Sdim else 1638276479Sdim handleShadowOr(I); 1639276479Sdim } 1640276479Sdim 1641249259Sdim void visitFAdd(BinaryOperator &I) { handleShadowOr(I); } 1642249259Sdim void visitFSub(BinaryOperator &I) { handleShadowOr(I); } 1643249259Sdim void visitFMul(BinaryOperator &I) { handleShadowOr(I); } 1644249259Sdim void visitAdd(BinaryOperator &I) { handleShadowOr(I); } 1645249259Sdim void visitSub(BinaryOperator &I) { handleShadowOr(I); } 1646249259Sdim void visitXor(BinaryOperator &I) { handleShadowOr(I); } 1647249259Sdim 1648249259Sdim void handleDiv(Instruction &I) { 1649249259Sdim IRBuilder<> IRB(&I); 1650249259Sdim // Strict on the second argument. 1651261991Sdim insertShadowCheck(I.getOperand(1), &I); 1652249259Sdim setShadow(&I, getShadow(&I, 0)); 1653249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1654249259Sdim } 1655249259Sdim 1656249259Sdim void visitUDiv(BinaryOperator &I) { handleDiv(I); } 1657249259Sdim void visitSDiv(BinaryOperator &I) { handleDiv(I); } 1658249259Sdim void visitFDiv(BinaryOperator &I) { handleDiv(I); } 1659249259Sdim void visitURem(BinaryOperator &I) { handleDiv(I); } 1660249259Sdim void visitSRem(BinaryOperator &I) { handleDiv(I); } 1661249259Sdim void visitFRem(BinaryOperator &I) { handleDiv(I); } 1662249259Sdim 1663249259Sdim /// \brief Instrument == and != comparisons. 1664249259Sdim /// 1665249259Sdim /// Sometimes the comparison result is known even if some of the bits of the 1666249259Sdim /// arguments are not. 1667249259Sdim void handleEqualityComparison(ICmpInst &I) { 1668249259Sdim IRBuilder<> IRB(&I); 1669249259Sdim Value *A = I.getOperand(0); 1670249259Sdim Value *B = I.getOperand(1); 1671249259Sdim Value *Sa = getShadow(A); 1672249259Sdim Value *Sb = getShadow(B); 1673249259Sdim 1674249259Sdim // Get rid of pointers and vectors of pointers. 1675249259Sdim // For ints (and vectors of ints), types of A and Sa match, 1676249259Sdim // and this is a no-op. 1677249259Sdim A = IRB.CreatePointerCast(A, Sa->getType()); 1678249259Sdim B = IRB.CreatePointerCast(B, Sb->getType()); 1679249259Sdim 1680249259Sdim // A == B <==> (C = A^B) == 0 1681249259Sdim // A != B <==> (C = A^B) != 0 1682249259Sdim // Sc = Sa | Sb 1683249259Sdim Value *C = IRB.CreateXor(A, B); 1684249259Sdim Value *Sc = IRB.CreateOr(Sa, Sb); 1685249259Sdim // Now dealing with i = (C == 0) comparison (or C != 0, does not matter now) 1686249259Sdim // Result is defined if one of the following is true 1687249259Sdim // * there is a defined 1 bit in C 1688249259Sdim // * C is fully defined 1689249259Sdim // Si = !(C & ~Sc) && Sc 1690249259Sdim Value *Zero = Constant::getNullValue(Sc->getType()); 1691249259Sdim Value *MinusOne = Constant::getAllOnesValue(Sc->getType()); 1692249259Sdim Value *Si = 1693249259Sdim IRB.CreateAnd(IRB.CreateICmpNE(Sc, Zero), 1694249259Sdim IRB.CreateICmpEQ( 1695249259Sdim IRB.CreateAnd(IRB.CreateXor(Sc, MinusOne), C), Zero)); 1696249259Sdim Si->setName("_msprop_icmp"); 1697249259Sdim setShadow(&I, Si); 1698249259Sdim setOriginForNaryOp(I); 1699249259Sdim } 1700249259Sdim 1701249259Sdim /// \brief Build the lowest possible value of V, taking into account V's 1702249259Sdim /// uninitialized bits. 1703249259Sdim Value *getLowestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa, 1704249259Sdim bool isSigned) { 1705249259Sdim if (isSigned) { 1706249259Sdim // Split shadow into sign bit and other bits. 1707249259Sdim Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1); 1708249259Sdim Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits); 1709249259Sdim // Maximise the undefined shadow bit, minimize other undefined bits. 1710249259Sdim return 1711249259Sdim IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaOtherBits)), SaSignBit); 1712249259Sdim } else { 1713249259Sdim // Minimize undefined bits. 1714249259Sdim return IRB.CreateAnd(A, IRB.CreateNot(Sa)); 1715249259Sdim } 1716249259Sdim } 1717249259Sdim 1718249259Sdim /// \brief Build the highest possible value of V, taking into account V's 1719249259Sdim /// uninitialized bits. 1720249259Sdim Value *getHighestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa, 1721249259Sdim bool isSigned) { 1722249259Sdim if (isSigned) { 1723249259Sdim // Split shadow into sign bit and other bits. 1724249259Sdim Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1); 1725249259Sdim Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits); 1726249259Sdim // Minimise the undefined shadow bit, maximise other undefined bits. 1727249259Sdim return 1728249259Sdim IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaSignBit)), SaOtherBits); 1729249259Sdim } else { 1730249259Sdim // Maximize undefined bits. 1731249259Sdim return IRB.CreateOr(A, Sa); 1732249259Sdim } 1733249259Sdim } 1734249259Sdim 1735249259Sdim /// \brief Instrument relational comparisons. 1736249259Sdim /// 1737249259Sdim /// This function does exact shadow propagation for all relational 1738249259Sdim /// comparisons of integers, pointers and vectors of those. 1739249259Sdim /// FIXME: output seems suboptimal when one of the operands is a constant 1740249259Sdim void handleRelationalComparisonExact(ICmpInst &I) { 1741249259Sdim IRBuilder<> IRB(&I); 1742249259Sdim Value *A = I.getOperand(0); 1743249259Sdim Value *B = I.getOperand(1); 1744249259Sdim Value *Sa = getShadow(A); 1745249259Sdim Value *Sb = getShadow(B); 1746249259Sdim 1747249259Sdim // Get rid of pointers and vectors of pointers. 1748249259Sdim // For ints (and vectors of ints), types of A and Sa match, 1749249259Sdim // and this is a no-op. 1750249259Sdim A = IRB.CreatePointerCast(A, Sa->getType()); 1751249259Sdim B = IRB.CreatePointerCast(B, Sb->getType()); 1752249259Sdim 1753249259Sdim // Let [a0, a1] be the interval of possible values of A, taking into account 1754249259Sdim // its undefined bits. Let [b0, b1] be the interval of possible values of B. 1755249259Sdim // Then (A cmp B) is defined iff (a0 cmp b1) == (a1 cmp b0). 1756249259Sdim bool IsSigned = I.isSigned(); 1757249259Sdim Value *S1 = IRB.CreateICmp(I.getPredicate(), 1758249259Sdim getLowestPossibleValue(IRB, A, Sa, IsSigned), 1759249259Sdim getHighestPossibleValue(IRB, B, Sb, IsSigned)); 1760249259Sdim Value *S2 = IRB.CreateICmp(I.getPredicate(), 1761249259Sdim getHighestPossibleValue(IRB, A, Sa, IsSigned), 1762249259Sdim getLowestPossibleValue(IRB, B, Sb, IsSigned)); 1763249259Sdim Value *Si = IRB.CreateXor(S1, S2); 1764249259Sdim setShadow(&I, Si); 1765249259Sdim setOriginForNaryOp(I); 1766249259Sdim } 1767249259Sdim 1768249259Sdim /// \brief Instrument signed relational comparisons. 1769249259Sdim /// 1770296417Sdim /// Handle sign bit tests: x<0, x>=0, x<=-1, x>-1 by propagating the highest 1771296417Sdim /// bit of the shadow. Everything else is delegated to handleShadowOr(). 1772249259Sdim void handleSignedRelationalComparison(ICmpInst &I) { 1773296417Sdim Constant *constOp; 1774296417Sdim Value *op = nullptr; 1775296417Sdim CmpInst::Predicate pre; 1776296417Sdim if ((constOp = dyn_cast<Constant>(I.getOperand(1)))) { 1777296417Sdim op = I.getOperand(0); 1778296417Sdim pre = I.getPredicate(); 1779296417Sdim } else if ((constOp = dyn_cast<Constant>(I.getOperand(0)))) { 1780249259Sdim op = I.getOperand(1); 1781296417Sdim pre = I.getSwappedPredicate(); 1782296417Sdim } else { 1783296417Sdim handleShadowOr(I); 1784296417Sdim return; 1785249259Sdim } 1786296417Sdim 1787296417Sdim if ((constOp->isNullValue() && 1788296417Sdim (pre == CmpInst::ICMP_SLT || pre == CmpInst::ICMP_SGE)) || 1789296417Sdim (constOp->isAllOnesValue() && 1790296417Sdim (pre == CmpInst::ICMP_SGT || pre == CmpInst::ICMP_SLE))) { 1791249259Sdim IRBuilder<> IRB(&I); 1792296417Sdim Value *Shadow = IRB.CreateICmpSLT(getShadow(op), getCleanShadow(op), 1793296417Sdim "_msprop_icmp_s"); 1794249259Sdim setShadow(&I, Shadow); 1795249259Sdim setOrigin(&I, getOrigin(op)); 1796249259Sdim } else { 1797249259Sdim handleShadowOr(I); 1798249259Sdim } 1799249259Sdim } 1800249259Sdim 1801249259Sdim void visitICmpInst(ICmpInst &I) { 1802249259Sdim if (!ClHandleICmp) { 1803249259Sdim handleShadowOr(I); 1804249259Sdim return; 1805249259Sdim } 1806249259Sdim if (I.isEquality()) { 1807249259Sdim handleEqualityComparison(I); 1808249259Sdim return; 1809249259Sdim } 1810249259Sdim 1811249259Sdim assert(I.isRelational()); 1812249259Sdim if (ClHandleICmpExact) { 1813249259Sdim handleRelationalComparisonExact(I); 1814249259Sdim return; 1815249259Sdim } 1816249259Sdim if (I.isSigned()) { 1817249259Sdim handleSignedRelationalComparison(I); 1818249259Sdim return; 1819249259Sdim } 1820249259Sdim 1821249259Sdim assert(I.isUnsigned()); 1822249259Sdim if ((isa<Constant>(I.getOperand(0)) || isa<Constant>(I.getOperand(1)))) { 1823249259Sdim handleRelationalComparisonExact(I); 1824249259Sdim return; 1825249259Sdim } 1826249259Sdim 1827249259Sdim handleShadowOr(I); 1828249259Sdim } 1829249259Sdim 1830249259Sdim void visitFCmpInst(FCmpInst &I) { 1831249259Sdim handleShadowOr(I); 1832249259Sdim } 1833249259Sdim 1834249259Sdim void handleShift(BinaryOperator &I) { 1835249259Sdim IRBuilder<> IRB(&I); 1836249259Sdim // If any of the S2 bits are poisoned, the whole thing is poisoned. 1837249259Sdim // Otherwise perform the same shift on S1. 1838249259Sdim Value *S1 = getShadow(&I, 0); 1839249259Sdim Value *S2 = getShadow(&I, 1); 1840249259Sdim Value *S2Conv = IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)), 1841249259Sdim S2->getType()); 1842249259Sdim Value *V2 = I.getOperand(1); 1843249259Sdim Value *Shift = IRB.CreateBinOp(I.getOpcode(), S1, V2); 1844249259Sdim setShadow(&I, IRB.CreateOr(Shift, S2Conv)); 1845249259Sdim setOriginForNaryOp(I); 1846249259Sdim } 1847249259Sdim 1848249259Sdim void visitShl(BinaryOperator &I) { handleShift(I); } 1849249259Sdim void visitAShr(BinaryOperator &I) { handleShift(I); } 1850249259Sdim void visitLShr(BinaryOperator &I) { handleShift(I); } 1851249259Sdim 1852249259Sdim /// \brief Instrument llvm.memmove 1853249259Sdim /// 1854249259Sdim /// At this point we don't know if llvm.memmove will be inlined or not. 1855249259Sdim /// If we don't instrument it and it gets inlined, 1856249259Sdim /// our interceptor will not kick in and we will lose the memmove. 1857249259Sdim /// If we instrument the call here, but it does not get inlined, 1858249259Sdim /// we will memove the shadow twice: which is bad in case 1859249259Sdim /// of overlapping regions. So, we simply lower the intrinsic to a call. 1860249259Sdim /// 1861249259Sdim /// Similar situation exists for memcpy and memset. 1862249259Sdim void visitMemMoveInst(MemMoveInst &I) { 1863249259Sdim IRBuilder<> IRB(&I); 1864288943Sdim IRB.CreateCall( 1865288943Sdim MS.MemmoveFn, 1866288943Sdim {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()), 1867288943Sdim IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()), 1868288943Sdim IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)}); 1869249259Sdim I.eraseFromParent(); 1870249259Sdim } 1871249259Sdim 1872249259Sdim // Similar to memmove: avoid copying shadow twice. 1873249259Sdim // This is somewhat unfortunate as it may slowdown small constant memcpys. 1874249259Sdim // FIXME: consider doing manual inline for small constant sizes and proper 1875249259Sdim // alignment. 1876249259Sdim void visitMemCpyInst(MemCpyInst &I) { 1877249259Sdim IRBuilder<> IRB(&I); 1878288943Sdim IRB.CreateCall( 1879288943Sdim MS.MemcpyFn, 1880288943Sdim {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()), 1881288943Sdim IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()), 1882288943Sdim IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)}); 1883249259Sdim I.eraseFromParent(); 1884249259Sdim } 1885249259Sdim 1886249259Sdim // Same as memcpy. 1887249259Sdim void visitMemSetInst(MemSetInst &I) { 1888249259Sdim IRBuilder<> IRB(&I); 1889288943Sdim IRB.CreateCall( 1890288943Sdim MS.MemsetFn, 1891288943Sdim {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()), 1892288943Sdim IRB.CreateIntCast(I.getArgOperand(1), IRB.getInt32Ty(), false), 1893288943Sdim IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)}); 1894249259Sdim I.eraseFromParent(); 1895249259Sdim } 1896249259Sdim 1897249259Sdim void visitVAStartInst(VAStartInst &I) { 1898249259Sdim VAHelper->visitVAStartInst(I); 1899249259Sdim } 1900249259Sdim 1901249259Sdim void visitVACopyInst(VACopyInst &I) { 1902249259Sdim VAHelper->visitVACopyInst(I); 1903249259Sdim } 1904249259Sdim 1905249259Sdim /// \brief Handle vector store-like intrinsics. 1906249259Sdim /// 1907249259Sdim /// Instrument intrinsics that look like a simple SIMD store: writes memory, 1908249259Sdim /// has 1 pointer argument and 1 vector argument, returns void. 1909249259Sdim bool handleVectorStoreIntrinsic(IntrinsicInst &I) { 1910249259Sdim IRBuilder<> IRB(&I); 1911249259Sdim Value* Addr = I.getArgOperand(0); 1912249259Sdim Value *Shadow = getShadow(&I, 1); 1913249259Sdim Value *ShadowPtr = getShadowPtr(Addr, Shadow->getType(), IRB); 1914249259Sdim 1915249259Sdim // We don't know the pointer alignment (could be unaligned SSE store!). 1916249259Sdim // Have to assume to worst case. 1917249259Sdim IRB.CreateAlignedStore(Shadow, ShadowPtr, 1); 1918249259Sdim 1919249259Sdim if (ClCheckAccessAddress) 1920261991Sdim insertShadowCheck(Addr, &I); 1921249259Sdim 1922249259Sdim // FIXME: use ClStoreCleanOrigin 1923249259Sdim // FIXME: factor out common code from materializeStores 1924249259Sdim if (MS.TrackOrigins) 1925280031Sdim IRB.CreateStore(getOrigin(&I, 1), getOriginPtr(Addr, IRB, 1)); 1926249259Sdim return true; 1927249259Sdim } 1928249259Sdim 1929249259Sdim /// \brief Handle vector load-like intrinsics. 1930249259Sdim /// 1931249259Sdim /// Instrument intrinsics that look like a simple SIMD load: reads memory, 1932249259Sdim /// has 1 pointer argument, returns a vector. 1933249259Sdim bool handleVectorLoadIntrinsic(IntrinsicInst &I) { 1934249259Sdim IRBuilder<> IRB(&I); 1935249259Sdim Value *Addr = I.getArgOperand(0); 1936249259Sdim 1937249259Sdim Type *ShadowTy = getShadowTy(&I); 1938276479Sdim if (PropagateShadow) { 1939249259Sdim Value *ShadowPtr = getShadowPtr(Addr, ShadowTy, IRB); 1940249259Sdim // We don't know the pointer alignment (could be unaligned SSE load!). 1941249259Sdim // Have to assume to worst case. 1942249259Sdim setShadow(&I, IRB.CreateAlignedLoad(ShadowPtr, 1, "_msld")); 1943249259Sdim } else { 1944249259Sdim setShadow(&I, getCleanShadow(&I)); 1945249259Sdim } 1946249259Sdim 1947249259Sdim if (ClCheckAccessAddress) 1948261991Sdim insertShadowCheck(Addr, &I); 1949249259Sdim 1950249259Sdim if (MS.TrackOrigins) { 1951276479Sdim if (PropagateShadow) 1952280031Sdim setOrigin(&I, IRB.CreateLoad(getOriginPtr(Addr, IRB, 1))); 1953249259Sdim else 1954249259Sdim setOrigin(&I, getCleanOrigin()); 1955249259Sdim } 1956249259Sdim return true; 1957249259Sdim } 1958249259Sdim 1959249259Sdim /// \brief Handle (SIMD arithmetic)-like intrinsics. 1960249259Sdim /// 1961249259Sdim /// Instrument intrinsics with any number of arguments of the same type, 1962249259Sdim /// equal to the return type. The type should be simple (no aggregates or 1963249259Sdim /// pointers; vectors are fine). 1964249259Sdim /// Caller guarantees that this intrinsic does not access memory. 1965249259Sdim bool maybeHandleSimpleNomemIntrinsic(IntrinsicInst &I) { 1966249259Sdim Type *RetTy = I.getType(); 1967249259Sdim if (!(RetTy->isIntOrIntVectorTy() || 1968249259Sdim RetTy->isFPOrFPVectorTy() || 1969249259Sdim RetTy->isX86_MMXTy())) 1970249259Sdim return false; 1971249259Sdim 1972249259Sdim unsigned NumArgOperands = I.getNumArgOperands(); 1973249259Sdim 1974249259Sdim for (unsigned i = 0; i < NumArgOperands; ++i) { 1975249259Sdim Type *Ty = I.getArgOperand(i)->getType(); 1976249259Sdim if (Ty != RetTy) 1977249259Sdim return false; 1978249259Sdim } 1979249259Sdim 1980249259Sdim IRBuilder<> IRB(&I); 1981249259Sdim ShadowAndOriginCombiner SC(this, IRB); 1982249259Sdim for (unsigned i = 0; i < NumArgOperands; ++i) 1983249259Sdim SC.Add(I.getArgOperand(i)); 1984249259Sdim SC.Done(&I); 1985249259Sdim 1986249259Sdim return true; 1987249259Sdim } 1988249259Sdim 1989249259Sdim /// \brief Heuristically instrument unknown intrinsics. 1990249259Sdim /// 1991249259Sdim /// The main purpose of this code is to do something reasonable with all 1992249259Sdim /// random intrinsics we might encounter, most importantly - SIMD intrinsics. 1993249259Sdim /// We recognize several classes of intrinsics by their argument types and 1994249259Sdim /// ModRefBehaviour and apply special intrumentation when we are reasonably 1995249259Sdim /// sure that we know what the intrinsic does. 1996249259Sdim /// 1997249259Sdim /// We special-case intrinsics where this approach fails. See llvm.bswap 1998249259Sdim /// handling as an example of that. 1999249259Sdim bool handleUnknownIntrinsic(IntrinsicInst &I) { 2000249259Sdim unsigned NumArgOperands = I.getNumArgOperands(); 2001249259Sdim if (NumArgOperands == 0) 2002249259Sdim return false; 2003249259Sdim 2004249259Sdim if (NumArgOperands == 2 && 2005249259Sdim I.getArgOperand(0)->getType()->isPointerTy() && 2006249259Sdim I.getArgOperand(1)->getType()->isVectorTy() && 2007249259Sdim I.getType()->isVoidTy() && 2008296417Sdim !I.onlyReadsMemory()) { 2009249259Sdim // This looks like a vector store. 2010249259Sdim return handleVectorStoreIntrinsic(I); 2011249259Sdim } 2012249259Sdim 2013249259Sdim if (NumArgOperands == 1 && 2014249259Sdim I.getArgOperand(0)->getType()->isPointerTy() && 2015249259Sdim I.getType()->isVectorTy() && 2016296417Sdim I.onlyReadsMemory()) { 2017249259Sdim // This looks like a vector load. 2018249259Sdim return handleVectorLoadIntrinsic(I); 2019249259Sdim } 2020249259Sdim 2021296417Sdim if (I.doesNotAccessMemory()) 2022249259Sdim if (maybeHandleSimpleNomemIntrinsic(I)) 2023249259Sdim return true; 2024249259Sdim 2025249259Sdim // FIXME: detect and handle SSE maskstore/maskload 2026249259Sdim return false; 2027249259Sdim } 2028249259Sdim 2029249259Sdim void handleBswap(IntrinsicInst &I) { 2030249259Sdim IRBuilder<> IRB(&I); 2031249259Sdim Value *Op = I.getArgOperand(0); 2032249259Sdim Type *OpType = Op->getType(); 2033249259Sdim Function *BswapFunc = Intrinsic::getDeclaration( 2034280031Sdim F.getParent(), Intrinsic::bswap, makeArrayRef(&OpType, 1)); 2035249259Sdim setShadow(&I, IRB.CreateCall(BswapFunc, getShadow(Op))); 2036249259Sdim setOrigin(&I, getOrigin(Op)); 2037249259Sdim } 2038249259Sdim 2039261991Sdim // \brief Instrument vector convert instrinsic. 2040261991Sdim // 2041261991Sdim // This function instruments intrinsics like cvtsi2ss: 2042261991Sdim // %Out = int_xxx_cvtyyy(%ConvertOp) 2043261991Sdim // or 2044261991Sdim // %Out = int_xxx_cvtyyy(%CopyOp, %ConvertOp) 2045261991Sdim // Intrinsic converts \p NumUsedElements elements of \p ConvertOp to the same 2046261991Sdim // number \p Out elements, and (if has 2 arguments) copies the rest of the 2047261991Sdim // elements from \p CopyOp. 2048261991Sdim // In most cases conversion involves floating-point value which may trigger a 2049261991Sdim // hardware exception when not fully initialized. For this reason we require 2050261991Sdim // \p ConvertOp[0:NumUsedElements] to be fully initialized and trap otherwise. 2051261991Sdim // We copy the shadow of \p CopyOp[NumUsedElements:] to \p 2052261991Sdim // Out[NumUsedElements:]. This means that intrinsics without \p CopyOp always 2053261991Sdim // return a fully initialized value. 2054261991Sdim void handleVectorConvertIntrinsic(IntrinsicInst &I, int NumUsedElements) { 2055261991Sdim IRBuilder<> IRB(&I); 2056261991Sdim Value *CopyOp, *ConvertOp; 2057261991Sdim 2058261991Sdim switch (I.getNumArgOperands()) { 2059288943Sdim case 3: 2060288943Sdim assert(isa<ConstantInt>(I.getArgOperand(2)) && "Invalid rounding mode"); 2061261991Sdim case 2: 2062261991Sdim CopyOp = I.getArgOperand(0); 2063261991Sdim ConvertOp = I.getArgOperand(1); 2064261991Sdim break; 2065261991Sdim case 1: 2066261991Sdim ConvertOp = I.getArgOperand(0); 2067276479Sdim CopyOp = nullptr; 2068261991Sdim break; 2069261991Sdim default: 2070261991Sdim llvm_unreachable("Cvt intrinsic with unsupported number of arguments."); 2071261991Sdim } 2072261991Sdim 2073261991Sdim // The first *NumUsedElements* elements of ConvertOp are converted to the 2074261991Sdim // same number of output elements. The rest of the output is copied from 2075261991Sdim // CopyOp, or (if not available) filled with zeroes. 2076261991Sdim // Combine shadow for elements of ConvertOp that are used in this operation, 2077261991Sdim // and insert a check. 2078261991Sdim // FIXME: consider propagating shadow of ConvertOp, at least in the case of 2079261991Sdim // int->any conversion. 2080261991Sdim Value *ConvertShadow = getShadow(ConvertOp); 2081276479Sdim Value *AggShadow = nullptr; 2082261991Sdim if (ConvertOp->getType()->isVectorTy()) { 2083261991Sdim AggShadow = IRB.CreateExtractElement( 2084261991Sdim ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), 0)); 2085261991Sdim for (int i = 1; i < NumUsedElements; ++i) { 2086261991Sdim Value *MoreShadow = IRB.CreateExtractElement( 2087261991Sdim ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), i)); 2088261991Sdim AggShadow = IRB.CreateOr(AggShadow, MoreShadow); 2089261991Sdim } 2090261991Sdim } else { 2091261991Sdim AggShadow = ConvertShadow; 2092261991Sdim } 2093261991Sdim assert(AggShadow->getType()->isIntegerTy()); 2094261991Sdim insertShadowCheck(AggShadow, getOrigin(ConvertOp), &I); 2095261991Sdim 2096261991Sdim // Build result shadow by zero-filling parts of CopyOp shadow that come from 2097261991Sdim // ConvertOp. 2098261991Sdim if (CopyOp) { 2099261991Sdim assert(CopyOp->getType() == I.getType()); 2100261991Sdim assert(CopyOp->getType()->isVectorTy()); 2101261991Sdim Value *ResultShadow = getShadow(CopyOp); 2102261991Sdim Type *EltTy = ResultShadow->getType()->getVectorElementType(); 2103261991Sdim for (int i = 0; i < NumUsedElements; ++i) { 2104261991Sdim ResultShadow = IRB.CreateInsertElement( 2105261991Sdim ResultShadow, ConstantInt::getNullValue(EltTy), 2106261991Sdim ConstantInt::get(IRB.getInt32Ty(), i)); 2107261991Sdim } 2108261991Sdim setShadow(&I, ResultShadow); 2109261991Sdim setOrigin(&I, getOrigin(CopyOp)); 2110261991Sdim } else { 2111261991Sdim setShadow(&I, getCleanShadow(&I)); 2112280031Sdim setOrigin(&I, getCleanOrigin()); 2113261991Sdim } 2114261991Sdim } 2115261991Sdim 2116276479Sdim // Given a scalar or vector, extract lower 64 bits (or less), and return all 2117276479Sdim // zeroes if it is zero, and all ones otherwise. 2118276479Sdim Value *Lower64ShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) { 2119276479Sdim if (S->getType()->isVectorTy()) 2120276479Sdim S = CreateShadowCast(IRB, S, IRB.getInt64Ty(), /* Signed */ true); 2121276479Sdim assert(S->getType()->getPrimitiveSizeInBits() <= 64); 2122276479Sdim Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S)); 2123276479Sdim return CreateShadowCast(IRB, S2, T, /* Signed */ true); 2124276479Sdim } 2125276479Sdim 2126276479Sdim Value *VariableShadowExtend(IRBuilder<> &IRB, Value *S) { 2127276479Sdim Type *T = S->getType(); 2128276479Sdim assert(T->isVectorTy()); 2129276479Sdim Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S)); 2130276479Sdim return IRB.CreateSExt(S2, T); 2131276479Sdim } 2132276479Sdim 2133276479Sdim // \brief Instrument vector shift instrinsic. 2134276479Sdim // 2135276479Sdim // This function instruments intrinsics like int_x86_avx2_psll_w. 2136276479Sdim // Intrinsic shifts %In by %ShiftSize bits. 2137276479Sdim // %ShiftSize may be a vector. In that case the lower 64 bits determine shift 2138276479Sdim // size, and the rest is ignored. Behavior is defined even if shift size is 2139276479Sdim // greater than register (or field) width. 2140276479Sdim void handleVectorShiftIntrinsic(IntrinsicInst &I, bool Variable) { 2141276479Sdim assert(I.getNumArgOperands() == 2); 2142276479Sdim IRBuilder<> IRB(&I); 2143276479Sdim // If any of the S2 bits are poisoned, the whole thing is poisoned. 2144276479Sdim // Otherwise perform the same shift on S1. 2145276479Sdim Value *S1 = getShadow(&I, 0); 2146276479Sdim Value *S2 = getShadow(&I, 1); 2147276479Sdim Value *S2Conv = Variable ? VariableShadowExtend(IRB, S2) 2148276479Sdim : Lower64ShadowExtend(IRB, S2, getShadowTy(&I)); 2149276479Sdim Value *V1 = I.getOperand(0); 2150276479Sdim Value *V2 = I.getOperand(1); 2151288943Sdim Value *Shift = IRB.CreateCall(I.getCalledValue(), 2152288943Sdim {IRB.CreateBitCast(S1, V1->getType()), V2}); 2153276479Sdim Shift = IRB.CreateBitCast(Shift, getShadowTy(&I)); 2154276479Sdim setShadow(&I, IRB.CreateOr(Shift, S2Conv)); 2155276479Sdim setOriginForNaryOp(I); 2156276479Sdim } 2157276479Sdim 2158276479Sdim // \brief Get an X86_MMX-sized vector type. 2159276479Sdim Type *getMMXVectorTy(unsigned EltSizeInBits) { 2160276479Sdim const unsigned X86_MMXSizeInBits = 64; 2161276479Sdim return VectorType::get(IntegerType::get(*MS.C, EltSizeInBits), 2162276479Sdim X86_MMXSizeInBits / EltSizeInBits); 2163276479Sdim } 2164276479Sdim 2165276479Sdim // \brief Returns a signed counterpart for an (un)signed-saturate-and-pack 2166276479Sdim // intrinsic. 2167276479Sdim Intrinsic::ID getSignedPackIntrinsic(Intrinsic::ID id) { 2168276479Sdim switch (id) { 2169276479Sdim case llvm::Intrinsic::x86_sse2_packsswb_128: 2170276479Sdim case llvm::Intrinsic::x86_sse2_packuswb_128: 2171276479Sdim return llvm::Intrinsic::x86_sse2_packsswb_128; 2172276479Sdim 2173276479Sdim case llvm::Intrinsic::x86_sse2_packssdw_128: 2174276479Sdim case llvm::Intrinsic::x86_sse41_packusdw: 2175276479Sdim return llvm::Intrinsic::x86_sse2_packssdw_128; 2176276479Sdim 2177276479Sdim case llvm::Intrinsic::x86_avx2_packsswb: 2178276479Sdim case llvm::Intrinsic::x86_avx2_packuswb: 2179276479Sdim return llvm::Intrinsic::x86_avx2_packsswb; 2180276479Sdim 2181276479Sdim case llvm::Intrinsic::x86_avx2_packssdw: 2182276479Sdim case llvm::Intrinsic::x86_avx2_packusdw: 2183276479Sdim return llvm::Intrinsic::x86_avx2_packssdw; 2184276479Sdim 2185276479Sdim case llvm::Intrinsic::x86_mmx_packsswb: 2186276479Sdim case llvm::Intrinsic::x86_mmx_packuswb: 2187276479Sdim return llvm::Intrinsic::x86_mmx_packsswb; 2188276479Sdim 2189276479Sdim case llvm::Intrinsic::x86_mmx_packssdw: 2190276479Sdim return llvm::Intrinsic::x86_mmx_packssdw; 2191276479Sdim default: 2192276479Sdim llvm_unreachable("unexpected intrinsic id"); 2193276479Sdim } 2194276479Sdim } 2195276479Sdim 2196276479Sdim // \brief Instrument vector pack instrinsic. 2197276479Sdim // 2198276479Sdim // This function instruments intrinsics like x86_mmx_packsswb, that 2199276479Sdim // packs elements of 2 input vectors into half as many bits with saturation. 2200276479Sdim // Shadow is propagated with the signed variant of the same intrinsic applied 2201276479Sdim // to sext(Sa != zeroinitializer), sext(Sb != zeroinitializer). 2202276479Sdim // EltSizeInBits is used only for x86mmx arguments. 2203276479Sdim void handleVectorPackIntrinsic(IntrinsicInst &I, unsigned EltSizeInBits = 0) { 2204276479Sdim assert(I.getNumArgOperands() == 2); 2205276479Sdim bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy(); 2206276479Sdim IRBuilder<> IRB(&I); 2207276479Sdim Value *S1 = getShadow(&I, 0); 2208276479Sdim Value *S2 = getShadow(&I, 1); 2209276479Sdim assert(isX86_MMX || S1->getType()->isVectorTy()); 2210276479Sdim 2211276479Sdim // SExt and ICmpNE below must apply to individual elements of input vectors. 2212276479Sdim // In case of x86mmx arguments, cast them to appropriate vector types and 2213276479Sdim // back. 2214276479Sdim Type *T = isX86_MMX ? getMMXVectorTy(EltSizeInBits) : S1->getType(); 2215276479Sdim if (isX86_MMX) { 2216276479Sdim S1 = IRB.CreateBitCast(S1, T); 2217276479Sdim S2 = IRB.CreateBitCast(S2, T); 2218276479Sdim } 2219276479Sdim Value *S1_ext = IRB.CreateSExt( 2220276479Sdim IRB.CreateICmpNE(S1, llvm::Constant::getNullValue(T)), T); 2221276479Sdim Value *S2_ext = IRB.CreateSExt( 2222276479Sdim IRB.CreateICmpNE(S2, llvm::Constant::getNullValue(T)), T); 2223276479Sdim if (isX86_MMX) { 2224276479Sdim Type *X86_MMXTy = Type::getX86_MMXTy(*MS.C); 2225276479Sdim S1_ext = IRB.CreateBitCast(S1_ext, X86_MMXTy); 2226276479Sdim S2_ext = IRB.CreateBitCast(S2_ext, X86_MMXTy); 2227276479Sdim } 2228276479Sdim 2229276479Sdim Function *ShadowFn = Intrinsic::getDeclaration( 2230276479Sdim F.getParent(), getSignedPackIntrinsic(I.getIntrinsicID())); 2231276479Sdim 2232288943Sdim Value *S = 2233288943Sdim IRB.CreateCall(ShadowFn, {S1_ext, S2_ext}, "_msprop_vector_pack"); 2234276479Sdim if (isX86_MMX) S = IRB.CreateBitCast(S, getShadowTy(&I)); 2235276479Sdim setShadow(&I, S); 2236276479Sdim setOriginForNaryOp(I); 2237276479Sdim } 2238276479Sdim 2239276479Sdim // \brief Instrument sum-of-absolute-differencies intrinsic. 2240276479Sdim void handleVectorSadIntrinsic(IntrinsicInst &I) { 2241276479Sdim const unsigned SignificantBitsPerResultElement = 16; 2242276479Sdim bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy(); 2243276479Sdim Type *ResTy = isX86_MMX ? IntegerType::get(*MS.C, 64) : I.getType(); 2244276479Sdim unsigned ZeroBitsPerResultElement = 2245276479Sdim ResTy->getScalarSizeInBits() - SignificantBitsPerResultElement; 2246276479Sdim 2247276479Sdim IRBuilder<> IRB(&I); 2248276479Sdim Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1)); 2249276479Sdim S = IRB.CreateBitCast(S, ResTy); 2250276479Sdim S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)), 2251276479Sdim ResTy); 2252276479Sdim S = IRB.CreateLShr(S, ZeroBitsPerResultElement); 2253276479Sdim S = IRB.CreateBitCast(S, getShadowTy(&I)); 2254276479Sdim setShadow(&I, S); 2255276479Sdim setOriginForNaryOp(I); 2256276479Sdim } 2257276479Sdim 2258276479Sdim // \brief Instrument multiply-add intrinsic. 2259276479Sdim void handleVectorPmaddIntrinsic(IntrinsicInst &I, 2260276479Sdim unsigned EltSizeInBits = 0) { 2261276479Sdim bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy(); 2262276479Sdim Type *ResTy = isX86_MMX ? getMMXVectorTy(EltSizeInBits * 2) : I.getType(); 2263276479Sdim IRBuilder<> IRB(&I); 2264276479Sdim Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1)); 2265276479Sdim S = IRB.CreateBitCast(S, ResTy); 2266276479Sdim S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)), 2267276479Sdim ResTy); 2268276479Sdim S = IRB.CreateBitCast(S, getShadowTy(&I)); 2269276479Sdim setShadow(&I, S); 2270276479Sdim setOriginForNaryOp(I); 2271276479Sdim } 2272276479Sdim 2273249259Sdim void visitIntrinsicInst(IntrinsicInst &I) { 2274249259Sdim switch (I.getIntrinsicID()) { 2275249259Sdim case llvm::Intrinsic::bswap: 2276249259Sdim handleBswap(I); 2277249259Sdim break; 2278261991Sdim case llvm::Intrinsic::x86_avx512_cvtsd2usi64: 2279261991Sdim case llvm::Intrinsic::x86_avx512_cvtsd2usi: 2280261991Sdim case llvm::Intrinsic::x86_avx512_cvtss2usi64: 2281261991Sdim case llvm::Intrinsic::x86_avx512_cvtss2usi: 2282261991Sdim case llvm::Intrinsic::x86_avx512_cvttss2usi64: 2283261991Sdim case llvm::Intrinsic::x86_avx512_cvttss2usi: 2284261991Sdim case llvm::Intrinsic::x86_avx512_cvttsd2usi64: 2285261991Sdim case llvm::Intrinsic::x86_avx512_cvttsd2usi: 2286261991Sdim case llvm::Intrinsic::x86_avx512_cvtusi2sd: 2287261991Sdim case llvm::Intrinsic::x86_avx512_cvtusi2ss: 2288261991Sdim case llvm::Intrinsic::x86_avx512_cvtusi642sd: 2289261991Sdim case llvm::Intrinsic::x86_avx512_cvtusi642ss: 2290261991Sdim case llvm::Intrinsic::x86_sse2_cvtsd2si64: 2291261991Sdim case llvm::Intrinsic::x86_sse2_cvtsd2si: 2292261991Sdim case llvm::Intrinsic::x86_sse2_cvtsd2ss: 2293261991Sdim case llvm::Intrinsic::x86_sse2_cvtsi2sd: 2294261991Sdim case llvm::Intrinsic::x86_sse2_cvtsi642sd: 2295261991Sdim case llvm::Intrinsic::x86_sse2_cvtss2sd: 2296261991Sdim case llvm::Intrinsic::x86_sse2_cvttsd2si64: 2297261991Sdim case llvm::Intrinsic::x86_sse2_cvttsd2si: 2298261991Sdim case llvm::Intrinsic::x86_sse_cvtsi2ss: 2299261991Sdim case llvm::Intrinsic::x86_sse_cvtsi642ss: 2300261991Sdim case llvm::Intrinsic::x86_sse_cvtss2si64: 2301261991Sdim case llvm::Intrinsic::x86_sse_cvtss2si: 2302261991Sdim case llvm::Intrinsic::x86_sse_cvttss2si64: 2303261991Sdim case llvm::Intrinsic::x86_sse_cvttss2si: 2304261991Sdim handleVectorConvertIntrinsic(I, 1); 2305261991Sdim break; 2306261991Sdim case llvm::Intrinsic::x86_sse2_cvtdq2pd: 2307261991Sdim case llvm::Intrinsic::x86_sse2_cvtps2pd: 2308261991Sdim case llvm::Intrinsic::x86_sse_cvtps2pi: 2309261991Sdim case llvm::Intrinsic::x86_sse_cvttps2pi: 2310261991Sdim handleVectorConvertIntrinsic(I, 2); 2311261991Sdim break; 2312276479Sdim case llvm::Intrinsic::x86_avx2_psll_w: 2313276479Sdim case llvm::Intrinsic::x86_avx2_psll_d: 2314276479Sdim case llvm::Intrinsic::x86_avx2_psll_q: 2315276479Sdim case llvm::Intrinsic::x86_avx2_pslli_w: 2316276479Sdim case llvm::Intrinsic::x86_avx2_pslli_d: 2317276479Sdim case llvm::Intrinsic::x86_avx2_pslli_q: 2318276479Sdim case llvm::Intrinsic::x86_avx2_psrl_w: 2319276479Sdim case llvm::Intrinsic::x86_avx2_psrl_d: 2320276479Sdim case llvm::Intrinsic::x86_avx2_psrl_q: 2321276479Sdim case llvm::Intrinsic::x86_avx2_psra_w: 2322276479Sdim case llvm::Intrinsic::x86_avx2_psra_d: 2323276479Sdim case llvm::Intrinsic::x86_avx2_psrli_w: 2324276479Sdim case llvm::Intrinsic::x86_avx2_psrli_d: 2325276479Sdim case llvm::Intrinsic::x86_avx2_psrli_q: 2326276479Sdim case llvm::Intrinsic::x86_avx2_psrai_w: 2327276479Sdim case llvm::Intrinsic::x86_avx2_psrai_d: 2328276479Sdim case llvm::Intrinsic::x86_sse2_psll_w: 2329276479Sdim case llvm::Intrinsic::x86_sse2_psll_d: 2330276479Sdim case llvm::Intrinsic::x86_sse2_psll_q: 2331276479Sdim case llvm::Intrinsic::x86_sse2_pslli_w: 2332276479Sdim case llvm::Intrinsic::x86_sse2_pslli_d: 2333276479Sdim case llvm::Intrinsic::x86_sse2_pslli_q: 2334276479Sdim case llvm::Intrinsic::x86_sse2_psrl_w: 2335276479Sdim case llvm::Intrinsic::x86_sse2_psrl_d: 2336276479Sdim case llvm::Intrinsic::x86_sse2_psrl_q: 2337276479Sdim case llvm::Intrinsic::x86_sse2_psra_w: 2338276479Sdim case llvm::Intrinsic::x86_sse2_psra_d: 2339276479Sdim case llvm::Intrinsic::x86_sse2_psrli_w: 2340276479Sdim case llvm::Intrinsic::x86_sse2_psrli_d: 2341276479Sdim case llvm::Intrinsic::x86_sse2_psrli_q: 2342276479Sdim case llvm::Intrinsic::x86_sse2_psrai_w: 2343276479Sdim case llvm::Intrinsic::x86_sse2_psrai_d: 2344276479Sdim case llvm::Intrinsic::x86_mmx_psll_w: 2345276479Sdim case llvm::Intrinsic::x86_mmx_psll_d: 2346276479Sdim case llvm::Intrinsic::x86_mmx_psll_q: 2347276479Sdim case llvm::Intrinsic::x86_mmx_pslli_w: 2348276479Sdim case llvm::Intrinsic::x86_mmx_pslli_d: 2349276479Sdim case llvm::Intrinsic::x86_mmx_pslli_q: 2350276479Sdim case llvm::Intrinsic::x86_mmx_psrl_w: 2351276479Sdim case llvm::Intrinsic::x86_mmx_psrl_d: 2352276479Sdim case llvm::Intrinsic::x86_mmx_psrl_q: 2353276479Sdim case llvm::Intrinsic::x86_mmx_psra_w: 2354276479Sdim case llvm::Intrinsic::x86_mmx_psra_d: 2355276479Sdim case llvm::Intrinsic::x86_mmx_psrli_w: 2356276479Sdim case llvm::Intrinsic::x86_mmx_psrli_d: 2357276479Sdim case llvm::Intrinsic::x86_mmx_psrli_q: 2358276479Sdim case llvm::Intrinsic::x86_mmx_psrai_w: 2359276479Sdim case llvm::Intrinsic::x86_mmx_psrai_d: 2360276479Sdim handleVectorShiftIntrinsic(I, /* Variable */ false); 2361276479Sdim break; 2362276479Sdim case llvm::Intrinsic::x86_avx2_psllv_d: 2363276479Sdim case llvm::Intrinsic::x86_avx2_psllv_d_256: 2364276479Sdim case llvm::Intrinsic::x86_avx2_psllv_q: 2365276479Sdim case llvm::Intrinsic::x86_avx2_psllv_q_256: 2366276479Sdim case llvm::Intrinsic::x86_avx2_psrlv_d: 2367276479Sdim case llvm::Intrinsic::x86_avx2_psrlv_d_256: 2368276479Sdim case llvm::Intrinsic::x86_avx2_psrlv_q: 2369276479Sdim case llvm::Intrinsic::x86_avx2_psrlv_q_256: 2370276479Sdim case llvm::Intrinsic::x86_avx2_psrav_d: 2371276479Sdim case llvm::Intrinsic::x86_avx2_psrav_d_256: 2372276479Sdim handleVectorShiftIntrinsic(I, /* Variable */ true); 2373276479Sdim break; 2374276479Sdim 2375276479Sdim case llvm::Intrinsic::x86_sse2_packsswb_128: 2376276479Sdim case llvm::Intrinsic::x86_sse2_packssdw_128: 2377276479Sdim case llvm::Intrinsic::x86_sse2_packuswb_128: 2378276479Sdim case llvm::Intrinsic::x86_sse41_packusdw: 2379276479Sdim case llvm::Intrinsic::x86_avx2_packsswb: 2380276479Sdim case llvm::Intrinsic::x86_avx2_packssdw: 2381276479Sdim case llvm::Intrinsic::x86_avx2_packuswb: 2382276479Sdim case llvm::Intrinsic::x86_avx2_packusdw: 2383276479Sdim handleVectorPackIntrinsic(I); 2384276479Sdim break; 2385276479Sdim 2386276479Sdim case llvm::Intrinsic::x86_mmx_packsswb: 2387276479Sdim case llvm::Intrinsic::x86_mmx_packuswb: 2388276479Sdim handleVectorPackIntrinsic(I, 16); 2389276479Sdim break; 2390276479Sdim 2391276479Sdim case llvm::Intrinsic::x86_mmx_packssdw: 2392276479Sdim handleVectorPackIntrinsic(I, 32); 2393276479Sdim break; 2394276479Sdim 2395276479Sdim case llvm::Intrinsic::x86_mmx_psad_bw: 2396276479Sdim case llvm::Intrinsic::x86_sse2_psad_bw: 2397276479Sdim case llvm::Intrinsic::x86_avx2_psad_bw: 2398276479Sdim handleVectorSadIntrinsic(I); 2399276479Sdim break; 2400276479Sdim 2401276479Sdim case llvm::Intrinsic::x86_sse2_pmadd_wd: 2402276479Sdim case llvm::Intrinsic::x86_avx2_pmadd_wd: 2403276479Sdim case llvm::Intrinsic::x86_ssse3_pmadd_ub_sw_128: 2404276479Sdim case llvm::Intrinsic::x86_avx2_pmadd_ub_sw: 2405276479Sdim handleVectorPmaddIntrinsic(I); 2406276479Sdim break; 2407276479Sdim 2408276479Sdim case llvm::Intrinsic::x86_ssse3_pmadd_ub_sw: 2409276479Sdim handleVectorPmaddIntrinsic(I, 8); 2410276479Sdim break; 2411276479Sdim 2412276479Sdim case llvm::Intrinsic::x86_mmx_pmadd_wd: 2413276479Sdim handleVectorPmaddIntrinsic(I, 16); 2414276479Sdim break; 2415276479Sdim 2416249259Sdim default: 2417249259Sdim if (!handleUnknownIntrinsic(I)) 2418249259Sdim visitInstruction(I); 2419249259Sdim break; 2420249259Sdim } 2421249259Sdim } 2422249259Sdim 2423249259Sdim void visitCallSite(CallSite CS) { 2424249259Sdim Instruction &I = *CS.getInstruction(); 2425249259Sdim assert((CS.isCall() || CS.isInvoke()) && "Unknown type of CallSite"); 2426249259Sdim if (CS.isCall()) { 2427249259Sdim CallInst *Call = cast<CallInst>(&I); 2428249259Sdim 2429249259Sdim // For inline asm, do the usual thing: check argument shadow and mark all 2430249259Sdim // outputs as clean. Note that any side effects of the inline asm that are 2431249259Sdim // not immediately visible in its constraints are not handled. 2432249259Sdim if (Call->isInlineAsm()) { 2433249259Sdim visitInstruction(I); 2434249259Sdim return; 2435249259Sdim } 2436249259Sdim 2437249259Sdim assert(!isa<IntrinsicInst>(&I) && "intrinsics are handled elsewhere"); 2438249259Sdim 2439249259Sdim // We are going to insert code that relies on the fact that the callee 2440249259Sdim // will become a non-readonly function after it is instrumented by us. To 2441249259Sdim // prevent this code from being optimized out, mark that function 2442249259Sdim // non-readonly in advance. 2443249259Sdim if (Function *Func = Call->getCalledFunction()) { 2444249259Sdim // Clear out readonly/readnone attributes. 2445249259Sdim AttrBuilder B; 2446249259Sdim B.addAttribute(Attribute::ReadOnly) 2447249259Sdim .addAttribute(Attribute::ReadNone); 2448249259Sdim Func->removeAttributes(AttributeSet::FunctionIndex, 2449249259Sdim AttributeSet::get(Func->getContext(), 2450249259Sdim AttributeSet::FunctionIndex, 2451249259Sdim B)); 2452249259Sdim } 2453249259Sdim } 2454249259Sdim IRBuilder<> IRB(&I); 2455261991Sdim 2456249259Sdim unsigned ArgOffset = 0; 2457249259Sdim DEBUG(dbgs() << " CallSite: " << I << "\n"); 2458249259Sdim for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end(); 2459249259Sdim ArgIt != End; ++ArgIt) { 2460249259Sdim Value *A = *ArgIt; 2461249259Sdim unsigned i = ArgIt - CS.arg_begin(); 2462249259Sdim if (!A->getType()->isSized()) { 2463249259Sdim DEBUG(dbgs() << "Arg " << i << " is not sized: " << I << "\n"); 2464249259Sdim continue; 2465249259Sdim } 2466249259Sdim unsigned Size = 0; 2467276479Sdim Value *Store = nullptr; 2468249259Sdim // Compute the Shadow for arg even if it is ByVal, because 2469249259Sdim // in that case getShadow() will copy the actual arg shadow to 2470249259Sdim // __msan_param_tls. 2471249259Sdim Value *ArgShadow = getShadow(A); 2472249259Sdim Value *ArgShadowBase = getShadowPtrForArgument(A, IRB, ArgOffset); 2473249259Sdim DEBUG(dbgs() << " Arg#" << i << ": " << *A << 2474249259Sdim " Shadow: " << *ArgShadow << "\n"); 2475276479Sdim bool ArgIsInitialized = false; 2476288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 2477249259Sdim if (CS.paramHasAttr(i + 1, Attribute::ByVal)) { 2478249259Sdim assert(A->getType()->isPointerTy() && 2479249259Sdim "ByVal argument is not a pointer!"); 2480288943Sdim Size = DL.getTypeAllocSize(A->getType()->getPointerElementType()); 2481280031Sdim if (ArgOffset + Size > kParamTLSSize) break; 2482280031Sdim unsigned ParamAlignment = CS.getParamAlignment(i + 1); 2483280031Sdim unsigned Alignment = std::min(ParamAlignment, kShadowTLSAlignment); 2484249259Sdim Store = IRB.CreateMemCpy(ArgShadowBase, 2485249259Sdim getShadowPtr(A, Type::getInt8Ty(*MS.C), IRB), 2486249259Sdim Size, Alignment); 2487249259Sdim } else { 2488288943Sdim Size = DL.getTypeAllocSize(A->getType()); 2489280031Sdim if (ArgOffset + Size > kParamTLSSize) break; 2490249259Sdim Store = IRB.CreateAlignedStore(ArgShadow, ArgShadowBase, 2491249259Sdim kShadowTLSAlignment); 2492276479Sdim Constant *Cst = dyn_cast<Constant>(ArgShadow); 2493276479Sdim if (Cst && Cst->isNullValue()) ArgIsInitialized = true; 2494249259Sdim } 2495276479Sdim if (MS.TrackOrigins && !ArgIsInitialized) 2496249259Sdim IRB.CreateStore(getOrigin(A), 2497249259Sdim getOriginPtrForArgument(A, IRB, ArgOffset)); 2498249259Sdim (void)Store; 2499276479Sdim assert(Size != 0 && Store != nullptr); 2500249259Sdim DEBUG(dbgs() << " Param:" << *Store << "\n"); 2501280031Sdim ArgOffset += RoundUpToAlignment(Size, 8); 2502249259Sdim } 2503249259Sdim DEBUG(dbgs() << " done with call args\n"); 2504249259Sdim 2505249259Sdim FunctionType *FT = 2506261991Sdim cast<FunctionType>(CS.getCalledValue()->getType()->getContainedType(0)); 2507249259Sdim if (FT->isVarArg()) { 2508249259Sdim VAHelper->visitCallSite(CS, IRB); 2509249259Sdim } 2510249259Sdim 2511249259Sdim // Now, get the shadow for the RetVal. 2512249259Sdim if (!I.getType()->isSized()) return; 2513296417Sdim // Don't emit the epilogue for musttail call returns. 2514296417Sdim if (CS.isCall() && cast<CallInst>(&I)->isMustTailCall()) return; 2515249259Sdim IRBuilder<> IRBBefore(&I); 2516276479Sdim // Until we have full dynamic coverage, make sure the retval shadow is 0. 2517249259Sdim Value *Base = getShadowPtrForRetval(&I, IRBBefore); 2518249259Sdim IRBBefore.CreateAlignedStore(getCleanShadow(&I), Base, kShadowTLSAlignment); 2519296417Sdim BasicBlock::iterator NextInsn; 2520249259Sdim if (CS.isCall()) { 2521296417Sdim NextInsn = ++I.getIterator(); 2522296417Sdim assert(NextInsn != I.getParent()->end()); 2523249259Sdim } else { 2524249259Sdim BasicBlock *NormalDest = cast<InvokeInst>(&I)->getNormalDest(); 2525249259Sdim if (!NormalDest->getSinglePredecessor()) { 2526249259Sdim // FIXME: this case is tricky, so we are just conservative here. 2527249259Sdim // Perhaps we need to split the edge between this BB and NormalDest, 2528249259Sdim // but a naive attempt to use SplitEdge leads to a crash. 2529249259Sdim setShadow(&I, getCleanShadow(&I)); 2530249259Sdim setOrigin(&I, getCleanOrigin()); 2531249259Sdim return; 2532249259Sdim } 2533249259Sdim NextInsn = NormalDest->getFirstInsertionPt(); 2534296417Sdim assert(NextInsn != NormalDest->end() && 2535249259Sdim "Could not find insertion point for retval shadow load"); 2536249259Sdim } 2537296417Sdim IRBuilder<> IRBAfter(&*NextInsn); 2538249259Sdim Value *RetvalShadow = 2539249259Sdim IRBAfter.CreateAlignedLoad(getShadowPtrForRetval(&I, IRBAfter), 2540249259Sdim kShadowTLSAlignment, "_msret"); 2541249259Sdim setShadow(&I, RetvalShadow); 2542249259Sdim if (MS.TrackOrigins) 2543249259Sdim setOrigin(&I, IRBAfter.CreateLoad(getOriginPtrForRetval(IRBAfter))); 2544249259Sdim } 2545249259Sdim 2546296417Sdim bool isAMustTailRetVal(Value *RetVal) { 2547296417Sdim if (auto *I = dyn_cast<BitCastInst>(RetVal)) { 2548296417Sdim RetVal = I->getOperand(0); 2549296417Sdim } 2550296417Sdim if (auto *I = dyn_cast<CallInst>(RetVal)) { 2551296417Sdim return I->isMustTailCall(); 2552296417Sdim } 2553296417Sdim return false; 2554296417Sdim } 2555296417Sdim 2556249259Sdim void visitReturnInst(ReturnInst &I) { 2557249259Sdim IRBuilder<> IRB(&I); 2558261991Sdim Value *RetVal = I.getReturnValue(); 2559261991Sdim if (!RetVal) return; 2560296417Sdim // Don't emit the epilogue for musttail call returns. 2561296417Sdim if (isAMustTailRetVal(RetVal)) return; 2562261991Sdim Value *ShadowPtr = getShadowPtrForRetval(RetVal, IRB); 2563261991Sdim if (CheckReturnValue) { 2564261991Sdim insertShadowCheck(RetVal, &I); 2565261991Sdim Value *Shadow = getCleanShadow(RetVal); 2566261991Sdim IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment); 2567261991Sdim } else { 2568249259Sdim Value *Shadow = getShadow(RetVal); 2569249259Sdim IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment); 2570261991Sdim // FIXME: make it conditional if ClStoreCleanOrigin==0 2571249259Sdim if (MS.TrackOrigins) 2572249259Sdim IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval(IRB)); 2573249259Sdim } 2574249259Sdim } 2575249259Sdim 2576249259Sdim void visitPHINode(PHINode &I) { 2577249259Sdim IRBuilder<> IRB(&I); 2578276479Sdim if (!PropagateShadow) { 2579276479Sdim setShadow(&I, getCleanShadow(&I)); 2580280031Sdim setOrigin(&I, getCleanOrigin()); 2581276479Sdim return; 2582276479Sdim } 2583276479Sdim 2584249259Sdim ShadowPHINodes.push_back(&I); 2585249259Sdim setShadow(&I, IRB.CreatePHI(getShadowTy(&I), I.getNumIncomingValues(), 2586249259Sdim "_msphi_s")); 2587249259Sdim if (MS.TrackOrigins) 2588249259Sdim setOrigin(&I, IRB.CreatePHI(MS.OriginTy, I.getNumIncomingValues(), 2589249259Sdim "_msphi_o")); 2590249259Sdim } 2591249259Sdim 2592249259Sdim void visitAllocaInst(AllocaInst &I) { 2593249259Sdim setShadow(&I, getCleanShadow(&I)); 2594280031Sdim setOrigin(&I, getCleanOrigin()); 2595249259Sdim IRBuilder<> IRB(I.getNextNode()); 2596288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 2597288943Sdim uint64_t Size = DL.getTypeAllocSize(I.getAllocatedType()); 2598261991Sdim if (PoisonStack && ClPoisonStackWithCall) { 2599288943Sdim IRB.CreateCall(MS.MsanPoisonStackFn, 2600288943Sdim {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), 2601288943Sdim ConstantInt::get(MS.IntptrTy, Size)}); 2602249259Sdim } else { 2603249259Sdim Value *ShadowBase = getShadowPtr(&I, Type::getInt8PtrTy(*MS.C), IRB); 2604261991Sdim Value *PoisonValue = IRB.getInt8(PoisonStack ? ClPoisonStackPattern : 0); 2605261991Sdim IRB.CreateMemSet(ShadowBase, PoisonValue, Size, I.getAlignment()); 2606249259Sdim } 2607249259Sdim 2608261991Sdim if (PoisonStack && MS.TrackOrigins) { 2609249259Sdim SmallString<2048> StackDescriptionStorage; 2610249259Sdim raw_svector_ostream StackDescription(StackDescriptionStorage); 2611249259Sdim // We create a string with a description of the stack allocation and 2612249259Sdim // pass it into __msan_set_alloca_origin. 2613249259Sdim // It will be printed by the run-time if stack-originated UMR is found. 2614249259Sdim // The first 4 bytes of the string are set to '----' and will be replaced 2615249259Sdim // by __msan_va_arg_overflow_size_tls at the first call. 2616249259Sdim StackDescription << "----" << I.getName() << "@" << F.getName(); 2617249259Sdim Value *Descr = 2618249259Sdim createPrivateNonConstGlobalForString(*F.getParent(), 2619249259Sdim StackDescription.str()); 2620261991Sdim 2621288943Sdim IRB.CreateCall(MS.MsanSetAllocaOrigin4Fn, 2622288943Sdim {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), 2623249259Sdim ConstantInt::get(MS.IntptrTy, Size), 2624261991Sdim IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy()), 2625288943Sdim IRB.CreatePointerCast(&F, MS.IntptrTy)}); 2626249259Sdim } 2627249259Sdim } 2628249259Sdim 2629249259Sdim void visitSelectInst(SelectInst& I) { 2630249259Sdim IRBuilder<> IRB(&I); 2631261991Sdim // a = select b, c, d 2632276479Sdim Value *B = I.getCondition(); 2633276479Sdim Value *C = I.getTrueValue(); 2634276479Sdim Value *D = I.getFalseValue(); 2635276479Sdim Value *Sb = getShadow(B); 2636276479Sdim Value *Sc = getShadow(C); 2637276479Sdim Value *Sd = getShadow(D); 2638276479Sdim 2639276479Sdim // Result shadow if condition shadow is 0. 2640276479Sdim Value *Sa0 = IRB.CreateSelect(B, Sc, Sd); 2641276479Sdim Value *Sa1; 2642261991Sdim if (I.getType()->isAggregateType()) { 2643261991Sdim // To avoid "sign extending" i1 to an arbitrary aggregate type, we just do 2644261991Sdim // an extra "select". This results in much more compact IR. 2645261991Sdim // Sa = select Sb, poisoned, (select b, Sc, Sd) 2646276479Sdim Sa1 = getPoisonedShadow(getShadowTy(I.getType())); 2647261991Sdim } else { 2648276479Sdim // Sa = select Sb, [ (c^d) | Sc | Sd ], [ b ? Sc : Sd ] 2649276479Sdim // If Sb (condition is poisoned), look for bits in c and d that are equal 2650276479Sdim // and both unpoisoned. 2651276479Sdim // If !Sb (condition is unpoisoned), simply pick one of Sc and Sd. 2652276479Sdim 2653276479Sdim // Cast arguments to shadow-compatible type. 2654276479Sdim C = CreateAppToShadowCast(IRB, C); 2655276479Sdim D = CreateAppToShadowCast(IRB, D); 2656276479Sdim 2657276479Sdim // Result shadow if condition shadow is 1. 2658276479Sdim Sa1 = IRB.CreateOr(IRB.CreateXor(C, D), IRB.CreateOr(Sc, Sd)); 2659261991Sdim } 2660276479Sdim Value *Sa = IRB.CreateSelect(Sb, Sa1, Sa0, "_msprop_select"); 2661276479Sdim setShadow(&I, Sa); 2662249259Sdim if (MS.TrackOrigins) { 2663249259Sdim // Origins are always i32, so any vector conditions must be flattened. 2664249259Sdim // FIXME: consider tracking vector origins for app vectors? 2665276479Sdim if (B->getType()->isVectorTy()) { 2666276479Sdim Type *FlatTy = getShadowTyNoVec(B->getType()); 2667276479Sdim B = IRB.CreateICmpNE(IRB.CreateBitCast(B, FlatTy), 2668276479Sdim ConstantInt::getNullValue(FlatTy)); 2669276479Sdim Sb = IRB.CreateICmpNE(IRB.CreateBitCast(Sb, FlatTy), 2670276479Sdim ConstantInt::getNullValue(FlatTy)); 2671249259Sdim } 2672276479Sdim // a = select b, c, d 2673276479Sdim // Oa = Sb ? Ob : (b ? Oc : Od) 2674280031Sdim setOrigin( 2675280031Sdim &I, IRB.CreateSelect(Sb, getOrigin(I.getCondition()), 2676280031Sdim IRB.CreateSelect(B, getOrigin(I.getTrueValue()), 2677280031Sdim getOrigin(I.getFalseValue())))); 2678249259Sdim } 2679249259Sdim } 2680249259Sdim 2681249259Sdim void visitLandingPadInst(LandingPadInst &I) { 2682249259Sdim // Do nothing. 2683249259Sdim // See http://code.google.com/p/memory-sanitizer/issues/detail?id=1 2684249259Sdim setShadow(&I, getCleanShadow(&I)); 2685249259Sdim setOrigin(&I, getCleanOrigin()); 2686249259Sdim } 2687249259Sdim 2688296417Sdim void visitCatchSwitchInst(CatchSwitchInst &I) { 2689296417Sdim setShadow(&I, getCleanShadow(&I)); 2690296417Sdim setOrigin(&I, getCleanOrigin()); 2691296417Sdim } 2692296417Sdim 2693296417Sdim void visitFuncletPadInst(FuncletPadInst &I) { 2694296417Sdim setShadow(&I, getCleanShadow(&I)); 2695296417Sdim setOrigin(&I, getCleanOrigin()); 2696296417Sdim } 2697296417Sdim 2698249259Sdim void visitGetElementPtrInst(GetElementPtrInst &I) { 2699249259Sdim handleShadowOr(I); 2700249259Sdim } 2701249259Sdim 2702249259Sdim void visitExtractValueInst(ExtractValueInst &I) { 2703249259Sdim IRBuilder<> IRB(&I); 2704249259Sdim Value *Agg = I.getAggregateOperand(); 2705249259Sdim DEBUG(dbgs() << "ExtractValue: " << I << "\n"); 2706249259Sdim Value *AggShadow = getShadow(Agg); 2707249259Sdim DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n"); 2708249259Sdim Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices()); 2709249259Sdim DEBUG(dbgs() << " ResShadow: " << *ResShadow << "\n"); 2710249259Sdim setShadow(&I, ResShadow); 2711261991Sdim setOriginForNaryOp(I); 2712249259Sdim } 2713249259Sdim 2714249259Sdim void visitInsertValueInst(InsertValueInst &I) { 2715249259Sdim IRBuilder<> IRB(&I); 2716249259Sdim DEBUG(dbgs() << "InsertValue: " << I << "\n"); 2717249259Sdim Value *AggShadow = getShadow(I.getAggregateOperand()); 2718249259Sdim Value *InsShadow = getShadow(I.getInsertedValueOperand()); 2719249259Sdim DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n"); 2720249259Sdim DEBUG(dbgs() << " InsShadow: " << *InsShadow << "\n"); 2721249259Sdim Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices()); 2722249259Sdim DEBUG(dbgs() << " Res: " << *Res << "\n"); 2723249259Sdim setShadow(&I, Res); 2724261991Sdim setOriginForNaryOp(I); 2725249259Sdim } 2726249259Sdim 2727249259Sdim void dumpInst(Instruction &I) { 2728249259Sdim if (CallInst *CI = dyn_cast<CallInst>(&I)) { 2729249259Sdim errs() << "ZZZ call " << CI->getCalledFunction()->getName() << "\n"; 2730249259Sdim } else { 2731249259Sdim errs() << "ZZZ " << I.getOpcodeName() << "\n"; 2732249259Sdim } 2733249259Sdim errs() << "QQQ " << I << "\n"; 2734249259Sdim } 2735249259Sdim 2736249259Sdim void visitResumeInst(ResumeInst &I) { 2737249259Sdim DEBUG(dbgs() << "Resume: " << I << "\n"); 2738249259Sdim // Nothing to do here. 2739249259Sdim } 2740249259Sdim 2741296417Sdim void visitCleanupReturnInst(CleanupReturnInst &CRI) { 2742296417Sdim DEBUG(dbgs() << "CleanupReturn: " << CRI << "\n"); 2743296417Sdim // Nothing to do here. 2744296417Sdim } 2745296417Sdim 2746296417Sdim void visitCatchReturnInst(CatchReturnInst &CRI) { 2747296417Sdim DEBUG(dbgs() << "CatchReturn: " << CRI << "\n"); 2748296417Sdim // Nothing to do here. 2749296417Sdim } 2750296417Sdim 2751249259Sdim void visitInstruction(Instruction &I) { 2752249259Sdim // Everything else: stop propagating and check for poisoned shadow. 2753249259Sdim if (ClDumpStrictInstructions) 2754249259Sdim dumpInst(I); 2755249259Sdim DEBUG(dbgs() << "DEFAULT: " << I << "\n"); 2756249259Sdim for (size_t i = 0, n = I.getNumOperands(); i < n; i++) 2757261991Sdim insertShadowCheck(I.getOperand(i), &I); 2758249259Sdim setShadow(&I, getCleanShadow(&I)); 2759249259Sdim setOrigin(&I, getCleanOrigin()); 2760249259Sdim } 2761249259Sdim}; 2762249259Sdim 2763249259Sdim/// \brief AMD64-specific implementation of VarArgHelper. 2764249259Sdimstruct VarArgAMD64Helper : public VarArgHelper { 2765249259Sdim // An unfortunate workaround for asymmetric lowering of va_arg stuff. 2766249259Sdim // See a comment in visitCallSite for more details. 2767249259Sdim static const unsigned AMD64GpEndOffset = 48; // AMD64 ABI Draft 0.99.6 p3.5.7 2768249259Sdim static const unsigned AMD64FpEndOffset = 176; 2769249259Sdim 2770249259Sdim Function &F; 2771249259Sdim MemorySanitizer &MS; 2772249259Sdim MemorySanitizerVisitor &MSV; 2773249259Sdim Value *VAArgTLSCopy; 2774249259Sdim Value *VAArgOverflowSize; 2775249259Sdim 2776249259Sdim SmallVector<CallInst*, 16> VAStartInstrumentationList; 2777249259Sdim 2778249259Sdim VarArgAMD64Helper(Function &F, MemorySanitizer &MS, 2779249259Sdim MemorySanitizerVisitor &MSV) 2780276479Sdim : F(F), MS(MS), MSV(MSV), VAArgTLSCopy(nullptr), 2781276479Sdim VAArgOverflowSize(nullptr) {} 2782249259Sdim 2783249259Sdim enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory }; 2784249259Sdim 2785249259Sdim ArgKind classifyArgument(Value* arg) { 2786249259Sdim // A very rough approximation of X86_64 argument classification rules. 2787249259Sdim Type *T = arg->getType(); 2788249259Sdim if (T->isFPOrFPVectorTy() || T->isX86_MMXTy()) 2789249259Sdim return AK_FloatingPoint; 2790249259Sdim if (T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64) 2791249259Sdim return AK_GeneralPurpose; 2792249259Sdim if (T->isPointerTy()) 2793249259Sdim return AK_GeneralPurpose; 2794249259Sdim return AK_Memory; 2795249259Sdim } 2796249259Sdim 2797249259Sdim // For VarArg functions, store the argument shadow in an ABI-specific format 2798249259Sdim // that corresponds to va_list layout. 2799249259Sdim // We do this because Clang lowers va_arg in the frontend, and this pass 2800249259Sdim // only sees the low level code that deals with va_list internals. 2801249259Sdim // A much easier alternative (provided that Clang emits va_arg instructions) 2802249259Sdim // would have been to associate each live instance of va_list with a copy of 2803249259Sdim // MSanParamTLS, and extract shadow on va_arg() call in the argument list 2804249259Sdim // order. 2805276479Sdim void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override { 2806249259Sdim unsigned GpOffset = 0; 2807249259Sdim unsigned FpOffset = AMD64GpEndOffset; 2808249259Sdim unsigned OverflowOffset = AMD64FpEndOffset; 2809288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 2810249259Sdim for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end(); 2811249259Sdim ArgIt != End; ++ArgIt) { 2812249259Sdim Value *A = *ArgIt; 2813276479Sdim unsigned ArgNo = CS.getArgumentNo(ArgIt); 2814276479Sdim bool IsByVal = CS.paramHasAttr(ArgNo + 1, Attribute::ByVal); 2815276479Sdim if (IsByVal) { 2816276479Sdim // ByVal arguments always go to the overflow area. 2817276479Sdim assert(A->getType()->isPointerTy()); 2818276479Sdim Type *RealTy = A->getType()->getPointerElementType(); 2819288943Sdim uint64_t ArgSize = DL.getTypeAllocSize(RealTy); 2820276479Sdim Value *Base = getShadowPtrForVAArgument(RealTy, IRB, OverflowOffset); 2821280031Sdim OverflowOffset += RoundUpToAlignment(ArgSize, 8); 2822276479Sdim IRB.CreateMemCpy(Base, MSV.getShadowPtr(A, IRB.getInt8Ty(), IRB), 2823276479Sdim ArgSize, kShadowTLSAlignment); 2824276479Sdim } else { 2825276479Sdim ArgKind AK = classifyArgument(A); 2826276479Sdim if (AK == AK_GeneralPurpose && GpOffset >= AMD64GpEndOffset) 2827276479Sdim AK = AK_Memory; 2828276479Sdim if (AK == AK_FloatingPoint && FpOffset >= AMD64FpEndOffset) 2829276479Sdim AK = AK_Memory; 2830276479Sdim Value *Base; 2831276479Sdim switch (AK) { 2832276479Sdim case AK_GeneralPurpose: 2833276479Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, GpOffset); 2834276479Sdim GpOffset += 8; 2835276479Sdim break; 2836276479Sdim case AK_FloatingPoint: 2837276479Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, FpOffset); 2838276479Sdim FpOffset += 16; 2839276479Sdim break; 2840276479Sdim case AK_Memory: 2841288943Sdim uint64_t ArgSize = DL.getTypeAllocSize(A->getType()); 2842276479Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset); 2843280031Sdim OverflowOffset += RoundUpToAlignment(ArgSize, 8); 2844276479Sdim } 2845276479Sdim IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment); 2846249259Sdim } 2847249259Sdim } 2848249259Sdim Constant *OverflowSize = 2849249259Sdim ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AMD64FpEndOffset); 2850249259Sdim IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS); 2851249259Sdim } 2852249259Sdim 2853249259Sdim /// \brief Compute the shadow address for a given va_arg. 2854276479Sdim Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB, 2855249259Sdim int ArgOffset) { 2856249259Sdim Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy); 2857249259Sdim Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset)); 2858276479Sdim return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0), 2859249259Sdim "_msarg"); 2860249259Sdim } 2861249259Sdim 2862276479Sdim void visitVAStartInst(VAStartInst &I) override { 2863296417Sdim if (F.getCallingConv() == CallingConv::X86_64_Win64) 2864296417Sdim return; 2865249259Sdim IRBuilder<> IRB(&I); 2866249259Sdim VAStartInstrumentationList.push_back(&I); 2867249259Sdim Value *VAListTag = I.getArgOperand(0); 2868249259Sdim Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB); 2869249259Sdim 2870249259Sdim // Unpoison the whole __va_list_tag. 2871249259Sdim // FIXME: magic ABI constants. 2872249259Sdim IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()), 2873249259Sdim /* size */24, /* alignment */8, false); 2874249259Sdim } 2875249259Sdim 2876276479Sdim void visitVACopyInst(VACopyInst &I) override { 2877296417Sdim if (F.getCallingConv() == CallingConv::X86_64_Win64) 2878296417Sdim return; 2879249259Sdim IRBuilder<> IRB(&I); 2880249259Sdim Value *VAListTag = I.getArgOperand(0); 2881249259Sdim Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB); 2882249259Sdim 2883249259Sdim // Unpoison the whole __va_list_tag. 2884249259Sdim // FIXME: magic ABI constants. 2885249259Sdim IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()), 2886249259Sdim /* size */24, /* alignment */8, false); 2887249259Sdim } 2888249259Sdim 2889276479Sdim void finalizeInstrumentation() override { 2890249259Sdim assert(!VAArgOverflowSize && !VAArgTLSCopy && 2891249259Sdim "finalizeInstrumentation called twice"); 2892249259Sdim if (!VAStartInstrumentationList.empty()) { 2893249259Sdim // If there is a va_start in this function, make a backup copy of 2894249259Sdim // va_arg_tls somewhere in the function entry block. 2895249259Sdim IRBuilder<> IRB(F.getEntryBlock().getFirstNonPHI()); 2896249259Sdim VAArgOverflowSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS); 2897249259Sdim Value *CopySize = 2898249259Sdim IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AMD64FpEndOffset), 2899249259Sdim VAArgOverflowSize); 2900249259Sdim VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize); 2901249259Sdim IRB.CreateMemCpy(VAArgTLSCopy, MS.VAArgTLS, CopySize, 8); 2902249259Sdim } 2903249259Sdim 2904249259Sdim // Instrument va_start. 2905249259Sdim // Copy va_list shadow from the backup copy of the TLS contents. 2906249259Sdim for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) { 2907249259Sdim CallInst *OrigInst = VAStartInstrumentationList[i]; 2908249259Sdim IRBuilder<> IRB(OrigInst->getNextNode()); 2909249259Sdim Value *VAListTag = OrigInst->getArgOperand(0); 2910249259Sdim 2911249259Sdim Value *RegSaveAreaPtrPtr = 2912249259Sdim IRB.CreateIntToPtr( 2913249259Sdim IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy), 2914249259Sdim ConstantInt::get(MS.IntptrTy, 16)), 2915249259Sdim Type::getInt64PtrTy(*MS.C)); 2916249259Sdim Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrPtr); 2917249259Sdim Value *RegSaveAreaShadowPtr = 2918249259Sdim MSV.getShadowPtr(RegSaveAreaPtr, IRB.getInt8Ty(), IRB); 2919249259Sdim IRB.CreateMemCpy(RegSaveAreaShadowPtr, VAArgTLSCopy, 2920249259Sdim AMD64FpEndOffset, 16); 2921249259Sdim 2922249259Sdim Value *OverflowArgAreaPtrPtr = 2923249259Sdim IRB.CreateIntToPtr( 2924249259Sdim IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy), 2925249259Sdim ConstantInt::get(MS.IntptrTy, 8)), 2926249259Sdim Type::getInt64PtrTy(*MS.C)); 2927249259Sdim Value *OverflowArgAreaPtr = IRB.CreateLoad(OverflowArgAreaPtrPtr); 2928249259Sdim Value *OverflowArgAreaShadowPtr = 2929249259Sdim MSV.getShadowPtr(OverflowArgAreaPtr, IRB.getInt8Ty(), IRB); 2930288943Sdim Value *SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSCopy, 2931288943Sdim AMD64FpEndOffset); 2932249259Sdim IRB.CreateMemCpy(OverflowArgAreaShadowPtr, SrcPtr, VAArgOverflowSize, 16); 2933249259Sdim } 2934249259Sdim } 2935249259Sdim}; 2936249259Sdim 2937288943Sdim/// \brief MIPS64-specific implementation of VarArgHelper. 2938288943Sdimstruct VarArgMIPS64Helper : public VarArgHelper { 2939288943Sdim Function &F; 2940288943Sdim MemorySanitizer &MS; 2941288943Sdim MemorySanitizerVisitor &MSV; 2942288943Sdim Value *VAArgTLSCopy; 2943288943Sdim Value *VAArgSize; 2944288943Sdim 2945288943Sdim SmallVector<CallInst*, 16> VAStartInstrumentationList; 2946288943Sdim 2947288943Sdim VarArgMIPS64Helper(Function &F, MemorySanitizer &MS, 2948288943Sdim MemorySanitizerVisitor &MSV) 2949288943Sdim : F(F), MS(MS), MSV(MSV), VAArgTLSCopy(nullptr), 2950288943Sdim VAArgSize(nullptr) {} 2951288943Sdim 2952288943Sdim void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override { 2953288943Sdim unsigned VAArgOffset = 0; 2954288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 2955288943Sdim for (CallSite::arg_iterator ArgIt = CS.arg_begin() + 1, End = CS.arg_end(); 2956288943Sdim ArgIt != End; ++ArgIt) { 2957288943Sdim Value *A = *ArgIt; 2958288943Sdim Value *Base; 2959288943Sdim uint64_t ArgSize = DL.getTypeAllocSize(A->getType()); 2960288943Sdim#if defined(__MIPSEB__) || defined(MIPSEB) 2961288943Sdim // Adjusting the shadow for argument with size < 8 to match the placement 2962288943Sdim // of bits in big endian system 2963288943Sdim if (ArgSize < 8) 2964288943Sdim VAArgOffset += (8 - ArgSize); 2965288943Sdim#endif 2966288943Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, VAArgOffset); 2967288943Sdim VAArgOffset += ArgSize; 2968288943Sdim VAArgOffset = RoundUpToAlignment(VAArgOffset, 8); 2969288943Sdim IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment); 2970288943Sdim } 2971288943Sdim 2972288943Sdim Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(), VAArgOffset); 2973288943Sdim // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of 2974288943Sdim // a new class member i.e. it is the total size of all VarArgs. 2975288943Sdim IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS); 2976288943Sdim } 2977288943Sdim 2978288943Sdim /// \brief Compute the shadow address for a given va_arg. 2979288943Sdim Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB, 2980288943Sdim int ArgOffset) { 2981288943Sdim Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy); 2982288943Sdim Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset)); 2983288943Sdim return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0), 2984288943Sdim "_msarg"); 2985288943Sdim } 2986288943Sdim 2987288943Sdim void visitVAStartInst(VAStartInst &I) override { 2988288943Sdim IRBuilder<> IRB(&I); 2989288943Sdim VAStartInstrumentationList.push_back(&I); 2990288943Sdim Value *VAListTag = I.getArgOperand(0); 2991288943Sdim Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB); 2992288943Sdim IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()), 2993288943Sdim /* size */8, /* alignment */8, false); 2994288943Sdim } 2995288943Sdim 2996288943Sdim void visitVACopyInst(VACopyInst &I) override { 2997288943Sdim IRBuilder<> IRB(&I); 2998288943Sdim Value *VAListTag = I.getArgOperand(0); 2999288943Sdim Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB); 3000288943Sdim // Unpoison the whole __va_list_tag. 3001288943Sdim // FIXME: magic ABI constants. 3002288943Sdim IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()), 3003288943Sdim /* size */8, /* alignment */8, false); 3004288943Sdim } 3005288943Sdim 3006288943Sdim void finalizeInstrumentation() override { 3007288943Sdim assert(!VAArgSize && !VAArgTLSCopy && 3008288943Sdim "finalizeInstrumentation called twice"); 3009288943Sdim IRBuilder<> IRB(F.getEntryBlock().getFirstNonPHI()); 3010288943Sdim VAArgSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS); 3011288943Sdim Value *CopySize = IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, 0), 3012288943Sdim VAArgSize); 3013288943Sdim 3014288943Sdim if (!VAStartInstrumentationList.empty()) { 3015288943Sdim // If there is a va_start in this function, make a backup copy of 3016288943Sdim // va_arg_tls somewhere in the function entry block. 3017288943Sdim VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize); 3018288943Sdim IRB.CreateMemCpy(VAArgTLSCopy, MS.VAArgTLS, CopySize, 8); 3019288943Sdim } 3020288943Sdim 3021288943Sdim // Instrument va_start. 3022288943Sdim // Copy va_list shadow from the backup copy of the TLS contents. 3023288943Sdim for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) { 3024288943Sdim CallInst *OrigInst = VAStartInstrumentationList[i]; 3025288943Sdim IRBuilder<> IRB(OrigInst->getNextNode()); 3026288943Sdim Value *VAListTag = OrigInst->getArgOperand(0); 3027288943Sdim Value *RegSaveAreaPtrPtr = 3028288943Sdim IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy), 3029288943Sdim Type::getInt64PtrTy(*MS.C)); 3030288943Sdim Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrPtr); 3031288943Sdim Value *RegSaveAreaShadowPtr = 3032288943Sdim MSV.getShadowPtr(RegSaveAreaPtr, IRB.getInt8Ty(), IRB); 3033288943Sdim IRB.CreateMemCpy(RegSaveAreaShadowPtr, VAArgTLSCopy, CopySize, 8); 3034288943Sdim } 3035288943Sdim } 3036288943Sdim}; 3037288943Sdim 3038296417Sdim 3039296417Sdim/// \brief AArch64-specific implementation of VarArgHelper. 3040296417Sdimstruct VarArgAArch64Helper : public VarArgHelper { 3041296417Sdim static const unsigned kAArch64GrArgSize = 56; 3042296417Sdim static const unsigned kAArch64VrArgSize = 128; 3043296417Sdim 3044296417Sdim static const unsigned AArch64GrBegOffset = 0; 3045296417Sdim static const unsigned AArch64GrEndOffset = kAArch64GrArgSize; 3046296417Sdim // Make VR space aligned to 16 bytes. 3047296417Sdim static const unsigned AArch64VrBegOffset = AArch64GrEndOffset + 8; 3048296417Sdim static const unsigned AArch64VrEndOffset = AArch64VrBegOffset 3049296417Sdim + kAArch64VrArgSize; 3050296417Sdim static const unsigned AArch64VAEndOffset = AArch64VrEndOffset; 3051296417Sdim 3052296417Sdim Function &F; 3053296417Sdim MemorySanitizer &MS; 3054296417Sdim MemorySanitizerVisitor &MSV; 3055296417Sdim Value *VAArgTLSCopy; 3056296417Sdim Value *VAArgOverflowSize; 3057296417Sdim 3058296417Sdim SmallVector<CallInst*, 16> VAStartInstrumentationList; 3059296417Sdim 3060296417Sdim VarArgAArch64Helper(Function &F, MemorySanitizer &MS, 3061296417Sdim MemorySanitizerVisitor &MSV) 3062296417Sdim : F(F), MS(MS), MSV(MSV), VAArgTLSCopy(nullptr), 3063296417Sdim VAArgOverflowSize(nullptr) {} 3064296417Sdim 3065296417Sdim enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory }; 3066296417Sdim 3067296417Sdim ArgKind classifyArgument(Value* arg) { 3068296417Sdim Type *T = arg->getType(); 3069296417Sdim if (T->isFPOrFPVectorTy()) 3070296417Sdim return AK_FloatingPoint; 3071296417Sdim if ((T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64) 3072296417Sdim || (T->isPointerTy())) 3073296417Sdim return AK_GeneralPurpose; 3074296417Sdim return AK_Memory; 3075296417Sdim } 3076296417Sdim 3077296417Sdim // The instrumentation stores the argument shadow in a non ABI-specific 3078296417Sdim // format because it does not know which argument is named (since Clang, 3079296417Sdim // like x86_64 case, lowers the va_args in the frontend and this pass only 3080296417Sdim // sees the low level code that deals with va_list internals). 3081296417Sdim // The first seven GR registers are saved in the first 56 bytes of the 3082296417Sdim // va_arg tls arra, followers by the first 8 FP/SIMD registers, and then 3083296417Sdim // the remaining arguments. 3084296417Sdim // Using constant offset within the va_arg TLS array allows fast copy 3085296417Sdim // in the finalize instrumentation. 3086296417Sdim void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override { 3087296417Sdim unsigned GrOffset = AArch64GrBegOffset; 3088296417Sdim unsigned VrOffset = AArch64VrBegOffset; 3089296417Sdim unsigned OverflowOffset = AArch64VAEndOffset; 3090296417Sdim 3091296417Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 3092296417Sdim for (CallSite::arg_iterator ArgIt = CS.arg_begin() + 1, End = CS.arg_end(); 3093296417Sdim ArgIt != End; ++ArgIt) { 3094296417Sdim Value *A = *ArgIt; 3095296417Sdim ArgKind AK = classifyArgument(A); 3096296417Sdim if (AK == AK_GeneralPurpose && GrOffset >= AArch64GrEndOffset) 3097296417Sdim AK = AK_Memory; 3098296417Sdim if (AK == AK_FloatingPoint && VrOffset >= AArch64VrEndOffset) 3099296417Sdim AK = AK_Memory; 3100296417Sdim Value *Base; 3101296417Sdim switch (AK) { 3102296417Sdim case AK_GeneralPurpose: 3103296417Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, GrOffset); 3104296417Sdim GrOffset += 8; 3105296417Sdim break; 3106296417Sdim case AK_FloatingPoint: 3107296417Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, VrOffset); 3108296417Sdim VrOffset += 16; 3109296417Sdim break; 3110296417Sdim case AK_Memory: 3111296417Sdim uint64_t ArgSize = DL.getTypeAllocSize(A->getType()); 3112296417Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset); 3113296417Sdim OverflowOffset += RoundUpToAlignment(ArgSize, 8); 3114296417Sdim break; 3115296417Sdim } 3116296417Sdim IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment); 3117296417Sdim } 3118296417Sdim Constant *OverflowSize = 3119296417Sdim ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AArch64VAEndOffset); 3120296417Sdim IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS); 3121296417Sdim } 3122296417Sdim 3123296417Sdim /// Compute the shadow address for a given va_arg. 3124296417Sdim Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB, 3125296417Sdim int ArgOffset) { 3126296417Sdim Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy); 3127296417Sdim Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset)); 3128296417Sdim return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0), 3129296417Sdim "_msarg"); 3130296417Sdim } 3131296417Sdim 3132296417Sdim void visitVAStartInst(VAStartInst &I) override { 3133296417Sdim IRBuilder<> IRB(&I); 3134296417Sdim VAStartInstrumentationList.push_back(&I); 3135296417Sdim Value *VAListTag = I.getArgOperand(0); 3136296417Sdim Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB); 3137296417Sdim // Unpoison the whole __va_list_tag. 3138296417Sdim // FIXME: magic ABI constants (size of va_list). 3139296417Sdim IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()), 3140296417Sdim /* size */32, /* alignment */8, false); 3141296417Sdim } 3142296417Sdim 3143296417Sdim void visitVACopyInst(VACopyInst &I) override { 3144296417Sdim IRBuilder<> IRB(&I); 3145296417Sdim Value *VAListTag = I.getArgOperand(0); 3146296417Sdim Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB); 3147296417Sdim // Unpoison the whole __va_list_tag. 3148296417Sdim // FIXME: magic ABI constants (size of va_list). 3149296417Sdim IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()), 3150296417Sdim /* size */32, /* alignment */8, false); 3151296417Sdim } 3152296417Sdim 3153296417Sdim // Retrieve a va_list field of 'void*' size. 3154296417Sdim Value* getVAField64(IRBuilder<> &IRB, Value *VAListTag, int offset) { 3155296417Sdim Value *SaveAreaPtrPtr = 3156296417Sdim IRB.CreateIntToPtr( 3157296417Sdim IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy), 3158296417Sdim ConstantInt::get(MS.IntptrTy, offset)), 3159296417Sdim Type::getInt64PtrTy(*MS.C)); 3160296417Sdim return IRB.CreateLoad(SaveAreaPtrPtr); 3161296417Sdim } 3162296417Sdim 3163296417Sdim // Retrieve a va_list field of 'int' size. 3164296417Sdim Value* getVAField32(IRBuilder<> &IRB, Value *VAListTag, int offset) { 3165296417Sdim Value *SaveAreaPtr = 3166296417Sdim IRB.CreateIntToPtr( 3167296417Sdim IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy), 3168296417Sdim ConstantInt::get(MS.IntptrTy, offset)), 3169296417Sdim Type::getInt32PtrTy(*MS.C)); 3170296417Sdim Value *SaveArea32 = IRB.CreateLoad(SaveAreaPtr); 3171296417Sdim return IRB.CreateSExt(SaveArea32, MS.IntptrTy); 3172296417Sdim } 3173296417Sdim 3174296417Sdim void finalizeInstrumentation() override { 3175296417Sdim assert(!VAArgOverflowSize && !VAArgTLSCopy && 3176296417Sdim "finalizeInstrumentation called twice"); 3177296417Sdim if (!VAStartInstrumentationList.empty()) { 3178296417Sdim // If there is a va_start in this function, make a backup copy of 3179296417Sdim // va_arg_tls somewhere in the function entry block. 3180296417Sdim IRBuilder<> IRB(F.getEntryBlock().getFirstNonPHI()); 3181296417Sdim VAArgOverflowSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS); 3182296417Sdim Value *CopySize = 3183296417Sdim IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AArch64VAEndOffset), 3184296417Sdim VAArgOverflowSize); 3185296417Sdim VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize); 3186296417Sdim IRB.CreateMemCpy(VAArgTLSCopy, MS.VAArgTLS, CopySize, 8); 3187296417Sdim } 3188296417Sdim 3189296417Sdim Value *GrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64GrArgSize); 3190296417Sdim Value *VrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64VrArgSize); 3191296417Sdim 3192296417Sdim // Instrument va_start, copy va_list shadow from the backup copy of 3193296417Sdim // the TLS contents. 3194296417Sdim for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) { 3195296417Sdim CallInst *OrigInst = VAStartInstrumentationList[i]; 3196296417Sdim IRBuilder<> IRB(OrigInst->getNextNode()); 3197296417Sdim 3198296417Sdim Value *VAListTag = OrigInst->getArgOperand(0); 3199296417Sdim 3200296417Sdim // The variadic ABI for AArch64 creates two areas to save the incoming 3201296417Sdim // argument registers (one for 64-bit general register xn-x7 and another 3202296417Sdim // for 128-bit FP/SIMD vn-v7). 3203296417Sdim // We need then to propagate the shadow arguments on both regions 3204296417Sdim // 'va::__gr_top + va::__gr_offs' and 'va::__vr_top + va::__vr_offs'. 3205296417Sdim // The remaning arguments are saved on shadow for 'va::stack'. 3206296417Sdim // One caveat is it requires only to propagate the non-named arguments, 3207296417Sdim // however on the call site instrumentation 'all' the arguments are 3208296417Sdim // saved. So to copy the shadow values from the va_arg TLS array 3209296417Sdim // we need to adjust the offset for both GR and VR fields based on 3210296417Sdim // the __{gr,vr}_offs value (since they are stores based on incoming 3211296417Sdim // named arguments). 3212296417Sdim 3213296417Sdim // Read the stack pointer from the va_list. 3214296417Sdim Value *StackSaveAreaPtr = getVAField64(IRB, VAListTag, 0); 3215296417Sdim 3216296417Sdim // Read both the __gr_top and __gr_off and add them up. 3217296417Sdim Value *GrTopSaveAreaPtr = getVAField64(IRB, VAListTag, 8); 3218296417Sdim Value *GrOffSaveArea = getVAField32(IRB, VAListTag, 24); 3219296417Sdim 3220296417Sdim Value *GrRegSaveAreaPtr = IRB.CreateAdd(GrTopSaveAreaPtr, GrOffSaveArea); 3221296417Sdim 3222296417Sdim // Read both the __vr_top and __vr_off and add them up. 3223296417Sdim Value *VrTopSaveAreaPtr = getVAField64(IRB, VAListTag, 16); 3224296417Sdim Value *VrOffSaveArea = getVAField32(IRB, VAListTag, 28); 3225296417Sdim 3226296417Sdim Value *VrRegSaveAreaPtr = IRB.CreateAdd(VrTopSaveAreaPtr, VrOffSaveArea); 3227296417Sdim 3228296417Sdim // It does not know how many named arguments is being used and, on the 3229296417Sdim // callsite all the arguments were saved. Since __gr_off is defined as 3230296417Sdim // '0 - ((8 - named_gr) * 8)', the idea is to just propagate the variadic 3231296417Sdim // argument by ignoring the bytes of shadow from named arguments. 3232296417Sdim Value *GrRegSaveAreaShadowPtrOff = 3233296417Sdim IRB.CreateAdd(GrArgSize, GrOffSaveArea); 3234296417Sdim 3235296417Sdim Value *GrRegSaveAreaShadowPtr = 3236296417Sdim MSV.getShadowPtr(GrRegSaveAreaPtr, IRB.getInt8Ty(), IRB); 3237296417Sdim 3238296417Sdim Value *GrSrcPtr = IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy, 3239296417Sdim GrRegSaveAreaShadowPtrOff); 3240296417Sdim Value *GrCopySize = IRB.CreateSub(GrArgSize, GrRegSaveAreaShadowPtrOff); 3241296417Sdim 3242296417Sdim IRB.CreateMemCpy(GrRegSaveAreaShadowPtr, GrSrcPtr, GrCopySize, 8); 3243296417Sdim 3244296417Sdim // Again, but for FP/SIMD values. 3245296417Sdim Value *VrRegSaveAreaShadowPtrOff = 3246296417Sdim IRB.CreateAdd(VrArgSize, VrOffSaveArea); 3247296417Sdim 3248296417Sdim Value *VrRegSaveAreaShadowPtr = 3249296417Sdim MSV.getShadowPtr(VrRegSaveAreaPtr, IRB.getInt8Ty(), IRB); 3250296417Sdim 3251296417Sdim Value *VrSrcPtr = IRB.CreateInBoundsGEP( 3252296417Sdim IRB.getInt8Ty(), 3253296417Sdim IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy, 3254296417Sdim IRB.getInt32(AArch64VrBegOffset)), 3255296417Sdim VrRegSaveAreaShadowPtrOff); 3256296417Sdim Value *VrCopySize = IRB.CreateSub(VrArgSize, VrRegSaveAreaShadowPtrOff); 3257296417Sdim 3258296417Sdim IRB.CreateMemCpy(VrRegSaveAreaShadowPtr, VrSrcPtr, VrCopySize, 8); 3259296417Sdim 3260296417Sdim // And finally for remaining arguments. 3261296417Sdim Value *StackSaveAreaShadowPtr = 3262296417Sdim MSV.getShadowPtr(StackSaveAreaPtr, IRB.getInt8Ty(), IRB); 3263296417Sdim 3264296417Sdim Value *StackSrcPtr = 3265296417Sdim IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy, 3266296417Sdim IRB.getInt32(AArch64VAEndOffset)); 3267296417Sdim 3268296417Sdim IRB.CreateMemCpy(StackSaveAreaShadowPtr, StackSrcPtr, 3269296417Sdim VAArgOverflowSize, 16); 3270296417Sdim } 3271296417Sdim } 3272296417Sdim}; 3273296417Sdim 3274261991Sdim/// \brief A no-op implementation of VarArgHelper. 3275261991Sdimstruct VarArgNoOpHelper : public VarArgHelper { 3276261991Sdim VarArgNoOpHelper(Function &F, MemorySanitizer &MS, 3277261991Sdim MemorySanitizerVisitor &MSV) {} 3278261991Sdim 3279276479Sdim void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {} 3280261991Sdim 3281276479Sdim void visitVAStartInst(VAStartInst &I) override {} 3282261991Sdim 3283276479Sdim void visitVACopyInst(VACopyInst &I) override {} 3284261991Sdim 3285276479Sdim void finalizeInstrumentation() override {} 3286261991Sdim}; 3287261991Sdim 3288261991SdimVarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan, 3289249259Sdim MemorySanitizerVisitor &Visitor) { 3290261991Sdim // VarArg handling is only implemented on AMD64. False positives are possible 3291261991Sdim // on other platforms. 3292261991Sdim llvm::Triple TargetTriple(Func.getParent()->getTargetTriple()); 3293261991Sdim if (TargetTriple.getArch() == llvm::Triple::x86_64) 3294261991Sdim return new VarArgAMD64Helper(Func, Msan, Visitor); 3295288943Sdim else if (TargetTriple.getArch() == llvm::Triple::mips64 || 3296288943Sdim TargetTriple.getArch() == llvm::Triple::mips64el) 3297288943Sdim return new VarArgMIPS64Helper(Func, Msan, Visitor); 3298296417Sdim else if (TargetTriple.getArch() == llvm::Triple::aarch64) 3299296417Sdim return new VarArgAArch64Helper(Func, Msan, Visitor); 3300261991Sdim else 3301261991Sdim return new VarArgNoOpHelper(Func, Msan, Visitor); 3302249259Sdim} 3303249259Sdim 3304296417Sdim} // anonymous namespace 3305249259Sdim 3306249259Sdimbool MemorySanitizer::runOnFunction(Function &F) { 3307288943Sdim if (&F == MsanCtorFunction) 3308288943Sdim return false; 3309249259Sdim MemorySanitizerVisitor Visitor(F, *this); 3310249259Sdim 3311249259Sdim // Clear out readonly/readnone attributes. 3312249259Sdim AttrBuilder B; 3313249259Sdim B.addAttribute(Attribute::ReadOnly) 3314249259Sdim .addAttribute(Attribute::ReadNone); 3315249259Sdim F.removeAttributes(AttributeSet::FunctionIndex, 3316249259Sdim AttributeSet::get(F.getContext(), 3317249259Sdim AttributeSet::FunctionIndex, B)); 3318249259Sdim 3319249259Sdim return Visitor.runOnFunction(); 3320249259Sdim} 3321