MemorySanitizer.cpp revision 288943
1249259Sdim//===-- MemorySanitizer.cpp - detector of uninitialized reads -------------===// 2249259Sdim// 3249259Sdim// The LLVM Compiler Infrastructure 4249259Sdim// 5249259Sdim// This file is distributed under the University of Illinois Open Source 6249259Sdim// License. See LICENSE.TXT for details. 7249259Sdim// 8249259Sdim//===----------------------------------------------------------------------===// 9249259Sdim/// \file 10249259Sdim/// This file is a part of MemorySanitizer, a detector of uninitialized 11249259Sdim/// reads. 12249259Sdim/// 13249259Sdim/// The algorithm of the tool is similar to Memcheck 14249259Sdim/// (http://goo.gl/QKbem). We associate a few shadow bits with every 15249259Sdim/// byte of the application memory, poison the shadow of the malloc-ed 16249259Sdim/// or alloca-ed memory, load the shadow bits on every memory read, 17249259Sdim/// propagate the shadow bits through some of the arithmetic 18249259Sdim/// instruction (including MOV), store the shadow bits on every memory 19249259Sdim/// write, report a bug on some other instructions (e.g. JMP) if the 20249259Sdim/// associated shadow is poisoned. 21249259Sdim/// 22249259Sdim/// But there are differences too. The first and the major one: 23249259Sdim/// compiler instrumentation instead of binary instrumentation. This 24249259Sdim/// gives us much better register allocation, possible compiler 25249259Sdim/// optimizations and a fast start-up. But this brings the major issue 26249259Sdim/// as well: msan needs to see all program events, including system 27249259Sdim/// calls and reads/writes in system libraries, so we either need to 28249259Sdim/// compile *everything* with msan or use a binary translation 29249259Sdim/// component (e.g. DynamoRIO) to instrument pre-built libraries. 30249259Sdim/// Another difference from Memcheck is that we use 8 shadow bits per 31249259Sdim/// byte of application memory and use a direct shadow mapping. This 32249259Sdim/// greatly simplifies the instrumentation code and avoids races on 33249259Sdim/// shadow updates (Memcheck is single-threaded so races are not a 34249259Sdim/// concern there. Memcheck uses 2 shadow bits per byte with a slow 35249259Sdim/// path storage that uses 8 bits per byte). 36249259Sdim/// 37249259Sdim/// The default value of shadow is 0, which means "clean" (not poisoned). 38249259Sdim/// 39249259Sdim/// Every module initializer should call __msan_init to ensure that the 40249259Sdim/// shadow memory is ready. On error, __msan_warning is called. Since 41249259Sdim/// parameters and return values may be passed via registers, we have a 42249259Sdim/// specialized thread-local shadow for return values 43249259Sdim/// (__msan_retval_tls) and parameters (__msan_param_tls). 44249259Sdim/// 45249259Sdim/// Origin tracking. 46249259Sdim/// 47249259Sdim/// MemorySanitizer can track origins (allocation points) of all uninitialized 48249259Sdim/// values. This behavior is controlled with a flag (msan-track-origins) and is 49249259Sdim/// disabled by default. 50249259Sdim/// 51249259Sdim/// Origins are 4-byte values created and interpreted by the runtime library. 52249259Sdim/// They are stored in a second shadow mapping, one 4-byte value for 4 bytes 53249259Sdim/// of application memory. Propagation of origins is basically a bunch of 54249259Sdim/// "select" instructions that pick the origin of a dirty argument, if an 55249259Sdim/// instruction has one. 56249259Sdim/// 57249259Sdim/// Every 4 aligned, consecutive bytes of application memory have one origin 58249259Sdim/// value associated with them. If these bytes contain uninitialized data 59249259Sdim/// coming from 2 different allocations, the last store wins. Because of this, 60249259Sdim/// MemorySanitizer reports can show unrelated origins, but this is unlikely in 61249259Sdim/// practice. 62249259Sdim/// 63249259Sdim/// Origins are meaningless for fully initialized values, so MemorySanitizer 64249259Sdim/// avoids storing origin to memory when a fully initialized value is stored. 65249259Sdim/// This way it avoids needless overwritting origin of the 4-byte region on 66249259Sdim/// a short (i.e. 1 byte) clean store, and it is also good for performance. 67261991Sdim/// 68261991Sdim/// Atomic handling. 69261991Sdim/// 70261991Sdim/// Ideally, every atomic store of application value should update the 71261991Sdim/// corresponding shadow location in an atomic way. Unfortunately, atomic store 72261991Sdim/// of two disjoint locations can not be done without severe slowdown. 73261991Sdim/// 74261991Sdim/// Therefore, we implement an approximation that may err on the safe side. 75261991Sdim/// In this implementation, every atomically accessed location in the program 76261991Sdim/// may only change from (partially) uninitialized to fully initialized, but 77261991Sdim/// not the other way around. We load the shadow _after_ the application load, 78261991Sdim/// and we store the shadow _before_ the app store. Also, we always store clean 79261991Sdim/// shadow (if the application store is atomic). This way, if the store-load 80261991Sdim/// pair constitutes a happens-before arc, shadow store and load are correctly 81261991Sdim/// ordered such that the load will get either the value that was stored, or 82261991Sdim/// some later value (which is always clean). 83261991Sdim/// 84261991Sdim/// This does not work very well with Compare-And-Swap (CAS) and 85261991Sdim/// Read-Modify-Write (RMW) operations. To follow the above logic, CAS and RMW 86261991Sdim/// must store the new shadow before the app operation, and load the shadow 87261991Sdim/// after the app operation. Computers don't work this way. Current 88261991Sdim/// implementation ignores the load aspect of CAS/RMW, always returning a clean 89261991Sdim/// value. It implements the store part as a simple atomic store by storing a 90261991Sdim/// clean shadow. 91261991Sdim 92249259Sdim//===----------------------------------------------------------------------===// 93249259Sdim 94249259Sdim#include "llvm/Transforms/Instrumentation.h" 95249259Sdim#include "llvm/ADT/DepthFirstIterator.h" 96249259Sdim#include "llvm/ADT/SmallString.h" 97249259Sdim#include "llvm/ADT/SmallVector.h" 98276479Sdim#include "llvm/ADT/StringExtras.h" 99261991Sdim#include "llvm/ADT/Triple.h" 100249259Sdim#include "llvm/IR/DataLayout.h" 101249259Sdim#include "llvm/IR/Function.h" 102249259Sdim#include "llvm/IR/IRBuilder.h" 103249259Sdim#include "llvm/IR/InlineAsm.h" 104276479Sdim#include "llvm/IR/InstVisitor.h" 105249259Sdim#include "llvm/IR/IntrinsicInst.h" 106249259Sdim#include "llvm/IR/LLVMContext.h" 107249259Sdim#include "llvm/IR/MDBuilder.h" 108249259Sdim#include "llvm/IR/Module.h" 109249259Sdim#include "llvm/IR/Type.h" 110276479Sdim#include "llvm/IR/ValueMap.h" 111249259Sdim#include "llvm/Support/CommandLine.h" 112249259Sdim#include "llvm/Support/Compiler.h" 113249259Sdim#include "llvm/Support/Debug.h" 114249259Sdim#include "llvm/Support/raw_ostream.h" 115249259Sdim#include "llvm/Transforms/Utils/BasicBlockUtils.h" 116249259Sdim#include "llvm/Transforms/Utils/Local.h" 117249259Sdim#include "llvm/Transforms/Utils/ModuleUtils.h" 118249259Sdim 119249259Sdimusing namespace llvm; 120249259Sdim 121276479Sdim#define DEBUG_TYPE "msan" 122276479Sdim 123288943Sdimstatic const unsigned kOriginSize = 4; 124249259Sdimstatic const unsigned kMinOriginAlignment = 4; 125249259Sdimstatic const unsigned kShadowTLSAlignment = 8; 126249259Sdim 127280031Sdim// These constants must be kept in sync with the ones in msan.h. 128280031Sdimstatic const unsigned kParamTLSSize = 800; 129280031Sdimstatic const unsigned kRetvalTLSSize = 800; 130280031Sdim 131276479Sdim// Accesses sizes are powers of two: 1, 2, 4, 8. 132276479Sdimstatic const size_t kNumberOfAccessSizes = 4; 133276479Sdim 134249259Sdim/// \brief Track origins of uninitialized values. 135249259Sdim/// 136249259Sdim/// Adds a section to MemorySanitizer report that points to the allocation 137249259Sdim/// (stack or heap) the uninitialized bits came from originally. 138276479Sdimstatic cl::opt<int> ClTrackOrigins("msan-track-origins", 139249259Sdim cl::desc("Track origins (allocation sites) of poisoned memory"), 140276479Sdim cl::Hidden, cl::init(0)); 141249259Sdimstatic cl::opt<bool> ClKeepGoing("msan-keep-going", 142249259Sdim cl::desc("keep going after reporting a UMR"), 143249259Sdim cl::Hidden, cl::init(false)); 144249259Sdimstatic cl::opt<bool> ClPoisonStack("msan-poison-stack", 145249259Sdim cl::desc("poison uninitialized stack variables"), 146249259Sdim cl::Hidden, cl::init(true)); 147249259Sdimstatic cl::opt<bool> ClPoisonStackWithCall("msan-poison-stack-with-call", 148249259Sdim cl::desc("poison uninitialized stack variables with a call"), 149249259Sdim cl::Hidden, cl::init(false)); 150249259Sdimstatic cl::opt<int> ClPoisonStackPattern("msan-poison-stack-pattern", 151249259Sdim cl::desc("poison uninitialized stack variables with the given patter"), 152249259Sdim cl::Hidden, cl::init(0xff)); 153249259Sdimstatic cl::opt<bool> ClPoisonUndef("msan-poison-undef", 154249259Sdim cl::desc("poison undef temps"), 155249259Sdim cl::Hidden, cl::init(true)); 156249259Sdim 157249259Sdimstatic cl::opt<bool> ClHandleICmp("msan-handle-icmp", 158249259Sdim cl::desc("propagate shadow through ICmpEQ and ICmpNE"), 159249259Sdim cl::Hidden, cl::init(true)); 160249259Sdim 161249259Sdimstatic cl::opt<bool> ClHandleICmpExact("msan-handle-icmp-exact", 162249259Sdim cl::desc("exact handling of relational integer ICmp"), 163249259Sdim cl::Hidden, cl::init(false)); 164249259Sdim 165249259Sdim// This flag controls whether we check the shadow of the address 166249259Sdim// operand of load or store. Such bugs are very rare, since load from 167249259Sdim// a garbage address typically results in SEGV, but still happen 168249259Sdim// (e.g. only lower bits of address are garbage, or the access happens 169249259Sdim// early at program startup where malloc-ed memory is more likely to 170249259Sdim// be zeroed. As of 2012-08-28 this flag adds 20% slowdown. 171249259Sdimstatic cl::opt<bool> ClCheckAccessAddress("msan-check-access-address", 172249259Sdim cl::desc("report accesses through a pointer which has poisoned shadow"), 173249259Sdim cl::Hidden, cl::init(true)); 174249259Sdim 175249259Sdimstatic cl::opt<bool> ClDumpStrictInstructions("msan-dump-strict-instructions", 176249259Sdim cl::desc("print out instructions with default strict semantics"), 177249259Sdim cl::Hidden, cl::init(false)); 178249259Sdim 179276479Sdimstatic cl::opt<int> ClInstrumentationWithCallThreshold( 180276479Sdim "msan-instrumentation-with-call-threshold", 181276479Sdim cl::desc( 182276479Sdim "If the function being instrumented requires more than " 183276479Sdim "this number of checks and origin stores, use callbacks instead of " 184276479Sdim "inline checks (-1 means never use callbacks)."), 185276479Sdim cl::Hidden, cl::init(3500)); 186249259Sdim 187280031Sdim// This is an experiment to enable handling of cases where shadow is a non-zero 188280031Sdim// compile-time constant. For some unexplainable reason they were silently 189280031Sdim// ignored in the instrumentation. 190280031Sdimstatic cl::opt<bool> ClCheckConstantShadow("msan-check-constant-shadow", 191280031Sdim cl::desc("Insert checks for constant shadow values"), 192280031Sdim cl::Hidden, cl::init(false)); 193261991Sdim 194288943Sdimstatic const char *const kMsanModuleCtorName = "msan.module_ctor"; 195288943Sdimstatic const char *const kMsanInitName = "__msan_init"; 196288943Sdim 197249259Sdimnamespace { 198249259Sdim 199280031Sdim// Memory map parameters used in application-to-shadow address calculation. 200280031Sdim// Offset = (Addr & ~AndMask) ^ XorMask 201280031Sdim// Shadow = ShadowBase + Offset 202280031Sdim// Origin = OriginBase + Offset 203280031Sdimstruct MemoryMapParams { 204280031Sdim uint64_t AndMask; 205280031Sdim uint64_t XorMask; 206280031Sdim uint64_t ShadowBase; 207280031Sdim uint64_t OriginBase; 208280031Sdim}; 209280031Sdim 210280031Sdimstruct PlatformMemoryMapParams { 211280031Sdim const MemoryMapParams *bits32; 212280031Sdim const MemoryMapParams *bits64; 213280031Sdim}; 214280031Sdim 215280031Sdim// i386 Linux 216288943Sdimstatic const MemoryMapParams Linux_I386_MemoryMapParams = { 217280031Sdim 0x000080000000, // AndMask 218280031Sdim 0, // XorMask (not used) 219280031Sdim 0, // ShadowBase (not used) 220280031Sdim 0x000040000000, // OriginBase 221280031Sdim}; 222280031Sdim 223280031Sdim// x86_64 Linux 224288943Sdimstatic const MemoryMapParams Linux_X86_64_MemoryMapParams = { 225280031Sdim 0x400000000000, // AndMask 226280031Sdim 0, // XorMask (not used) 227280031Sdim 0, // ShadowBase (not used) 228280031Sdim 0x200000000000, // OriginBase 229280031Sdim}; 230280031Sdim 231288943Sdim// mips64 Linux 232288943Sdimstatic const MemoryMapParams Linux_MIPS64_MemoryMapParams = { 233288943Sdim 0x004000000000, // AndMask 234288943Sdim 0, // XorMask (not used) 235288943Sdim 0, // ShadowBase (not used) 236288943Sdim 0x002000000000, // OriginBase 237288943Sdim}; 238288943Sdim 239288943Sdim// ppc64 Linux 240288943Sdimstatic const MemoryMapParams Linux_PowerPC64_MemoryMapParams = { 241288943Sdim 0x200000000000, // AndMask 242288943Sdim 0x100000000000, // XorMask 243288943Sdim 0x080000000000, // ShadowBase 244288943Sdim 0x1C0000000000, // OriginBase 245288943Sdim}; 246288943Sdim 247280031Sdim// i386 FreeBSD 248288943Sdimstatic const MemoryMapParams FreeBSD_I386_MemoryMapParams = { 249280031Sdim 0x000180000000, // AndMask 250280031Sdim 0x000040000000, // XorMask 251280031Sdim 0x000020000000, // ShadowBase 252280031Sdim 0x000700000000, // OriginBase 253280031Sdim}; 254280031Sdim 255280031Sdim// x86_64 FreeBSD 256288943Sdimstatic const MemoryMapParams FreeBSD_X86_64_MemoryMapParams = { 257280031Sdim 0xc00000000000, // AndMask 258280031Sdim 0x200000000000, // XorMask 259280031Sdim 0x100000000000, // ShadowBase 260280031Sdim 0x380000000000, // OriginBase 261280031Sdim}; 262280031Sdim 263288943Sdimstatic const PlatformMemoryMapParams Linux_X86_MemoryMapParams = { 264288943Sdim &Linux_I386_MemoryMapParams, 265288943Sdim &Linux_X86_64_MemoryMapParams, 266280031Sdim}; 267280031Sdim 268288943Sdimstatic const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams = { 269288943Sdim NULL, 270288943Sdim &Linux_MIPS64_MemoryMapParams, 271280031Sdim}; 272280031Sdim 273288943Sdimstatic const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams = { 274288943Sdim NULL, 275288943Sdim &Linux_PowerPC64_MemoryMapParams, 276288943Sdim}; 277288943Sdim 278288943Sdimstatic const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams = { 279288943Sdim &FreeBSD_I386_MemoryMapParams, 280288943Sdim &FreeBSD_X86_64_MemoryMapParams, 281288943Sdim}; 282288943Sdim 283249259Sdim/// \brief An instrumentation pass implementing detection of uninitialized 284249259Sdim/// reads. 285249259Sdim/// 286249259Sdim/// MemorySanitizer: instrument the code in module to find 287249259Sdim/// uninitialized reads. 288249259Sdimclass MemorySanitizer : public FunctionPass { 289249259Sdim public: 290276479Sdim MemorySanitizer(int TrackOrigins = 0) 291261991Sdim : FunctionPass(ID), 292276479Sdim TrackOrigins(std::max(TrackOrigins, (int)ClTrackOrigins)), 293280031Sdim WarningFn(nullptr) {} 294276479Sdim const char *getPassName() const override { return "MemorySanitizer"; } 295276479Sdim bool runOnFunction(Function &F) override; 296276479Sdim bool doInitialization(Module &M) override; 297249259Sdim static char ID; // Pass identification, replacement for typeid. 298249259Sdim 299249259Sdim private: 300249259Sdim void initializeCallbacks(Module &M); 301249259Sdim 302249259Sdim /// \brief Track origins (allocation points) of uninitialized values. 303276479Sdim int TrackOrigins; 304249259Sdim 305249259Sdim LLVMContext *C; 306249259Sdim Type *IntptrTy; 307249259Sdim Type *OriginTy; 308249259Sdim /// \brief Thread-local shadow storage for function parameters. 309249259Sdim GlobalVariable *ParamTLS; 310249259Sdim /// \brief Thread-local origin storage for function parameters. 311249259Sdim GlobalVariable *ParamOriginTLS; 312249259Sdim /// \brief Thread-local shadow storage for function return value. 313249259Sdim GlobalVariable *RetvalTLS; 314249259Sdim /// \brief Thread-local origin storage for function return value. 315249259Sdim GlobalVariable *RetvalOriginTLS; 316249259Sdim /// \brief Thread-local shadow storage for in-register va_arg function 317249259Sdim /// parameters (x86_64-specific). 318249259Sdim GlobalVariable *VAArgTLS; 319249259Sdim /// \brief Thread-local shadow storage for va_arg overflow area 320249259Sdim /// (x86_64-specific). 321249259Sdim GlobalVariable *VAArgOverflowSizeTLS; 322249259Sdim /// \brief Thread-local space used to pass origin value to the UMR reporting 323249259Sdim /// function. 324249259Sdim GlobalVariable *OriginTLS; 325249259Sdim 326249259Sdim /// \brief The run-time callback to print a warning. 327249259Sdim Value *WarningFn; 328276479Sdim // These arrays are indexed by log2(AccessSize). 329276479Sdim Value *MaybeWarningFn[kNumberOfAccessSizes]; 330276479Sdim Value *MaybeStoreOriginFn[kNumberOfAccessSizes]; 331276479Sdim 332249259Sdim /// \brief Run-time helper that generates a new origin value for a stack 333249259Sdim /// allocation. 334261991Sdim Value *MsanSetAllocaOrigin4Fn; 335249259Sdim /// \brief Run-time helper that poisons stack on function entry. 336249259Sdim Value *MsanPoisonStackFn; 337276479Sdim /// \brief Run-time helper that records a store (or any event) of an 338276479Sdim /// uninitialized value and returns an updated origin id encoding this info. 339276479Sdim Value *MsanChainOriginFn; 340249259Sdim /// \brief MSan runtime replacements for memmove, memcpy and memset. 341249259Sdim Value *MemmoveFn, *MemcpyFn, *MemsetFn; 342249259Sdim 343280031Sdim /// \brief Memory map parameters used in application-to-shadow calculation. 344280031Sdim const MemoryMapParams *MapParams; 345280031Sdim 346249259Sdim MDNode *ColdCallWeights; 347249259Sdim /// \brief Branch weights for origin store. 348249259Sdim MDNode *OriginStoreWeights; 349249259Sdim /// \brief An empty volatile inline asm that prevents callback merge. 350249259Sdim InlineAsm *EmptyAsm; 351288943Sdim Function *MsanCtorFunction; 352249259Sdim 353249259Sdim friend struct MemorySanitizerVisitor; 354249259Sdim friend struct VarArgAMD64Helper; 355288943Sdim friend struct VarArgMIPS64Helper; 356249259Sdim}; 357249259Sdim} // namespace 358249259Sdim 359249259Sdimchar MemorySanitizer::ID = 0; 360249259SdimINITIALIZE_PASS(MemorySanitizer, "msan", 361249259Sdim "MemorySanitizer: detects uninitialized reads.", 362249259Sdim false, false) 363249259Sdim 364276479SdimFunctionPass *llvm::createMemorySanitizerPass(int TrackOrigins) { 365276479Sdim return new MemorySanitizer(TrackOrigins); 366249259Sdim} 367249259Sdim 368249259Sdim/// \brief Create a non-const global initialized with the given string. 369249259Sdim/// 370249259Sdim/// Creates a writable global for Str so that we can pass it to the 371249259Sdim/// run-time lib. Runtime uses first 4 bytes of the string to store the 372249259Sdim/// frame ID, so the string needs to be mutable. 373249259Sdimstatic GlobalVariable *createPrivateNonConstGlobalForString(Module &M, 374249259Sdim StringRef Str) { 375249259Sdim Constant *StrConst = ConstantDataArray::getString(M.getContext(), Str); 376249259Sdim return new GlobalVariable(M, StrConst->getType(), /*isConstant=*/false, 377249259Sdim GlobalValue::PrivateLinkage, StrConst, ""); 378249259Sdim} 379249259Sdim 380249259Sdim 381249259Sdim/// \brief Insert extern declaration of runtime-provided functions and globals. 382249259Sdimvoid MemorySanitizer::initializeCallbacks(Module &M) { 383249259Sdim // Only do this once. 384249259Sdim if (WarningFn) 385249259Sdim return; 386249259Sdim 387249259Sdim IRBuilder<> IRB(*C); 388249259Sdim // Create the callback. 389249259Sdim // FIXME: this function should have "Cold" calling conv, 390249259Sdim // which is not yet implemented. 391249259Sdim StringRef WarningFnName = ClKeepGoing ? "__msan_warning" 392249259Sdim : "__msan_warning_noreturn"; 393280031Sdim WarningFn = M.getOrInsertFunction(WarningFnName, IRB.getVoidTy(), nullptr); 394249259Sdim 395276479Sdim for (size_t AccessSizeIndex = 0; AccessSizeIndex < kNumberOfAccessSizes; 396276479Sdim AccessSizeIndex++) { 397276479Sdim unsigned AccessSize = 1 << AccessSizeIndex; 398276479Sdim std::string FunctionName = "__msan_maybe_warning_" + itostr(AccessSize); 399276479Sdim MaybeWarningFn[AccessSizeIndex] = M.getOrInsertFunction( 400276479Sdim FunctionName, IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), 401280031Sdim IRB.getInt32Ty(), nullptr); 402276479Sdim 403276479Sdim FunctionName = "__msan_maybe_store_origin_" + itostr(AccessSize); 404276479Sdim MaybeStoreOriginFn[AccessSizeIndex] = M.getOrInsertFunction( 405276479Sdim FunctionName, IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), 406280031Sdim IRB.getInt8PtrTy(), IRB.getInt32Ty(), nullptr); 407276479Sdim } 408276479Sdim 409261991Sdim MsanSetAllocaOrigin4Fn = M.getOrInsertFunction( 410261991Sdim "__msan_set_alloca_origin4", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy, 411280031Sdim IRB.getInt8PtrTy(), IntptrTy, nullptr); 412280031Sdim MsanPoisonStackFn = 413280031Sdim M.getOrInsertFunction("__msan_poison_stack", IRB.getVoidTy(), 414280031Sdim IRB.getInt8PtrTy(), IntptrTy, nullptr); 415276479Sdim MsanChainOriginFn = M.getOrInsertFunction( 416280031Sdim "__msan_chain_origin", IRB.getInt32Ty(), IRB.getInt32Ty(), nullptr); 417249259Sdim MemmoveFn = M.getOrInsertFunction( 418249259Sdim "__msan_memmove", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), 419280031Sdim IRB.getInt8PtrTy(), IntptrTy, nullptr); 420249259Sdim MemcpyFn = M.getOrInsertFunction( 421249259Sdim "__msan_memcpy", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), 422280031Sdim IntptrTy, nullptr); 423249259Sdim MemsetFn = M.getOrInsertFunction( 424249259Sdim "__msan_memset", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt32Ty(), 425280031Sdim IntptrTy, nullptr); 426249259Sdim 427249259Sdim // Create globals. 428249259Sdim RetvalTLS = new GlobalVariable( 429280031Sdim M, ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8), false, 430276479Sdim GlobalVariable::ExternalLinkage, nullptr, "__msan_retval_tls", nullptr, 431261991Sdim GlobalVariable::InitialExecTLSModel); 432249259Sdim RetvalOriginTLS = new GlobalVariable( 433276479Sdim M, OriginTy, false, GlobalVariable::ExternalLinkage, nullptr, 434276479Sdim "__msan_retval_origin_tls", nullptr, GlobalVariable::InitialExecTLSModel); 435249259Sdim 436249259Sdim ParamTLS = new GlobalVariable( 437280031Sdim M, ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), false, 438276479Sdim GlobalVariable::ExternalLinkage, nullptr, "__msan_param_tls", nullptr, 439261991Sdim GlobalVariable::InitialExecTLSModel); 440249259Sdim ParamOriginTLS = new GlobalVariable( 441280031Sdim M, ArrayType::get(OriginTy, kParamTLSSize / 4), false, 442280031Sdim GlobalVariable::ExternalLinkage, nullptr, "__msan_param_origin_tls", 443280031Sdim nullptr, GlobalVariable::InitialExecTLSModel); 444249259Sdim 445249259Sdim VAArgTLS = new GlobalVariable( 446280031Sdim M, ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), false, 447276479Sdim GlobalVariable::ExternalLinkage, nullptr, "__msan_va_arg_tls", nullptr, 448261991Sdim GlobalVariable::InitialExecTLSModel); 449249259Sdim VAArgOverflowSizeTLS = new GlobalVariable( 450276479Sdim M, IRB.getInt64Ty(), false, GlobalVariable::ExternalLinkage, nullptr, 451276479Sdim "__msan_va_arg_overflow_size_tls", nullptr, 452261991Sdim GlobalVariable::InitialExecTLSModel); 453249259Sdim OriginTLS = new GlobalVariable( 454276479Sdim M, IRB.getInt32Ty(), false, GlobalVariable::ExternalLinkage, nullptr, 455276479Sdim "__msan_origin_tls", nullptr, GlobalVariable::InitialExecTLSModel); 456249259Sdim 457249259Sdim // We insert an empty inline asm after __msan_report* to avoid callback merge. 458249259Sdim EmptyAsm = InlineAsm::get(FunctionType::get(IRB.getVoidTy(), false), 459249259Sdim StringRef(""), StringRef(""), 460249259Sdim /*hasSideEffects=*/true); 461249259Sdim} 462249259Sdim 463249259Sdim/// \brief Module-level initialization. 464249259Sdim/// 465249259Sdim/// inserts a call to __msan_init to the module's constructor list. 466249259Sdimbool MemorySanitizer::doInitialization(Module &M) { 467288943Sdim auto &DL = M.getDataLayout(); 468276479Sdim 469280031Sdim Triple TargetTriple(M.getTargetTriple()); 470288943Sdim switch (TargetTriple.getOS()) { 471288943Sdim case Triple::FreeBSD: 472288943Sdim switch (TargetTriple.getArch()) { 473288943Sdim case Triple::x86_64: 474288943Sdim MapParams = FreeBSD_X86_MemoryMapParams.bits64; 475288943Sdim break; 476288943Sdim case Triple::x86: 477288943Sdim MapParams = FreeBSD_X86_MemoryMapParams.bits32; 478288943Sdim break; 479288943Sdim default: 480288943Sdim report_fatal_error("unsupported architecture"); 481288943Sdim } 482249259Sdim break; 483288943Sdim case Triple::Linux: 484288943Sdim switch (TargetTriple.getArch()) { 485288943Sdim case Triple::x86_64: 486288943Sdim MapParams = Linux_X86_MemoryMapParams.bits64; 487288943Sdim break; 488288943Sdim case Triple::x86: 489288943Sdim MapParams = Linux_X86_MemoryMapParams.bits32; 490288943Sdim break; 491288943Sdim case Triple::mips64: 492288943Sdim case Triple::mips64el: 493288943Sdim MapParams = Linux_MIPS_MemoryMapParams.bits64; 494288943Sdim break; 495288943Sdim case Triple::ppc64: 496288943Sdim case Triple::ppc64le: 497288943Sdim MapParams = Linux_PowerPC_MemoryMapParams.bits64; 498288943Sdim break; 499288943Sdim default: 500288943Sdim report_fatal_error("unsupported architecture"); 501288943Sdim } 502249259Sdim break; 503249259Sdim default: 504288943Sdim report_fatal_error("unsupported operating system"); 505249259Sdim } 506249259Sdim 507288943Sdim C = &(M.getContext()); 508249259Sdim IRBuilder<> IRB(*C); 509276479Sdim IntptrTy = IRB.getIntPtrTy(DL); 510249259Sdim OriginTy = IRB.getInt32Ty(); 511249259Sdim 512249259Sdim ColdCallWeights = MDBuilder(*C).createBranchWeights(1, 1000); 513249259Sdim OriginStoreWeights = MDBuilder(*C).createBranchWeights(1, 1000); 514249259Sdim 515288943Sdim std::tie(MsanCtorFunction, std::ignore) = 516288943Sdim createSanitizerCtorAndInitFunctions(M, kMsanModuleCtorName, kMsanInitName, 517288943Sdim /*InitArgTypes=*/{}, 518288943Sdim /*InitArgs=*/{}); 519249259Sdim 520288943Sdim appendToGlobalCtors(M, MsanCtorFunction, 0); 521288943Sdim 522261991Sdim if (TrackOrigins) 523261991Sdim new GlobalVariable(M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage, 524261991Sdim IRB.getInt32(TrackOrigins), "__msan_track_origins"); 525249259Sdim 526261991Sdim if (ClKeepGoing) 527261991Sdim new GlobalVariable(M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage, 528261991Sdim IRB.getInt32(ClKeepGoing), "__msan_keep_going"); 529249259Sdim 530249259Sdim return true; 531249259Sdim} 532249259Sdim 533249259Sdimnamespace { 534249259Sdim 535249259Sdim/// \brief A helper class that handles instrumentation of VarArg 536249259Sdim/// functions on a particular platform. 537249259Sdim/// 538249259Sdim/// Implementations are expected to insert the instrumentation 539249259Sdim/// necessary to propagate argument shadow through VarArg function 540249259Sdim/// calls. Visit* methods are called during an InstVisitor pass over 541249259Sdim/// the function, and should avoid creating new basic blocks. A new 542249259Sdim/// instance of this class is created for each instrumented function. 543249259Sdimstruct VarArgHelper { 544249259Sdim /// \brief Visit a CallSite. 545249259Sdim virtual void visitCallSite(CallSite &CS, IRBuilder<> &IRB) = 0; 546249259Sdim 547249259Sdim /// \brief Visit a va_start call. 548249259Sdim virtual void visitVAStartInst(VAStartInst &I) = 0; 549249259Sdim 550249259Sdim /// \brief Visit a va_copy call. 551249259Sdim virtual void visitVACopyInst(VACopyInst &I) = 0; 552249259Sdim 553249259Sdim /// \brief Finalize function instrumentation. 554249259Sdim /// 555249259Sdim /// This method is called after visiting all interesting (see above) 556249259Sdim /// instructions in a function. 557249259Sdim virtual void finalizeInstrumentation() = 0; 558249259Sdim 559249259Sdim virtual ~VarArgHelper() {} 560249259Sdim}; 561249259Sdim 562249259Sdimstruct MemorySanitizerVisitor; 563249259Sdim 564249259SdimVarArgHelper* 565249259SdimCreateVarArgHelper(Function &Func, MemorySanitizer &Msan, 566249259Sdim MemorySanitizerVisitor &Visitor); 567249259Sdim 568276479Sdimunsigned TypeSizeToSizeIndex(unsigned TypeSize) { 569276479Sdim if (TypeSize <= 8) return 0; 570276479Sdim return Log2_32_Ceil(TypeSize / 8); 571276479Sdim} 572276479Sdim 573249259Sdim/// This class does all the work for a given function. Store and Load 574249259Sdim/// instructions store and load corresponding shadow and origin 575249259Sdim/// values. Most instructions propagate shadow from arguments to their 576249259Sdim/// return values. Certain instructions (most importantly, BranchInst) 577249259Sdim/// test their argument shadow and print reports (with a runtime call) if it's 578249259Sdim/// non-zero. 579249259Sdimstruct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> { 580249259Sdim Function &F; 581249259Sdim MemorySanitizer &MS; 582249259Sdim SmallVector<PHINode *, 16> ShadowPHINodes, OriginPHINodes; 583249259Sdim ValueMap<Value*, Value*> ShadowMap, OriginMap; 584276479Sdim std::unique_ptr<VarArgHelper> VAHelper; 585261991Sdim 586261991Sdim // The following flags disable parts of MSan instrumentation based on 587261991Sdim // blacklist contents and command-line options. 588249259Sdim bool InsertChecks; 589276479Sdim bool PropagateShadow; 590261991Sdim bool PoisonStack; 591261991Sdim bool PoisonUndef; 592261991Sdim bool CheckReturnValue; 593249259Sdim 594249259Sdim struct ShadowOriginAndInsertPoint { 595261991Sdim Value *Shadow; 596261991Sdim Value *Origin; 597249259Sdim Instruction *OrigIns; 598261991Sdim ShadowOriginAndInsertPoint(Value *S, Value *O, Instruction *I) 599249259Sdim : Shadow(S), Origin(O), OrigIns(I) { } 600249259Sdim }; 601249259Sdim SmallVector<ShadowOriginAndInsertPoint, 16> InstrumentationList; 602249259Sdim SmallVector<Instruction*, 16> StoreList; 603249259Sdim 604249259Sdim MemorySanitizerVisitor(Function &F, MemorySanitizer &MS) 605249259Sdim : F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)) { 606288943Sdim bool SanitizeFunction = F.hasFnAttribute(Attribute::SanitizeMemory); 607261991Sdim InsertChecks = SanitizeFunction; 608276479Sdim PropagateShadow = SanitizeFunction; 609261991Sdim PoisonStack = SanitizeFunction && ClPoisonStack; 610261991Sdim PoisonUndef = SanitizeFunction && ClPoisonUndef; 611261991Sdim // FIXME: Consider using SpecialCaseList to specify a list of functions that 612261991Sdim // must always return fully initialized values. For now, we hardcode "main". 613261991Sdim CheckReturnValue = SanitizeFunction && (F.getName() == "main"); 614249259Sdim 615249259Sdim DEBUG(if (!InsertChecks) 616249259Sdim dbgs() << "MemorySanitizer is not inserting checks into '" 617249259Sdim << F.getName() << "'\n"); 618249259Sdim } 619249259Sdim 620276479Sdim Value *updateOrigin(Value *V, IRBuilder<> &IRB) { 621276479Sdim if (MS.TrackOrigins <= 1) return V; 622276479Sdim return IRB.CreateCall(MS.MsanChainOriginFn, V); 623276479Sdim } 624249259Sdim 625288943Sdim Value *originToIntptr(IRBuilder<> &IRB, Value *Origin) { 626288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 627288943Sdim unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy); 628288943Sdim if (IntptrSize == kOriginSize) return Origin; 629288943Sdim assert(IntptrSize == kOriginSize * 2); 630288943Sdim Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false); 631288943Sdim return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8)); 632288943Sdim } 633288943Sdim 634288943Sdim /// \brief Fill memory range with the given origin value. 635288943Sdim void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr, 636288943Sdim unsigned Size, unsigned Alignment) { 637288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 638288943Sdim unsigned IntptrAlignment = DL.getABITypeAlignment(MS.IntptrTy); 639288943Sdim unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy); 640288943Sdim assert(IntptrAlignment >= kMinOriginAlignment); 641288943Sdim assert(IntptrSize >= kOriginSize); 642288943Sdim 643288943Sdim unsigned Ofs = 0; 644288943Sdim unsigned CurrentAlignment = Alignment; 645288943Sdim if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) { 646288943Sdim Value *IntptrOrigin = originToIntptr(IRB, Origin); 647288943Sdim Value *IntptrOriginPtr = 648288943Sdim IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0)); 649288943Sdim for (unsigned i = 0; i < Size / IntptrSize; ++i) { 650288943Sdim Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i) 651288943Sdim : IntptrOriginPtr; 652288943Sdim IRB.CreateAlignedStore(IntptrOrigin, Ptr, CurrentAlignment); 653288943Sdim Ofs += IntptrSize / kOriginSize; 654288943Sdim CurrentAlignment = IntptrAlignment; 655288943Sdim } 656288943Sdim } 657288943Sdim 658288943Sdim for (unsigned i = Ofs; i < (Size + kOriginSize - 1) / kOriginSize; ++i) { 659288943Sdim Value *GEP = 660288943Sdim i ? IRB.CreateConstGEP1_32(nullptr, OriginPtr, i) : OriginPtr; 661288943Sdim IRB.CreateAlignedStore(Origin, GEP, CurrentAlignment); 662288943Sdim CurrentAlignment = kMinOriginAlignment; 663288943Sdim } 664288943Sdim } 665288943Sdim 666276479Sdim void storeOrigin(IRBuilder<> &IRB, Value *Addr, Value *Shadow, Value *Origin, 667276479Sdim unsigned Alignment, bool AsCall) { 668288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 669280031Sdim unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment); 670288943Sdim unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType()); 671276479Sdim if (isa<StructType>(Shadow->getType())) { 672288943Sdim paintOrigin(IRB, updateOrigin(Origin, IRB), 673288943Sdim getOriginPtr(Addr, IRB, Alignment), StoreSize, 674288943Sdim OriginAlignment); 675276479Sdim } else { 676276479Sdim Value *ConvertedShadow = convertToShadowTyNoVec(Shadow, IRB); 677288943Sdim Constant *ConstantShadow = dyn_cast_or_null<Constant>(ConvertedShadow); 678288943Sdim if (ConstantShadow) { 679288943Sdim if (ClCheckConstantShadow && !ConstantShadow->isZeroValue()) 680288943Sdim paintOrigin(IRB, updateOrigin(Origin, IRB), 681288943Sdim getOriginPtr(Addr, IRB, Alignment), StoreSize, 682288943Sdim OriginAlignment); 683288943Sdim return; 684288943Sdim } 685288943Sdim 686276479Sdim unsigned TypeSizeInBits = 687288943Sdim DL.getTypeSizeInBits(ConvertedShadow->getType()); 688276479Sdim unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits); 689276479Sdim if (AsCall && SizeIndex < kNumberOfAccessSizes) { 690276479Sdim Value *Fn = MS.MaybeStoreOriginFn[SizeIndex]; 691276479Sdim Value *ConvertedShadow2 = IRB.CreateZExt( 692276479Sdim ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex))); 693288943Sdim IRB.CreateCall(Fn, {ConvertedShadow2, 694288943Sdim IRB.CreatePointerCast(Addr, IRB.getInt8PtrTy()), 695288943Sdim Origin}); 696276479Sdim } else { 697276479Sdim Value *Cmp = IRB.CreateICmpNE( 698276479Sdim ConvertedShadow, getCleanShadow(ConvertedShadow), "_mscmp"); 699276479Sdim Instruction *CheckTerm = SplitBlockAndInsertIfThen( 700276479Sdim Cmp, IRB.GetInsertPoint(), false, MS.OriginStoreWeights); 701276479Sdim IRBuilder<> IRBNew(CheckTerm); 702288943Sdim paintOrigin(IRBNew, updateOrigin(Origin, IRBNew), 703288943Sdim getOriginPtr(Addr, IRBNew, Alignment), StoreSize, 704288943Sdim OriginAlignment); 705276479Sdim } 706276479Sdim } 707276479Sdim } 708276479Sdim 709276479Sdim void materializeStores(bool InstrumentWithCalls) { 710276479Sdim for (auto Inst : StoreList) { 711276479Sdim StoreInst &SI = *dyn_cast<StoreInst>(Inst); 712276479Sdim 713276479Sdim IRBuilder<> IRB(&SI); 714276479Sdim Value *Val = SI.getValueOperand(); 715276479Sdim Value *Addr = SI.getPointerOperand(); 716276479Sdim Value *Shadow = SI.isAtomic() ? getCleanShadow(Val) : getShadow(Val); 717249259Sdim Value *ShadowPtr = getShadowPtr(Addr, Shadow->getType(), IRB); 718249259Sdim 719249259Sdim StoreInst *NewSI = 720276479Sdim IRB.CreateAlignedStore(Shadow, ShadowPtr, SI.getAlignment()); 721249259Sdim DEBUG(dbgs() << " STORE: " << *NewSI << "\n"); 722249259Sdim (void)NewSI; 723249259Sdim 724276479Sdim if (ClCheckAccessAddress) insertShadowCheck(Addr, &SI); 725249259Sdim 726276479Sdim if (SI.isAtomic()) SI.setOrdering(addReleaseOrdering(SI.getOrdering())); 727261991Sdim 728280031Sdim if (MS.TrackOrigins && !SI.isAtomic()) 729280031Sdim storeOrigin(IRB, Addr, Shadow, getOrigin(Val), SI.getAlignment(), 730276479Sdim InstrumentWithCalls); 731249259Sdim } 732249259Sdim } 733249259Sdim 734276479Sdim void materializeOneCheck(Instruction *OrigIns, Value *Shadow, Value *Origin, 735276479Sdim bool AsCall) { 736276479Sdim IRBuilder<> IRB(OrigIns); 737276479Sdim DEBUG(dbgs() << " SHAD0 : " << *Shadow << "\n"); 738276479Sdim Value *ConvertedShadow = convertToShadowTyNoVec(Shadow, IRB); 739276479Sdim DEBUG(dbgs() << " SHAD1 : " << *ConvertedShadow << "\n"); 740288943Sdim 741288943Sdim Constant *ConstantShadow = dyn_cast_or_null<Constant>(ConvertedShadow); 742288943Sdim if (ConstantShadow) { 743288943Sdim if (ClCheckConstantShadow && !ConstantShadow->isZeroValue()) { 744288943Sdim if (MS.TrackOrigins) { 745288943Sdim IRB.CreateStore(Origin ? (Value *)Origin : (Value *)IRB.getInt32(0), 746288943Sdim MS.OriginTLS); 747288943Sdim } 748288943Sdim IRB.CreateCall(MS.WarningFn, {}); 749288943Sdim IRB.CreateCall(MS.EmptyAsm, {}); 750288943Sdim // FIXME: Insert UnreachableInst if !ClKeepGoing? 751288943Sdim // This may invalidate some of the following checks and needs to be done 752288943Sdim // at the very end. 753288943Sdim } 754288943Sdim return; 755288943Sdim } 756288943Sdim 757288943Sdim const DataLayout &DL = OrigIns->getModule()->getDataLayout(); 758288943Sdim 759288943Sdim unsigned TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType()); 760276479Sdim unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits); 761276479Sdim if (AsCall && SizeIndex < kNumberOfAccessSizes) { 762276479Sdim Value *Fn = MS.MaybeWarningFn[SizeIndex]; 763276479Sdim Value *ConvertedShadow2 = 764276479Sdim IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex))); 765288943Sdim IRB.CreateCall(Fn, {ConvertedShadow2, MS.TrackOrigins && Origin 766276479Sdim ? Origin 767288943Sdim : (Value *)IRB.getInt32(0)}); 768276479Sdim } else { 769249259Sdim Value *Cmp = IRB.CreateICmpNE(ConvertedShadow, 770249259Sdim getCleanShadow(ConvertedShadow), "_mscmp"); 771276479Sdim Instruction *CheckTerm = SplitBlockAndInsertIfThen( 772276479Sdim Cmp, OrigIns, 773276479Sdim /* Unreachable */ !ClKeepGoing, MS.ColdCallWeights); 774249259Sdim 775249259Sdim IRB.SetInsertPoint(CheckTerm); 776249259Sdim if (MS.TrackOrigins) { 777276479Sdim IRB.CreateStore(Origin ? (Value *)Origin : (Value *)IRB.getInt32(0), 778249259Sdim MS.OriginTLS); 779249259Sdim } 780288943Sdim IRB.CreateCall(MS.WarningFn, {}); 781288943Sdim IRB.CreateCall(MS.EmptyAsm, {}); 782249259Sdim DEBUG(dbgs() << " CHECK: " << *Cmp << "\n"); 783249259Sdim } 784276479Sdim } 785276479Sdim 786276479Sdim void materializeChecks(bool InstrumentWithCalls) { 787276479Sdim for (const auto &ShadowData : InstrumentationList) { 788276479Sdim Instruction *OrigIns = ShadowData.OrigIns; 789276479Sdim Value *Shadow = ShadowData.Shadow; 790276479Sdim Value *Origin = ShadowData.Origin; 791276479Sdim materializeOneCheck(OrigIns, Shadow, Origin, InstrumentWithCalls); 792276479Sdim } 793249259Sdim DEBUG(dbgs() << "DONE:\n" << F); 794249259Sdim } 795249259Sdim 796249259Sdim /// \brief Add MemorySanitizer instrumentation to a function. 797249259Sdim bool runOnFunction() { 798249259Sdim MS.initializeCallbacks(*F.getParent()); 799249259Sdim 800249259Sdim // In the presence of unreachable blocks, we may see Phi nodes with 801249259Sdim // incoming nodes from such blocks. Since InstVisitor skips unreachable 802249259Sdim // blocks, such nodes will not have any shadow value associated with them. 803249259Sdim // It's easier to remove unreachable blocks than deal with missing shadow. 804249259Sdim removeUnreachableBlocks(F); 805249259Sdim 806249259Sdim // Iterate all BBs in depth-first order and create shadow instructions 807249259Sdim // for all instructions (where applicable). 808249259Sdim // For PHI nodes we create dummy shadow PHIs which will be finalized later. 809276479Sdim for (BasicBlock *BB : depth_first(&F.getEntryBlock())) 810249259Sdim visit(*BB); 811249259Sdim 812276479Sdim 813249259Sdim // Finalize PHI nodes. 814276479Sdim for (PHINode *PN : ShadowPHINodes) { 815249259Sdim PHINode *PNS = cast<PHINode>(getShadow(PN)); 816276479Sdim PHINode *PNO = MS.TrackOrigins ? cast<PHINode>(getOrigin(PN)) : nullptr; 817249259Sdim size_t NumValues = PN->getNumIncomingValues(); 818249259Sdim for (size_t v = 0; v < NumValues; v++) { 819249259Sdim PNS->addIncoming(getShadow(PN, v), PN->getIncomingBlock(v)); 820276479Sdim if (PNO) PNO->addIncoming(getOrigin(PN, v), PN->getIncomingBlock(v)); 821249259Sdim } 822249259Sdim } 823249259Sdim 824249259Sdim VAHelper->finalizeInstrumentation(); 825249259Sdim 826276479Sdim bool InstrumentWithCalls = ClInstrumentationWithCallThreshold >= 0 && 827276479Sdim InstrumentationList.size() + StoreList.size() > 828276479Sdim (unsigned)ClInstrumentationWithCallThreshold; 829276479Sdim 830249259Sdim // Delayed instrumentation of StoreInst. 831249259Sdim // This may add new checks to be inserted later. 832276479Sdim materializeStores(InstrumentWithCalls); 833249259Sdim 834249259Sdim // Insert shadow value checks. 835276479Sdim materializeChecks(InstrumentWithCalls); 836249259Sdim 837249259Sdim return true; 838249259Sdim } 839249259Sdim 840249259Sdim /// \brief Compute the shadow type that corresponds to a given Value. 841249259Sdim Type *getShadowTy(Value *V) { 842249259Sdim return getShadowTy(V->getType()); 843249259Sdim } 844249259Sdim 845249259Sdim /// \brief Compute the shadow type that corresponds to a given Type. 846249259Sdim Type *getShadowTy(Type *OrigTy) { 847249259Sdim if (!OrigTy->isSized()) { 848276479Sdim return nullptr; 849249259Sdim } 850249259Sdim // For integer type, shadow is the same as the original type. 851249259Sdim // This may return weird-sized types like i1. 852249259Sdim if (IntegerType *IT = dyn_cast<IntegerType>(OrigTy)) 853249259Sdim return IT; 854288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 855249259Sdim if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) { 856288943Sdim uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType()); 857249259Sdim return VectorType::get(IntegerType::get(*MS.C, EltSize), 858249259Sdim VT->getNumElements()); 859249259Sdim } 860280031Sdim if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) { 861280031Sdim return ArrayType::get(getShadowTy(AT->getElementType()), 862280031Sdim AT->getNumElements()); 863280031Sdim } 864249259Sdim if (StructType *ST = dyn_cast<StructType>(OrigTy)) { 865249259Sdim SmallVector<Type*, 4> Elements; 866249259Sdim for (unsigned i = 0, n = ST->getNumElements(); i < n; i++) 867249259Sdim Elements.push_back(getShadowTy(ST->getElementType(i))); 868249259Sdim StructType *Res = StructType::get(*MS.C, Elements, ST->isPacked()); 869249259Sdim DEBUG(dbgs() << "getShadowTy: " << *ST << " ===> " << *Res << "\n"); 870249259Sdim return Res; 871249259Sdim } 872288943Sdim uint32_t TypeSize = DL.getTypeSizeInBits(OrigTy); 873249259Sdim return IntegerType::get(*MS.C, TypeSize); 874249259Sdim } 875249259Sdim 876249259Sdim /// \brief Flatten a vector type. 877249259Sdim Type *getShadowTyNoVec(Type *ty) { 878249259Sdim if (VectorType *vt = dyn_cast<VectorType>(ty)) 879249259Sdim return IntegerType::get(*MS.C, vt->getBitWidth()); 880249259Sdim return ty; 881249259Sdim } 882249259Sdim 883249259Sdim /// \brief Convert a shadow value to it's flattened variant. 884249259Sdim Value *convertToShadowTyNoVec(Value *V, IRBuilder<> &IRB) { 885249259Sdim Type *Ty = V->getType(); 886249259Sdim Type *NoVecTy = getShadowTyNoVec(Ty); 887249259Sdim if (Ty == NoVecTy) return V; 888249259Sdim return IRB.CreateBitCast(V, NoVecTy); 889249259Sdim } 890249259Sdim 891280031Sdim /// \brief Compute the integer shadow offset that corresponds to a given 892280031Sdim /// application address. 893280031Sdim /// 894280031Sdim /// Offset = (Addr & ~AndMask) ^ XorMask 895280031Sdim Value *getShadowPtrOffset(Value *Addr, IRBuilder<> &IRB) { 896280031Sdim uint64_t AndMask = MS.MapParams->AndMask; 897280031Sdim assert(AndMask != 0 && "AndMask shall be specified"); 898280031Sdim Value *OffsetLong = 899280031Sdim IRB.CreateAnd(IRB.CreatePointerCast(Addr, MS.IntptrTy), 900280031Sdim ConstantInt::get(MS.IntptrTy, ~AndMask)); 901280031Sdim 902280031Sdim uint64_t XorMask = MS.MapParams->XorMask; 903280031Sdim if (XorMask != 0) 904280031Sdim OffsetLong = IRB.CreateXor(OffsetLong, 905280031Sdim ConstantInt::get(MS.IntptrTy, XorMask)); 906280031Sdim return OffsetLong; 907280031Sdim } 908280031Sdim 909249259Sdim /// \brief Compute the shadow address that corresponds to a given application 910249259Sdim /// address. 911249259Sdim /// 912280031Sdim /// Shadow = ShadowBase + Offset 913249259Sdim Value *getShadowPtr(Value *Addr, Type *ShadowTy, 914249259Sdim IRBuilder<> &IRB) { 915280031Sdim Value *ShadowLong = getShadowPtrOffset(Addr, IRB); 916280031Sdim uint64_t ShadowBase = MS.MapParams->ShadowBase; 917280031Sdim if (ShadowBase != 0) 918280031Sdim ShadowLong = 919280031Sdim IRB.CreateAdd(ShadowLong, 920280031Sdim ConstantInt::get(MS.IntptrTy, ShadowBase)); 921249259Sdim return IRB.CreateIntToPtr(ShadowLong, PointerType::get(ShadowTy, 0)); 922249259Sdim } 923249259Sdim 924249259Sdim /// \brief Compute the origin address that corresponds to a given application 925249259Sdim /// address. 926249259Sdim /// 927280031Sdim /// OriginAddr = (OriginBase + Offset) & ~3ULL 928280031Sdim Value *getOriginPtr(Value *Addr, IRBuilder<> &IRB, unsigned Alignment) { 929280031Sdim Value *OriginLong = getShadowPtrOffset(Addr, IRB); 930280031Sdim uint64_t OriginBase = MS.MapParams->OriginBase; 931280031Sdim if (OriginBase != 0) 932280031Sdim OriginLong = 933280031Sdim IRB.CreateAdd(OriginLong, 934280031Sdim ConstantInt::get(MS.IntptrTy, OriginBase)); 935280031Sdim if (Alignment < kMinOriginAlignment) { 936280031Sdim uint64_t Mask = kMinOriginAlignment - 1; 937280031Sdim OriginLong = IRB.CreateAnd(OriginLong, 938280031Sdim ConstantInt::get(MS.IntptrTy, ~Mask)); 939280031Sdim } 940280031Sdim return IRB.CreateIntToPtr(OriginLong, 941280031Sdim PointerType::get(IRB.getInt32Ty(), 0)); 942249259Sdim } 943249259Sdim 944249259Sdim /// \brief Compute the shadow address for a given function argument. 945249259Sdim /// 946249259Sdim /// Shadow = ParamTLS+ArgOffset. 947249259Sdim Value *getShadowPtrForArgument(Value *A, IRBuilder<> &IRB, 948249259Sdim int ArgOffset) { 949249259Sdim Value *Base = IRB.CreatePointerCast(MS.ParamTLS, MS.IntptrTy); 950249259Sdim Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset)); 951249259Sdim return IRB.CreateIntToPtr(Base, PointerType::get(getShadowTy(A), 0), 952249259Sdim "_msarg"); 953249259Sdim } 954249259Sdim 955249259Sdim /// \brief Compute the origin address for a given function argument. 956249259Sdim Value *getOriginPtrForArgument(Value *A, IRBuilder<> &IRB, 957249259Sdim int ArgOffset) { 958276479Sdim if (!MS.TrackOrigins) return nullptr; 959249259Sdim Value *Base = IRB.CreatePointerCast(MS.ParamOriginTLS, MS.IntptrTy); 960249259Sdim Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset)); 961249259Sdim return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0), 962249259Sdim "_msarg_o"); 963249259Sdim } 964249259Sdim 965249259Sdim /// \brief Compute the shadow address for a retval. 966249259Sdim Value *getShadowPtrForRetval(Value *A, IRBuilder<> &IRB) { 967249259Sdim Value *Base = IRB.CreatePointerCast(MS.RetvalTLS, MS.IntptrTy); 968249259Sdim return IRB.CreateIntToPtr(Base, PointerType::get(getShadowTy(A), 0), 969249259Sdim "_msret"); 970249259Sdim } 971249259Sdim 972249259Sdim /// \brief Compute the origin address for a retval. 973249259Sdim Value *getOriginPtrForRetval(IRBuilder<> &IRB) { 974249259Sdim // We keep a single origin for the entire retval. Might be too optimistic. 975249259Sdim return MS.RetvalOriginTLS; 976249259Sdim } 977249259Sdim 978249259Sdim /// \brief Set SV to be the shadow value for V. 979249259Sdim void setShadow(Value *V, Value *SV) { 980249259Sdim assert(!ShadowMap.count(V) && "Values may only have one shadow"); 981276479Sdim ShadowMap[V] = PropagateShadow ? SV : getCleanShadow(V); 982249259Sdim } 983249259Sdim 984249259Sdim /// \brief Set Origin to be the origin value for V. 985249259Sdim void setOrigin(Value *V, Value *Origin) { 986249259Sdim if (!MS.TrackOrigins) return; 987249259Sdim assert(!OriginMap.count(V) && "Values may only have one origin"); 988249259Sdim DEBUG(dbgs() << "ORIGIN: " << *V << " ==> " << *Origin << "\n"); 989249259Sdim OriginMap[V] = Origin; 990249259Sdim } 991249259Sdim 992249259Sdim /// \brief Create a clean shadow value for a given value. 993249259Sdim /// 994249259Sdim /// Clean shadow (all zeroes) means all bits of the value are defined 995249259Sdim /// (initialized). 996249259Sdim Constant *getCleanShadow(Value *V) { 997249259Sdim Type *ShadowTy = getShadowTy(V); 998249259Sdim if (!ShadowTy) 999276479Sdim return nullptr; 1000249259Sdim return Constant::getNullValue(ShadowTy); 1001249259Sdim } 1002249259Sdim 1003249259Sdim /// \brief Create a dirty shadow of a given shadow type. 1004249259Sdim Constant *getPoisonedShadow(Type *ShadowTy) { 1005249259Sdim assert(ShadowTy); 1006249259Sdim if (isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy)) 1007249259Sdim return Constant::getAllOnesValue(ShadowTy); 1008280031Sdim if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy)) { 1009280031Sdim SmallVector<Constant *, 4> Vals(AT->getNumElements(), 1010280031Sdim getPoisonedShadow(AT->getElementType())); 1011280031Sdim return ConstantArray::get(AT, Vals); 1012280031Sdim } 1013280031Sdim if (StructType *ST = dyn_cast<StructType>(ShadowTy)) { 1014280031Sdim SmallVector<Constant *, 4> Vals; 1015280031Sdim for (unsigned i = 0, n = ST->getNumElements(); i < n; i++) 1016280031Sdim Vals.push_back(getPoisonedShadow(ST->getElementType(i))); 1017280031Sdim return ConstantStruct::get(ST, Vals); 1018280031Sdim } 1019280031Sdim llvm_unreachable("Unexpected shadow type"); 1020249259Sdim } 1021249259Sdim 1022249259Sdim /// \brief Create a dirty shadow for a given value. 1023249259Sdim Constant *getPoisonedShadow(Value *V) { 1024249259Sdim Type *ShadowTy = getShadowTy(V); 1025249259Sdim if (!ShadowTy) 1026276479Sdim return nullptr; 1027249259Sdim return getPoisonedShadow(ShadowTy); 1028249259Sdim } 1029249259Sdim 1030249259Sdim /// \brief Create a clean (zero) origin. 1031249259Sdim Value *getCleanOrigin() { 1032249259Sdim return Constant::getNullValue(MS.OriginTy); 1033249259Sdim } 1034249259Sdim 1035249259Sdim /// \brief Get the shadow value for a given Value. 1036249259Sdim /// 1037249259Sdim /// This function either returns the value set earlier with setShadow, 1038249259Sdim /// or extracts if from ParamTLS (for function arguments). 1039249259Sdim Value *getShadow(Value *V) { 1040276479Sdim if (!PropagateShadow) return getCleanShadow(V); 1041249259Sdim if (Instruction *I = dyn_cast<Instruction>(V)) { 1042249259Sdim // For instructions the shadow is already stored in the map. 1043249259Sdim Value *Shadow = ShadowMap[V]; 1044249259Sdim if (!Shadow) { 1045249259Sdim DEBUG(dbgs() << "No shadow: " << *V << "\n" << *(I->getParent())); 1046249259Sdim (void)I; 1047249259Sdim assert(Shadow && "No shadow for a value"); 1048249259Sdim } 1049249259Sdim return Shadow; 1050249259Sdim } 1051249259Sdim if (UndefValue *U = dyn_cast<UndefValue>(V)) { 1052261991Sdim Value *AllOnes = PoisonUndef ? getPoisonedShadow(V) : getCleanShadow(V); 1053249259Sdim DEBUG(dbgs() << "Undef: " << *U << " ==> " << *AllOnes << "\n"); 1054249259Sdim (void)U; 1055249259Sdim return AllOnes; 1056249259Sdim } 1057249259Sdim if (Argument *A = dyn_cast<Argument>(V)) { 1058249259Sdim // For arguments we compute the shadow on demand and store it in the map. 1059249259Sdim Value **ShadowPtr = &ShadowMap[V]; 1060249259Sdim if (*ShadowPtr) 1061249259Sdim return *ShadowPtr; 1062249259Sdim Function *F = A->getParent(); 1063249259Sdim IRBuilder<> EntryIRB(F->getEntryBlock().getFirstNonPHI()); 1064249259Sdim unsigned ArgOffset = 0; 1065288943Sdim const DataLayout &DL = F->getParent()->getDataLayout(); 1066276479Sdim for (auto &FArg : F->args()) { 1067276479Sdim if (!FArg.getType()->isSized()) { 1068249259Sdim DEBUG(dbgs() << "Arg is not sized\n"); 1069249259Sdim continue; 1070249259Sdim } 1071288943Sdim unsigned Size = 1072288943Sdim FArg.hasByValAttr() 1073288943Sdim ? DL.getTypeAllocSize(FArg.getType()->getPointerElementType()) 1074288943Sdim : DL.getTypeAllocSize(FArg.getType()); 1075276479Sdim if (A == &FArg) { 1076280031Sdim bool Overflow = ArgOffset + Size > kParamTLSSize; 1077276479Sdim Value *Base = getShadowPtrForArgument(&FArg, EntryIRB, ArgOffset); 1078276479Sdim if (FArg.hasByValAttr()) { 1079249259Sdim // ByVal pointer itself has clean shadow. We copy the actual 1080249259Sdim // argument shadow to the underlying memory. 1081261991Sdim // Figure out maximal valid memcpy alignment. 1082276479Sdim unsigned ArgAlign = FArg.getParamAlignment(); 1083261991Sdim if (ArgAlign == 0) { 1084261991Sdim Type *EltType = A->getType()->getPointerElementType(); 1085288943Sdim ArgAlign = DL.getABITypeAlignment(EltType); 1086261991Sdim } 1087280031Sdim if (Overflow) { 1088280031Sdim // ParamTLS overflow. 1089280031Sdim EntryIRB.CreateMemSet( 1090280031Sdim getShadowPtr(V, EntryIRB.getInt8Ty(), EntryIRB), 1091280031Sdim Constant::getNullValue(EntryIRB.getInt8Ty()), Size, ArgAlign); 1092280031Sdim } else { 1093280031Sdim unsigned CopyAlign = std::min(ArgAlign, kShadowTLSAlignment); 1094280031Sdim Value *Cpy = EntryIRB.CreateMemCpy( 1095280031Sdim getShadowPtr(V, EntryIRB.getInt8Ty(), EntryIRB), Base, Size, 1096280031Sdim CopyAlign); 1097280031Sdim DEBUG(dbgs() << " ByValCpy: " << *Cpy << "\n"); 1098280031Sdim (void)Cpy; 1099280031Sdim } 1100249259Sdim *ShadowPtr = getCleanShadow(V); 1101249259Sdim } else { 1102280031Sdim if (Overflow) { 1103280031Sdim // ParamTLS overflow. 1104280031Sdim *ShadowPtr = getCleanShadow(V); 1105280031Sdim } else { 1106280031Sdim *ShadowPtr = 1107280031Sdim EntryIRB.CreateAlignedLoad(Base, kShadowTLSAlignment); 1108280031Sdim } 1109249259Sdim } 1110276479Sdim DEBUG(dbgs() << " ARG: " << FArg << " ==> " << 1111249259Sdim **ShadowPtr << "\n"); 1112280031Sdim if (MS.TrackOrigins && !Overflow) { 1113276479Sdim Value *OriginPtr = 1114276479Sdim getOriginPtrForArgument(&FArg, EntryIRB, ArgOffset); 1115249259Sdim setOrigin(A, EntryIRB.CreateLoad(OriginPtr)); 1116280031Sdim } else { 1117280031Sdim setOrigin(A, getCleanOrigin()); 1118249259Sdim } 1119249259Sdim } 1120280031Sdim ArgOffset += RoundUpToAlignment(Size, kShadowTLSAlignment); 1121249259Sdim } 1122249259Sdim assert(*ShadowPtr && "Could not find shadow for an argument"); 1123249259Sdim return *ShadowPtr; 1124249259Sdim } 1125249259Sdim // For everything else the shadow is zero. 1126249259Sdim return getCleanShadow(V); 1127249259Sdim } 1128249259Sdim 1129249259Sdim /// \brief Get the shadow for i-th argument of the instruction I. 1130249259Sdim Value *getShadow(Instruction *I, int i) { 1131249259Sdim return getShadow(I->getOperand(i)); 1132249259Sdim } 1133249259Sdim 1134249259Sdim /// \brief Get the origin for a value. 1135249259Sdim Value *getOrigin(Value *V) { 1136276479Sdim if (!MS.TrackOrigins) return nullptr; 1137280031Sdim if (!PropagateShadow) return getCleanOrigin(); 1138280031Sdim if (isa<Constant>(V)) return getCleanOrigin(); 1139280031Sdim assert((isa<Instruction>(V) || isa<Argument>(V)) && 1140280031Sdim "Unexpected value type in getOrigin()"); 1141280031Sdim Value *Origin = OriginMap[V]; 1142280031Sdim assert(Origin && "Missing origin"); 1143280031Sdim return Origin; 1144249259Sdim } 1145249259Sdim 1146249259Sdim /// \brief Get the origin for i-th argument of the instruction I. 1147249259Sdim Value *getOrigin(Instruction *I, int i) { 1148249259Sdim return getOrigin(I->getOperand(i)); 1149249259Sdim } 1150249259Sdim 1151249259Sdim /// \brief Remember the place where a shadow check should be inserted. 1152249259Sdim /// 1153249259Sdim /// This location will be later instrumented with a check that will print a 1154261991Sdim /// UMR warning in runtime if the shadow value is not 0. 1155261991Sdim void insertShadowCheck(Value *Shadow, Value *Origin, Instruction *OrigIns) { 1156261991Sdim assert(Shadow); 1157249259Sdim if (!InsertChecks) return; 1158249259Sdim#ifndef NDEBUG 1159249259Sdim Type *ShadowTy = Shadow->getType(); 1160249259Sdim assert((isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy)) && 1161249259Sdim "Can only insert checks for integer and vector shadow types"); 1162249259Sdim#endif 1163249259Sdim InstrumentationList.push_back( 1164261991Sdim ShadowOriginAndInsertPoint(Shadow, Origin, OrigIns)); 1165249259Sdim } 1166249259Sdim 1167261991Sdim /// \brief Remember the place where a shadow check should be inserted. 1168261991Sdim /// 1169261991Sdim /// This location will be later instrumented with a check that will print a 1170261991Sdim /// UMR warning in runtime if the value is not fully defined. 1171261991Sdim void insertShadowCheck(Value *Val, Instruction *OrigIns) { 1172261991Sdim assert(Val); 1173280031Sdim Value *Shadow, *Origin; 1174280031Sdim if (ClCheckConstantShadow) { 1175280031Sdim Shadow = getShadow(Val); 1176280031Sdim if (!Shadow) return; 1177280031Sdim Origin = getOrigin(Val); 1178280031Sdim } else { 1179280031Sdim Shadow = dyn_cast_or_null<Instruction>(getShadow(Val)); 1180280031Sdim if (!Shadow) return; 1181280031Sdim Origin = dyn_cast_or_null<Instruction>(getOrigin(Val)); 1182280031Sdim } 1183261991Sdim insertShadowCheck(Shadow, Origin, OrigIns); 1184261991Sdim } 1185261991Sdim 1186261991Sdim AtomicOrdering addReleaseOrdering(AtomicOrdering a) { 1187261991Sdim switch (a) { 1188261991Sdim case NotAtomic: 1189261991Sdim return NotAtomic; 1190261991Sdim case Unordered: 1191261991Sdim case Monotonic: 1192261991Sdim case Release: 1193261991Sdim return Release; 1194261991Sdim case Acquire: 1195261991Sdim case AcquireRelease: 1196261991Sdim return AcquireRelease; 1197261991Sdim case SequentiallyConsistent: 1198261991Sdim return SequentiallyConsistent; 1199261991Sdim } 1200261991Sdim llvm_unreachable("Unknown ordering"); 1201261991Sdim } 1202261991Sdim 1203261991Sdim AtomicOrdering addAcquireOrdering(AtomicOrdering a) { 1204261991Sdim switch (a) { 1205261991Sdim case NotAtomic: 1206261991Sdim return NotAtomic; 1207261991Sdim case Unordered: 1208261991Sdim case Monotonic: 1209261991Sdim case Acquire: 1210261991Sdim return Acquire; 1211261991Sdim case Release: 1212261991Sdim case AcquireRelease: 1213261991Sdim return AcquireRelease; 1214261991Sdim case SequentiallyConsistent: 1215261991Sdim return SequentiallyConsistent; 1216261991Sdim } 1217261991Sdim llvm_unreachable("Unknown ordering"); 1218261991Sdim } 1219261991Sdim 1220249259Sdim // ------------------- Visitors. 1221249259Sdim 1222249259Sdim /// \brief Instrument LoadInst 1223249259Sdim /// 1224249259Sdim /// Loads the corresponding shadow and (optionally) origin. 1225249259Sdim /// Optionally, checks that the load address is fully defined. 1226249259Sdim void visitLoadInst(LoadInst &I) { 1227249259Sdim assert(I.getType()->isSized() && "Load type must have size"); 1228261991Sdim IRBuilder<> IRB(I.getNextNode()); 1229249259Sdim Type *ShadowTy = getShadowTy(&I); 1230249259Sdim Value *Addr = I.getPointerOperand(); 1231280031Sdim if (PropagateShadow && !I.getMetadata("nosanitize")) { 1232249259Sdim Value *ShadowPtr = getShadowPtr(Addr, ShadowTy, IRB); 1233249259Sdim setShadow(&I, 1234249259Sdim IRB.CreateAlignedLoad(ShadowPtr, I.getAlignment(), "_msld")); 1235249259Sdim } else { 1236249259Sdim setShadow(&I, getCleanShadow(&I)); 1237249259Sdim } 1238249259Sdim 1239249259Sdim if (ClCheckAccessAddress) 1240261991Sdim insertShadowCheck(I.getPointerOperand(), &I); 1241249259Sdim 1242261991Sdim if (I.isAtomic()) 1243261991Sdim I.setOrdering(addAcquireOrdering(I.getOrdering())); 1244261991Sdim 1245249259Sdim if (MS.TrackOrigins) { 1246276479Sdim if (PropagateShadow) { 1247280031Sdim unsigned Alignment = I.getAlignment(); 1248280031Sdim unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment); 1249280031Sdim setOrigin(&I, IRB.CreateAlignedLoad(getOriginPtr(Addr, IRB, Alignment), 1250280031Sdim OriginAlignment)); 1251249259Sdim } else { 1252249259Sdim setOrigin(&I, getCleanOrigin()); 1253249259Sdim } 1254249259Sdim } 1255249259Sdim } 1256249259Sdim 1257249259Sdim /// \brief Instrument StoreInst 1258249259Sdim /// 1259249259Sdim /// Stores the corresponding shadow and (optionally) origin. 1260249259Sdim /// Optionally, checks that the store address is fully defined. 1261249259Sdim void visitStoreInst(StoreInst &I) { 1262249259Sdim StoreList.push_back(&I); 1263249259Sdim } 1264249259Sdim 1265261991Sdim void handleCASOrRMW(Instruction &I) { 1266261991Sdim assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I)); 1267261991Sdim 1268261991Sdim IRBuilder<> IRB(&I); 1269261991Sdim Value *Addr = I.getOperand(0); 1270261991Sdim Value *ShadowPtr = getShadowPtr(Addr, I.getType(), IRB); 1271261991Sdim 1272261991Sdim if (ClCheckAccessAddress) 1273261991Sdim insertShadowCheck(Addr, &I); 1274261991Sdim 1275261991Sdim // Only test the conditional argument of cmpxchg instruction. 1276261991Sdim // The other argument can potentially be uninitialized, but we can not 1277261991Sdim // detect this situation reliably without possible false positives. 1278261991Sdim if (isa<AtomicCmpXchgInst>(I)) 1279261991Sdim insertShadowCheck(I.getOperand(1), &I); 1280261991Sdim 1281261991Sdim IRB.CreateStore(getCleanShadow(&I), ShadowPtr); 1282261991Sdim 1283261991Sdim setShadow(&I, getCleanShadow(&I)); 1284280031Sdim setOrigin(&I, getCleanOrigin()); 1285261991Sdim } 1286261991Sdim 1287261991Sdim void visitAtomicRMWInst(AtomicRMWInst &I) { 1288261991Sdim handleCASOrRMW(I); 1289261991Sdim I.setOrdering(addReleaseOrdering(I.getOrdering())); 1290261991Sdim } 1291261991Sdim 1292261991Sdim void visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) { 1293261991Sdim handleCASOrRMW(I); 1294276479Sdim I.setSuccessOrdering(addReleaseOrdering(I.getSuccessOrdering())); 1295261991Sdim } 1296261991Sdim 1297249259Sdim // Vector manipulation. 1298249259Sdim void visitExtractElementInst(ExtractElementInst &I) { 1299261991Sdim insertShadowCheck(I.getOperand(1), &I); 1300249259Sdim IRBuilder<> IRB(&I); 1301249259Sdim setShadow(&I, IRB.CreateExtractElement(getShadow(&I, 0), I.getOperand(1), 1302249259Sdim "_msprop")); 1303249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1304249259Sdim } 1305249259Sdim 1306249259Sdim void visitInsertElementInst(InsertElementInst &I) { 1307261991Sdim insertShadowCheck(I.getOperand(2), &I); 1308249259Sdim IRBuilder<> IRB(&I); 1309249259Sdim setShadow(&I, IRB.CreateInsertElement(getShadow(&I, 0), getShadow(&I, 1), 1310249259Sdim I.getOperand(2), "_msprop")); 1311249259Sdim setOriginForNaryOp(I); 1312249259Sdim } 1313249259Sdim 1314249259Sdim void visitShuffleVectorInst(ShuffleVectorInst &I) { 1315261991Sdim insertShadowCheck(I.getOperand(2), &I); 1316249259Sdim IRBuilder<> IRB(&I); 1317249259Sdim setShadow(&I, IRB.CreateShuffleVector(getShadow(&I, 0), getShadow(&I, 1), 1318249259Sdim I.getOperand(2), "_msprop")); 1319249259Sdim setOriginForNaryOp(I); 1320249259Sdim } 1321249259Sdim 1322249259Sdim // Casts. 1323249259Sdim void visitSExtInst(SExtInst &I) { 1324249259Sdim IRBuilder<> IRB(&I); 1325249259Sdim setShadow(&I, IRB.CreateSExt(getShadow(&I, 0), I.getType(), "_msprop")); 1326249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1327249259Sdim } 1328249259Sdim 1329249259Sdim void visitZExtInst(ZExtInst &I) { 1330249259Sdim IRBuilder<> IRB(&I); 1331249259Sdim setShadow(&I, IRB.CreateZExt(getShadow(&I, 0), I.getType(), "_msprop")); 1332249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1333249259Sdim } 1334249259Sdim 1335249259Sdim void visitTruncInst(TruncInst &I) { 1336249259Sdim IRBuilder<> IRB(&I); 1337249259Sdim setShadow(&I, IRB.CreateTrunc(getShadow(&I, 0), I.getType(), "_msprop")); 1338249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1339249259Sdim } 1340249259Sdim 1341249259Sdim void visitBitCastInst(BitCastInst &I) { 1342249259Sdim IRBuilder<> IRB(&I); 1343249259Sdim setShadow(&I, IRB.CreateBitCast(getShadow(&I, 0), getShadowTy(&I))); 1344249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1345249259Sdim } 1346249259Sdim 1347249259Sdim void visitPtrToIntInst(PtrToIntInst &I) { 1348249259Sdim IRBuilder<> IRB(&I); 1349249259Sdim setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false, 1350249259Sdim "_msprop_ptrtoint")); 1351249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1352249259Sdim } 1353249259Sdim 1354249259Sdim void visitIntToPtrInst(IntToPtrInst &I) { 1355249259Sdim IRBuilder<> IRB(&I); 1356249259Sdim setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false, 1357249259Sdim "_msprop_inttoptr")); 1358249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1359249259Sdim } 1360249259Sdim 1361249259Sdim void visitFPToSIInst(CastInst& I) { handleShadowOr(I); } 1362249259Sdim void visitFPToUIInst(CastInst& I) { handleShadowOr(I); } 1363249259Sdim void visitSIToFPInst(CastInst& I) { handleShadowOr(I); } 1364249259Sdim void visitUIToFPInst(CastInst& I) { handleShadowOr(I); } 1365249259Sdim void visitFPExtInst(CastInst& I) { handleShadowOr(I); } 1366249259Sdim void visitFPTruncInst(CastInst& I) { handleShadowOr(I); } 1367249259Sdim 1368249259Sdim /// \brief Propagate shadow for bitwise AND. 1369249259Sdim /// 1370249259Sdim /// This code is exact, i.e. if, for example, a bit in the left argument 1371249259Sdim /// is defined and 0, then neither the value not definedness of the 1372249259Sdim /// corresponding bit in B don't affect the resulting shadow. 1373249259Sdim void visitAnd(BinaryOperator &I) { 1374249259Sdim IRBuilder<> IRB(&I); 1375249259Sdim // "And" of 0 and a poisoned value results in unpoisoned value. 1376249259Sdim // 1&1 => 1; 0&1 => 0; p&1 => p; 1377249259Sdim // 1&0 => 0; 0&0 => 0; p&0 => 0; 1378249259Sdim // 1&p => p; 0&p => 0; p&p => p; 1379249259Sdim // S = (S1 & S2) | (V1 & S2) | (S1 & V2) 1380249259Sdim Value *S1 = getShadow(&I, 0); 1381249259Sdim Value *S2 = getShadow(&I, 1); 1382249259Sdim Value *V1 = I.getOperand(0); 1383249259Sdim Value *V2 = I.getOperand(1); 1384249259Sdim if (V1->getType() != S1->getType()) { 1385249259Sdim V1 = IRB.CreateIntCast(V1, S1->getType(), false); 1386249259Sdim V2 = IRB.CreateIntCast(V2, S2->getType(), false); 1387249259Sdim } 1388249259Sdim Value *S1S2 = IRB.CreateAnd(S1, S2); 1389249259Sdim Value *V1S2 = IRB.CreateAnd(V1, S2); 1390249259Sdim Value *S1V2 = IRB.CreateAnd(S1, V2); 1391249259Sdim setShadow(&I, IRB.CreateOr(S1S2, IRB.CreateOr(V1S2, S1V2))); 1392249259Sdim setOriginForNaryOp(I); 1393249259Sdim } 1394249259Sdim 1395249259Sdim void visitOr(BinaryOperator &I) { 1396249259Sdim IRBuilder<> IRB(&I); 1397249259Sdim // "Or" of 1 and a poisoned value results in unpoisoned value. 1398249259Sdim // 1|1 => 1; 0|1 => 1; p|1 => 1; 1399249259Sdim // 1|0 => 1; 0|0 => 0; p|0 => p; 1400249259Sdim // 1|p => 1; 0|p => p; p|p => p; 1401249259Sdim // S = (S1 & S2) | (~V1 & S2) | (S1 & ~V2) 1402249259Sdim Value *S1 = getShadow(&I, 0); 1403249259Sdim Value *S2 = getShadow(&I, 1); 1404249259Sdim Value *V1 = IRB.CreateNot(I.getOperand(0)); 1405249259Sdim Value *V2 = IRB.CreateNot(I.getOperand(1)); 1406249259Sdim if (V1->getType() != S1->getType()) { 1407249259Sdim V1 = IRB.CreateIntCast(V1, S1->getType(), false); 1408249259Sdim V2 = IRB.CreateIntCast(V2, S2->getType(), false); 1409249259Sdim } 1410249259Sdim Value *S1S2 = IRB.CreateAnd(S1, S2); 1411249259Sdim Value *V1S2 = IRB.CreateAnd(V1, S2); 1412249259Sdim Value *S1V2 = IRB.CreateAnd(S1, V2); 1413249259Sdim setShadow(&I, IRB.CreateOr(S1S2, IRB.CreateOr(V1S2, S1V2))); 1414249259Sdim setOriginForNaryOp(I); 1415249259Sdim } 1416249259Sdim 1417249259Sdim /// \brief Default propagation of shadow and/or origin. 1418249259Sdim /// 1419249259Sdim /// This class implements the general case of shadow propagation, used in all 1420249259Sdim /// cases where we don't know and/or don't care about what the operation 1421249259Sdim /// actually does. It converts all input shadow values to a common type 1422249259Sdim /// (extending or truncating as necessary), and bitwise OR's them. 1423249259Sdim /// 1424249259Sdim /// This is much cheaper than inserting checks (i.e. requiring inputs to be 1425249259Sdim /// fully initialized), and less prone to false positives. 1426249259Sdim /// 1427249259Sdim /// This class also implements the general case of origin propagation. For a 1428249259Sdim /// Nary operation, result origin is set to the origin of an argument that is 1429249259Sdim /// not entirely initialized. If there is more than one such arguments, the 1430249259Sdim /// rightmost of them is picked. It does not matter which one is picked if all 1431249259Sdim /// arguments are initialized. 1432249259Sdim template <bool CombineShadow> 1433249259Sdim class Combiner { 1434249259Sdim Value *Shadow; 1435249259Sdim Value *Origin; 1436249259Sdim IRBuilder<> &IRB; 1437249259Sdim MemorySanitizerVisitor *MSV; 1438249259Sdim 1439249259Sdim public: 1440249259Sdim Combiner(MemorySanitizerVisitor *MSV, IRBuilder<> &IRB) : 1441276479Sdim Shadow(nullptr), Origin(nullptr), IRB(IRB), MSV(MSV) {} 1442249259Sdim 1443249259Sdim /// \brief Add a pair of shadow and origin values to the mix. 1444249259Sdim Combiner &Add(Value *OpShadow, Value *OpOrigin) { 1445249259Sdim if (CombineShadow) { 1446249259Sdim assert(OpShadow); 1447249259Sdim if (!Shadow) 1448249259Sdim Shadow = OpShadow; 1449249259Sdim else { 1450249259Sdim OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType()); 1451249259Sdim Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop"); 1452249259Sdim } 1453249259Sdim } 1454249259Sdim 1455249259Sdim if (MSV->MS.TrackOrigins) { 1456249259Sdim assert(OpOrigin); 1457249259Sdim if (!Origin) { 1458249259Sdim Origin = OpOrigin; 1459249259Sdim } else { 1460276479Sdim Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin); 1461276479Sdim // No point in adding something that might result in 0 origin value. 1462276479Sdim if (!ConstOrigin || !ConstOrigin->isNullValue()) { 1463276479Sdim Value *FlatShadow = MSV->convertToShadowTyNoVec(OpShadow, IRB); 1464276479Sdim Value *Cond = 1465276479Sdim IRB.CreateICmpNE(FlatShadow, MSV->getCleanShadow(FlatShadow)); 1466276479Sdim Origin = IRB.CreateSelect(Cond, OpOrigin, Origin); 1467276479Sdim } 1468249259Sdim } 1469249259Sdim } 1470249259Sdim return *this; 1471249259Sdim } 1472249259Sdim 1473249259Sdim /// \brief Add an application value to the mix. 1474249259Sdim Combiner &Add(Value *V) { 1475249259Sdim Value *OpShadow = MSV->getShadow(V); 1476276479Sdim Value *OpOrigin = MSV->MS.TrackOrigins ? MSV->getOrigin(V) : nullptr; 1477249259Sdim return Add(OpShadow, OpOrigin); 1478249259Sdim } 1479249259Sdim 1480249259Sdim /// \brief Set the current combined values as the given instruction's shadow 1481249259Sdim /// and origin. 1482249259Sdim void Done(Instruction *I) { 1483249259Sdim if (CombineShadow) { 1484249259Sdim assert(Shadow); 1485249259Sdim Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I)); 1486249259Sdim MSV->setShadow(I, Shadow); 1487249259Sdim } 1488249259Sdim if (MSV->MS.TrackOrigins) { 1489249259Sdim assert(Origin); 1490249259Sdim MSV->setOrigin(I, Origin); 1491249259Sdim } 1492249259Sdim } 1493249259Sdim }; 1494249259Sdim 1495249259Sdim typedef Combiner<true> ShadowAndOriginCombiner; 1496249259Sdim typedef Combiner<false> OriginCombiner; 1497249259Sdim 1498249259Sdim /// \brief Propagate origin for arbitrary operation. 1499249259Sdim void setOriginForNaryOp(Instruction &I) { 1500249259Sdim if (!MS.TrackOrigins) return; 1501249259Sdim IRBuilder<> IRB(&I); 1502249259Sdim OriginCombiner OC(this, IRB); 1503249259Sdim for (Instruction::op_iterator OI = I.op_begin(); OI != I.op_end(); ++OI) 1504249259Sdim OC.Add(OI->get()); 1505249259Sdim OC.Done(&I); 1506249259Sdim } 1507249259Sdim 1508249259Sdim size_t VectorOrPrimitiveTypeSizeInBits(Type *Ty) { 1509249259Sdim assert(!(Ty->isVectorTy() && Ty->getScalarType()->isPointerTy()) && 1510249259Sdim "Vector of pointers is not a valid shadow type"); 1511249259Sdim return Ty->isVectorTy() ? 1512249259Sdim Ty->getVectorNumElements() * Ty->getScalarSizeInBits() : 1513249259Sdim Ty->getPrimitiveSizeInBits(); 1514249259Sdim } 1515249259Sdim 1516249259Sdim /// \brief Cast between two shadow types, extending or truncating as 1517249259Sdim /// necessary. 1518261991Sdim Value *CreateShadowCast(IRBuilder<> &IRB, Value *V, Type *dstTy, 1519261991Sdim bool Signed = false) { 1520249259Sdim Type *srcTy = V->getType(); 1521249259Sdim if (dstTy->isIntegerTy() && srcTy->isIntegerTy()) 1522261991Sdim return IRB.CreateIntCast(V, dstTy, Signed); 1523249259Sdim if (dstTy->isVectorTy() && srcTy->isVectorTy() && 1524249259Sdim dstTy->getVectorNumElements() == srcTy->getVectorNumElements()) 1525261991Sdim return IRB.CreateIntCast(V, dstTy, Signed); 1526249259Sdim size_t srcSizeInBits = VectorOrPrimitiveTypeSizeInBits(srcTy); 1527249259Sdim size_t dstSizeInBits = VectorOrPrimitiveTypeSizeInBits(dstTy); 1528249259Sdim Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits)); 1529249259Sdim Value *V2 = 1530261991Sdim IRB.CreateIntCast(V1, Type::getIntNTy(*MS.C, dstSizeInBits), Signed); 1531249259Sdim return IRB.CreateBitCast(V2, dstTy); 1532249259Sdim // TODO: handle struct types. 1533249259Sdim } 1534249259Sdim 1535276479Sdim /// \brief Cast an application value to the type of its own shadow. 1536276479Sdim Value *CreateAppToShadowCast(IRBuilder<> &IRB, Value *V) { 1537276479Sdim Type *ShadowTy = getShadowTy(V); 1538276479Sdim if (V->getType() == ShadowTy) 1539276479Sdim return V; 1540276479Sdim if (V->getType()->isPtrOrPtrVectorTy()) 1541276479Sdim return IRB.CreatePtrToInt(V, ShadowTy); 1542276479Sdim else 1543276479Sdim return IRB.CreateBitCast(V, ShadowTy); 1544276479Sdim } 1545276479Sdim 1546249259Sdim /// \brief Propagate shadow for arbitrary operation. 1547249259Sdim void handleShadowOr(Instruction &I) { 1548249259Sdim IRBuilder<> IRB(&I); 1549249259Sdim ShadowAndOriginCombiner SC(this, IRB); 1550249259Sdim for (Instruction::op_iterator OI = I.op_begin(); OI != I.op_end(); ++OI) 1551249259Sdim SC.Add(OI->get()); 1552249259Sdim SC.Done(&I); 1553249259Sdim } 1554249259Sdim 1555276479Sdim // \brief Handle multiplication by constant. 1556276479Sdim // 1557276479Sdim // Handle a special case of multiplication by constant that may have one or 1558276479Sdim // more zeros in the lower bits. This makes corresponding number of lower bits 1559276479Sdim // of the result zero as well. We model it by shifting the other operand 1560276479Sdim // shadow left by the required number of bits. Effectively, we transform 1561276479Sdim // (X * (A * 2**B)) to ((X << B) * A) and instrument (X << B) as (Sx << B). 1562276479Sdim // We use multiplication by 2**N instead of shift to cover the case of 1563276479Sdim // multiplication by 0, which may occur in some elements of a vector operand. 1564276479Sdim void handleMulByConstant(BinaryOperator &I, Constant *ConstArg, 1565276479Sdim Value *OtherArg) { 1566276479Sdim Constant *ShadowMul; 1567276479Sdim Type *Ty = ConstArg->getType(); 1568276479Sdim if (Ty->isVectorTy()) { 1569276479Sdim unsigned NumElements = Ty->getVectorNumElements(); 1570276479Sdim Type *EltTy = Ty->getSequentialElementType(); 1571276479Sdim SmallVector<Constant *, 16> Elements; 1572276479Sdim for (unsigned Idx = 0; Idx < NumElements; ++Idx) { 1573276479Sdim ConstantInt *Elt = 1574276479Sdim dyn_cast<ConstantInt>(ConstArg->getAggregateElement(Idx)); 1575276479Sdim APInt V = Elt->getValue(); 1576276479Sdim APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros(); 1577276479Sdim Elements.push_back(ConstantInt::get(EltTy, V2)); 1578276479Sdim } 1579276479Sdim ShadowMul = ConstantVector::get(Elements); 1580276479Sdim } else { 1581276479Sdim ConstantInt *Elt = dyn_cast<ConstantInt>(ConstArg); 1582276479Sdim APInt V = Elt->getValue(); 1583276479Sdim APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros(); 1584276479Sdim ShadowMul = ConstantInt::get(Elt->getType(), V2); 1585276479Sdim } 1586276479Sdim 1587276479Sdim IRBuilder<> IRB(&I); 1588276479Sdim setShadow(&I, 1589276479Sdim IRB.CreateMul(getShadow(OtherArg), ShadowMul, "msprop_mul_cst")); 1590276479Sdim setOrigin(&I, getOrigin(OtherArg)); 1591276479Sdim } 1592276479Sdim 1593276479Sdim void visitMul(BinaryOperator &I) { 1594276479Sdim Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0)); 1595276479Sdim Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1)); 1596276479Sdim if (constOp0 && !constOp1) 1597276479Sdim handleMulByConstant(I, constOp0, I.getOperand(1)); 1598276479Sdim else if (constOp1 && !constOp0) 1599276479Sdim handleMulByConstant(I, constOp1, I.getOperand(0)); 1600276479Sdim else 1601276479Sdim handleShadowOr(I); 1602276479Sdim } 1603276479Sdim 1604249259Sdim void visitFAdd(BinaryOperator &I) { handleShadowOr(I); } 1605249259Sdim void visitFSub(BinaryOperator &I) { handleShadowOr(I); } 1606249259Sdim void visitFMul(BinaryOperator &I) { handleShadowOr(I); } 1607249259Sdim void visitAdd(BinaryOperator &I) { handleShadowOr(I); } 1608249259Sdim void visitSub(BinaryOperator &I) { handleShadowOr(I); } 1609249259Sdim void visitXor(BinaryOperator &I) { handleShadowOr(I); } 1610249259Sdim 1611249259Sdim void handleDiv(Instruction &I) { 1612249259Sdim IRBuilder<> IRB(&I); 1613249259Sdim // Strict on the second argument. 1614261991Sdim insertShadowCheck(I.getOperand(1), &I); 1615249259Sdim setShadow(&I, getShadow(&I, 0)); 1616249259Sdim setOrigin(&I, getOrigin(&I, 0)); 1617249259Sdim } 1618249259Sdim 1619249259Sdim void visitUDiv(BinaryOperator &I) { handleDiv(I); } 1620249259Sdim void visitSDiv(BinaryOperator &I) { handleDiv(I); } 1621249259Sdim void visitFDiv(BinaryOperator &I) { handleDiv(I); } 1622249259Sdim void visitURem(BinaryOperator &I) { handleDiv(I); } 1623249259Sdim void visitSRem(BinaryOperator &I) { handleDiv(I); } 1624249259Sdim void visitFRem(BinaryOperator &I) { handleDiv(I); } 1625249259Sdim 1626249259Sdim /// \brief Instrument == and != comparisons. 1627249259Sdim /// 1628249259Sdim /// Sometimes the comparison result is known even if some of the bits of the 1629249259Sdim /// arguments are not. 1630249259Sdim void handleEqualityComparison(ICmpInst &I) { 1631249259Sdim IRBuilder<> IRB(&I); 1632249259Sdim Value *A = I.getOperand(0); 1633249259Sdim Value *B = I.getOperand(1); 1634249259Sdim Value *Sa = getShadow(A); 1635249259Sdim Value *Sb = getShadow(B); 1636249259Sdim 1637249259Sdim // Get rid of pointers and vectors of pointers. 1638249259Sdim // For ints (and vectors of ints), types of A and Sa match, 1639249259Sdim // and this is a no-op. 1640249259Sdim A = IRB.CreatePointerCast(A, Sa->getType()); 1641249259Sdim B = IRB.CreatePointerCast(B, Sb->getType()); 1642249259Sdim 1643249259Sdim // A == B <==> (C = A^B) == 0 1644249259Sdim // A != B <==> (C = A^B) != 0 1645249259Sdim // Sc = Sa | Sb 1646249259Sdim Value *C = IRB.CreateXor(A, B); 1647249259Sdim Value *Sc = IRB.CreateOr(Sa, Sb); 1648249259Sdim // Now dealing with i = (C == 0) comparison (or C != 0, does not matter now) 1649249259Sdim // Result is defined if one of the following is true 1650249259Sdim // * there is a defined 1 bit in C 1651249259Sdim // * C is fully defined 1652249259Sdim // Si = !(C & ~Sc) && Sc 1653249259Sdim Value *Zero = Constant::getNullValue(Sc->getType()); 1654249259Sdim Value *MinusOne = Constant::getAllOnesValue(Sc->getType()); 1655249259Sdim Value *Si = 1656249259Sdim IRB.CreateAnd(IRB.CreateICmpNE(Sc, Zero), 1657249259Sdim IRB.CreateICmpEQ( 1658249259Sdim IRB.CreateAnd(IRB.CreateXor(Sc, MinusOne), C), Zero)); 1659249259Sdim Si->setName("_msprop_icmp"); 1660249259Sdim setShadow(&I, Si); 1661249259Sdim setOriginForNaryOp(I); 1662249259Sdim } 1663249259Sdim 1664249259Sdim /// \brief Build the lowest possible value of V, taking into account V's 1665249259Sdim /// uninitialized bits. 1666249259Sdim Value *getLowestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa, 1667249259Sdim bool isSigned) { 1668249259Sdim if (isSigned) { 1669249259Sdim // Split shadow into sign bit and other bits. 1670249259Sdim Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1); 1671249259Sdim Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits); 1672249259Sdim // Maximise the undefined shadow bit, minimize other undefined bits. 1673249259Sdim return 1674249259Sdim IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaOtherBits)), SaSignBit); 1675249259Sdim } else { 1676249259Sdim // Minimize undefined bits. 1677249259Sdim return IRB.CreateAnd(A, IRB.CreateNot(Sa)); 1678249259Sdim } 1679249259Sdim } 1680249259Sdim 1681249259Sdim /// \brief Build the highest possible value of V, taking into account V's 1682249259Sdim /// uninitialized bits. 1683249259Sdim Value *getHighestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa, 1684249259Sdim bool isSigned) { 1685249259Sdim if (isSigned) { 1686249259Sdim // Split shadow into sign bit and other bits. 1687249259Sdim Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1); 1688249259Sdim Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits); 1689249259Sdim // Minimise the undefined shadow bit, maximise other undefined bits. 1690249259Sdim return 1691249259Sdim IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaSignBit)), SaOtherBits); 1692249259Sdim } else { 1693249259Sdim // Maximize undefined bits. 1694249259Sdim return IRB.CreateOr(A, Sa); 1695249259Sdim } 1696249259Sdim } 1697249259Sdim 1698249259Sdim /// \brief Instrument relational comparisons. 1699249259Sdim /// 1700249259Sdim /// This function does exact shadow propagation for all relational 1701249259Sdim /// comparisons of integers, pointers and vectors of those. 1702249259Sdim /// FIXME: output seems suboptimal when one of the operands is a constant 1703249259Sdim void handleRelationalComparisonExact(ICmpInst &I) { 1704249259Sdim IRBuilder<> IRB(&I); 1705249259Sdim Value *A = I.getOperand(0); 1706249259Sdim Value *B = I.getOperand(1); 1707249259Sdim Value *Sa = getShadow(A); 1708249259Sdim Value *Sb = getShadow(B); 1709249259Sdim 1710249259Sdim // Get rid of pointers and vectors of pointers. 1711249259Sdim // For ints (and vectors of ints), types of A and Sa match, 1712249259Sdim // and this is a no-op. 1713249259Sdim A = IRB.CreatePointerCast(A, Sa->getType()); 1714249259Sdim B = IRB.CreatePointerCast(B, Sb->getType()); 1715249259Sdim 1716249259Sdim // Let [a0, a1] be the interval of possible values of A, taking into account 1717249259Sdim // its undefined bits. Let [b0, b1] be the interval of possible values of B. 1718249259Sdim // Then (A cmp B) is defined iff (a0 cmp b1) == (a1 cmp b0). 1719249259Sdim bool IsSigned = I.isSigned(); 1720249259Sdim Value *S1 = IRB.CreateICmp(I.getPredicate(), 1721249259Sdim getLowestPossibleValue(IRB, A, Sa, IsSigned), 1722249259Sdim getHighestPossibleValue(IRB, B, Sb, IsSigned)); 1723249259Sdim Value *S2 = IRB.CreateICmp(I.getPredicate(), 1724249259Sdim getHighestPossibleValue(IRB, A, Sa, IsSigned), 1725249259Sdim getLowestPossibleValue(IRB, B, Sb, IsSigned)); 1726249259Sdim Value *Si = IRB.CreateXor(S1, S2); 1727249259Sdim setShadow(&I, Si); 1728249259Sdim setOriginForNaryOp(I); 1729249259Sdim } 1730249259Sdim 1731249259Sdim /// \brief Instrument signed relational comparisons. 1732249259Sdim /// 1733249259Sdim /// Handle (x<0) and (x>=0) comparisons (essentially, sign bit tests) by 1734249259Sdim /// propagating the highest bit of the shadow. Everything else is delegated 1735249259Sdim /// to handleShadowOr(). 1736249259Sdim void handleSignedRelationalComparison(ICmpInst &I) { 1737249259Sdim Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0)); 1738249259Sdim Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1)); 1739276479Sdim Value* op = nullptr; 1740249259Sdim CmpInst::Predicate pre = I.getPredicate(); 1741249259Sdim if (constOp0 && constOp0->isNullValue() && 1742249259Sdim (pre == CmpInst::ICMP_SGT || pre == CmpInst::ICMP_SLE)) { 1743249259Sdim op = I.getOperand(1); 1744249259Sdim } else if (constOp1 && constOp1->isNullValue() && 1745249259Sdim (pre == CmpInst::ICMP_SLT || pre == CmpInst::ICMP_SGE)) { 1746249259Sdim op = I.getOperand(0); 1747249259Sdim } 1748249259Sdim if (op) { 1749249259Sdim IRBuilder<> IRB(&I); 1750249259Sdim Value* Shadow = 1751249259Sdim IRB.CreateICmpSLT(getShadow(op), getCleanShadow(op), "_msprop_icmpslt"); 1752249259Sdim setShadow(&I, Shadow); 1753249259Sdim setOrigin(&I, getOrigin(op)); 1754249259Sdim } else { 1755249259Sdim handleShadowOr(I); 1756249259Sdim } 1757249259Sdim } 1758249259Sdim 1759249259Sdim void visitICmpInst(ICmpInst &I) { 1760249259Sdim if (!ClHandleICmp) { 1761249259Sdim handleShadowOr(I); 1762249259Sdim return; 1763249259Sdim } 1764249259Sdim if (I.isEquality()) { 1765249259Sdim handleEqualityComparison(I); 1766249259Sdim return; 1767249259Sdim } 1768249259Sdim 1769249259Sdim assert(I.isRelational()); 1770249259Sdim if (ClHandleICmpExact) { 1771249259Sdim handleRelationalComparisonExact(I); 1772249259Sdim return; 1773249259Sdim } 1774249259Sdim if (I.isSigned()) { 1775249259Sdim handleSignedRelationalComparison(I); 1776249259Sdim return; 1777249259Sdim } 1778249259Sdim 1779249259Sdim assert(I.isUnsigned()); 1780249259Sdim if ((isa<Constant>(I.getOperand(0)) || isa<Constant>(I.getOperand(1)))) { 1781249259Sdim handleRelationalComparisonExact(I); 1782249259Sdim return; 1783249259Sdim } 1784249259Sdim 1785249259Sdim handleShadowOr(I); 1786249259Sdim } 1787249259Sdim 1788249259Sdim void visitFCmpInst(FCmpInst &I) { 1789249259Sdim handleShadowOr(I); 1790249259Sdim } 1791249259Sdim 1792249259Sdim void handleShift(BinaryOperator &I) { 1793249259Sdim IRBuilder<> IRB(&I); 1794249259Sdim // If any of the S2 bits are poisoned, the whole thing is poisoned. 1795249259Sdim // Otherwise perform the same shift on S1. 1796249259Sdim Value *S1 = getShadow(&I, 0); 1797249259Sdim Value *S2 = getShadow(&I, 1); 1798249259Sdim Value *S2Conv = IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)), 1799249259Sdim S2->getType()); 1800249259Sdim Value *V2 = I.getOperand(1); 1801249259Sdim Value *Shift = IRB.CreateBinOp(I.getOpcode(), S1, V2); 1802249259Sdim setShadow(&I, IRB.CreateOr(Shift, S2Conv)); 1803249259Sdim setOriginForNaryOp(I); 1804249259Sdim } 1805249259Sdim 1806249259Sdim void visitShl(BinaryOperator &I) { handleShift(I); } 1807249259Sdim void visitAShr(BinaryOperator &I) { handleShift(I); } 1808249259Sdim void visitLShr(BinaryOperator &I) { handleShift(I); } 1809249259Sdim 1810249259Sdim /// \brief Instrument llvm.memmove 1811249259Sdim /// 1812249259Sdim /// At this point we don't know if llvm.memmove will be inlined or not. 1813249259Sdim /// If we don't instrument it and it gets inlined, 1814249259Sdim /// our interceptor will not kick in and we will lose the memmove. 1815249259Sdim /// If we instrument the call here, but it does not get inlined, 1816249259Sdim /// we will memove the shadow twice: which is bad in case 1817249259Sdim /// of overlapping regions. So, we simply lower the intrinsic to a call. 1818249259Sdim /// 1819249259Sdim /// Similar situation exists for memcpy and memset. 1820249259Sdim void visitMemMoveInst(MemMoveInst &I) { 1821249259Sdim IRBuilder<> IRB(&I); 1822288943Sdim IRB.CreateCall( 1823288943Sdim MS.MemmoveFn, 1824288943Sdim {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()), 1825288943Sdim IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()), 1826288943Sdim IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)}); 1827249259Sdim I.eraseFromParent(); 1828249259Sdim } 1829249259Sdim 1830249259Sdim // Similar to memmove: avoid copying shadow twice. 1831249259Sdim // This is somewhat unfortunate as it may slowdown small constant memcpys. 1832249259Sdim // FIXME: consider doing manual inline for small constant sizes and proper 1833249259Sdim // alignment. 1834249259Sdim void visitMemCpyInst(MemCpyInst &I) { 1835249259Sdim IRBuilder<> IRB(&I); 1836288943Sdim IRB.CreateCall( 1837288943Sdim MS.MemcpyFn, 1838288943Sdim {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()), 1839288943Sdim IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()), 1840288943Sdim IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)}); 1841249259Sdim I.eraseFromParent(); 1842249259Sdim } 1843249259Sdim 1844249259Sdim // Same as memcpy. 1845249259Sdim void visitMemSetInst(MemSetInst &I) { 1846249259Sdim IRBuilder<> IRB(&I); 1847288943Sdim IRB.CreateCall( 1848288943Sdim MS.MemsetFn, 1849288943Sdim {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()), 1850288943Sdim IRB.CreateIntCast(I.getArgOperand(1), IRB.getInt32Ty(), false), 1851288943Sdim IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)}); 1852249259Sdim I.eraseFromParent(); 1853249259Sdim } 1854249259Sdim 1855249259Sdim void visitVAStartInst(VAStartInst &I) { 1856249259Sdim VAHelper->visitVAStartInst(I); 1857249259Sdim } 1858249259Sdim 1859249259Sdim void visitVACopyInst(VACopyInst &I) { 1860249259Sdim VAHelper->visitVACopyInst(I); 1861249259Sdim } 1862249259Sdim 1863249259Sdim enum IntrinsicKind { 1864249259Sdim IK_DoesNotAccessMemory, 1865249259Sdim IK_OnlyReadsMemory, 1866249259Sdim IK_WritesMemory 1867249259Sdim }; 1868249259Sdim 1869249259Sdim static IntrinsicKind getIntrinsicKind(Intrinsic::ID iid) { 1870249259Sdim const int DoesNotAccessMemory = IK_DoesNotAccessMemory; 1871249259Sdim const int OnlyReadsArgumentPointees = IK_OnlyReadsMemory; 1872249259Sdim const int OnlyReadsMemory = IK_OnlyReadsMemory; 1873249259Sdim const int OnlyAccessesArgumentPointees = IK_WritesMemory; 1874249259Sdim const int UnknownModRefBehavior = IK_WritesMemory; 1875249259Sdim#define GET_INTRINSIC_MODREF_BEHAVIOR 1876249259Sdim#define ModRefBehavior IntrinsicKind 1877249259Sdim#include "llvm/IR/Intrinsics.gen" 1878249259Sdim#undef ModRefBehavior 1879249259Sdim#undef GET_INTRINSIC_MODREF_BEHAVIOR 1880249259Sdim } 1881249259Sdim 1882249259Sdim /// \brief Handle vector store-like intrinsics. 1883249259Sdim /// 1884249259Sdim /// Instrument intrinsics that look like a simple SIMD store: writes memory, 1885249259Sdim /// has 1 pointer argument and 1 vector argument, returns void. 1886249259Sdim bool handleVectorStoreIntrinsic(IntrinsicInst &I) { 1887249259Sdim IRBuilder<> IRB(&I); 1888249259Sdim Value* Addr = I.getArgOperand(0); 1889249259Sdim Value *Shadow = getShadow(&I, 1); 1890249259Sdim Value *ShadowPtr = getShadowPtr(Addr, Shadow->getType(), IRB); 1891249259Sdim 1892249259Sdim // We don't know the pointer alignment (could be unaligned SSE store!). 1893249259Sdim // Have to assume to worst case. 1894249259Sdim IRB.CreateAlignedStore(Shadow, ShadowPtr, 1); 1895249259Sdim 1896249259Sdim if (ClCheckAccessAddress) 1897261991Sdim insertShadowCheck(Addr, &I); 1898249259Sdim 1899249259Sdim // FIXME: use ClStoreCleanOrigin 1900249259Sdim // FIXME: factor out common code from materializeStores 1901249259Sdim if (MS.TrackOrigins) 1902280031Sdim IRB.CreateStore(getOrigin(&I, 1), getOriginPtr(Addr, IRB, 1)); 1903249259Sdim return true; 1904249259Sdim } 1905249259Sdim 1906249259Sdim /// \brief Handle vector load-like intrinsics. 1907249259Sdim /// 1908249259Sdim /// Instrument intrinsics that look like a simple SIMD load: reads memory, 1909249259Sdim /// has 1 pointer argument, returns a vector. 1910249259Sdim bool handleVectorLoadIntrinsic(IntrinsicInst &I) { 1911249259Sdim IRBuilder<> IRB(&I); 1912249259Sdim Value *Addr = I.getArgOperand(0); 1913249259Sdim 1914249259Sdim Type *ShadowTy = getShadowTy(&I); 1915276479Sdim if (PropagateShadow) { 1916249259Sdim Value *ShadowPtr = getShadowPtr(Addr, ShadowTy, IRB); 1917249259Sdim // We don't know the pointer alignment (could be unaligned SSE load!). 1918249259Sdim // Have to assume to worst case. 1919249259Sdim setShadow(&I, IRB.CreateAlignedLoad(ShadowPtr, 1, "_msld")); 1920249259Sdim } else { 1921249259Sdim setShadow(&I, getCleanShadow(&I)); 1922249259Sdim } 1923249259Sdim 1924249259Sdim if (ClCheckAccessAddress) 1925261991Sdim insertShadowCheck(Addr, &I); 1926249259Sdim 1927249259Sdim if (MS.TrackOrigins) { 1928276479Sdim if (PropagateShadow) 1929280031Sdim setOrigin(&I, IRB.CreateLoad(getOriginPtr(Addr, IRB, 1))); 1930249259Sdim else 1931249259Sdim setOrigin(&I, getCleanOrigin()); 1932249259Sdim } 1933249259Sdim return true; 1934249259Sdim } 1935249259Sdim 1936249259Sdim /// \brief Handle (SIMD arithmetic)-like intrinsics. 1937249259Sdim /// 1938249259Sdim /// Instrument intrinsics with any number of arguments of the same type, 1939249259Sdim /// equal to the return type. The type should be simple (no aggregates or 1940249259Sdim /// pointers; vectors are fine). 1941249259Sdim /// Caller guarantees that this intrinsic does not access memory. 1942249259Sdim bool maybeHandleSimpleNomemIntrinsic(IntrinsicInst &I) { 1943249259Sdim Type *RetTy = I.getType(); 1944249259Sdim if (!(RetTy->isIntOrIntVectorTy() || 1945249259Sdim RetTy->isFPOrFPVectorTy() || 1946249259Sdim RetTy->isX86_MMXTy())) 1947249259Sdim return false; 1948249259Sdim 1949249259Sdim unsigned NumArgOperands = I.getNumArgOperands(); 1950249259Sdim 1951249259Sdim for (unsigned i = 0; i < NumArgOperands; ++i) { 1952249259Sdim Type *Ty = I.getArgOperand(i)->getType(); 1953249259Sdim if (Ty != RetTy) 1954249259Sdim return false; 1955249259Sdim } 1956249259Sdim 1957249259Sdim IRBuilder<> IRB(&I); 1958249259Sdim ShadowAndOriginCombiner SC(this, IRB); 1959249259Sdim for (unsigned i = 0; i < NumArgOperands; ++i) 1960249259Sdim SC.Add(I.getArgOperand(i)); 1961249259Sdim SC.Done(&I); 1962249259Sdim 1963249259Sdim return true; 1964249259Sdim } 1965249259Sdim 1966249259Sdim /// \brief Heuristically instrument unknown intrinsics. 1967249259Sdim /// 1968249259Sdim /// The main purpose of this code is to do something reasonable with all 1969249259Sdim /// random intrinsics we might encounter, most importantly - SIMD intrinsics. 1970249259Sdim /// We recognize several classes of intrinsics by their argument types and 1971249259Sdim /// ModRefBehaviour and apply special intrumentation when we are reasonably 1972249259Sdim /// sure that we know what the intrinsic does. 1973249259Sdim /// 1974249259Sdim /// We special-case intrinsics where this approach fails. See llvm.bswap 1975249259Sdim /// handling as an example of that. 1976249259Sdim bool handleUnknownIntrinsic(IntrinsicInst &I) { 1977249259Sdim unsigned NumArgOperands = I.getNumArgOperands(); 1978249259Sdim if (NumArgOperands == 0) 1979249259Sdim return false; 1980249259Sdim 1981249259Sdim Intrinsic::ID iid = I.getIntrinsicID(); 1982249259Sdim IntrinsicKind IK = getIntrinsicKind(iid); 1983249259Sdim bool OnlyReadsMemory = IK == IK_OnlyReadsMemory; 1984249259Sdim bool WritesMemory = IK == IK_WritesMemory; 1985249259Sdim assert(!(OnlyReadsMemory && WritesMemory)); 1986249259Sdim 1987249259Sdim if (NumArgOperands == 2 && 1988249259Sdim I.getArgOperand(0)->getType()->isPointerTy() && 1989249259Sdim I.getArgOperand(1)->getType()->isVectorTy() && 1990249259Sdim I.getType()->isVoidTy() && 1991249259Sdim WritesMemory) { 1992249259Sdim // This looks like a vector store. 1993249259Sdim return handleVectorStoreIntrinsic(I); 1994249259Sdim } 1995249259Sdim 1996249259Sdim if (NumArgOperands == 1 && 1997249259Sdim I.getArgOperand(0)->getType()->isPointerTy() && 1998249259Sdim I.getType()->isVectorTy() && 1999249259Sdim OnlyReadsMemory) { 2000249259Sdim // This looks like a vector load. 2001249259Sdim return handleVectorLoadIntrinsic(I); 2002249259Sdim } 2003249259Sdim 2004249259Sdim if (!OnlyReadsMemory && !WritesMemory) 2005249259Sdim if (maybeHandleSimpleNomemIntrinsic(I)) 2006249259Sdim return true; 2007249259Sdim 2008249259Sdim // FIXME: detect and handle SSE maskstore/maskload 2009249259Sdim return false; 2010249259Sdim } 2011249259Sdim 2012249259Sdim void handleBswap(IntrinsicInst &I) { 2013249259Sdim IRBuilder<> IRB(&I); 2014249259Sdim Value *Op = I.getArgOperand(0); 2015249259Sdim Type *OpType = Op->getType(); 2016249259Sdim Function *BswapFunc = Intrinsic::getDeclaration( 2017280031Sdim F.getParent(), Intrinsic::bswap, makeArrayRef(&OpType, 1)); 2018249259Sdim setShadow(&I, IRB.CreateCall(BswapFunc, getShadow(Op))); 2019249259Sdim setOrigin(&I, getOrigin(Op)); 2020249259Sdim } 2021249259Sdim 2022261991Sdim // \brief Instrument vector convert instrinsic. 2023261991Sdim // 2024261991Sdim // This function instruments intrinsics like cvtsi2ss: 2025261991Sdim // %Out = int_xxx_cvtyyy(%ConvertOp) 2026261991Sdim // or 2027261991Sdim // %Out = int_xxx_cvtyyy(%CopyOp, %ConvertOp) 2028261991Sdim // Intrinsic converts \p NumUsedElements elements of \p ConvertOp to the same 2029261991Sdim // number \p Out elements, and (if has 2 arguments) copies the rest of the 2030261991Sdim // elements from \p CopyOp. 2031261991Sdim // In most cases conversion involves floating-point value which may trigger a 2032261991Sdim // hardware exception when not fully initialized. For this reason we require 2033261991Sdim // \p ConvertOp[0:NumUsedElements] to be fully initialized and trap otherwise. 2034261991Sdim // We copy the shadow of \p CopyOp[NumUsedElements:] to \p 2035261991Sdim // Out[NumUsedElements:]. This means that intrinsics without \p CopyOp always 2036261991Sdim // return a fully initialized value. 2037261991Sdim void handleVectorConvertIntrinsic(IntrinsicInst &I, int NumUsedElements) { 2038261991Sdim IRBuilder<> IRB(&I); 2039261991Sdim Value *CopyOp, *ConvertOp; 2040261991Sdim 2041261991Sdim switch (I.getNumArgOperands()) { 2042288943Sdim case 3: 2043288943Sdim assert(isa<ConstantInt>(I.getArgOperand(2)) && "Invalid rounding mode"); 2044261991Sdim case 2: 2045261991Sdim CopyOp = I.getArgOperand(0); 2046261991Sdim ConvertOp = I.getArgOperand(1); 2047261991Sdim break; 2048261991Sdim case 1: 2049261991Sdim ConvertOp = I.getArgOperand(0); 2050276479Sdim CopyOp = nullptr; 2051261991Sdim break; 2052261991Sdim default: 2053261991Sdim llvm_unreachable("Cvt intrinsic with unsupported number of arguments."); 2054261991Sdim } 2055261991Sdim 2056261991Sdim // The first *NumUsedElements* elements of ConvertOp are converted to the 2057261991Sdim // same number of output elements. The rest of the output is copied from 2058261991Sdim // CopyOp, or (if not available) filled with zeroes. 2059261991Sdim // Combine shadow for elements of ConvertOp that are used in this operation, 2060261991Sdim // and insert a check. 2061261991Sdim // FIXME: consider propagating shadow of ConvertOp, at least in the case of 2062261991Sdim // int->any conversion. 2063261991Sdim Value *ConvertShadow = getShadow(ConvertOp); 2064276479Sdim Value *AggShadow = nullptr; 2065261991Sdim if (ConvertOp->getType()->isVectorTy()) { 2066261991Sdim AggShadow = IRB.CreateExtractElement( 2067261991Sdim ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), 0)); 2068261991Sdim for (int i = 1; i < NumUsedElements; ++i) { 2069261991Sdim Value *MoreShadow = IRB.CreateExtractElement( 2070261991Sdim ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), i)); 2071261991Sdim AggShadow = IRB.CreateOr(AggShadow, MoreShadow); 2072261991Sdim } 2073261991Sdim } else { 2074261991Sdim AggShadow = ConvertShadow; 2075261991Sdim } 2076261991Sdim assert(AggShadow->getType()->isIntegerTy()); 2077261991Sdim insertShadowCheck(AggShadow, getOrigin(ConvertOp), &I); 2078261991Sdim 2079261991Sdim // Build result shadow by zero-filling parts of CopyOp shadow that come from 2080261991Sdim // ConvertOp. 2081261991Sdim if (CopyOp) { 2082261991Sdim assert(CopyOp->getType() == I.getType()); 2083261991Sdim assert(CopyOp->getType()->isVectorTy()); 2084261991Sdim Value *ResultShadow = getShadow(CopyOp); 2085261991Sdim Type *EltTy = ResultShadow->getType()->getVectorElementType(); 2086261991Sdim for (int i = 0; i < NumUsedElements; ++i) { 2087261991Sdim ResultShadow = IRB.CreateInsertElement( 2088261991Sdim ResultShadow, ConstantInt::getNullValue(EltTy), 2089261991Sdim ConstantInt::get(IRB.getInt32Ty(), i)); 2090261991Sdim } 2091261991Sdim setShadow(&I, ResultShadow); 2092261991Sdim setOrigin(&I, getOrigin(CopyOp)); 2093261991Sdim } else { 2094261991Sdim setShadow(&I, getCleanShadow(&I)); 2095280031Sdim setOrigin(&I, getCleanOrigin()); 2096261991Sdim } 2097261991Sdim } 2098261991Sdim 2099276479Sdim // Given a scalar or vector, extract lower 64 bits (or less), and return all 2100276479Sdim // zeroes if it is zero, and all ones otherwise. 2101276479Sdim Value *Lower64ShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) { 2102276479Sdim if (S->getType()->isVectorTy()) 2103276479Sdim S = CreateShadowCast(IRB, S, IRB.getInt64Ty(), /* Signed */ true); 2104276479Sdim assert(S->getType()->getPrimitiveSizeInBits() <= 64); 2105276479Sdim Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S)); 2106276479Sdim return CreateShadowCast(IRB, S2, T, /* Signed */ true); 2107276479Sdim } 2108276479Sdim 2109276479Sdim Value *VariableShadowExtend(IRBuilder<> &IRB, Value *S) { 2110276479Sdim Type *T = S->getType(); 2111276479Sdim assert(T->isVectorTy()); 2112276479Sdim Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S)); 2113276479Sdim return IRB.CreateSExt(S2, T); 2114276479Sdim } 2115276479Sdim 2116276479Sdim // \brief Instrument vector shift instrinsic. 2117276479Sdim // 2118276479Sdim // This function instruments intrinsics like int_x86_avx2_psll_w. 2119276479Sdim // Intrinsic shifts %In by %ShiftSize bits. 2120276479Sdim // %ShiftSize may be a vector. In that case the lower 64 bits determine shift 2121276479Sdim // size, and the rest is ignored. Behavior is defined even if shift size is 2122276479Sdim // greater than register (or field) width. 2123276479Sdim void handleVectorShiftIntrinsic(IntrinsicInst &I, bool Variable) { 2124276479Sdim assert(I.getNumArgOperands() == 2); 2125276479Sdim IRBuilder<> IRB(&I); 2126276479Sdim // If any of the S2 bits are poisoned, the whole thing is poisoned. 2127276479Sdim // Otherwise perform the same shift on S1. 2128276479Sdim Value *S1 = getShadow(&I, 0); 2129276479Sdim Value *S2 = getShadow(&I, 1); 2130276479Sdim Value *S2Conv = Variable ? VariableShadowExtend(IRB, S2) 2131276479Sdim : Lower64ShadowExtend(IRB, S2, getShadowTy(&I)); 2132276479Sdim Value *V1 = I.getOperand(0); 2133276479Sdim Value *V2 = I.getOperand(1); 2134288943Sdim Value *Shift = IRB.CreateCall(I.getCalledValue(), 2135288943Sdim {IRB.CreateBitCast(S1, V1->getType()), V2}); 2136276479Sdim Shift = IRB.CreateBitCast(Shift, getShadowTy(&I)); 2137276479Sdim setShadow(&I, IRB.CreateOr(Shift, S2Conv)); 2138276479Sdim setOriginForNaryOp(I); 2139276479Sdim } 2140276479Sdim 2141276479Sdim // \brief Get an X86_MMX-sized vector type. 2142276479Sdim Type *getMMXVectorTy(unsigned EltSizeInBits) { 2143276479Sdim const unsigned X86_MMXSizeInBits = 64; 2144276479Sdim return VectorType::get(IntegerType::get(*MS.C, EltSizeInBits), 2145276479Sdim X86_MMXSizeInBits / EltSizeInBits); 2146276479Sdim } 2147276479Sdim 2148276479Sdim // \brief Returns a signed counterpart for an (un)signed-saturate-and-pack 2149276479Sdim // intrinsic. 2150276479Sdim Intrinsic::ID getSignedPackIntrinsic(Intrinsic::ID id) { 2151276479Sdim switch (id) { 2152276479Sdim case llvm::Intrinsic::x86_sse2_packsswb_128: 2153276479Sdim case llvm::Intrinsic::x86_sse2_packuswb_128: 2154276479Sdim return llvm::Intrinsic::x86_sse2_packsswb_128; 2155276479Sdim 2156276479Sdim case llvm::Intrinsic::x86_sse2_packssdw_128: 2157276479Sdim case llvm::Intrinsic::x86_sse41_packusdw: 2158276479Sdim return llvm::Intrinsic::x86_sse2_packssdw_128; 2159276479Sdim 2160276479Sdim case llvm::Intrinsic::x86_avx2_packsswb: 2161276479Sdim case llvm::Intrinsic::x86_avx2_packuswb: 2162276479Sdim return llvm::Intrinsic::x86_avx2_packsswb; 2163276479Sdim 2164276479Sdim case llvm::Intrinsic::x86_avx2_packssdw: 2165276479Sdim case llvm::Intrinsic::x86_avx2_packusdw: 2166276479Sdim return llvm::Intrinsic::x86_avx2_packssdw; 2167276479Sdim 2168276479Sdim case llvm::Intrinsic::x86_mmx_packsswb: 2169276479Sdim case llvm::Intrinsic::x86_mmx_packuswb: 2170276479Sdim return llvm::Intrinsic::x86_mmx_packsswb; 2171276479Sdim 2172276479Sdim case llvm::Intrinsic::x86_mmx_packssdw: 2173276479Sdim return llvm::Intrinsic::x86_mmx_packssdw; 2174276479Sdim default: 2175276479Sdim llvm_unreachable("unexpected intrinsic id"); 2176276479Sdim } 2177276479Sdim } 2178276479Sdim 2179276479Sdim // \brief Instrument vector pack instrinsic. 2180276479Sdim // 2181276479Sdim // This function instruments intrinsics like x86_mmx_packsswb, that 2182276479Sdim // packs elements of 2 input vectors into half as many bits with saturation. 2183276479Sdim // Shadow is propagated with the signed variant of the same intrinsic applied 2184276479Sdim // to sext(Sa != zeroinitializer), sext(Sb != zeroinitializer). 2185276479Sdim // EltSizeInBits is used only for x86mmx arguments. 2186276479Sdim void handleVectorPackIntrinsic(IntrinsicInst &I, unsigned EltSizeInBits = 0) { 2187276479Sdim assert(I.getNumArgOperands() == 2); 2188276479Sdim bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy(); 2189276479Sdim IRBuilder<> IRB(&I); 2190276479Sdim Value *S1 = getShadow(&I, 0); 2191276479Sdim Value *S2 = getShadow(&I, 1); 2192276479Sdim assert(isX86_MMX || S1->getType()->isVectorTy()); 2193276479Sdim 2194276479Sdim // SExt and ICmpNE below must apply to individual elements of input vectors. 2195276479Sdim // In case of x86mmx arguments, cast them to appropriate vector types and 2196276479Sdim // back. 2197276479Sdim Type *T = isX86_MMX ? getMMXVectorTy(EltSizeInBits) : S1->getType(); 2198276479Sdim if (isX86_MMX) { 2199276479Sdim S1 = IRB.CreateBitCast(S1, T); 2200276479Sdim S2 = IRB.CreateBitCast(S2, T); 2201276479Sdim } 2202276479Sdim Value *S1_ext = IRB.CreateSExt( 2203276479Sdim IRB.CreateICmpNE(S1, llvm::Constant::getNullValue(T)), T); 2204276479Sdim Value *S2_ext = IRB.CreateSExt( 2205276479Sdim IRB.CreateICmpNE(S2, llvm::Constant::getNullValue(T)), T); 2206276479Sdim if (isX86_MMX) { 2207276479Sdim Type *X86_MMXTy = Type::getX86_MMXTy(*MS.C); 2208276479Sdim S1_ext = IRB.CreateBitCast(S1_ext, X86_MMXTy); 2209276479Sdim S2_ext = IRB.CreateBitCast(S2_ext, X86_MMXTy); 2210276479Sdim } 2211276479Sdim 2212276479Sdim Function *ShadowFn = Intrinsic::getDeclaration( 2213276479Sdim F.getParent(), getSignedPackIntrinsic(I.getIntrinsicID())); 2214276479Sdim 2215288943Sdim Value *S = 2216288943Sdim IRB.CreateCall(ShadowFn, {S1_ext, S2_ext}, "_msprop_vector_pack"); 2217276479Sdim if (isX86_MMX) S = IRB.CreateBitCast(S, getShadowTy(&I)); 2218276479Sdim setShadow(&I, S); 2219276479Sdim setOriginForNaryOp(I); 2220276479Sdim } 2221276479Sdim 2222276479Sdim // \brief Instrument sum-of-absolute-differencies intrinsic. 2223276479Sdim void handleVectorSadIntrinsic(IntrinsicInst &I) { 2224276479Sdim const unsigned SignificantBitsPerResultElement = 16; 2225276479Sdim bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy(); 2226276479Sdim Type *ResTy = isX86_MMX ? IntegerType::get(*MS.C, 64) : I.getType(); 2227276479Sdim unsigned ZeroBitsPerResultElement = 2228276479Sdim ResTy->getScalarSizeInBits() - SignificantBitsPerResultElement; 2229276479Sdim 2230276479Sdim IRBuilder<> IRB(&I); 2231276479Sdim Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1)); 2232276479Sdim S = IRB.CreateBitCast(S, ResTy); 2233276479Sdim S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)), 2234276479Sdim ResTy); 2235276479Sdim S = IRB.CreateLShr(S, ZeroBitsPerResultElement); 2236276479Sdim S = IRB.CreateBitCast(S, getShadowTy(&I)); 2237276479Sdim setShadow(&I, S); 2238276479Sdim setOriginForNaryOp(I); 2239276479Sdim } 2240276479Sdim 2241276479Sdim // \brief Instrument multiply-add intrinsic. 2242276479Sdim void handleVectorPmaddIntrinsic(IntrinsicInst &I, 2243276479Sdim unsigned EltSizeInBits = 0) { 2244276479Sdim bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy(); 2245276479Sdim Type *ResTy = isX86_MMX ? getMMXVectorTy(EltSizeInBits * 2) : I.getType(); 2246276479Sdim IRBuilder<> IRB(&I); 2247276479Sdim Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1)); 2248276479Sdim S = IRB.CreateBitCast(S, ResTy); 2249276479Sdim S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)), 2250276479Sdim ResTy); 2251276479Sdim S = IRB.CreateBitCast(S, getShadowTy(&I)); 2252276479Sdim setShadow(&I, S); 2253276479Sdim setOriginForNaryOp(I); 2254276479Sdim } 2255276479Sdim 2256249259Sdim void visitIntrinsicInst(IntrinsicInst &I) { 2257249259Sdim switch (I.getIntrinsicID()) { 2258249259Sdim case llvm::Intrinsic::bswap: 2259249259Sdim handleBswap(I); 2260249259Sdim break; 2261261991Sdim case llvm::Intrinsic::x86_avx512_cvtsd2usi64: 2262261991Sdim case llvm::Intrinsic::x86_avx512_cvtsd2usi: 2263261991Sdim case llvm::Intrinsic::x86_avx512_cvtss2usi64: 2264261991Sdim case llvm::Intrinsic::x86_avx512_cvtss2usi: 2265261991Sdim case llvm::Intrinsic::x86_avx512_cvttss2usi64: 2266261991Sdim case llvm::Intrinsic::x86_avx512_cvttss2usi: 2267261991Sdim case llvm::Intrinsic::x86_avx512_cvttsd2usi64: 2268261991Sdim case llvm::Intrinsic::x86_avx512_cvttsd2usi: 2269261991Sdim case llvm::Intrinsic::x86_avx512_cvtusi2sd: 2270261991Sdim case llvm::Intrinsic::x86_avx512_cvtusi2ss: 2271261991Sdim case llvm::Intrinsic::x86_avx512_cvtusi642sd: 2272261991Sdim case llvm::Intrinsic::x86_avx512_cvtusi642ss: 2273261991Sdim case llvm::Intrinsic::x86_sse2_cvtsd2si64: 2274261991Sdim case llvm::Intrinsic::x86_sse2_cvtsd2si: 2275261991Sdim case llvm::Intrinsic::x86_sse2_cvtsd2ss: 2276261991Sdim case llvm::Intrinsic::x86_sse2_cvtsi2sd: 2277261991Sdim case llvm::Intrinsic::x86_sse2_cvtsi642sd: 2278261991Sdim case llvm::Intrinsic::x86_sse2_cvtss2sd: 2279261991Sdim case llvm::Intrinsic::x86_sse2_cvttsd2si64: 2280261991Sdim case llvm::Intrinsic::x86_sse2_cvttsd2si: 2281261991Sdim case llvm::Intrinsic::x86_sse_cvtsi2ss: 2282261991Sdim case llvm::Intrinsic::x86_sse_cvtsi642ss: 2283261991Sdim case llvm::Intrinsic::x86_sse_cvtss2si64: 2284261991Sdim case llvm::Intrinsic::x86_sse_cvtss2si: 2285261991Sdim case llvm::Intrinsic::x86_sse_cvttss2si64: 2286261991Sdim case llvm::Intrinsic::x86_sse_cvttss2si: 2287261991Sdim handleVectorConvertIntrinsic(I, 1); 2288261991Sdim break; 2289261991Sdim case llvm::Intrinsic::x86_sse2_cvtdq2pd: 2290261991Sdim case llvm::Intrinsic::x86_sse2_cvtps2pd: 2291261991Sdim case llvm::Intrinsic::x86_sse_cvtps2pi: 2292261991Sdim case llvm::Intrinsic::x86_sse_cvttps2pi: 2293261991Sdim handleVectorConvertIntrinsic(I, 2); 2294261991Sdim break; 2295276479Sdim case llvm::Intrinsic::x86_avx2_psll_w: 2296276479Sdim case llvm::Intrinsic::x86_avx2_psll_d: 2297276479Sdim case llvm::Intrinsic::x86_avx2_psll_q: 2298276479Sdim case llvm::Intrinsic::x86_avx2_pslli_w: 2299276479Sdim case llvm::Intrinsic::x86_avx2_pslli_d: 2300276479Sdim case llvm::Intrinsic::x86_avx2_pslli_q: 2301276479Sdim case llvm::Intrinsic::x86_avx2_psrl_w: 2302276479Sdim case llvm::Intrinsic::x86_avx2_psrl_d: 2303276479Sdim case llvm::Intrinsic::x86_avx2_psrl_q: 2304276479Sdim case llvm::Intrinsic::x86_avx2_psra_w: 2305276479Sdim case llvm::Intrinsic::x86_avx2_psra_d: 2306276479Sdim case llvm::Intrinsic::x86_avx2_psrli_w: 2307276479Sdim case llvm::Intrinsic::x86_avx2_psrli_d: 2308276479Sdim case llvm::Intrinsic::x86_avx2_psrli_q: 2309276479Sdim case llvm::Intrinsic::x86_avx2_psrai_w: 2310276479Sdim case llvm::Intrinsic::x86_avx2_psrai_d: 2311276479Sdim case llvm::Intrinsic::x86_sse2_psll_w: 2312276479Sdim case llvm::Intrinsic::x86_sse2_psll_d: 2313276479Sdim case llvm::Intrinsic::x86_sse2_psll_q: 2314276479Sdim case llvm::Intrinsic::x86_sse2_pslli_w: 2315276479Sdim case llvm::Intrinsic::x86_sse2_pslli_d: 2316276479Sdim case llvm::Intrinsic::x86_sse2_pslli_q: 2317276479Sdim case llvm::Intrinsic::x86_sse2_psrl_w: 2318276479Sdim case llvm::Intrinsic::x86_sse2_psrl_d: 2319276479Sdim case llvm::Intrinsic::x86_sse2_psrl_q: 2320276479Sdim case llvm::Intrinsic::x86_sse2_psra_w: 2321276479Sdim case llvm::Intrinsic::x86_sse2_psra_d: 2322276479Sdim case llvm::Intrinsic::x86_sse2_psrli_w: 2323276479Sdim case llvm::Intrinsic::x86_sse2_psrli_d: 2324276479Sdim case llvm::Intrinsic::x86_sse2_psrli_q: 2325276479Sdim case llvm::Intrinsic::x86_sse2_psrai_w: 2326276479Sdim case llvm::Intrinsic::x86_sse2_psrai_d: 2327276479Sdim case llvm::Intrinsic::x86_mmx_psll_w: 2328276479Sdim case llvm::Intrinsic::x86_mmx_psll_d: 2329276479Sdim case llvm::Intrinsic::x86_mmx_psll_q: 2330276479Sdim case llvm::Intrinsic::x86_mmx_pslli_w: 2331276479Sdim case llvm::Intrinsic::x86_mmx_pslli_d: 2332276479Sdim case llvm::Intrinsic::x86_mmx_pslli_q: 2333276479Sdim case llvm::Intrinsic::x86_mmx_psrl_w: 2334276479Sdim case llvm::Intrinsic::x86_mmx_psrl_d: 2335276479Sdim case llvm::Intrinsic::x86_mmx_psrl_q: 2336276479Sdim case llvm::Intrinsic::x86_mmx_psra_w: 2337276479Sdim case llvm::Intrinsic::x86_mmx_psra_d: 2338276479Sdim case llvm::Intrinsic::x86_mmx_psrli_w: 2339276479Sdim case llvm::Intrinsic::x86_mmx_psrli_d: 2340276479Sdim case llvm::Intrinsic::x86_mmx_psrli_q: 2341276479Sdim case llvm::Intrinsic::x86_mmx_psrai_w: 2342276479Sdim case llvm::Intrinsic::x86_mmx_psrai_d: 2343276479Sdim handleVectorShiftIntrinsic(I, /* Variable */ false); 2344276479Sdim break; 2345276479Sdim case llvm::Intrinsic::x86_avx2_psllv_d: 2346276479Sdim case llvm::Intrinsic::x86_avx2_psllv_d_256: 2347276479Sdim case llvm::Intrinsic::x86_avx2_psllv_q: 2348276479Sdim case llvm::Intrinsic::x86_avx2_psllv_q_256: 2349276479Sdim case llvm::Intrinsic::x86_avx2_psrlv_d: 2350276479Sdim case llvm::Intrinsic::x86_avx2_psrlv_d_256: 2351276479Sdim case llvm::Intrinsic::x86_avx2_psrlv_q: 2352276479Sdim case llvm::Intrinsic::x86_avx2_psrlv_q_256: 2353276479Sdim case llvm::Intrinsic::x86_avx2_psrav_d: 2354276479Sdim case llvm::Intrinsic::x86_avx2_psrav_d_256: 2355276479Sdim handleVectorShiftIntrinsic(I, /* Variable */ true); 2356276479Sdim break; 2357276479Sdim 2358276479Sdim case llvm::Intrinsic::x86_sse2_packsswb_128: 2359276479Sdim case llvm::Intrinsic::x86_sse2_packssdw_128: 2360276479Sdim case llvm::Intrinsic::x86_sse2_packuswb_128: 2361276479Sdim case llvm::Intrinsic::x86_sse41_packusdw: 2362276479Sdim case llvm::Intrinsic::x86_avx2_packsswb: 2363276479Sdim case llvm::Intrinsic::x86_avx2_packssdw: 2364276479Sdim case llvm::Intrinsic::x86_avx2_packuswb: 2365276479Sdim case llvm::Intrinsic::x86_avx2_packusdw: 2366276479Sdim handleVectorPackIntrinsic(I); 2367276479Sdim break; 2368276479Sdim 2369276479Sdim case llvm::Intrinsic::x86_mmx_packsswb: 2370276479Sdim case llvm::Intrinsic::x86_mmx_packuswb: 2371276479Sdim handleVectorPackIntrinsic(I, 16); 2372276479Sdim break; 2373276479Sdim 2374276479Sdim case llvm::Intrinsic::x86_mmx_packssdw: 2375276479Sdim handleVectorPackIntrinsic(I, 32); 2376276479Sdim break; 2377276479Sdim 2378276479Sdim case llvm::Intrinsic::x86_mmx_psad_bw: 2379276479Sdim case llvm::Intrinsic::x86_sse2_psad_bw: 2380276479Sdim case llvm::Intrinsic::x86_avx2_psad_bw: 2381276479Sdim handleVectorSadIntrinsic(I); 2382276479Sdim break; 2383276479Sdim 2384276479Sdim case llvm::Intrinsic::x86_sse2_pmadd_wd: 2385276479Sdim case llvm::Intrinsic::x86_avx2_pmadd_wd: 2386276479Sdim case llvm::Intrinsic::x86_ssse3_pmadd_ub_sw_128: 2387276479Sdim case llvm::Intrinsic::x86_avx2_pmadd_ub_sw: 2388276479Sdim handleVectorPmaddIntrinsic(I); 2389276479Sdim break; 2390276479Sdim 2391276479Sdim case llvm::Intrinsic::x86_ssse3_pmadd_ub_sw: 2392276479Sdim handleVectorPmaddIntrinsic(I, 8); 2393276479Sdim break; 2394276479Sdim 2395276479Sdim case llvm::Intrinsic::x86_mmx_pmadd_wd: 2396276479Sdim handleVectorPmaddIntrinsic(I, 16); 2397276479Sdim break; 2398276479Sdim 2399249259Sdim default: 2400249259Sdim if (!handleUnknownIntrinsic(I)) 2401249259Sdim visitInstruction(I); 2402249259Sdim break; 2403249259Sdim } 2404249259Sdim } 2405249259Sdim 2406249259Sdim void visitCallSite(CallSite CS) { 2407249259Sdim Instruction &I = *CS.getInstruction(); 2408249259Sdim assert((CS.isCall() || CS.isInvoke()) && "Unknown type of CallSite"); 2409249259Sdim if (CS.isCall()) { 2410249259Sdim CallInst *Call = cast<CallInst>(&I); 2411249259Sdim 2412249259Sdim // For inline asm, do the usual thing: check argument shadow and mark all 2413249259Sdim // outputs as clean. Note that any side effects of the inline asm that are 2414249259Sdim // not immediately visible in its constraints are not handled. 2415249259Sdim if (Call->isInlineAsm()) { 2416249259Sdim visitInstruction(I); 2417249259Sdim return; 2418249259Sdim } 2419249259Sdim 2420249259Sdim assert(!isa<IntrinsicInst>(&I) && "intrinsics are handled elsewhere"); 2421249259Sdim 2422249259Sdim // We are going to insert code that relies on the fact that the callee 2423249259Sdim // will become a non-readonly function after it is instrumented by us. To 2424249259Sdim // prevent this code from being optimized out, mark that function 2425249259Sdim // non-readonly in advance. 2426249259Sdim if (Function *Func = Call->getCalledFunction()) { 2427249259Sdim // Clear out readonly/readnone attributes. 2428249259Sdim AttrBuilder B; 2429249259Sdim B.addAttribute(Attribute::ReadOnly) 2430249259Sdim .addAttribute(Attribute::ReadNone); 2431249259Sdim Func->removeAttributes(AttributeSet::FunctionIndex, 2432249259Sdim AttributeSet::get(Func->getContext(), 2433249259Sdim AttributeSet::FunctionIndex, 2434249259Sdim B)); 2435249259Sdim } 2436249259Sdim } 2437249259Sdim IRBuilder<> IRB(&I); 2438261991Sdim 2439249259Sdim unsigned ArgOffset = 0; 2440249259Sdim DEBUG(dbgs() << " CallSite: " << I << "\n"); 2441249259Sdim for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end(); 2442249259Sdim ArgIt != End; ++ArgIt) { 2443249259Sdim Value *A = *ArgIt; 2444249259Sdim unsigned i = ArgIt - CS.arg_begin(); 2445249259Sdim if (!A->getType()->isSized()) { 2446249259Sdim DEBUG(dbgs() << "Arg " << i << " is not sized: " << I << "\n"); 2447249259Sdim continue; 2448249259Sdim } 2449249259Sdim unsigned Size = 0; 2450276479Sdim Value *Store = nullptr; 2451249259Sdim // Compute the Shadow for arg even if it is ByVal, because 2452249259Sdim // in that case getShadow() will copy the actual arg shadow to 2453249259Sdim // __msan_param_tls. 2454249259Sdim Value *ArgShadow = getShadow(A); 2455249259Sdim Value *ArgShadowBase = getShadowPtrForArgument(A, IRB, ArgOffset); 2456249259Sdim DEBUG(dbgs() << " Arg#" << i << ": " << *A << 2457249259Sdim " Shadow: " << *ArgShadow << "\n"); 2458276479Sdim bool ArgIsInitialized = false; 2459288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 2460249259Sdim if (CS.paramHasAttr(i + 1, Attribute::ByVal)) { 2461249259Sdim assert(A->getType()->isPointerTy() && 2462249259Sdim "ByVal argument is not a pointer!"); 2463288943Sdim Size = DL.getTypeAllocSize(A->getType()->getPointerElementType()); 2464280031Sdim if (ArgOffset + Size > kParamTLSSize) break; 2465280031Sdim unsigned ParamAlignment = CS.getParamAlignment(i + 1); 2466280031Sdim unsigned Alignment = std::min(ParamAlignment, kShadowTLSAlignment); 2467249259Sdim Store = IRB.CreateMemCpy(ArgShadowBase, 2468249259Sdim getShadowPtr(A, Type::getInt8Ty(*MS.C), IRB), 2469249259Sdim Size, Alignment); 2470249259Sdim } else { 2471288943Sdim Size = DL.getTypeAllocSize(A->getType()); 2472280031Sdim if (ArgOffset + Size > kParamTLSSize) break; 2473249259Sdim Store = IRB.CreateAlignedStore(ArgShadow, ArgShadowBase, 2474249259Sdim kShadowTLSAlignment); 2475276479Sdim Constant *Cst = dyn_cast<Constant>(ArgShadow); 2476276479Sdim if (Cst && Cst->isNullValue()) ArgIsInitialized = true; 2477249259Sdim } 2478276479Sdim if (MS.TrackOrigins && !ArgIsInitialized) 2479249259Sdim IRB.CreateStore(getOrigin(A), 2480249259Sdim getOriginPtrForArgument(A, IRB, ArgOffset)); 2481249259Sdim (void)Store; 2482276479Sdim assert(Size != 0 && Store != nullptr); 2483249259Sdim DEBUG(dbgs() << " Param:" << *Store << "\n"); 2484280031Sdim ArgOffset += RoundUpToAlignment(Size, 8); 2485249259Sdim } 2486249259Sdim DEBUG(dbgs() << " done with call args\n"); 2487249259Sdim 2488249259Sdim FunctionType *FT = 2489261991Sdim cast<FunctionType>(CS.getCalledValue()->getType()->getContainedType(0)); 2490249259Sdim if (FT->isVarArg()) { 2491249259Sdim VAHelper->visitCallSite(CS, IRB); 2492249259Sdim } 2493249259Sdim 2494249259Sdim // Now, get the shadow for the RetVal. 2495249259Sdim if (!I.getType()->isSized()) return; 2496249259Sdim IRBuilder<> IRBBefore(&I); 2497276479Sdim // Until we have full dynamic coverage, make sure the retval shadow is 0. 2498249259Sdim Value *Base = getShadowPtrForRetval(&I, IRBBefore); 2499249259Sdim IRBBefore.CreateAlignedStore(getCleanShadow(&I), Base, kShadowTLSAlignment); 2500276479Sdim Instruction *NextInsn = nullptr; 2501249259Sdim if (CS.isCall()) { 2502249259Sdim NextInsn = I.getNextNode(); 2503249259Sdim } else { 2504249259Sdim BasicBlock *NormalDest = cast<InvokeInst>(&I)->getNormalDest(); 2505249259Sdim if (!NormalDest->getSinglePredecessor()) { 2506249259Sdim // FIXME: this case is tricky, so we are just conservative here. 2507249259Sdim // Perhaps we need to split the edge between this BB and NormalDest, 2508249259Sdim // but a naive attempt to use SplitEdge leads to a crash. 2509249259Sdim setShadow(&I, getCleanShadow(&I)); 2510249259Sdim setOrigin(&I, getCleanOrigin()); 2511249259Sdim return; 2512249259Sdim } 2513249259Sdim NextInsn = NormalDest->getFirstInsertionPt(); 2514249259Sdim assert(NextInsn && 2515249259Sdim "Could not find insertion point for retval shadow load"); 2516249259Sdim } 2517249259Sdim IRBuilder<> IRBAfter(NextInsn); 2518249259Sdim Value *RetvalShadow = 2519249259Sdim IRBAfter.CreateAlignedLoad(getShadowPtrForRetval(&I, IRBAfter), 2520249259Sdim kShadowTLSAlignment, "_msret"); 2521249259Sdim setShadow(&I, RetvalShadow); 2522249259Sdim if (MS.TrackOrigins) 2523249259Sdim setOrigin(&I, IRBAfter.CreateLoad(getOriginPtrForRetval(IRBAfter))); 2524249259Sdim } 2525249259Sdim 2526249259Sdim void visitReturnInst(ReturnInst &I) { 2527249259Sdim IRBuilder<> IRB(&I); 2528261991Sdim Value *RetVal = I.getReturnValue(); 2529261991Sdim if (!RetVal) return; 2530261991Sdim Value *ShadowPtr = getShadowPtrForRetval(RetVal, IRB); 2531261991Sdim if (CheckReturnValue) { 2532261991Sdim insertShadowCheck(RetVal, &I); 2533261991Sdim Value *Shadow = getCleanShadow(RetVal); 2534261991Sdim IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment); 2535261991Sdim } else { 2536249259Sdim Value *Shadow = getShadow(RetVal); 2537249259Sdim IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment); 2538261991Sdim // FIXME: make it conditional if ClStoreCleanOrigin==0 2539249259Sdim if (MS.TrackOrigins) 2540249259Sdim IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval(IRB)); 2541249259Sdim } 2542249259Sdim } 2543249259Sdim 2544249259Sdim void visitPHINode(PHINode &I) { 2545249259Sdim IRBuilder<> IRB(&I); 2546276479Sdim if (!PropagateShadow) { 2547276479Sdim setShadow(&I, getCleanShadow(&I)); 2548280031Sdim setOrigin(&I, getCleanOrigin()); 2549276479Sdim return; 2550276479Sdim } 2551276479Sdim 2552249259Sdim ShadowPHINodes.push_back(&I); 2553249259Sdim setShadow(&I, IRB.CreatePHI(getShadowTy(&I), I.getNumIncomingValues(), 2554249259Sdim "_msphi_s")); 2555249259Sdim if (MS.TrackOrigins) 2556249259Sdim setOrigin(&I, IRB.CreatePHI(MS.OriginTy, I.getNumIncomingValues(), 2557249259Sdim "_msphi_o")); 2558249259Sdim } 2559249259Sdim 2560249259Sdim void visitAllocaInst(AllocaInst &I) { 2561249259Sdim setShadow(&I, getCleanShadow(&I)); 2562280031Sdim setOrigin(&I, getCleanOrigin()); 2563249259Sdim IRBuilder<> IRB(I.getNextNode()); 2564288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 2565288943Sdim uint64_t Size = DL.getTypeAllocSize(I.getAllocatedType()); 2566261991Sdim if (PoisonStack && ClPoisonStackWithCall) { 2567288943Sdim IRB.CreateCall(MS.MsanPoisonStackFn, 2568288943Sdim {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), 2569288943Sdim ConstantInt::get(MS.IntptrTy, Size)}); 2570249259Sdim } else { 2571249259Sdim Value *ShadowBase = getShadowPtr(&I, Type::getInt8PtrTy(*MS.C), IRB); 2572261991Sdim Value *PoisonValue = IRB.getInt8(PoisonStack ? ClPoisonStackPattern : 0); 2573261991Sdim IRB.CreateMemSet(ShadowBase, PoisonValue, Size, I.getAlignment()); 2574249259Sdim } 2575249259Sdim 2576261991Sdim if (PoisonStack && MS.TrackOrigins) { 2577249259Sdim SmallString<2048> StackDescriptionStorage; 2578249259Sdim raw_svector_ostream StackDescription(StackDescriptionStorage); 2579249259Sdim // We create a string with a description of the stack allocation and 2580249259Sdim // pass it into __msan_set_alloca_origin. 2581249259Sdim // It will be printed by the run-time if stack-originated UMR is found. 2582249259Sdim // The first 4 bytes of the string are set to '----' and will be replaced 2583249259Sdim // by __msan_va_arg_overflow_size_tls at the first call. 2584249259Sdim StackDescription << "----" << I.getName() << "@" << F.getName(); 2585249259Sdim Value *Descr = 2586249259Sdim createPrivateNonConstGlobalForString(*F.getParent(), 2587249259Sdim StackDescription.str()); 2588261991Sdim 2589288943Sdim IRB.CreateCall(MS.MsanSetAllocaOrigin4Fn, 2590288943Sdim {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), 2591249259Sdim ConstantInt::get(MS.IntptrTy, Size), 2592261991Sdim IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy()), 2593288943Sdim IRB.CreatePointerCast(&F, MS.IntptrTy)}); 2594249259Sdim } 2595249259Sdim } 2596249259Sdim 2597249259Sdim void visitSelectInst(SelectInst& I) { 2598249259Sdim IRBuilder<> IRB(&I); 2599261991Sdim // a = select b, c, d 2600276479Sdim Value *B = I.getCondition(); 2601276479Sdim Value *C = I.getTrueValue(); 2602276479Sdim Value *D = I.getFalseValue(); 2603276479Sdim Value *Sb = getShadow(B); 2604276479Sdim Value *Sc = getShadow(C); 2605276479Sdim Value *Sd = getShadow(D); 2606276479Sdim 2607276479Sdim // Result shadow if condition shadow is 0. 2608276479Sdim Value *Sa0 = IRB.CreateSelect(B, Sc, Sd); 2609276479Sdim Value *Sa1; 2610261991Sdim if (I.getType()->isAggregateType()) { 2611261991Sdim // To avoid "sign extending" i1 to an arbitrary aggregate type, we just do 2612261991Sdim // an extra "select". This results in much more compact IR. 2613261991Sdim // Sa = select Sb, poisoned, (select b, Sc, Sd) 2614276479Sdim Sa1 = getPoisonedShadow(getShadowTy(I.getType())); 2615261991Sdim } else { 2616276479Sdim // Sa = select Sb, [ (c^d) | Sc | Sd ], [ b ? Sc : Sd ] 2617276479Sdim // If Sb (condition is poisoned), look for bits in c and d that are equal 2618276479Sdim // and both unpoisoned. 2619276479Sdim // If !Sb (condition is unpoisoned), simply pick one of Sc and Sd. 2620276479Sdim 2621276479Sdim // Cast arguments to shadow-compatible type. 2622276479Sdim C = CreateAppToShadowCast(IRB, C); 2623276479Sdim D = CreateAppToShadowCast(IRB, D); 2624276479Sdim 2625276479Sdim // Result shadow if condition shadow is 1. 2626276479Sdim Sa1 = IRB.CreateOr(IRB.CreateXor(C, D), IRB.CreateOr(Sc, Sd)); 2627261991Sdim } 2628276479Sdim Value *Sa = IRB.CreateSelect(Sb, Sa1, Sa0, "_msprop_select"); 2629276479Sdim setShadow(&I, Sa); 2630249259Sdim if (MS.TrackOrigins) { 2631249259Sdim // Origins are always i32, so any vector conditions must be flattened. 2632249259Sdim // FIXME: consider tracking vector origins for app vectors? 2633276479Sdim if (B->getType()->isVectorTy()) { 2634276479Sdim Type *FlatTy = getShadowTyNoVec(B->getType()); 2635276479Sdim B = IRB.CreateICmpNE(IRB.CreateBitCast(B, FlatTy), 2636276479Sdim ConstantInt::getNullValue(FlatTy)); 2637276479Sdim Sb = IRB.CreateICmpNE(IRB.CreateBitCast(Sb, FlatTy), 2638276479Sdim ConstantInt::getNullValue(FlatTy)); 2639249259Sdim } 2640276479Sdim // a = select b, c, d 2641276479Sdim // Oa = Sb ? Ob : (b ? Oc : Od) 2642280031Sdim setOrigin( 2643280031Sdim &I, IRB.CreateSelect(Sb, getOrigin(I.getCondition()), 2644280031Sdim IRB.CreateSelect(B, getOrigin(I.getTrueValue()), 2645280031Sdim getOrigin(I.getFalseValue())))); 2646249259Sdim } 2647249259Sdim } 2648249259Sdim 2649249259Sdim void visitLandingPadInst(LandingPadInst &I) { 2650249259Sdim // Do nothing. 2651249259Sdim // See http://code.google.com/p/memory-sanitizer/issues/detail?id=1 2652249259Sdim setShadow(&I, getCleanShadow(&I)); 2653249259Sdim setOrigin(&I, getCleanOrigin()); 2654249259Sdim } 2655249259Sdim 2656249259Sdim void visitGetElementPtrInst(GetElementPtrInst &I) { 2657249259Sdim handleShadowOr(I); 2658249259Sdim } 2659249259Sdim 2660249259Sdim void visitExtractValueInst(ExtractValueInst &I) { 2661249259Sdim IRBuilder<> IRB(&I); 2662249259Sdim Value *Agg = I.getAggregateOperand(); 2663249259Sdim DEBUG(dbgs() << "ExtractValue: " << I << "\n"); 2664249259Sdim Value *AggShadow = getShadow(Agg); 2665249259Sdim DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n"); 2666249259Sdim Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices()); 2667249259Sdim DEBUG(dbgs() << " ResShadow: " << *ResShadow << "\n"); 2668249259Sdim setShadow(&I, ResShadow); 2669261991Sdim setOriginForNaryOp(I); 2670249259Sdim } 2671249259Sdim 2672249259Sdim void visitInsertValueInst(InsertValueInst &I) { 2673249259Sdim IRBuilder<> IRB(&I); 2674249259Sdim DEBUG(dbgs() << "InsertValue: " << I << "\n"); 2675249259Sdim Value *AggShadow = getShadow(I.getAggregateOperand()); 2676249259Sdim Value *InsShadow = getShadow(I.getInsertedValueOperand()); 2677249259Sdim DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n"); 2678249259Sdim DEBUG(dbgs() << " InsShadow: " << *InsShadow << "\n"); 2679249259Sdim Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices()); 2680249259Sdim DEBUG(dbgs() << " Res: " << *Res << "\n"); 2681249259Sdim setShadow(&I, Res); 2682261991Sdim setOriginForNaryOp(I); 2683249259Sdim } 2684249259Sdim 2685249259Sdim void dumpInst(Instruction &I) { 2686249259Sdim if (CallInst *CI = dyn_cast<CallInst>(&I)) { 2687249259Sdim errs() << "ZZZ call " << CI->getCalledFunction()->getName() << "\n"; 2688249259Sdim } else { 2689249259Sdim errs() << "ZZZ " << I.getOpcodeName() << "\n"; 2690249259Sdim } 2691249259Sdim errs() << "QQQ " << I << "\n"; 2692249259Sdim } 2693249259Sdim 2694249259Sdim void visitResumeInst(ResumeInst &I) { 2695249259Sdim DEBUG(dbgs() << "Resume: " << I << "\n"); 2696249259Sdim // Nothing to do here. 2697249259Sdim } 2698249259Sdim 2699249259Sdim void visitInstruction(Instruction &I) { 2700249259Sdim // Everything else: stop propagating and check for poisoned shadow. 2701249259Sdim if (ClDumpStrictInstructions) 2702249259Sdim dumpInst(I); 2703249259Sdim DEBUG(dbgs() << "DEFAULT: " << I << "\n"); 2704249259Sdim for (size_t i = 0, n = I.getNumOperands(); i < n; i++) 2705261991Sdim insertShadowCheck(I.getOperand(i), &I); 2706249259Sdim setShadow(&I, getCleanShadow(&I)); 2707249259Sdim setOrigin(&I, getCleanOrigin()); 2708249259Sdim } 2709249259Sdim}; 2710249259Sdim 2711249259Sdim/// \brief AMD64-specific implementation of VarArgHelper. 2712249259Sdimstruct VarArgAMD64Helper : public VarArgHelper { 2713249259Sdim // An unfortunate workaround for asymmetric lowering of va_arg stuff. 2714249259Sdim // See a comment in visitCallSite for more details. 2715249259Sdim static const unsigned AMD64GpEndOffset = 48; // AMD64 ABI Draft 0.99.6 p3.5.7 2716249259Sdim static const unsigned AMD64FpEndOffset = 176; 2717249259Sdim 2718249259Sdim Function &F; 2719249259Sdim MemorySanitizer &MS; 2720249259Sdim MemorySanitizerVisitor &MSV; 2721249259Sdim Value *VAArgTLSCopy; 2722249259Sdim Value *VAArgOverflowSize; 2723249259Sdim 2724249259Sdim SmallVector<CallInst*, 16> VAStartInstrumentationList; 2725249259Sdim 2726249259Sdim VarArgAMD64Helper(Function &F, MemorySanitizer &MS, 2727249259Sdim MemorySanitizerVisitor &MSV) 2728276479Sdim : F(F), MS(MS), MSV(MSV), VAArgTLSCopy(nullptr), 2729276479Sdim VAArgOverflowSize(nullptr) {} 2730249259Sdim 2731249259Sdim enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory }; 2732249259Sdim 2733249259Sdim ArgKind classifyArgument(Value* arg) { 2734249259Sdim // A very rough approximation of X86_64 argument classification rules. 2735249259Sdim Type *T = arg->getType(); 2736249259Sdim if (T->isFPOrFPVectorTy() || T->isX86_MMXTy()) 2737249259Sdim return AK_FloatingPoint; 2738249259Sdim if (T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64) 2739249259Sdim return AK_GeneralPurpose; 2740249259Sdim if (T->isPointerTy()) 2741249259Sdim return AK_GeneralPurpose; 2742249259Sdim return AK_Memory; 2743249259Sdim } 2744249259Sdim 2745249259Sdim // For VarArg functions, store the argument shadow in an ABI-specific format 2746249259Sdim // that corresponds to va_list layout. 2747249259Sdim // We do this because Clang lowers va_arg in the frontend, and this pass 2748249259Sdim // only sees the low level code that deals with va_list internals. 2749249259Sdim // A much easier alternative (provided that Clang emits va_arg instructions) 2750249259Sdim // would have been to associate each live instance of va_list with a copy of 2751249259Sdim // MSanParamTLS, and extract shadow on va_arg() call in the argument list 2752249259Sdim // order. 2753276479Sdim void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override { 2754249259Sdim unsigned GpOffset = 0; 2755249259Sdim unsigned FpOffset = AMD64GpEndOffset; 2756249259Sdim unsigned OverflowOffset = AMD64FpEndOffset; 2757288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 2758249259Sdim for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end(); 2759249259Sdim ArgIt != End; ++ArgIt) { 2760249259Sdim Value *A = *ArgIt; 2761276479Sdim unsigned ArgNo = CS.getArgumentNo(ArgIt); 2762276479Sdim bool IsByVal = CS.paramHasAttr(ArgNo + 1, Attribute::ByVal); 2763276479Sdim if (IsByVal) { 2764276479Sdim // ByVal arguments always go to the overflow area. 2765276479Sdim assert(A->getType()->isPointerTy()); 2766276479Sdim Type *RealTy = A->getType()->getPointerElementType(); 2767288943Sdim uint64_t ArgSize = DL.getTypeAllocSize(RealTy); 2768276479Sdim Value *Base = getShadowPtrForVAArgument(RealTy, IRB, OverflowOffset); 2769280031Sdim OverflowOffset += RoundUpToAlignment(ArgSize, 8); 2770276479Sdim IRB.CreateMemCpy(Base, MSV.getShadowPtr(A, IRB.getInt8Ty(), IRB), 2771276479Sdim ArgSize, kShadowTLSAlignment); 2772276479Sdim } else { 2773276479Sdim ArgKind AK = classifyArgument(A); 2774276479Sdim if (AK == AK_GeneralPurpose && GpOffset >= AMD64GpEndOffset) 2775276479Sdim AK = AK_Memory; 2776276479Sdim if (AK == AK_FloatingPoint && FpOffset >= AMD64FpEndOffset) 2777276479Sdim AK = AK_Memory; 2778276479Sdim Value *Base; 2779276479Sdim switch (AK) { 2780276479Sdim case AK_GeneralPurpose: 2781276479Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, GpOffset); 2782276479Sdim GpOffset += 8; 2783276479Sdim break; 2784276479Sdim case AK_FloatingPoint: 2785276479Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, FpOffset); 2786276479Sdim FpOffset += 16; 2787276479Sdim break; 2788276479Sdim case AK_Memory: 2789288943Sdim uint64_t ArgSize = DL.getTypeAllocSize(A->getType()); 2790276479Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset); 2791280031Sdim OverflowOffset += RoundUpToAlignment(ArgSize, 8); 2792276479Sdim } 2793276479Sdim IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment); 2794249259Sdim } 2795249259Sdim } 2796249259Sdim Constant *OverflowSize = 2797249259Sdim ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AMD64FpEndOffset); 2798249259Sdim IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS); 2799249259Sdim } 2800249259Sdim 2801249259Sdim /// \brief Compute the shadow address for a given va_arg. 2802276479Sdim Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB, 2803249259Sdim int ArgOffset) { 2804249259Sdim Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy); 2805249259Sdim Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset)); 2806276479Sdim return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0), 2807249259Sdim "_msarg"); 2808249259Sdim } 2809249259Sdim 2810276479Sdim void visitVAStartInst(VAStartInst &I) override { 2811249259Sdim IRBuilder<> IRB(&I); 2812249259Sdim VAStartInstrumentationList.push_back(&I); 2813249259Sdim Value *VAListTag = I.getArgOperand(0); 2814249259Sdim Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB); 2815249259Sdim 2816249259Sdim // Unpoison the whole __va_list_tag. 2817249259Sdim // FIXME: magic ABI constants. 2818249259Sdim IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()), 2819249259Sdim /* size */24, /* alignment */8, false); 2820249259Sdim } 2821249259Sdim 2822276479Sdim void visitVACopyInst(VACopyInst &I) override { 2823249259Sdim IRBuilder<> IRB(&I); 2824249259Sdim Value *VAListTag = I.getArgOperand(0); 2825249259Sdim Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB); 2826249259Sdim 2827249259Sdim // Unpoison the whole __va_list_tag. 2828249259Sdim // FIXME: magic ABI constants. 2829249259Sdim IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()), 2830249259Sdim /* size */24, /* alignment */8, false); 2831249259Sdim } 2832249259Sdim 2833276479Sdim void finalizeInstrumentation() override { 2834249259Sdim assert(!VAArgOverflowSize && !VAArgTLSCopy && 2835249259Sdim "finalizeInstrumentation called twice"); 2836249259Sdim if (!VAStartInstrumentationList.empty()) { 2837249259Sdim // If there is a va_start in this function, make a backup copy of 2838249259Sdim // va_arg_tls somewhere in the function entry block. 2839249259Sdim IRBuilder<> IRB(F.getEntryBlock().getFirstNonPHI()); 2840249259Sdim VAArgOverflowSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS); 2841249259Sdim Value *CopySize = 2842249259Sdim IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AMD64FpEndOffset), 2843249259Sdim VAArgOverflowSize); 2844249259Sdim VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize); 2845249259Sdim IRB.CreateMemCpy(VAArgTLSCopy, MS.VAArgTLS, CopySize, 8); 2846249259Sdim } 2847249259Sdim 2848249259Sdim // Instrument va_start. 2849249259Sdim // Copy va_list shadow from the backup copy of the TLS contents. 2850249259Sdim for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) { 2851249259Sdim CallInst *OrigInst = VAStartInstrumentationList[i]; 2852249259Sdim IRBuilder<> IRB(OrigInst->getNextNode()); 2853249259Sdim Value *VAListTag = OrigInst->getArgOperand(0); 2854249259Sdim 2855249259Sdim Value *RegSaveAreaPtrPtr = 2856249259Sdim IRB.CreateIntToPtr( 2857249259Sdim IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy), 2858249259Sdim ConstantInt::get(MS.IntptrTy, 16)), 2859249259Sdim Type::getInt64PtrTy(*MS.C)); 2860249259Sdim Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrPtr); 2861249259Sdim Value *RegSaveAreaShadowPtr = 2862249259Sdim MSV.getShadowPtr(RegSaveAreaPtr, IRB.getInt8Ty(), IRB); 2863249259Sdim IRB.CreateMemCpy(RegSaveAreaShadowPtr, VAArgTLSCopy, 2864249259Sdim AMD64FpEndOffset, 16); 2865249259Sdim 2866249259Sdim Value *OverflowArgAreaPtrPtr = 2867249259Sdim IRB.CreateIntToPtr( 2868249259Sdim IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy), 2869249259Sdim ConstantInt::get(MS.IntptrTy, 8)), 2870249259Sdim Type::getInt64PtrTy(*MS.C)); 2871249259Sdim Value *OverflowArgAreaPtr = IRB.CreateLoad(OverflowArgAreaPtrPtr); 2872249259Sdim Value *OverflowArgAreaShadowPtr = 2873249259Sdim MSV.getShadowPtr(OverflowArgAreaPtr, IRB.getInt8Ty(), IRB); 2874288943Sdim Value *SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSCopy, 2875288943Sdim AMD64FpEndOffset); 2876249259Sdim IRB.CreateMemCpy(OverflowArgAreaShadowPtr, SrcPtr, VAArgOverflowSize, 16); 2877249259Sdim } 2878249259Sdim } 2879249259Sdim}; 2880249259Sdim 2881288943Sdim/// \brief MIPS64-specific implementation of VarArgHelper. 2882288943Sdimstruct VarArgMIPS64Helper : public VarArgHelper { 2883288943Sdim Function &F; 2884288943Sdim MemorySanitizer &MS; 2885288943Sdim MemorySanitizerVisitor &MSV; 2886288943Sdim Value *VAArgTLSCopy; 2887288943Sdim Value *VAArgSize; 2888288943Sdim 2889288943Sdim SmallVector<CallInst*, 16> VAStartInstrumentationList; 2890288943Sdim 2891288943Sdim VarArgMIPS64Helper(Function &F, MemorySanitizer &MS, 2892288943Sdim MemorySanitizerVisitor &MSV) 2893288943Sdim : F(F), MS(MS), MSV(MSV), VAArgTLSCopy(nullptr), 2894288943Sdim VAArgSize(nullptr) {} 2895288943Sdim 2896288943Sdim void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override { 2897288943Sdim unsigned VAArgOffset = 0; 2898288943Sdim const DataLayout &DL = F.getParent()->getDataLayout(); 2899288943Sdim for (CallSite::arg_iterator ArgIt = CS.arg_begin() + 1, End = CS.arg_end(); 2900288943Sdim ArgIt != End; ++ArgIt) { 2901288943Sdim Value *A = *ArgIt; 2902288943Sdim Value *Base; 2903288943Sdim uint64_t ArgSize = DL.getTypeAllocSize(A->getType()); 2904288943Sdim#if defined(__MIPSEB__) || defined(MIPSEB) 2905288943Sdim // Adjusting the shadow for argument with size < 8 to match the placement 2906288943Sdim // of bits in big endian system 2907288943Sdim if (ArgSize < 8) 2908288943Sdim VAArgOffset += (8 - ArgSize); 2909288943Sdim#endif 2910288943Sdim Base = getShadowPtrForVAArgument(A->getType(), IRB, VAArgOffset); 2911288943Sdim VAArgOffset += ArgSize; 2912288943Sdim VAArgOffset = RoundUpToAlignment(VAArgOffset, 8); 2913288943Sdim IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment); 2914288943Sdim } 2915288943Sdim 2916288943Sdim Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(), VAArgOffset); 2917288943Sdim // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of 2918288943Sdim // a new class member i.e. it is the total size of all VarArgs. 2919288943Sdim IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS); 2920288943Sdim } 2921288943Sdim 2922288943Sdim /// \brief Compute the shadow address for a given va_arg. 2923288943Sdim Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB, 2924288943Sdim int ArgOffset) { 2925288943Sdim Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy); 2926288943Sdim Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset)); 2927288943Sdim return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0), 2928288943Sdim "_msarg"); 2929288943Sdim } 2930288943Sdim 2931288943Sdim void visitVAStartInst(VAStartInst &I) override { 2932288943Sdim IRBuilder<> IRB(&I); 2933288943Sdim VAStartInstrumentationList.push_back(&I); 2934288943Sdim Value *VAListTag = I.getArgOperand(0); 2935288943Sdim Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB); 2936288943Sdim IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()), 2937288943Sdim /* size */8, /* alignment */8, false); 2938288943Sdim } 2939288943Sdim 2940288943Sdim void visitVACopyInst(VACopyInst &I) override { 2941288943Sdim IRBuilder<> IRB(&I); 2942288943Sdim Value *VAListTag = I.getArgOperand(0); 2943288943Sdim Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB); 2944288943Sdim // Unpoison the whole __va_list_tag. 2945288943Sdim // FIXME: magic ABI constants. 2946288943Sdim IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()), 2947288943Sdim /* size */8, /* alignment */8, false); 2948288943Sdim } 2949288943Sdim 2950288943Sdim void finalizeInstrumentation() override { 2951288943Sdim assert(!VAArgSize && !VAArgTLSCopy && 2952288943Sdim "finalizeInstrumentation called twice"); 2953288943Sdim IRBuilder<> IRB(F.getEntryBlock().getFirstNonPHI()); 2954288943Sdim VAArgSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS); 2955288943Sdim Value *CopySize = IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, 0), 2956288943Sdim VAArgSize); 2957288943Sdim 2958288943Sdim if (!VAStartInstrumentationList.empty()) { 2959288943Sdim // If there is a va_start in this function, make a backup copy of 2960288943Sdim // va_arg_tls somewhere in the function entry block. 2961288943Sdim VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize); 2962288943Sdim IRB.CreateMemCpy(VAArgTLSCopy, MS.VAArgTLS, CopySize, 8); 2963288943Sdim } 2964288943Sdim 2965288943Sdim // Instrument va_start. 2966288943Sdim // Copy va_list shadow from the backup copy of the TLS contents. 2967288943Sdim for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) { 2968288943Sdim CallInst *OrigInst = VAStartInstrumentationList[i]; 2969288943Sdim IRBuilder<> IRB(OrigInst->getNextNode()); 2970288943Sdim Value *VAListTag = OrigInst->getArgOperand(0); 2971288943Sdim Value *RegSaveAreaPtrPtr = 2972288943Sdim IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy), 2973288943Sdim Type::getInt64PtrTy(*MS.C)); 2974288943Sdim Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrPtr); 2975288943Sdim Value *RegSaveAreaShadowPtr = 2976288943Sdim MSV.getShadowPtr(RegSaveAreaPtr, IRB.getInt8Ty(), IRB); 2977288943Sdim IRB.CreateMemCpy(RegSaveAreaShadowPtr, VAArgTLSCopy, CopySize, 8); 2978288943Sdim } 2979288943Sdim } 2980288943Sdim}; 2981288943Sdim 2982261991Sdim/// \brief A no-op implementation of VarArgHelper. 2983261991Sdimstruct VarArgNoOpHelper : public VarArgHelper { 2984261991Sdim VarArgNoOpHelper(Function &F, MemorySanitizer &MS, 2985261991Sdim MemorySanitizerVisitor &MSV) {} 2986261991Sdim 2987276479Sdim void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {} 2988261991Sdim 2989276479Sdim void visitVAStartInst(VAStartInst &I) override {} 2990261991Sdim 2991276479Sdim void visitVACopyInst(VACopyInst &I) override {} 2992261991Sdim 2993276479Sdim void finalizeInstrumentation() override {} 2994261991Sdim}; 2995261991Sdim 2996261991SdimVarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan, 2997249259Sdim MemorySanitizerVisitor &Visitor) { 2998261991Sdim // VarArg handling is only implemented on AMD64. False positives are possible 2999261991Sdim // on other platforms. 3000261991Sdim llvm::Triple TargetTriple(Func.getParent()->getTargetTriple()); 3001261991Sdim if (TargetTriple.getArch() == llvm::Triple::x86_64) 3002261991Sdim return new VarArgAMD64Helper(Func, Msan, Visitor); 3003288943Sdim else if (TargetTriple.getArch() == llvm::Triple::mips64 || 3004288943Sdim TargetTriple.getArch() == llvm::Triple::mips64el) 3005288943Sdim return new VarArgMIPS64Helper(Func, Msan, Visitor); 3006261991Sdim else 3007261991Sdim return new VarArgNoOpHelper(Func, Msan, Visitor); 3008249259Sdim} 3009249259Sdim 3010249259Sdim} // namespace 3011249259Sdim 3012249259Sdimbool MemorySanitizer::runOnFunction(Function &F) { 3013288943Sdim if (&F == MsanCtorFunction) 3014288943Sdim return false; 3015249259Sdim MemorySanitizerVisitor Visitor(F, *this); 3016249259Sdim 3017249259Sdim // Clear out readonly/readnone attributes. 3018249259Sdim AttrBuilder B; 3019249259Sdim B.addAttribute(Attribute::ReadOnly) 3020249259Sdim .addAttribute(Attribute::ReadNone); 3021249259Sdim F.removeAttributes(AttributeSet::FunctionIndex, 3022249259Sdim AttributeSet::get(F.getContext(), 3023249259Sdim AttributeSet::FunctionIndex, B)); 3024249259Sdim 3025249259Sdim return Visitor.runOnFunction(); 3026249259Sdim} 3027