1Freezing of tasks 2 (C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL 3 4I. What is the freezing of tasks? 5 6The freezing of tasks is a mechanism by which user space processes and some 7kernel threads are controlled during hibernation or system-wide suspend (on some 8architectures). 9 10II. How does it work? 11 12There are four per-task flags used for that, PF_NOFREEZE, PF_FROZEN, TIF_FREEZE 13and PF_FREEZER_SKIP (the last one is auxiliary). The tasks that have 14PF_NOFREEZE unset (all user space processes and some kernel threads) are 15regarded as 'freezable' and treated in a special way before the system enters a 16suspend state as well as before a hibernation image is created (in what follows 17we only consider hibernation, but the description also applies to suspend). 18 19Namely, as the first step of the hibernation procedure the function 20freeze_processes() (defined in kernel/power/process.c) is called. It executes 21try_to_freeze_tasks() that sets TIF_FREEZE for all of the freezable tasks and 22either wakes them up, if they are kernel threads, or sends fake signals to them, 23if they are user space processes. A task that has TIF_FREEZE set, should react 24to it by calling the function called refrigerator() (defined in 25kernel/power/process.c), which sets the task's PF_FROZEN flag, changes its state 26to TASK_UNINTERRUPTIBLE and makes it loop until PF_FROZEN is cleared for it. 27Then, we say that the task is 'frozen' and therefore the set of functions 28handling this mechanism is referred to as 'the freezer' (these functions are 29defined in kernel/power/process.c and include/linux/freezer.h). User space 30processes are generally frozen before kernel threads. 31 32It is not recommended to call refrigerator() directly. Instead, it is 33recommended to use the try_to_freeze() function (defined in 34include/linux/freezer.h), that checks the task's TIF_FREEZE flag and makes the 35task enter refrigerator() if the flag is set. 36 37For user space processes try_to_freeze() is called automatically from the 38signal-handling code, but the freezable kernel threads need to call it 39explicitly in suitable places or use the wait_event_freezable() or 40wait_event_freezable_timeout() macros (defined in include/linux/freezer.h) 41that combine interruptible sleep with checking if TIF_FREEZE is set and calling 42try_to_freeze(). The main loop of a freezable kernel thread may look like the 43following one: 44 45 set_freezable(); 46 do { 47 hub_events(); 48 wait_event_freezable(khubd_wait, 49 !list_empty(&hub_event_list) || 50 kthread_should_stop()); 51 } while (!kthread_should_stop() || !list_empty(&hub_event_list)); 52 53(from drivers/usb/core/hub.c::hub_thread()). 54 55If a freezable kernel thread fails to call try_to_freeze() after the freezer has 56set TIF_FREEZE for it, the freezing of tasks will fail and the entire 57hibernation operation will be cancelled. For this reason, freezable kernel 58threads must call try_to_freeze() somewhere or use one of the 59wait_event_freezable() and wait_event_freezable_timeout() macros. 60 61After the system memory state has been restored from a hibernation image and 62devices have been reinitialized, the function thaw_processes() is called in 63order to clear the PF_FROZEN flag for each frozen task. Then, the tasks that 64have been frozen leave refrigerator() and continue running. 65 66III. Which kernel threads are freezable? 67 68Kernel threads are not freezable by default. However, a kernel thread may clear 69PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE 70directly is strongly discouraged). From this point it is regarded as freezable 71and must call try_to_freeze() in a suitable place. 72 73IV. Why do we do that? 74 75Generally speaking, there is a couple of reasons to use the freezing of tasks: 76 771. The principal reason is to prevent filesystems from being damaged after 78hibernation. At the moment we have no simple means of checkpointing 79filesystems, so if there are any modifications made to filesystem data and/or 80metadata on disks, we cannot bring them back to the state from before the 81modifications. At the same time each hibernation image contains some 82filesystem-related information that must be consistent with the state of the 83on-disk data and metadata after the system memory state has been restored from 84the image (otherwise the filesystems will be damaged in a nasty way, usually 85making them almost impossible to repair). We therefore freeze tasks that might 86cause the on-disk filesystems' data and metadata to be modified after the 87hibernation image has been created and before the system is finally powered off. 88The majority of these are user space processes, but if any of the kernel threads 89may cause something like this to happen, they have to be freezable. 90 912. Next, to create the hibernation image we need to free a sufficient amount of 92memory (approximately 50% of available RAM) and we need to do that before 93devices are deactivated, because we generally need them for swapping out. Then, 94after the memory for the image has been freed, we don't want tasks to allocate 95additional memory and we prevent them from doing that by freezing them earlier. 96[Of course, this also means that device drivers should not allocate substantial 97amounts of memory from their .suspend() callbacks before hibernation, but this 98is e separate issue.] 99 1003. The third reason is to prevent user space processes and some kernel threads 101from interfering with the suspending and resuming of devices. A user space 102process running on a second CPU while we are suspending devices may, for 103example, be troublesome and without the freezing of tasks we would need some 104safeguards against race conditions that might occur in such a case. 105 106Although Linus Torvalds doesn't like the freezing of tasks, he said this in one 107of the discussions on LKML (http://lkml.org/lkml/2007/4/27/608): 108 109"RJW:> Why we freeze tasks at all or why we freeze kernel threads? 110 111Linus: In many ways, 'at all'. 112 113I _do_ realize the IO request queue issues, and that we cannot actually do 114s2ram with some devices in the middle of a DMA. So we want to be able to 115avoid *that*, there's no question about that. And I suspect that stopping 116user threads and then waiting for a sync is practically one of the easier 117ways to do so. 118 119So in practice, the 'at all' may become a 'why freeze kernel threads?' and 120freezing user threads I don't find really objectionable." 121 122Still, there are kernel threads that may want to be freezable. For example, if 123a kernel that belongs to a device driver accesses the device directly, it in 124principle needs to know when the device is suspended, so that it doesn't try to 125access it at that time. However, if the kernel thread is freezable, it will be 126frozen before the driver's .suspend() callback is executed and it will be 127thawed after the driver's .resume() callback has run, so it won't be accessing 128the device while it's suspended. 129 1304. Another reason for freezing tasks is to prevent user space processes from 131realizing that hibernation (or suspend) operation takes place. Ideally, user 132space processes should not notice that such a system-wide operation has occurred 133and should continue running without any problems after the restore (or resume 134from suspend). Unfortunately, in the most general case this is quite difficult 135to achieve without the freezing of tasks. Consider, for example, a process 136that depends on all CPUs being online while it's running. Since we need to 137disable nonboot CPUs during the hibernation, if this process is not frozen, it 138may notice that the number of CPUs has changed and may start to work incorrectly 139because of that. 140 141V. Are there any problems related to the freezing of tasks? 142 143Yes, there are. 144 145First of all, the freezing of kernel threads may be tricky if they depend one 146on another. For example, if kernel thread A waits for a completion (in the 147TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B 148and B is frozen in the meantime, then A will be blocked until B is thawed, which 149may be undesirable. That's why kernel threads are not freezable by default. 150 151Second, there are the following two problems related to the freezing of user 152space processes: 1531. Putting processes into an uninterruptible sleep distorts the load average. 1542. Now that we have FUSE, plus the framework for doing device drivers in 155userspace, it gets even more complicated because some userspace processes are 156now doing the sorts of things that kernel threads do 157(https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html). 158 159The problem 1. seems to be fixable, although it hasn't been fixed so far. The 160other one is more serious, but it seems that we can work around it by using 161hibernation (and suspend) notifiers (in that case, though, we won't be able to 162avoid the realization by the user space processes that the hibernation is taking 163place). 164 165There are also problems that the freezing of tasks tends to expose, although 166they are not directly related to it. For example, if request_firmware() is 167called from a device driver's .resume() routine, it will timeout and eventually 168fail, because the user land process that should respond to the request is frozen 169at this point. So, seemingly, the failure is due to the freezing of tasks. 170Suppose, however, that the firmware file is located on a filesystem accessible 171only through another device that hasn't been resumed yet. In that case, 172request_firmware() will fail regardless of whether or not the freezing of tasks 173is used. Consequently, the problem is not really related to the freezing of 174tasks, since it generally exists anyway. 175 176A driver must have all firmwares it may need in RAM before suspend() is called. 177If keeping them is not practical, for example due to their size, they must be 178requested early enough using the suspend notifier API described in notifiers.txt. 179