# CDDL HEADER START # # The contents of this file are subject to the terms of the # Common Development and Distribution License (the "License"). # You may not use this file except in compliance with the License. # # You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE # or http://www.opensolaris.org/os/licensing. # See the License for the specific language governing permissions # and limitations under the License. # # When distributing Covered Code, include this CDDL HEADER in each # file and include the License file at usr/src/OPENSOLARIS.LICENSE. # If applicable, add the following below this CDDL HEADER, with the # fields enclosed by brackets "[]" replaced with your own identifying # information: Portions Copyright [yyyy] [name of copyright owner] # # CDDL HEADER END # # # Copyright 2008 Sun Microsystems, Inc. All rights reserved. # Use is subject to license terms. # TITLE: Dynamic Memory Implementation Overview DATE: 10/13/2000 AUTHOR: Jim Guerrera (james.guerrera@east) 1.0 Dynamic Memory Implementation in the SCM Module The system memory allocation required by the Storage Cache Manager (SCM) has been modified to more fully conform to the requirements of the Solaris OS. The previous implementation required that the total memory requirements of the package be allocated 'up front' during bootup and was never released. The current implementation performs 'on demand' allocations at the time memory is required in a piecemeal manner. In addition the requisitioned memory will be released back to the system at some later time. 2.0 Implementation 2.1 Memory Allocation The memory allocation involves modifications primarily to sd_alloc_buf() in module sd_bcache.c. When a request is received for cache and system resources it is broken down and each piece catagorized both as an independent entity and as a member of a group with close neighbors. Cache resources comprise cache control entries (ccent), write control entries (wctrl for FWC support) and sytem memory. The current allocation algorithim for ccent and wrctl remains the same. The memory allocation has been modified and falls into two general catagories - single page and multi-page allocations. 2.1.1 A single page allocation means exactly that - the ccent points to and owns one page of system memory. If two or more ccent are requisitioned to support the caching request then only the first entry in the group actually owns the the allocated memory of two or more pages. The secondary entries simply point to page boundaries within this larger piece of contiguous memory. The first entry is termed a host and the secondaries are termed parasites. The process for determining what is a host, a parasite or anything else is done in three phases. Phase one simply determines whether the caching request references a disk area already in cache and marks it as such. If it is not in cache it is typed as eligible - i.e. needing memory allocation. Phase two scans this list of typed cache entries and based on immediate neighbors is catagorized as host, pest or downgraded to other. A host can only exist if there is one or more eligible entries immediately following it and it itself either starts the list or immediately follows a non-eligible entry. If either condition proves false the catagory remains as eligible (i.e. needs memory allocation) but the type is cleared to not host (i.e. other). The next phase is simply a matter of scanning the cache entry list and allocating multipage memory for hosts, single page entries for others or simply setting up pointers in the parasitic entries into it's corresponding host multipage memory allocation block. 2.1.2 The maximum number of parasitic entries following a host memory allocation is adjustable by the system administrator. The details of this are under the description of the KSTAT interface (Sec 3.0). 2.2 Memory Deallocation Memory deallocation is implemented in sd_dealloc_dm() in module sd_io.c. This possibly overly complicated routine works as follows: In general the routine sleeps a specified amount of time then wakes and examines the entire centry list. If an entry is available (i.e. not in use by another thread and has memory which may be deallocated) it takes possession and ages the centry by one tick. It then determines if the centry has aged sufficiently to have its memory deallocated and for it to be placed at the top of the lru. 2.3 There are two general deallocation schemes in place depending on whether the centry is a single page allocation centry or it is a member of a host/parasite multipage allocation chain. 2.3.1 The behavior for a single page allocation centry is as follows: If the given centry is selected as a 'holdover' it will age normally however at full aging it will only be placed at the head of the lru. It's memory will not be deallocated until a further aging level has been reached. The entries selected for this behavior are governed by counting the number of these holdovers in existence on each wakeup and comparing it to a specified percentage. This comparision is always one cycle out of date and will float in the relative vicinity of the specified number. In addition there is a placeholder for centries identified as 'sticky meta-data' with its own aging counter. It operates exactly as the holdover entries as regards to aging but is absolute - i.e. no percentage governs the number of such entries. 2.3.2 The percentage and additional aging count are adjustable by the system administrator. The details of this are under the description of the KSTAT interface (Sec. 3.0). 2.3.3 The behavior for a host/parasite chain is as follows: The host/parasite subchain is examined. If all entries are fully aged the entire chain is removed - i.e memory is deallocated from the host centry and all centry fields are cleared and each entry requeued on to the lru. There are three sleep times and two percentage levels specifiable by the system administrator. A meaningful relationship between these variables is: sleeptime1 >= sleeptime2 >= sleeptime2 and 100% >= pcntfree1 >= pcntfree2 >= 0% sleeptime1 is honored between 100% free and pcntfree1. sleeptime2 is honored between pcntfree1 and pcntfree2. sleeptime3 is honored between pcntfree2 and 0% free. The general thrust here is to automatically adjust sleep time to centry load. In addition there exist an accelerated aging flag which mimics hysterisis behavior. If the available centrys fall between pcntfree1 and pcntfree2 an 8 bit counter is switched on. The effect is to keep the timer value at sleeptime2 for 8 cycles even if the number available cache entries drifts above pcntfree1. If it falls below pcntfree2 an additional 8 bit counter is switched on. This causes the sleep timer to remain at sleeptime3 for at least 8 cycles even if it floats above pcntfree2 or even pcntfree1. The overall effect of this is to accelerate the release of system resources under what the thread thinks is a heavy load as measured by the number of used cache entries. 3.0 Dynamic Memory Tuning A number of behavior modification variables are accessible via system calls to the kstat library. A sample program exercising the various features can be found in ./src/cmd/ns/sdbc/sdbc_dynmem.c. In addition the behavior variable identifiers can be placed in the sdbc.conf file and will take effect on bootup. There is also a number of dynamic memory statistics available to gauge its current state. 3.1 Behavior Variables sdbc_monitor_dynmem --- D0=monitor thread shutdown in the console window D1=print deallocation thread stats to the console window D2=print more deallocation thread stats to the console window (usage: setting a value of 6 = 2+4 sets D1 and D2) sdbc_max_dyn_list ----- 1 to ?: sets the maximum host/parasite list length (A length of 1 prevents any multipage allocations from occuring and effectively removes the concept of host/parasite.) sdbc_cache_aging_ct1 -- 1 to 255: fully aged count (everything but meta and holdover) sdbc_cache_aging_ct2 -- 1 to 255: fully aged count for meta-data entries sdbc_cache_aging_ct3 -- 1 to 255: fully aged count for holdovers sdbc_cache_aging_sec1 - 1 to 255: sleep level 1 for 100% to pcnt1 free cache entries sdbc_cache_aging_sec2 - 1 to 255: sleep level 2 for pcnt1 to pcnt2 free cache entries sdbc_cache_aging_sec3 - 1 to 255: sleep level 3 for pcnt2 to 0% free cache entries sdbc_cache_aging_pcnt1- 0 to 100: cache free percent for transition from sleep1 to sleep2 sdbc_cache_aging_pcnt2- 0 to 100: cache free percent for transition from sleep2 to sleep3 sdbc_max_holds_pcnt --- 0 to 100: max percent of cache entries to be maintained as holdovers 3.2 Statistical Variables Cache Stats (per wake cycle) (r/w): sdbc_alloc_ct --------- total allocations performed sdbc_dealloc_ct ------- total deallocations performed sdbc_history ---------- current hysterisis flag setting sdbc_nodatas ---------- cache entries w/o memory assigned sdbc_candidates ------- cache entries ready to be aged or released sdbc_deallocs --------- cache entries w/memory deallocated and requeued sdbc_hosts ------------ number of host cache entries sdbc_pests ------------ number of parasitic cache entries sdbc_metas ------------ number of meta-data cache entries sdbc_holds ------------ number of holdovers (fully aged w/memory and requeued) sdbc_others ----------- number of not [host, pests or metas] sdbc_notavail --------- number of cache entries to bypass (nodatas+'in use by other processes') sdbc_process_directive- D0=1 wake thread D1=1 temporaily accelerate aging (set the hysterisis flag) sdbc_simplect --------- simple count of the number of times the kstat update routine has been called 3.3 Range Checks and Limits Only range limits are checked. Internal inconsistencies are not checked (e.g. pcnt2 > pcnt1). Inconsistencies won't break the system you just won't get meaningful behavior. The aging counter and sleep timer limits are arbitrarily limited to a byte wide counter. This can be expanded. However max'ing the values under the current implementation yields about 18 hours for full aging. 3.4 Kstat Lookup Name The kstat_lookup() module name is "sdbc:dynmem" with an instance of 0. 3.5 Defaults Default values are: sdbc_max_dyn_list = 8 sdbc_monitor_dynmem = 0 sdbc_cache_aging_ct1 = 3 sdbc_cache_aging_ct2 = 3 sdbc_cache_aging_ct3 = 3 sdbc_cache_aging_sec1 = 10 sdbc_cache_aging_sec2 = 5 sdbc_cache_aging_sec3 = 1 sdbc_cache_aging_pcnt1 = 50 sdbc_cache_aging_pcnt2 = 25 sdbc_max_holds_pcnt = 0 To make the dynmem act for all intents and purposes like the static model beyond the inital startup the appropriate values are: sdbc_max_dyn_list = 1, sdbc_cache_aging_ct1/2/3=255, sdbc_cache_aging_sec1/2/3=255 The remaining variables are irrelevant. 4.0 KSTAT Implementation for Existing Statistics The existing cache statistical reporting mechanism has been replaced by the kstat library reporting mechanism. In general the statistics fall into two general catagories - global and shared. The global stats reflect gross behavior over all cached volumes and shared reflects behavior particular to each cached volume. 4.1 Global KSTAT lookup_name The kstat_lookup() module name is "sdbc:gstats" with an instance of 0. The identifying ascii strings and associated values matching the sd_stats driver structure are: sdbc_dirty -------- net_dirty sdbc_pending ------ net_pending sdbc_free --------- net_free sdbc_count -------- st_count - number of opens for device sdbc_loc_count ---- st_loc_count - number of open devices sdbc_rdhits ------- st_rdhits - number of read hits sdbc_rdmiss ------- st_rdmiss - number of read misses sdbc_wrhits ------- st_wrhits - number of write hits sdbc_wrmiss ------- st_wrmiss - number of write misses sdbc_blksize ------ st_blksize - cache block size sdbc_num_memsize -- SD_MAX_MEM - number of defined blocks (currently 6) To find the size of each memory blocks append the numbers 0 to 5 to 'sdbc_memsize'. sdbc_memsize0 ----- local memory sdbc_memsize1 ----- cache memory sdbc_memsize2 ----- iobuf memory sdbc_memsize3 ----- hash memory sdbc_memsize4 ----- global memory sdbc_memsize5 ----- stats memory sdbc_total_cmem --- st_total_cmem - memory used by cache structs sdbc_total_smem --- st_total_smem - memory used by stat structs sdbc_lru_blocks --- st_lru_blocks sdbc_lru_noreq ---- st_lru_noreq sdbc_lru_req ------ st_lru_req sdbc_num_wlru_inq - MAX_CACHE_NET - number of net (currently 4) To find the size of the least recently used write cache per net append the numbers 0-3 to sdbc_wlru_inq sdbc_wlru_inq0 ---- net 0 sdbc_wlru_inq1 ---- net 1 sdbc_wlru_inq2 ---- net 2 sdbc_wlru_inq3 ---- net 3 sdbc_cachesize ---- st_cachesize - cache size sdbc_numblocks ---- st_numblocks - cache blocks sdbc_num_shared --- MAXFILES*2 - number of shared structures (one for each cached volume) This number dictates the maximum index size for shared stats and names given below. sdbc_simplect ----- simple count of the number of times the kstat update routine has been called All fields are read only. 4.2 Shared Structures KSTAT lookup_name The kstat_lookup() module name is "sdbc:shstats" and "sdbc:shname" both with an instance of 0. The identifying ascii strings and associated values matching the sd_shared driver structure are: sdbc:shstats module sdbc_index ------- structure index number sdbc_alloc ------- sh_alloc - is this allocated? sdbc_failed ------ sh_failed - Disk failure status (0=ok,1= /o error ,2= open failed) sdbc_cd ---------- sh_cd - the cache descriptor. (for stats) sdbc_cache_read -- sh_cache_read - Number of bytes read from cache sdbc_cache_write - sh_cache_write - Number of bytes written to cache sdbc_disk_read --- sh_disk_read - Number of bytes read from disk sdbc_disk_write -- sh_disk_write - Number of bytes written to disk sdbc_filesize ---- sh_filesize - Filesize sdbc_numdirty ---- sh_numdirty - Number of dirty blocks sdbc_numio ------- sh_numio - Number of blocks on way to disk sdbc_numfail ----- sh_numfail - Number of blocks failed sdbc_flushloop --- sh_flushloop - Loops delayed so far sdbc_flag -------- sh_flag - Flags visible to user programs sdbc_simplect ---- simple count of the number of times the kstat update routine has been called sdbc:shname module read in as raw bytes and interpreted as a nul terminated assci string. These two modules operate hand in hand based on information obtained from the "sdbc:gstats" module. "sdbc:gstats - sdbc_num_shared" gives the maximum number possible of shared devices. It does not tell how many devices are actually cached - just the maximum possible. In order to determine the number present and retrieve the statistics for each device the user must: 1. open and read "sdbc:shstats" 2. set the index "sdbc_index" to a starting value (presumably 0) 3. write the kstat module ( the only item in the module is sdbc_index) What this does is set a starting index for all subsequent reads. 4. to get the device count and associated statistics the user now simply reads each module "sdbc:shstats" and "sdbc:shname" as a group repeatedly - the index will auto increment To reset the index set "sdbc:shstats - sdbc_index" to the required value and write the module. The first entry returning a nul string to "sdbc:shname" signifies no more configured devices.