ORA-04031案例一则
ORA-04031这个错误,几乎每一个专业的DBA都遇到过。这是一个相当严重的错误,Oracle进程在向SGA申请内存时,如果申请失败,则会报这个错误。大部分情况下是在向SGA中的shared pool申请内存时失败,而少有向large pool等池中申请内存失败。比如下面的报错:
Wed Apr 27 16:00:25 2011 Errors in file /oracle/app/oracle/admin/zxin/bdump/zxin1_ora_2052294.trc: ORA-04031: unable to allocate 4128 bytes of shared memory ("shared pool","unknown object","sga heap(3,0)","kgllk hash table")
这里很清楚地表示出来,是在向shared pool申请内存时失败。
shared pool内存申请(分配)失败,通常有如下的几种可能:
本文中的案例,其数据库是运行在AIX 5.3系统中的10.2.0.4 RAC,RAC节点数为2。数据库是从9i升级到10g,而目前处于正式升级前的测试阶段。数据库报的ORA-04031错误信息如本文前面所示(其中的数据库名称已经做了处理)。
在继续讲解案例之前,不得不提到MOS上的几篇关于ORA-04031错误的文档:
其实分析ORA-04031错误,通常有以下几个要点:
上面的步骤写得比较粗略,关于分析和解决ORA-04031问题,这里也有一篇不错的文章:Simplified Approach to Resolve ORA-4031
这里关键的是分析Shared Pool的内存数据。ORA-04031错误发生后如果有条件可以马上连接到数据库中查询相应的x$表和v$视图得到相应的数据,否则只能通过ORA-4031错误发生时产生的trace文件。_4031_dump_bitvec这个隐含参数用于控制发生ORA-04031错误时对SGA的dump行为,而trace文件的分析就不像使用SQL那样简单了。
下面再来详细地分析案例:
从错误信息来看,很显然,是向shared pool的第3个subpool申请内存时出错。
以下的数据是shared pool的数据:
SQL> select sum(bytes)/1024/1024 mb from v$sgastat where pool=\'shared pool\'; MB ---------- 4096.53062 SQL> SELECT KSMCHCLS CLASS, COUNT(KSMCHCLS) NUM, SUM(KSMCHSIZ) 2 SIZ, 3 To_char( ((SUM(KSMCHSIZ)/COUNT(KSMCHCLS)/1024)),\'999,999.00\')| 4 |\'k\' "AVG SIZE" 5 FROM X$KSMSP GROUP BY KSMCHCLS; CLASS NUM SIZ AVG SIZE -------- ---------- ---------- ------------ R-freea 512 24576 .05k freeabl 807395 1643969848 1.99k recr 530728 662065240 1.22k R-free 256 214910976 819.82k free 43063 100605496 2.28k perm 140 1673368632 11,672.49k
虽然free的数量不是太多,但是freeable的数量还是很多的。
下面是各个子池更详细的数据:
SQL> SELECT KSMCHIDX,KSMCHDUR, KSMCHCLS CLASS, COUNT(KSMCHCLS) NUM, SUM(KSMCHSIZ) 2 SIZ, 3 To_char( ((SUM(KSMCHSIZ)/COUNT(KSMCHCLS)/1024)),\'999,999.00\')| 4 |\'k\' "AVG SIZE" 5 FROM X$KSMSP GROUP BY KSMCHIDX,KSMCHDUR, KSMCHCLS 6 order by 1,2,3; KSMCHIDX KSMCHDUR CLASS NUM SIZ AVG SIZE ---------- ---------- -------- ---------- ---------- ------------ 1 1 R-free 27 22666392 819.82k R-freea 54 2592 .05k free 26 14024 .53k perm 32 430299448 13,131.70k 2 R-free 12 10073952 819.82k R-freea 24 1152 .05k free 10531 24519112 2.27k freeabl 44922 32457736 .71k recr 163177 134273584 .80k 3 R-free 9 7555464 819.82k R-freea 18 864 .05k free 1678 4555704 2.65k freeabl 79815 102514024 1.25k recr 32689 36368096 1.09k 4 R-free 20 16789920 819.82k R-freea 40 1920 .05k free 2182 5810056 2.60k freeabl 66235 254656184 3.75k recr 16245 58284480 3.50k 2 1 R-free 25 20987400 819.82k R-freea 50 2400 .05k free 23 20016 .85k perm 35 398418384 11,116.58k 2 R-free 4 3357984 819.82k R-freea 8 384 .05k free 5137 6604176 1.26k freeabl 40377 12140944 .29k recr 54942 45005024 .80k 3 R-free 9 7555464 819.82k R-freea 18 864 .05k free 1477 5524568 3.65k freeabl 79548 101879808 1.25k recr 32380 36033448 1.09k 4 R-free 21 17629416 819.82k R-freea 42 2016 .05k free 2540 7092424 2.73k freeabl 70133 270332800 3.76k recr 15924 57263032 3.51k 3 1 R-free 26 21826896 819.82k R-freea 52 2496 .05k free 26 20520 .77k perm 33 414355416 12,261.94k 2 R-free 4 3357984 819.82k R-freea 8 384 .05k free 4693 7053032 1.47k freeabl 49723 14339800 .28k recr 52771 42357312 .78k 3 R-free 11 9234456 819.82k R-freea 22 1056 .05k free 3594 9280904 2.52k freeabl 95823 121934488 1.24k recr 39643 44097504 1.09k 4 R-free 25 20987400 819.82k R-freea 50 2400 .05k free 2822 7291680 2.52k freeabl 84443 323149712 3.74k recr 19148 67997008 3.47k 4 1 R-free 27 22666392 819.82k R-freea 54 2592 .05k free 26 18088 .68k perm 40 430295384 10,505.26k 2 R-free 6 5036976 819.82k R-freea 12 576 .05k free 4818 11473920 2.33k freeabl 46089 39963224 .85k recr 54061 44188072 .80k 3 R-free 9 7555464 819.82k R-freea 18 864 .05k free 1427 4109504 2.81k freeabl 80026 102379080 1.25k recr 33217 36949240 1.09k 4 R-free 21 17629416 819.82k R-freea 42 2016 .05k free 1993 6228464 3.05k freeabl 70581 269087136 3.72k recr 16614 59372656 3.49k
KSMCHDUR是什么意思?在9i里面这个列的值通常为1。实际上,Oracle从9i开始,将shared pool划分为多个sub pool。而在10g以上的版本中(具体开始的版本号已经不记得),每个sub pool又分了4个更小的池,我们暂且称之为mini heap。每个mini heap有其自己的free list。KSMCHDUR这一列即表示mini heap的编号。”heap(3,0)”中的0是指第1个mini heap。
在上面的数据中,可以看到这个子池的第1个mini heap的free已经很少,只有10来K。另外,我们可以观察到,perm类型的内存块只存在于每个sub pool的第1个min heap中。这个是一个重点,在后面会有解释。
这里本应该有通过查询v$sgastat得到shared pool的各个组件占用的内存分布,只是写BLOG时找不到了….但是我们可以在trace文件中找到数据,下面只列出sub pool 3的数据:
============================== Memory Utilization of Subpool 3 ================================ Allocation Name Size _________________________ __________ "free memory " 81466568 "miscellaneous " 0 "dpslut_kfdsg " 512 "trace buffer " 737280 "trace_knlasg " 504 "gcs res hash bucket " 1048576 "gcs res latch table " 12288 "evaluation con " 0 "sql area " 344545752 "UNDO STAT INFO " 59904 "txncallback " 141744 "transaction " 2103264 "ges resource pools " 3968 "sessions " 4526488 "dlo fib struct " 128032 "KJCTS process batching st" 240 "row cache " 3272 "KCB where statistics arra" 25888 "KCB buffer wait statistic" 32000 "KCB incremental ckpt entr" 512 "invalid low rba queue " 1024 "table definiti " 108704 "temporary tabl " 4136 "KCL instance cache transf" 131072 "resumable " 2720 "KESTB existence bitvec se" 128 "type object de " 392448 "enqueue_hash " 318960 "KSXR pending consumption " 20192 "KTI SGA freeable small po" 0 "trigger defini " 885472 "trigger source " 99264 "trigger inform " 960 "KTCN: Obj Invalidation Se" 2336 "kmgsb circular statistics" 108800 "kgl lock hash table state" 45360 "kglsim size of pinned mem" 8024 "kelr system metrics table" 280 "kzctxgjsi ksuseclid memor" 117360 "kzctxgjci ksuseclid memo" 0 "CCursor " 95912048 "ksr message pool free que" 188960 "ksb ci process list (each" 144 "ksunfy: nodes of hierarch" 320 "ksuloi: long op free list" 256 "kwqmncal: allocate buffer" 4048 "ksim group query request " 0 "ksuxds ksuseclid memory " 0 "call " 87304 "dictionary cache " 0 "KSXR pending reply queue " 255488 "hng: All sessions data fo" 0 "ksfv subheap descriptor " 184 "gcs resources " 169082312 "gcs affinity " 8320 "gcs opaque in " 12312 "PCursor " 50743128 "ges resource " 539376 "fdrec_kffsg " 24 "work area tab " 80640 "kglsim main lru count " 38400 "plwpil:wa " 4264 "grptab_kfgsg " 2464 "AW SGA " 40 "KEWS sesstat seg tbl " 8 "kebm slave descriptors " 1456 "kglsim hash table bkts " 1048576 "KSXR global channels " 1288 "ges enqueues " 17333720 "PLS chunk " 352 "KSQ event description " 1440 "KESTB existence bitvec " 4096 "gcs shadows " 101246344 "qmtb_init_data " 224 "Core dump directory " 264 "sort segment handle " 7480 "SERVICE NAME ENTRY " 48 "PQ/BizCard " 1536 "qtree_kwqbspse " 40 "latch descriptor table " 1576 "recovery domain " 29856 "parameters " 30056 "SHARED SERVERS INFO " 240 "qtree_kwqbsgn " 40 "post agent " 0 "pspool_kfsg " 80 "plwsppwp:wa " 0 "PL/SQL DIANA " 14050624 "segmented arrays " 2072 "Checkpoint queue " 4097024 "sim lru segments " 2560 "sim segment hits " 2560 "sim state object " 40 "partitioning d " 199616 "ASH buffers " 8388608 "message pool freequeue " 276336 "PL/SQL MPCODE " 4499360 "PL/SQL PPCODE " 3984944 "procs: ksunfy " 1512000 "primem_kfmdsg " 1032 "SYSTEM PARAMETERS " 76624 "object queue hash buckets" 262656 "object queue hash table d" 7552 "object level stats hash t" 512 "object stat dummy statprv" 144 "sim cache sizes " 320 "logout storm management " 24000 "pl/sql source " 21256 "sys event stats " 199136 "parameter handle " 67896 "Parameter Handle " 1656 "channel handle " 828672 "API blockers array " 64 "PARAMETER TABLE " 2048 "PARAMETER ENTRY " 8 "LGWR post requested array" 24 "bloom filter " 3104 "param hash values " 5984 "sql area:PLSQL " 11477776 "PX subheap desc " 256 "repository " 213544 "sql area:KOKA " 16192 "archive_lag_target " 9624 "state objects " 640 "latch nowait fails or sle" 116832 "sched job slv " 5952 "pso tbs: ksunfy " 390000 "dummy " 269928 "Sort Segment " 37440 "Cursor Stats " 6095760 "Banner Storage " 2048 "quiescing session " 3872 "API data buffer " 16 "buffer handles " 1020000 "prmtzdini tz region " 408320 "sga node map " 16 "savepoints " 0 "Managed Standby Proc Arra" 24576 "OS proc request holder " 4664 "db_files " 416576 "PX server msg stats " 2288 "KQR M PO " 283376 "kks stats " 40 "parameter table block " 483168 "KSFV SGA " 824 "plugin datafile array " 36016 "plwda:PLW_STR_NEW_RVAL " 24 "plwspv:PLW_STR_NEW_VAL " 16 "KGKP sga " 32 "BRANCH TABLE SEGMENTED AR" 70176 "mvobj part des " 306544 "parameter value memory " 216 "multiblock re " 98496 "parameter text value " 1080 "parallel_max_servers " 8192 "KGLS heap " 13290800 "KGSKI sga " 80 "resize request state obje" 368000 "MTTR advisory " 1462832 "monitoring co " 12480 "rules engine aggregate st" 1416 "krbmror " 36400 "joxs heap " 136 "krbmrsr " 152 "ksfqpar " 4008 "SGA - SWRF DrvMet Runtime" 2656 "SGA - SWRF Metrics ksuTim" 72 "SGA - SWRF RawMet Runtime" 1408 "SGA - SWRF Metrics WCTime" 32 "SQL Memory Manager Base W" 13400 "change notification regis" 4096 "simulator latch/bucket st" 59392 "Prefetch history buffer " 2832 "change notification obj m" 4096 "KQR ENQ " 16512 "kksss " 16464 "API data buffer length " 0 "kokcd " 0 "kohsg " 8 "Sequence Background Insta" 88 "ksfqpn " 416 "KGLS SP " 4704 "knstsg " 48 "latch classes " 352 "system default language h" 568 "name-service entry " 2592 "API data buffer array " 0 "kzull " 4096 "kzulu " 392 "kfgsga " 104 "library cache " 46604712 "kcrrny " 25320 "spfile cleanup structure " 16760 "xssinfo " 5952 "buffer_pool_desc_array " 3384 "row cache child latch " 3360 "rm request queue link " 5320 "SCHEDULING POLICY TABLE " 160 "namhsh_kfdsg " 4104 "Closed Thread SCN Bitvec " 8448 "Client ID trace settings " 3872 "osp allocation " 21104 "os statistics " 9192 "plwppwp:PLW_STR_NEW_LEN_V" 16 "plwgc: plwgc_garbage_clea" 0 "plwiiw: kglpql warnings " 0 "object queue " 808080 "obj stat memo " 599184 "obj htab chun " 122960 "object level " 111888 "XCT XGA " 0 "SGA - SWRF Metric Eidbuf " 900840 "Processor group descripto" 64 "Prefetch client count per" 32 "X$SKGXPIA " 2680 "simulator hash buckets " 2101248 "State object subpools " 896 "API data buffer length ma" 0 "AWR Table Info (KEW layer" 872 "character set memory " 4856 "sim segment num bufs " 1280 "character set object " 129728 "session idle latches " 2560 "qesmmaInitialize: " 112 "returns from remote ops " 49152 "name-service " 4080 "SGA - SWRF Metric CHBs " 10912 "listener addresses " 32 "db_block_hash_buckets " 67108864 "KSI resource types " 2704 "kglsim object batch " 4196304 "trigger condition node " 72 "ksws service events " 18560 "Heap0: KGL " 11642128 "fixed allocation callback" 392 "kqlpWrntoStr:value " 0 "KEWS statistic name " 424 "KEWS statistic maps " 1096 "KCL partition table " 131072 "kebm slave message " 88 "kcbl state objects " 12800 "free rm request queue lin" 0 "xsoqsehift " 3104 "DBWR event stats array " 192 "kgllk hash table " 659456 "event descriptor table " 192 "kpssnfy: kpsssgct " 32 "kpscad: kpscscon " 1952 "dbwriter coalesce buffer " 3158016 "kglsim hash table " 8208 "gcs resource freelist dyn" 256 "gcs shadow locks dyn seg " 256 "kks stats latch " 160 "KTC latch cleanup " 576 "ges enqueue max. usage pe" 64 "ges lmd process descripto" 2760 "KTU latch cleanup " 2496 "kscdnfyinithead " 16 "X$KSVIT table " 512 "kqlpaac:value-1 " 64 "KCL buffer header " 192064 "kxfpdp pointers " 28800 "kodosgi kopfdo " 104 "kglsim latches " 136 "TXN TABLE SEGMENTED ARRAY" 54784 "KJCT remote i " 1640 "KKJ SHRD WRQS " 288 "KJC dest ctx " 3560 "kwrsnfy: kwrs " 1624 "kwqmn:tskdata " 0 "KKKI consumer " 4136 "dbwr suspend/resume ptr a" 16 "dbwr actual working sets " 64 "KGSKI schedule " 0 "temp lob duration state o" 3296 "ges regular msg buffers " 3078008 "jsksncb: 9 " 28672 "Transportable DB Converte" 2552 "KTU lat struct " 800 "kks stats hds " 256 "KSFD SGA I/O b " 4190248 "HTTP fixed headers " 72 "UNDO INFO SEGMENTED ARRAY" 649856 "ges process hash table " 132000 "jsksncb-latch " 1280 "kfkid hrec " 24 "KTCCC OBJECT " 0 "KTPR HIST TB " 2808 "KTF MAPPINGS " 12288 "kksss-heap " 35136 "kglsim heap " 3431232 "event statistics per sess" 7665280 "eventlist to post commits" 16
从上面的数据可以看到,第3个sub pool中,占用较多的内存是gcs resources、gcs shadows以及sql area。但是没有明显的异常。
下面是第3个sub pool中第1个mini-heap中free memory的更详细数据:
SQL> break on ksmchidx on ksmchdur SQL> select 2 ksmchidx,ksmchdur, 3 case 4 when ksmchsiz < 1672 then trunc((ksmchsiz-32)/8) 5 when ksmchsiz < 4120 then trunc((ksmchsiz+7928)/48) 6 when ksmchsiz < 8216 then 250 7 when ksmchsiz < 16408 then 251 8 when ksmchsiz < 32792 then 252 9 when ksmchsiz < 65560 then 253 10 when ksmchsiz >= 65560 then 253 11 end bucket, 12 sum(ksmchsiz) free_space, 13 count(*) free_chunks, 14 trunc(avg(ksmchsiz)) average_size, 15 max(ksmchsiz) biggest 16 from 17 sys.x$ksmsp 18 where 19 inst_id = userenv(\'Instance\') and 20 ksmchcls = \'free\' 21 group by 22 case 23 when ksmchsiz < 1672 then trunc((ksmchsiz-32)/8) 24 when ksmchsiz < 4120 then trunc((ksmchsiz+7928)/48) 25 when ksmchsiz < 8216 then 250 26 when ksmchsiz < 16408 then 251 27 when ksmchsiz < 32792 then 252 28 when ksmchsiz < 65560 then 253 29 when ksmchsiz >= 65560 then 253 30 end , 31 ksmchidx, ksmchdur 32 order by ksmchidx , ksmchdur 33 / KSMCHIDX KSMCHDUR BUCKET FREE_SPACE FREE_CHUNKS AVERAGE_SIZE BIGGEST ---------- ---------- ---------- ---------- ----------- ------------ ---------- 3 1 5 72 1 72 72 13 136 1 136 136 27 248 1 248 248 48 416 1 416 416 56 1920 4 480 480 66 1680 3 560 560 68 4608 8 576 576 164 1344 1 1344 1344 180 1472 1 1472 1472 188 1536 1 1536 1536 190 1552 1 1552 1552 199 1624 1 1624 1624 204 1880 1 1880 1880 207 2032 1 2032 2032
可以看到,最大的free memory块才2032字节,而报错中提到的申请的内存大小为4128字节。由于在第3个sub pool的第1个mini heap中没有4128字节的连续free memory,所以导致内存申请失败。
那么这里的问题是,为什么这个mini heap中的free memory那么少?正如前面提及,为什么这个mini heap中的已经使用的类型全是perm类型?这个问题的答案就在于”DURATION”。Oracle在启用了SGA自动管理的模式下,为了便于在shared pool与buffer cache或其他内存之间动态调整大小,规定了在每一个mini heap中分配内存按照duration来进行。这里duration可以理解为内存块的持久时间。perm类型的内存块,就是分配后不能释放,只能用于相同组件的重用。比如gcs resources这种组件的内存是perm类型,这种内存被分配后,不能释放给sql area使用,也不能给gcs shadows使用,只能给其他的gcs resource使用。按DURATION分配内存时,perm类型的内存就只能从每个sub pool的第1个mini heap中分配。而其他类型的内存通常在sub pool的第2-4个mini heap中分配。由于perm类型的内存不能释放,也不能被其他组件的内存重用,所以里面的内存会越用越少,如果没有了free memory怎么办?前面说到,这种模式主要是工作在SGA自动管理模式下,如果free memory没有了,就会从SGA中的其他部分,比如buffer cache中取得memory chunk,加入到缺少内存的mini heap中。正常情况下这种机制没有问题。
完全使用SGA自动管理有一个缺陷就是,如果应用系统绑定变量做得不好,或者由于BUG,child cursor过多,导致shared pool会变得很大,甚至超过10G,严重的比buffer cache还大,另一方面,在buffer cache和shared pool之间频繁地调整大小,可能会导致严重的解析问题和其他性能问题。针对这个问题,通常有2种解决办法:一种就是关闭SGA自动管理,即将SGA_TARGET设置为0,以9i的方式来设置shared_pool_size,db_cache_size这些参数,来手动管理SGA;第二种就是sga_target仍然大于0,即自动管理SGA,但是通过设置shared_pool_size,db_cache_size等参数限制这些内存组件的最小大小,而只留给系统极少的自动调整空间。
而出现问题的这套系统,正是使用了第二种方式,开启了SGA自动调整,但是留给自动调整的空间极少。SGA_TARGET为35G,buffer_cache_size为30G,shared_pool_size为4G,再加上large_pool等组件,几乎没有什么可自动调整的余地。这种方式下,就存在了问题。下面来做一个按时间的分析:
在上面的时间点T5那里,如果SGA有较大的自动调整空间,比如说完全没有限制,即buffer_cache_size等参数很少或为0,这样在请求第1个mini heap中的内存时,完全可以从buffer cache中占用,这样的后果是使shared pool越来越大。
而本文案例的ORA-04031,正是由于SGA自动管理,而自动调整的余地又太小,最终使sub pool的第1个mini heap空间用光。当然我们可以分析为什么会用光,这个就显得更为复杂,这跟数据量、应用系统都有很大的关系。而系统中第1次出现ORA-04031错误的进程,是一个job进程,而此后大部分出现的错误均是job进程,能检查job代码,发现在做大量的表的大量数据的UPDATE操作,这可能是引起gcs shadows和gcs resources大量内存使用的原因。在一套RAC数据库中,gcs和ges相关的perm内存占用可能会比较大。
那么除了调整应用,应该怎么样解决这样问题?这里的解决方法是增加shared_pool_size参数到6G,同时将sga_target设置为0,再重启。
而另一种可能的办法是将参数“_enable_shared_pool_durations”设置为FALSE。这一参数为FALSE,将会使shared pool内存分配时,不再使某一类型的内存(比如perm)必须要求在一个固定的mini heap中。而实际上,sga_target设置为0之后,这一个参数自动会设为FALSE(由于这一参数是静态参数,所以修改了sga_target之后需要重启才会使这个隐含参数改变),所以建议的解决办法是设置sga_target参数,而不建议修改隐含参数。当然还有一种办法是完全让Oracle自动管理SGA,将buffer_cache_size和shared_pool_size等参数设置为0,但是正如前面所说,这种方法有比较大的缺陷。
扫一扫订阅我的微信号:IT技术博客大学习
- 作者:老熊 来源: 老熊的三分地-Oracle、UNIX、数据恢复
- 标签: ORA-04031
- 发布时间:2012-04-09 12:24:59
- [55] Oracle MTS模式下 进程地址与会话信
- [55] IOS安全–浅谈关于IOS加固的几种方法
- [54] 如何拿下简短的域名
- [53] android 开发入门
- [52] Go Reflect 性能
- [52] 图书馆的世界纪录
- [49] 读书笔记-壹百度:百度十年千倍的29条法则
- [48] 【社会化设计】自我(self)部分――欢迎区
- [38] 程序员技术练级攻略
- [32] 视觉调整-设计师 vs. 逻辑