Today I’ve came across strange error related to srvctl add service.
$ srvctl add service -d db -s SERVICE_NAME -preferred INST1,INST2
PRCR-1006 : Failed to add resource ora.db.db_service.svc for SERVICE_NAME
PRCT-1011 : Failed to run "osdbagrp". Detailed error:
You might wonder how CSSD, which is required to start the clustered ASM instance, can be started if voting disks are stored in ASM? This sound like a chicken-and-egg problem. Without access to the voting disks there is no CSS, hence the node can’t join the cluster. But without begin part of the cluster, CSSD can’t start the ASM instance. To solve this problem the ASM disk headers have metadata since 11.2. You can use kfed to read the headers of ASM disks containing a voting disk. The kfdhdb.vfstart and kfdhdb.vfend fields tell CSS where to find the voting file. This does not require the ASM instance to be up. Once the voting disks are located, CSS can access them and joins the cluster.
SQL> alter diskgroup data add failgroup FG01 disk 'ORCL:T01L209' rebalance power 10;
alter diskgroup data add failgroup FG01 disk 'ORCL:T01L209' rebalance power 10
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15137: cluster in rolling patch
[root@node1 ~]# /opt/oracle/product/grid/126.96.36.199/bin/crsctl query crs softwarepatch
Oracle Clusterware patch level on node node1 is .
[root@node2 ~]# /opt/oracle/product/grid/188.8.131.52/bin/crsctl query crs softwarepatch
Oracle Clusterware patch level on node node2 is .
[root@node1 ~]# /opt/oracle/product/grid/184.108.40.206/bin/clscfg -patch
clscfg: -patch mode specified
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
[root@node1 ~]# /opt/oracle/product/grid/220.127.116.11/bin/crsctl stop rollingpatch
CRS-1161: The cluster was successfully patched to patch level .
ONE MORE THING
If you’ve got more than one node, and in some reason you have different patch levels between nodes.
For example after unseccessful patching action. You need to use clscfg -localpatch for patching OLR, and ater that clscfg -patch for patching OCR.
So, as you know if you have problem with your diskgroup in ASM which contain OCR/Voting. You can re-create diskgroup and restore ocr from backup and make new voting files. But there are smaller nuance. You need to set attribute “compatible.asm” equal or greater 18.104.22.168 otherwise you will get an error.
# /opt/oracle/product/grid/22.214.171.124/bin/ocrconfig -restore /opt/oracle/product/grid/126.96.36.199/cdata/cluster-name/backup00.ocr
PROT-35: The configured OCR locations are not accessible.
# /opt/oracle/product/grid/188.8.131.52/bin/crsctl replace votedisk +OCRVT
Oracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage
ASMCMD> lsattr -l -G ocrvt
ASMCMD> setattr -G ocrvt compatible.asm 184.108.40.206.0