PITFALL with rootupgrade.sh in 12c

Some day ago I’m stumbled upon problem while has upgraded 12c cluster. In our lab we have next plan for upgrade.

1. Install new GI soft without configuring.
2. Configure new GI.

So, after executed config.sh we need to execute rootupgrade.sh and here we stumbled upon problem.

# /opt/oracle/product/grid/12.1.0.2/rootupgrade.sh 
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/oracle/product/grid/12.1.0.2

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/oracle/product/grid/12.1.0.2/crs/install/crsconfig_params
2015/03/04 16:08:52 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2015/03/04 16:08:52 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.

2015/03/04 16:09:00 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.

2015/03/04 16:09:11 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2015/03/04 16:09:11 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil2.log' does not exist

2015/03/04 16:09:11 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil2.log' does not exist

2015/03/04 16:09:11 CLSRSC-175: Failed to write the checkpoint 'ROOTCRS_STACK' with status 'START' (error code 1)

2015/03/04 16:09:11 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil3.log' does not exist

2015/03/04 16:09:11 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil3.log' does not exist

2015/03/04 16:09:11 CLSRSC-177: Failed to add (property/value):('VERSION'/'12.1.0.2.0') for checkpoint 'ROOTCRS_STACK' (error code 1)

2015/03/04 16:09:11 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil4.log' does not exist

2015/03/04 16:09:11 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil4.log' does not exist

2015/03/04 16:09:11 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil5.log' does not exist
...
...
...
2015/03/04 17:13:08 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil7.log' does not exist

2015/03/04 17:13:08 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil7.log' does not exist

2015/03/04 17:13:08 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil8.log' does not exist

2015/03/04 17:13:08 CLSRSC-46: Error: '/opt/oracle/base/crsdata/balancebckp2e/crsconfig/cluutil8.log' does not exist

2015/03/04 17:13:08 CLSRSC-175: Failed to write the checkpoint 'ROOTCRS_STACK' with status 'FAIL' (error code 1)
Died at /opt/oracle/product/grid/12.1.0.2/crs/install/crsutils.pm line 11802.

Now, we need to go into exact script and exact line (/opt/oracle/product/grid/12.1.0.2/crs/install/crsutils.pm line 11802.)

sub writeOcrKeyPair
{
  my $keyName = $_[0];
  my $keyValue = $_[1];
  my $fullKeyName = "SYSTEM.rootcrs.$keyName";
  my $crshome  = $CFG->ORA_CRS_HOME;
  my @program = ('-exec', '-ocrsetval', '-key', $fullKeyName, '-value', $keyValue);
  my $args_str = join(' ', @program);

  my ($rc, @capout) = cluutil(FALSE, $args_str, $crshome); #run as root

  if (0 != $rc)
   {
     trace("Failed to write the key pair ($fullKeyName:$keyValue) ".
           "to OCR. \n Error message is : @capout.  Error code is $rc");
     die(dieformat(175, $fullKeyName, $keyValue, $rc))
       if((CKPTFAIL ne $keyValue) && (CKPTSTART ne $keyValue));
     return FAILED;
   }
   else
   {
     trace("Succeeded in writing the key pair ".
           "($fullKeyName:$keyValue) to OCR");
     return SUCCESS;
   }
}

After some time for investigating, I’ve figured out that we have incorrect value in $rc variable and condition “if (0 != $rc)” isn’t work properly.

So, we need to drill down to procedure cluutil.

if ($run_as_owner)
  {
    if (-e $cluutil_trc_file)
    {
      s_set_ownergroup($CFG->params('ORACLE_OWNER'),
                       $CFG->params('ORA_DBA_GROUP'),
                       $cluutil_trc_file);
      s_set_perms("0755", $cluutil_trc_file);
    }
    $rc = run_as_user2($CFG->params('ORACLE_OWNER'), \@capout, $cmd);
  }

And again we have not correct value for $rc variable. Going to the next procedure run_as_user2. But this procedure just only has call another procedure.

return $CFG->s_run_as_user2e($user, $aref, @_);

Ok, we should go deeper 🙂 into s_run_as_user2e procedure. Had Spend some time for investigation how it works. I’m decide that I want to know what we have in buffer @_. Then I’ve added strings with stars (***) into script.

sub s_run_as_user2e {
  my $cfg = shift;
  my $user = shift;
  my $aref = shift;
  my $rc;
  my $fuck; --***
  my @args = (\@_, $user, $aref);
  if (!$cfg->s_run_as_user2_v2p) {
    my $pgm  = $cfg->s_run_as_user2p;
    $rc = &$pgm(\@_, $user, $aref);
    open($fuck, '>', '/opt/oracle/base/crsdata/some/crsconfig/cluutil1.log'); --***
    print $fuck join(", ","@_"); --***
    close $fuck; --***
  }
  else {
    my $pgm = $cfg->s_run_as_user2_v2p;
    my @out = &$pgm($user, @_);
    $rc = shift @out;
    @{$aref} = @out;
  }
  return $rc;
}

And after that in the log file I’ve seen this string.

echo CLSRSC_START; /opt/oracle/product/grid/12.1.0.2/bin/cluutil ..........

Ok, lets call it.

$ /opt/oracle/product/grid/12.1.0.2/bin/cluutil -h
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

Oh, I see.

$ export _JAVA_OPTIONS="-Xmx256M"

$ /opt/oracle/product/grid/12.1.0.2/bin/cluutil -h
Picked up _JAVA_OPTIONS: -Xmx256M
oracle.ops.mgmt.cluster.ClusterException: PRKU-1001 : Unexpected argument -h.
usage: cluutil [-copy] -sourcefile <Source file location> [-fromnodesfile <source nodelist file>] -destfile <destination file location> {-nodelist node1[,node2,...] | -nodelistfile <destination nodelist file>} [-oraclehome <oracle home path>]

Now we can re-run rootupgrade.sh

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s