ORA-00304: requested INSTANCE_NUMBER is busy

Problem: Creating a copy of 2 databases using netapp flexclones on a sandpit cluster. Both databases were to be mounted on the same node as a colleague was using node 2 for a different database clone. One of the databases came up fine, but the other always gave the following error:

ORA-00304: requested INSTANCE_NUMBER is busy

The Alert log showed the following error:

USER (ospid: 1110): terminating the instance due to error 304
Instance terminated by USER, pid = 1110

Error 304 is:

00304, 00000, "requested INSTANCE_NUMBER is busy"

// *Cause: An instance tried to start by using a value of the

// initialization parameter INSTANCE_NUMBER that is already in use.

// *Action: Either

// a) specify another INSTANCE_NUMBER,

// b) shut down the running instance with this number

// c) wait for instance recovery to complete on the instance with this number.

Tried several different things including checking shared memory /dev/shm for any memory blocks relating to the instance i was trying to start (which have memory attached), but no memory for that instance.

Also tried rebooting the machine which didn’t help so started looking into the startup parameters.

Looked into this blog article but this wasn’t the solution for me: http://arup.blogspot.co.uk/2009/12/instancenumber-is-busy-message-during.html

Solution

As we were cloning and starting up several databases on Node 1, and a colleague was working on a separate database on Node 2. There was a point when his database on Node 2 was mounted as instance number 2 on the cluster, and my database was trying to be the same instance number on Node 1.

As CRS was still the underlying architecture holding the nodes together, the fact that we were mounting databases which to CRS were on the same cluster, it was causing an error.

Make sure to check all databases on the cluster when working in sandpit scenarios and in cluster aware situations! Once we shut down the database on Node 2, we were able to mount without issues.

CRS Commands

Some basic CRS Commands for 11.1

--start and stop
sudo su

cd /etc/init.d/
[HOST]host:/etc/init.d : ./init.crs start
Startup will be queued to init within 30 seconds.

cd /etc/init.d/
[HOST]host:/etc/init.d : ./init.crs stop

To view the current status of CRS

crs_stat -t

Full documentation list for CRS 11.1:

http://docs.oracle.com/cd/B28359_01/rac.111/b28255/crsref.htm