Backend WLS or EM application seems to be down

A few days ago, the server, where OMS and its DB reside, crushed.
After we restarted it, whenever we tried to access the grid console, we got error "Backend WLS or EM application seems to be down".
Agents failed to upload XML files to OMS and "emctl pingOMS" was giving an error "EMD pingOMS error: No response header from OMS".
We checked WebLogic and OMS .trc, .log and .out files, but there was no error recorded, neither before nor after the crush.

we had to restart database and restarted grid control 12c.

To correct this issue:

    1. Stop OMS 

     /u00/app/gc/oms12cr3/oms/bin/emctl stop oms -all

    2. Kill -9 all WebLogic and OMS processes still running after the stop. You can find these processes, using ps.

     ps -ef | grep EMGC_ADMINSERVER
     ps -ef | grep EMGC_OMS1
     ps -ef | grep oms

    3. Delete every .lok file you find under WebLogic Domain

cd /u00/app/gc/gc_inst/user_projects/domains/GCDomain

find . -name "*.lok"

    These files were:

    ../gc_inst/user_projects/domains/GCDomain/config/config.lok
    ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/ldap/ldapfiles/EmbeddedLDAP.lok
    ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/EMGC_OMS1.lok
    ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles/EmbeddedLDAP.lok
    ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/tmp/EMGC_ADMINSERVER.lok

    4. Start OMS

     /u00/app/gc/oms12cr3/oms/bin/emctl start oms

   5. Check OMS log file after restart

tail -f /u00/app/gc/gc_inst/NodeManager/emnodemanager/nodemanager.log
tail -f /u00/app/gc/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out    

tail -f /u00/app/gc/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out

The best matching Oracle Documents about this incident are:
The best matching Oracle Documents about this incident are:

• ID 943790.1: What are the .lok Files Used For in a WebLogic Server (WLS) Domain? In general, these files are a mechanism to ensure file and server locks and to prevent a server from being booted twice.
• ID 957377.1: Weblogic Fails To Start With Error "Unable To Obtain Lock"
• ID 1235753.1: 11g Grid Control: OMS Startup Shows "AdminServer Could Not Be Started" but OMS is able to Startup

JAVA GARBAGE COLLECTION LOGGING

Java’s built-in garbage collection logging provides a quick, easy, and free way to profile or troubleshoot your Java application.  It can help you understand your application in terms of:

• memory usage (object count and size)
• heap size (initial and over time)
• GC metrics (frequency, pause time, and amount freed)

So, now you can answer questions like “how much memory is my app using,” “how much time is being spent doing garbage collection,” and “how big is the heap over time.” For help analyzing and visualizing a GC log file, be sure to check out GCViewer, a garbage collection and heap analysis tool.

GC logging is enabled using JVM arguments; below are the arguments I use. (Note: the log file specified as file is reset each time the VM starts.)

-verbose:gc -Xloggc:file

Dan Evans, in his garbage collection talk at DevNexus 2013, recommends the following GC flags in addition to the above.

-XX:+PrintGCDetails -XX:+PrintTenuringDistribution

Here’s an example GC log file which shows three GC events including one full GC.

[GC 19062K->7172K(60800K), 0.0087190 secs]
[GC 9346K->7525K(60800K), 0.0052810 secs]
[Full GC 7525K->7477K(60800K), 0.0615190 secs]

In the first event, garbage collection takes just under nine thousandths of a second to collect 11,890K memory used by objects (19,062K to 7,172K) while the heap is 60,800K.  The last event is a full GC lasting just over six hundredths of a second to free up 48K with no change in the heap.

install jdk on linux using alternatives

For installing Java, make that file executable by becoming root and type

# chmod a+x jdk-6urpm.bin
#./jdk-6urpm.bin

Check your default Java installation

# /usr/sbin/alternatives –config java

If latest Java installation is not avaliable, you must update alternatives with

# alternatives –install “usr/bin/java” “java” “/usr/java/jkd1.6.0_/bin/java” 2

Note that number 2 means that this Java installation will be shown as second when you run config java( if you have icedtea or open jdk for example, if not, type 1 ).
Now, check again Java with

# /usr/sbin/alternatives –config java

And select new JDK as default.

SRVCTL Commands 11g R2 RAC

SRVCTL Commands 11g R2 RAC

OBJECTS

1. Database 
2. Instance
3. Listener
4. VIP 
5. Scan
6. Scan_Listener
7. Service
8. File System
9. Disk group
10. Server
11. srvpool
12. Nodeapps
13. ons
14. eons
15. gns
16. oc4j

SRVCTL help command

$ srvctl -help or $ srvctl -v

$ srvctl -V  (To check the version)

To get complete help of  all commands with objects

$ srvctl -h

To get the help for specific command

$ srvctl  

Ex: $ srvctl add database -h 

I.  Object - Database:

Syntax for 10g RAC

srvctl add database -d db_name -o ORACLE_HOME [-m domain_name][-p spfile] [-A name | ip/netmask]

[-r {PRIMARY|PHYSICAL_STANDBY|LOGICAL_STANDBY|SNAPSHOT_STANDBY}]

[-s start_options] [-n db_name] [-y {AUTOMATIC|MANUAL}]

$ srvctl add database -d orcl -o /u01/oracle/product/10.2/dbs

srvctl remove database -d db_name [-f]

$ srvctl remove database -d orcl

srvctl start database -d db_name [-o start_options] [-c connect_str | -q]

srvctl start database -d db_name [-o open]

srvctl start database -d db_name -o nomount

srvctl start database -d db_name -o mount

$ srvctl start db -d orcl
$ srvctl start database -d orcl -o open

srvctl stop database -d db_name [-o stop_options] [-c connect_str | -q]

srvctl stop database -d db_name [-o normal]

srvctl stop database -d db_name -o transactional

srvctl stop database -d db_name -o immediate

srvctl stop database -d db_name -o abort

$ srvctl stop db -d orcl -o immediate

srvctl status database -d db_name [-f] [-v] [-S level]

srvctl status database -d db_name -v service_name

$ srvctl status database -d orcl

srvctl enable database -d db_name$ srvctl enable database -d orcl

srvctl disable database -d db_name

$ srvctl disable db -d orcl

srvctl config database

srvctl config database -d db_name [-a] [-t]

$ srvctl config database
$ srvctl config database -d orcl -t  (for 10g RAC)

$ srvctl config database -d orcl -a (for 11g R2 RAC)

srvctl modify database -d db_name [-n db_name] [-o ORACLE_HOME] [-m domain_name] [-p spfile]

[-r {PRIMARY|PHYSICAL_STANDBY|LOGICAL_STANDBY|SNAPSHOT_STANDBY}] [-s start_options] [-y {AUTOMATIC|MANUAL}]

$ srvctl modify database -d orcl -r primary

$ srvctl modify db -d orcl -p /u01/ oracle/product/10.2/dbs/spfileRAC.ora -- moves the pfile
$ srvctl modify database –d orcl –o /u01/ oracle/product/10.2/dbs –s open

srvctl getenv database -d db_name [-t name_list]$ srvctl getenv database -d orcl

srvctl setenv database -d db_name {-t name=val[,name=val,...]|-T name=val}$ srvctl setenv database –d orcl –t “TNS_ADMIN=/u01/app/oracle/product/11.1/asm/network/admin”
$ srvctl setenv db -d orcl -t LANG=en

srvctl unsetenv database -d db_name [-t name_list]

$ srvctl unsetenv database -d orcl -t CLASSPATH

Syntax for 11g R2 RAC

srvctl add database -d db_unique_name -o ORACLE_HOME [-x node_name] [-m domain_name] [-p spfile] [-r {PRIMARY|PHYSICAL_STANDBY|LOGICAL_STANDBY|SNAPSHOT_STANDBY}] [-s start_options] [-t stop_options] [-n db_name] [-y {AUTOMATIC|MANUAL}] [-g server_pool_list] [-a "diskgroup_list"]
$ srvctl add database -d orcl -o /u01/oracle/product/11.2/dbs -m oracle.com -p +data/orcl/spfileorcl.ora -r PRIMARY -s open -t normal -n db2 -y AUTOMATIC -g svrpool1,svrpool2 -a "dg1,dg2"

srvctl remove database -d db_unique_name [-f  Force remove] [-y To Suppress Prompt] [-v Verbose output]
$ srvctl remove database -d orcl -y

srvctl stop database -d db_unique_name [-o stop_options] [-f]
$ srvctl stop database -d orcl -f

srvctl status database -d db_unique_name [-f] [-v]
$ srvctl status db -d orcl -v

srvctl enable database -d db_unique_name [-n node_name]
$ srvctl enable database -d orcl -n rac1

srvctl disable database -d db_unique_name [-n node_name]
$ srvctl disable db -d orcl -n rac3

srvctl config database [-d db_unique_name [-a To list options]
$ srvctl config db -d orcl -a

srvctl modify database -d db_unique_name [-n db_name] [-o ORACLE_HOME] [-u oracle_user] [-m domain] [-p spfile] [-r {PRIMARY|PHYSICAL_STANDBY|LOGICAL_STANDBY|SNAPSHOT_STANDBY}] [-s start_options] [-t stop_options] [-y {AUTOMATIC|MANUAL}] [-g "server_pool_list"] [-a "diskgroup_list"|-z]
$ srvctl modify db -d orcl -r logical_standby
$ srvctl modify database -d orcl -a "SYSFILES,LOGS,OLTP"
$ srvctl modify database -d orcl -e rac1,rac2

srvctl relocate database -d db_unique_name {[-n target_node] [-w timeout] | -a [-r]} [-v]

$ srvctl relocate database -d orcl -n rac1
$ srvctl relocate database -d orcl -n rac1 -w 120 -v 

srvctl convert database -d

$ srvctl convert database -d orcl -c RAC -n rac1
$ srvctl convert database -d orcl -c RACONENODE -i RoneDB


 

2.   Object - Instance:

Syntax for 10g RAC

srvctl add instance –d db_name –i inst_name -n node_name$ srvctl add instance -d orcl -i inst1 -n rac1

srvctl remove instance –d db_name –i inst_name [-f]

$ srvctl remove instance -d orcl -i inst1

srvctl start instance -d db_name -i inst_names [-o start_options] [-c connect_str | -q]

srvctl start instance –d db_name –i inst_names [-o open]

srvctl start instance –d db_name –i inst_names -o nomount

srvctl start instance –d db_name –i inst_names -o mount

$ srvctl start instance –d orcl -i inst2

srvctl stop instance -d db_name -i inst_names [-o stop_options] [-c connect_str | -q]

srvctl stop instance –d db_name –i inst_names [-o normal]

srvctl stop instance –d db_name –i inst_names -o transactional

srvctl stop instance –d db_name –i inst_names -o immediate

srvctl stop instance –d db_name –i inst_names -o abort

$ srvctl stop inst –d orcl -i inst1

srvctl status instance –d db_name –i inst_names [-f] [-v] [-S level]

$ srvctl status inst –d orcl -i inst1

srvctl enable instance –d db_name –i inst_names$ srvctl enable instance -d orcl -i "inst1,inst2"

srvctl disable instance –d db_name –i inst_names

$ srvctl disable inst -d orcl -i "inst1,inst3"

srvctl modify instance -d db_name -i inst_name {-s asm_inst_name|-r} -- set dependency of instance to ASM

srvctl modify instance -d db_name -i inst_name -n node_name -- move the instance

srvctl modify instance -d db_name -i inst_name -r -- remove the instance

srvctl getenv instance –d db_name –i inst_name [-t name_list]

srvctl setenv instance –d db_name [–i inst_name] {-t "name=val[,name=val,...]" | -T "name=val"}

srvctl unsetenv instance –d db_name [–i inst_name] [-t name_list]
 

Syntax for 11g R2 RAC

srvctl start instance -d db_unique_name {-n node_name -i "instance_name_list"} [-o start_options]
$ srvctl start instance -d orcl -n rac1
$ srvctl start inst -d orcl -i "inst1,inst2"

srvctl stop instance -d db_unique_name {[-n node_name]|[-i "instance_name_list"]} [-o stop_options] [-f]
$ srvctl stop inst -d orcl -n rac1
$ srvctl stop instance -d orcl -i inst1

srvctl status instance -d db_unique_name {-n node_name | -i "instance_name_list"} [-f Include disabled applications] [-v verbose output]
$ srvctl status instance -d orcl -i "inst2,inst3" -v

srvctl modify instance -d db_unique_name -i instance_name {-n node_name | -z To remove existing node instance mapping for policy-managed database}
$ srvctl modify instance -d orcl -i inst1 -n rac1
$ srvctl modify inst -d orcl -i inst1 -z

3.  Object - Listener:

Syntax for 10g RAC

srvctl start listener -n node_name [-l listener_names]$ srvctl start listener -n rac1

srvctl stop listener -n node_name [-l listener_names]

$ srvctl stop listener -n rac1

srvctl config listener -n node_name
Syntax for 11g R1 RAC

srvctl add listener -n node_name -o ORACLE_HOME [-l listener_name]  

srvctl remove listener -n node_name [-l listener_name]

srvctl status listener [-n node_name] [-l listener_names] 
$ srvctl status listener -n rac2

srvctl modify listener -n node_name [-l listener_names] -o ORACLE_HOME

$ srvctl modify listener -n rac1 -o /u01/app/oracle/product/11.1/asm -l "LISTENER_RAC1"
Syntax for 11g R2 RAC

srvctl add listener [-l lsnr_name] [-s] [-p "[TCP:]port[, ...][/IPC:key][/NMP:pipe_name][/TCPS:s_port] [/SDP:port]"] [-k network_number] [-o ORACLE_HOME]
$ srvctl add listener -l list_prod1 -p "TCP:1522" -o $ORACLE_HOME
$ srvctl add listener -l list_prod1 -p 1341 -o /ora/ora112
srvctl remove listener [-l lsnr_name|-a] [-f]
$ srvctl remove listener -l list_prod1
srvctl stop listener [-n node_name] [-l lsnr_name] [-f]
srvctl enable listener [-l lsnr_name] [-n node_name]
$ srvctl enable listener -l list_prod2 -n rac2
srvctl disable listener [-l lsnr_name] [-n node_name]
$ srvctl disable listener -l list_prod2 -n node1
srvctl config listener [-l lsnr_name] [-a]
$ srvctl config listener
srvctl modify listener [-l listener_name] [-o oracle_home] [-u user_name] [-p "[TCP:]port_list[/IPC:key][/NMP:pipe_name][/TCPS:s_port][/SDP:port]"] [-k network_number]
$ srvctl modify listener -n rac1 -p "TCP:1521,1522"
srvctl getenv listener [-l lsnr_name] [-t name[, ...]]
$ srvctl getenv listener
srvctl setenv listener [-l lsnr_name] {-t "name=val [,...]" | -T "name=value"}
$ srvctl setenv listener -t LANG=en
srvctl unsetenv listener [-l lsnr_name] -t "name[, ...]"
$ srvctl unsetenv listener -t "TNS_ADMIN"

  

4.  Object - VIP (Virtual Internet Protocol):

Syntax for 11g R2 RAC (New SRVCTL commands in 11g R2)

srvctl add vip -n node_name -A {name|ip}/netmask[/if1[if2|...]] [-k network_number] [-v]

#srvctl add vip -n rac1 -A 192.168.1.108/255.255.255.0 -k 2

srvctl remove vip -i "vip_name_list" [-f] [-y] [-v]

$ srvctl remove vip -i "vip1,vip2,vip3" -f -y -v

srvctl start vip {-n node_name|-i vip_name} [-v]

$ srvctl start vip -i rac1-vip -v

srvctl stop vip {-n node_name|-i vip_name} [-r] [-v]

$ srvctl stop vip -n rac1 -v

srvctl status vip {-n node_name|-i vip_name}

$ srvctl status vip -i rac1-vip

srvctl enable vip -i vip_name [-v]

$ srvctl enable vip -i rac1-vip -v

srvctl disable vip -i vip_name [-v]

$ srvctl disable vip -i vip3 -v

srvctl config vip {-n node_name|-i vip_name}

$ srvctl config vip -n rac2

srvctl getenv vip -i vip_name [-t "name_list"] [-v]

$ srvctl getenv vip -i rac1-vip

srvctl setenv vip -i vip_name {-t "name=val[,name=val,...]" | -T "name=val"}

$ srvctl setenv vip -i rac1-vip -t LANG=en

srvctl unsetenv vip -i vip_name -t "name_list" [-v]

$ srvctl unsetenv vip -i rac2 -t CLASSPATH

5.   Scan (Single Client Access Name):

Syntax for 11g R2 RAC (New SRVCTL command in 11g R2)

srvctl add scan -n scan_name [-k network_number] [-S subnet/netmask[/if1[|if2|...]]]

#srvctl add scan -n oracle.com

srvctl remove scan [-f]

$ srvctl remove scan
$ srvctl remove scan -f

srvctl start scan [-i ordinal_number] [-n node_name]

$ srvctl start scan
$ srvctl start scan -i 1 -n rac1

srvctl stop scan [-i ordinal_number] [-f]

$ srvctl stop scan
$ srvctl stop scan -i 1

srvctl status scan [-i ordinal_number]

$ srvctl status scan
$ srvctl status scan -i 1

srvctl enable scan [-i ordinal_number]

$ srvctl enable scan
$ srvctl enable scan -i 1

srvctl disable scan [-i ordinal_number]

$ srvctl disable scan
$ srvctl disable scan -i 1

srvctl config scan [-i ordinal_number]

$ srvctl config scan
$ srvctl config scan -i 1

srvctl modify scan -n scan_name

$ srvctl modify scan
$ srvctl modify scan -n scan2

srvctl relocate scan -i ordinal_number [-n node_name]

srvctl relocate scan -i 2 -n rac1

ordinal_number=1,2,3

6.  Object - Scan_Listener:

Syntax for 11g R2 RAC (New SRVCTL command in 11g R2)

srvctl add scan_listener [-l lsnr_name_prefix] [-s] [-p "[TCP:]port_list[/IPC:key][/NMP:pipe_name][/TCPS:s_port] [/SDP:port]"]

#srvctl add scan_listener -l list_prod1

srvctl remove scan_listener [-f]

$ srvctl remove scan_listener
$ srvctl remove scan_listener -f

srvctl start scan_listener [-n node_name] [-i ordinal_number]

$ srvctl start scan_listener
$ srvctl start scan_listener -i 1

srvctl stop scan_listener [-i ordinal_number] [-f]

$ srvctl stop scan_listener -i 2

srvctl status scan_listener [-i ordinal_number]

$ srvctl status scan_listener
$ srvctl status scan_listener -i 3

srvctl enable scan_listener [-i ordinal_number]

$ srvctl enable scan_listener
$ srvctl enable scan_listener -i 1

srvctl disable scan_listener [-i ordinal_number]

$ srvctl disable scan_listener
$ srvctl disable scan_listener -i 2

srvctl config scan_listener [-i ordinal_number]

$ srvctl config scan_listener
$ srvctl config scan_listener -i 3

srvctl modify scan_listener {-p [TCP:]port[/IPC:key][/NMP:pipe_name] [/TCPS:s_port][/SDP:port] | -u }

$ srvctl modify scan_listener -u

srvctl relocate scan_listener -i ordinal_number [-n node_name]

$ srvctl relocate scan_listener -i 1

ordinal_number=1,2,3

7.  Object - Service:

Syntax for 10g RAC

srvctl add service -d db_name -s service_name -r pref_insts [-a avail_insts] [-P TAF_policy]

srvctl add service -d db_name -s service_name -u {-r "new_pref_inst" | -a "new_avail_inst"}

$ srvctl add service -d orcl -s serv1 -r inst1,inst2 -a inst3,inst4

$ srvctl add serv -d orcl -s serv1 -r inst1 -a inst3 -P basic

srvctl remove service -d db_name -s service_name [-i inst_name] [-f]

$ srvctl remove serv -d orclv -s serv1
$ srvctl remove service -d orcl -s serv1 -i inst2,inst3

srvctl start service -d db_name [-s service_names [-i inst_name]] [-o start_options]

srvctl start service -d db_name -s service_names [-o open]

srvctl start service -d db_name -s service_names -o nomount

srvctl start service -d db_name -s service_names -o mount

$ srvctl start serv -d orcl -s serv1

srvctl stop service -d db_name [-s service_names [-i inst_name]] [-f]

$ srvctl stop serv -d orcl -s serv1

srvctl status service -d db_name [-s service_names] [-f] [-v] [-S level]$ srvctl status service -d orcl -s serv2

srvctl enable service -d db_name -s service_names [–i inst_name]$ srvctl enable service -d orcl -s serv1

srvctl disable service -d db_name -s service_names [–i inst_name]

$ srvctl disable serv -d orcl -s serv1 -i inst1

srvctl config service -d db_name [-s service_name] [-a] [-S level]

srvctl config service -d db_name -a -- -a shows TAF configuration

$ srvctl config service -d orcl -s serv1 PREF:serv2 AVAIL:serv3

srvctl modify service -d db_name -s service_name -i old_inst_name -t new_inst_name [-f]

srvctl modify service -d db_name -s service_name -i avail_inst_name -r [-f]

srvctl modify service -d db_name -s service_name -n -i preferred_list [-a available_list] [-f]

srvctl modify service -d db_name -s service_name -i old_inst_name -a avail_inst -P TAF_policy

$ srvctl modify serv -d orcl -s serv1 -n -i inst1,inst2,inst3 -a inst4,inst5

srvctl relocate service -d db_name -s service_name -i old_inst_name -t target_inst [-f Disconnect all sessions during stop or relocate service options]

-s              Service name

-i             Old instance name

-t             New instance name

-c         Node name to relocate service from

-n          Node name to relocate service to

-f                       Disconnect all sessions during stop or relocate service operations

srvctl getenv service -d db_name -s service_name -t name_list

srvctl setenv service -d db_name [-s service_name] {-t "name=val[,name=val,...]" | -T "name=val"}

srvctl unsetenv service -d db_name -s service_name -t name_list 

SYNTAX for 11g R2 RAC

srvctl add service -d db_unique_name -s service_name [-l [PRIMARY][,PHYSICAL_STANDBY][,LOGICAL_STANDBY][,SNAPSHOT_STANDBY]] [-y {AUTOMATIC|MANUAL}] [-q {true|false}] [-j {SHORT|LONG}] [-B {NONE|SERVICE_TIME|THROUGHPUT}][-e {NONE|SESSION|SELECT}] [-m {NONE|BASIC}][-z failover_retries] [-w failover_delay]$ srvctl add service -d orcl -s serv1 -q TRUE -m BASIC -e SELECT -z 180 -w 5 -j LONG

-d       Unique name for the database

-s              Service name

-r ""    Comma separated list of preferred instances

-a ""    Comma separated list of available instances

-g           Server pool name

-c {UNIFORM | SINGLETON} Service runs on every active server in the server pool hosting this service (UNIFORM) or just one server (SINGLETON)

-k              VIP network number (default number is 1)

-P {NONE | BASIC | PRECONNECT}        TAF policy specification

-l                 Role of the service (primary, physical_standby, logical_standby, snapshot_standby)

-y               Management policy for the service (AUTOMATIC or MANUAL)

-e        Failover type (NONE, SESSION, or SELECT)

-m      Failover method (NONE or BASIC)

-w              Failover delay

-z              Failover retries

-j   Connection Load Balancing Goal (SHORT or LONG). Default is LONG.

-B      Runtime Load Balancing Goal (SERVICE_TIME, THROUGHPUT, or

-x   Distributed Transaction Processing (TRUE or FALSE)

-q AQ HA notifications (TRUE or FALSE)

Usage: srvctl add service -d -s -u {-r "" | -a ""}

-d       Unique name for the database

-s              Service name

-u                       Add a new instance to service configuration

-r        Name of new preferred instance

-a       Name of new available instance

-h                       Print usage
srvctl add service -d db_unique_name -s service_name -u {-r preferred_list | -a available_list}

srvctl add service -d db_unique_name -s service_name
-g server_pool [-c {UNIFORM|SINGLETON}] [-k network_number]
[-l [PRIMARY|PHYSICAL_STANDBY|LOGICAL_STANDBY|SNAPSHOT_STANDBY]
[-y {AUTOMATIC|MANUAL}] [-q {TRUE|FALSE}] [-j {SHORT|LONG}]
[-B {NONE|SERVICE_TIME|THROUGHPUT}] [-e {NONE|SESSION|SELECT}]
[-m {NONE|BASIC}] [-P {BASIC|NONE|PRECONNECT}] [-x {TRUE|FALSE}]
[-z failover_retries] [-w failover_delay]

srvctl add service -d db_unique_name -s service_name -r preferred_list [-a available_list] [-P {BASIC|NONE|PRECONNECT}]
[-l [PRIMARY|PHYSICAL_STANDBY|LOGICAL_STANDBY|SNAPSHOT_STANDBY]
[-y {AUTOMATIC|MANUAL}] [-q {TRUE|FALSE}] [-j {SHORT|LONG}]
[-B {NONE|SERVICE_TIME|THROUGHPUT}] [-e {NONE|SESSION|SELECT}]
[-m {NONE|BASIC}] [-x {TRUE|FALSE}] [-z failover_retries] [-w failover_delay]$ srvctl add serv -d orcl -s serv1 -r serv2,serv3 -a serv4 -P PRECONNECT
srvctl start service -d db_unique_name [-s "service_name_list" [-n node_name | -i instance_name]] [-o start_options]
$ srvctl start serv -d orcl -s serv1
$ srvctl start service -d orcl -s serv1 -i inst1
srvctl stop service -d db_unique_name [-s "service_name_list"] [-n node_name | -i instance_name] [-f]
$ srvctl stop service -d orcl -s serv1
$ srvctl stop serv -d orcl -s serv1 -i inst1
srvctl status service -d db_unique_name [-s "service_name_list"] [-f] [-v]
$ srvctl status service -d orcl -s serv1 -v
srvctl enable service -d db_unique_name -s "service_name_list" [-i instance_name | -n node_name]
$ srvctl enable service -d orcl -s serv1
$ srvctl enable serv -d orcl -s serv1 -i inst1
srvctl disable service -d db_unique_name -s "service_name_list" [-i instance_name | -n node_name]
$ srvctl disable service -d orcl -s "serv1"
$ srvctl disable serv -d orcl -s serv1 -i inst1
srvctl config service -d db_unique_name [-s service_name] [-a]
$ srvctl config service -d orcl -s serv1
srvctl modify service -d db_unique_name -s service_name
[-c {UNIFORM|SINGLETON}] [-P {BASIC|PRECONNECT|NONE}]
[-l {[PRIMARY]|[PHYSICAL_STANDBY]|[LOGICAL_STANDBY]|[SNAPSHOT_STANDBY]} [-q {TRUE|FALSE}] [-x {TRUE|FALSE}] [-j {SHORT|LONG}] [-B {NONE|SERVICE_TIME|THROUGHPUT}] [-e {NONE|SESSION|SELECT}] [-m {NONE|BASIC}] [-z failover_retries] [-w failover_delay] [-y {AUTOMATIC|MANUAL}]
srvctl modify service -d db_unique_name -s service_name -i old_instance_name -t new_instance_name [-f]
srvctl modify service -d db_unique_name -s service_name -i avail_inst_name -r [-f]
srvctl modify service -d db_unique_name -s service_name -n -i preferred_list [-a available_list] [-f]
$ srvctl modify service -d orcl -s serv1 -i inst1 -t inst2
$ srvctl modify serv -d orcl -s serv1 -i inst1 -r
$ srvctl modify service -d orcl -s serv1 -n -i inst1 -a inst2
srvctl relocate service -d db_unique_name -s service_name {-c source_node -n target_node|-i old_instance_name -t new_instance_name} [-f]
$ srvctl relocate service -d orcl -s serv1 -i inst1 -t inst3

9.  Object - File System:

Syntax for 11g R2 RAC (New SRVCTL command in 11g R2)

srvctl add filesystem -d volume_device -v volume_name -g diskgroup_name [-m mountpoint_path] [-u user_name]

$ srvctl add filesystem -d /dev/asm/d1volume1 -v VOLUME1 -d RAC_DATA -m /oracle/cluster1/acfs1

srvctl remove filesystem -d volume_device_name [-f]

$ srvctl remove filesystem -d /dev/asm/racvol1

srvctl start filesystem -d volume_device_name [-n node_name]

$ srvctl start filesystem -d /dev/asm/racvol3

srvctl stop filesystem -d volume_device_name [-n node_name] [-f]

$ srvctl stop filesystem -d /dev/asm/racvol1 -f

srvctl status filesystem -d volume_device_name

$ srvctl status filesystem -d /dev/asm/racvol2

srvctl enable filesystem -d volume_device_name

$ srvctl enable filesystem -d /dev/asm/racvol9

srvctl disable filesystem -d volume_device_name

$ srvctl disable filesystem -d /dev/asm/racvol1
srvctl config filesystem -d volume_device_path

srvctl modify filesystem -d volume_device_name -u user_name

$ srvctl modify filesystem -d /dev/asm/racvol1 -u sysadmin

9.  Object - File System:

Syntax for 11g R2 RAC (New SRVCTL command in 11g R2)

srvctl remove diskgroup -g diskgroup_name [-n node_list] [-f]

$ srvctl remove diskgroup -g diskg1 -f

srvctl start diskgroup -g diskgroup_name [-n node_list]

$ srvctl start diskgroup -g diskg1 -n rac1,rac2

srvctl stop diskgroup -g diskgroup_name [-n node_list] [-f]

$ srvctl stop diskgroup -g diskg1

$ srvctl stop diskgroup -g diskg2 -n rac2,rac3 -f

srvctl status diskgroup -g diskgroup_name [-n node_list] [-a]

$ srvctl status diskgroup -g diskg1 -n rac1,rac2 -a

srvctl enable diskgroup -g diskgroup_name [-n node_list]

$ srvctl enable diskgroup -g diskgroup1 -n node1,node2

srvctl disable diskgroup -g diskgroup_name [-n node_list]

$ srvctl disable diskgroup -g dg_fra -n node1, node2

10.  Object - Server:

Syntax for 11g R2 RAC (New SRVCTL command in 11g R2)

srvctl status server -n "server_name_list" [-a]

$ srvctl status server -n serv1 -a

srvctl relocate server -n "server_name_list" -g server_pool_name [-f]

$ srvctl relocate server -n "serv1, serv2" -g sp2

11.  Object - Srvpool (Server Pool):

Syntax for 11g R2 RAC (New SRVCTL command in 11g R2)

srvctl add srvpool -g server_pool [-i importance] [-l min_size] [-u max_size] [-n node_list] [-f]

$ srvctl add srvpool -g srvp1 -i 1 -l 3 -u 7 -n rac1,rac2

srvctl remove srvpool -g server_pool

$ srvctl remove srvpool -g srvp1

srvctl status srvpool [-g server_pool] [-a]

$ srvctl status srvpool -g srvp1 -a

srvctl config srvpool [-g server_pool]

$ srvctl config srvpool -g srvp2

srvctl modify srvpool -g server_pool [-i importance] [-l min_size] [-u max_size] [-n node_name_list] [-f]

$ srvctl modify srvpool -g srvp2 -i 0 -l 2 -u 4 -n rac1, rac2

  

12.  Object - Nodeapps:

Syntax for 10g RAC

#srvctl add nodeapps -n node_name -o ORACLE_HOME -A name|ip/netmask[/if1[|if2|...]]

#srvctl add nodeapps -n lnx02 -o $ORACLE_HOME -A 192.168.0.151/255.255.0.0/eth0

#srvctl remove nodeapps -n node_names [-f]

#srvctl start nodeapps -n node_name     -- Starts GSD, VIP, listener & ONS

#srvctl stop nodeapps -n node_name [-r] -- Stops GSD, VIP, listener & ONS

#srvctl status nodeapps -n node_name

#srvctl config nodeapps -n node_name [-a] [-g] [-o] [-s] [-l]

-a Display VIP configuration
-g Display GSD configuration
-s Display ONS daemon configuration
-l Display listener configuration

#srvctl modify nodeapps -n node_name [-A new_vip_address]

#srvctl modify nodeapps -n lnx06 -A 10.50.99.43/255.255.252.0/eth0

#srvctl getenv nodeapps -n node_name [-t name_list]

#srvctl setenv nodeapps -n node_name {-t "name=val[,name=val,...]"|-T "name=val"}#srvctl setenv nodeapps –n adcracdbq3 –t “TNS_ADMIN=/u01/app/oracle/product/11.1/asm/network/admin”

#srvctl unsetenv nodeapps -n node_name [-t name_list]

Syntax for 11g R2 RAC

srvctl add nodeapps -n node_name -A {name|ip}/netmask[/if1[|if2|...]] [-m multicast_ip_address] [-p multicast_port_number] [-l ons_local_port] [-r ons_remote-port] [-t host[:port][,host[:port],...]] [-v]
srvctl add nodeapps -S subnet/netmask[/if1[|if2|...]] [-d dhcp_server_type] [-m multicast_ip_address] [-p multicast_port_number] [-l ons_local_port] [-r ons_remote-port] [-t host[:port][,host[:port],...]] [-v]
#srvctl add nodeapps -n rac1 -A 1.2.3.4/255.255.255.0
srvctl remove nodeapps [-f] [-y] [-v]
$ srvctl remove nodeapps
srvctl start nodeapps [-n node_name] [-v]
$ srvctl start nodeapps
srvctl stop nodeapps [-n node_name] [-r] [-v]
$ srvctl stop nodeapps
srvctl status nodeapps

srvctl enable nodeapps [-g] [-v]
$ srvctl enable nodeapps -g -v
srvctl disable nodeapps [-g] [-v]
$ srvctl disable nodeapps -g -v
srvctl config nodeapps [-a] [-g] [-s] [-e]
$ srvctl config nodeapps -a -g -s -e
srvctl modify nodeapps [-n node_name -A new_vip_address] [-S subnet/netmask[/if1[|if2|...]] [-m multicast_ip_address] [-p multicast_port_number] [-e eons_listen_port] [-l ons_local_port] [-r ons_remote_port] [-t host[:port][,host:port,...]] [-v]
$ srvctl modify nodeapps -n rac1 -A 100.200.300.40/255.255.255.0/eth0
srvctl getenv nodeapps [-a] [-g] [-s] [-e] [-t "name_list"] [-v]
$ srvctl getenv nodeapps -a
srvctl setenv nodeapps {-t "name=val[,name=val][...]" | -T "name=val"} [-v]
$ srvctl setenv nodeapps -T "CLASSPATH=/usr/local/jdk/jre/rt.jar" -v
srvctl unsetenv nodeapps -t "name_list" [-v]
$ srvctl unsetenv nodeapps -t "test_var1,test_var2"

13.  Object - ONS (Oracle Notification Service):

Syntax for 11g R2 RAC (New SRVCTL command in 11g R2)

srvctl add ons [-l ons-local-port] [-r ons-remote-port] [-t host[:port][,host[:port]...]] [-v]

$ srvctl add ons -l 6200

srvctl remove ons [-f] [-v]

$ srvctl remove ons -f

srvctl start ons [-v]

$ srvctl start ons -v

srvctl stop ons [-v]

$ srvctl stop ons -v

srvctl status ons

srvctl enable ons [-v]

$ srvctl enable ons

srvctl disable ons [-v]

$ srvctl disable ons

srvctl config ons

srvctl modify ons [-l ons-local-port] [-r ons-remote-port] [-t host[:port][,host[:port]...]] [-v]

$ srvctl modify ons

13.  Object -E ONS (E Oracle Notification Service):

Syntax for 11g R2 RAC (New SRVCTL command in 11g R2)

srvctl add eons [-p portnum] [-m multicast-ip-address] [-e eons-listen-port] [-v]

#srvctl add eons -p 2018

srvctl remove eons [-f] [-v]

$ srvctl remove eons -f

srvctl start eons [-v]

$ srvctl start eons

srvctl stop eons [-f] [-v]

$ srvctl stop eons -f

srvctl status eons

srvctl enable eons [-v]

$ srvctl enable eons

srvctl disable eons [-v]

$ srvctl disable eons

srvctl config eons

srvctl modify eons [-m multicast_ip_address] [-p multicast_port_number] [-e eons_listen_port] [-v]

$ srvctl modify eons -p 2018

15.  Object -GNS (Grid Naming Service):

Syntax for 11g R2 RAC (New SRVCTL command in 11g R2)

srvctl add gns -i ip_address -d domain

$ srvctl add gns -i 192.124.16.96 -d oracle.com

srvctl remove gns [-f]

$ srvctl remove gns

srvctl start gns [-l log_level] [-n node_name]

$ srvctl start gns

srvctl stop gns [-n node_name [-v] [-f]

$ srvctl stop gns

srvctl status gns [-n node_name]

$ srvctl status gns

srvctl enable gns [-n node_name]

$ srvctl enable gns
srvctl disable gns [-n node_name]

$ srvctl disable gns -n rac2

srvctl config gns [-a] [-d] [-k] [-m] [-n node_name] [-p] [-s] [-V] [-q name] [-l] [-v]

$ srvctl config gns -n rac1

srvctl modify gns [-i ip_address] [-d domain]

$ srvctl modify gns -i 192.168.1.108

srvctl relocate gns [-n node_name]

$ srvctl relocate gns -n node2

 
 

 16.  Object - OC4J (Oracle Container for Java):

Syntax for 11g R2 RAC (New SRVCTL command in 11g R2)

srvctl add oc4j [-v]

$ srvctl add oc4j

srvctl remove oc4j [-f] [-v]

$ srvctl remove oc4j

srvctl start ocj4 [-v]

$ srvctl start ocj4 -v

srvctl stop oc4j [-f] [-v]

$ srvctl stop oc4j -f -v

srvctl status oc4j [-n node_name]

$ srvctl status oc4j -n rac1

srvctl enable oc4j [-n node_name] [-v]

$ srvctl enable oc4j -n rac3

srvctl disable oc4j [-n node_name] [-v]

$ srvctl disable oc4j -n rac2

srvctl config oc4j

srvctl modify oc4j -p oc4j_rmi_port [-v]

$ srvctl modify oc4j -p 5385

srvctl relocate oc4j [-n node_name] [-v]

$ srvctl relocate oc4j -n rac1