Tuesday, May 17, 2011

Unix basic troubleshooting command

Troubleshooting
------------------------



Files System

How to find a name in a file/directory:

# find . –type f –exec grep –ls {} \;

Check Files Sizes

Search for the 5 largest directories (MB) on a Unix System:

Command: # du –sk *
sort –nr
head

Find the largest files on a Unix System:

Command: # du –k
sort –nr

Check the size of a directory on a Unix System:

Command: # du –sk .

Core Files

Command: File

Description: Shows which process is responsible for the core dump.

Compare file1 and file2 and put result in file3

Command: sdiff –l

Compress big files to readable format (octal dump)

Command: od –ad –t cd1

How to remove ^M out of a file

# dos2unix

Read binary files

Command: strings

Count the number of files in a directory

Command: ls –1
wc –l

How to delete a directory with too many files

When you get the message: “rm * Arguments too long.”

This means that there are too many files in the directory.

Delete all files with XXXX:

Command: # find .
grep “XXXX”
xargs rm –f

Remove all files in the directory and sub-directories:

Command: # find .
xargs rm –f



UNIX Processes

Show only the process id

Command: ps –ef
grep
nawk ‘{print $2}’

Monitor the performance of processor/server

Command: prstat

Command: top

Command: truss –wall –f –o -p

How to kill a series of processes

Export pids = ‘ps –ef
grep
nawk ‘{print $2}’ kill –9 pids

Shows which processes are using which port numbers

Command: netstat –a



Configuration

Link related commands:

Check logical channels

Command: vcstat

Get number to use for error message

Command: pad +0.

Get error message:

Command: X25diags



To determine which parameters the hme drivers support:

Command: ndd (-set)(-get) /dev/hme

Determine state of IP forwarding (quick method):

Command: ndd (-set) (-get) /dev/ip ip_forwarding

Analysing output:

0 ---> the system is not forwarding IP packets between its interfaces.

1---> the system is forwarding IP packets between its interfaces

Determine the interface speed:

Command: ndd (-set) (-get) /dev/hme link_speed

Analysing output:

0---> indicates that the interface is running at 10 MBits per second.

1---> indicates that the interface is running at 100 MBits per second

Determine if the hme interface is running in full-duplex or half-duplex mode:

Command: ndd (-set) (-get) /dev/hme link_

Analysing output:

0---> indicates that the interface is running in half-duplex mode.

1---> indicates that the interface is running in full-duplex mode.

Query if interface is up or down by using the link_status parameter:

Command: ndd (-set) (-get) /dev/hme link_status

Analysing output:

0---> the link is down

1---> the link is up

Determines if the interface supports auto-negotiation:

Command: ndd (-set) (-get) /dev/hme adv_autoneg_cap 0--->with auto negotiation

Analysing output:

0--->with auto negotiation

1--->without auto negotiation

4.3.4 Check LAN speed between two terminals

cd /var/tmp (on terminal 1)

mkfile 10M

ftp to

cd /var/tmp

bin

hash

put

Monday, May 2, 2011

RBAC for rebooting a server

RBAC  rebooting a server





So, in a followup to the Solaris RBAC configuration post, I wanted to show how quick and easy it is to configure RBAC. As an example, I’m going to be working with the Solaris reboot command, on the basis that many developers want to reboot their environments, but you don’t always want to give them root.



So, the basic steps are:



define a Profile

assign a command to the Profile

define a Role

assign the Profile to the Role

allow a user to use the Role

Easy stuff. First stage, let’s create the profile. Profiles live in /etc/security/prof_attr, and are a way to group together similar commands. If you look in that file, you’ll see a lot of existing profiles, which tie together common groups of Solaris commands.



Adding a new profile is easy - we just add an extra line to the end of that file:



# echo "Reboot:::Profile to reboot Solaris:help=" >> /etc/security/prof_attr

Breaking it down - the first field is the profile name, and the fourth field is the description. The rest of the fields don’t matter at this stage, for what we’re doing.



The new profile is useless without a command, so let’s add the Solaris reboot command. Commands associated with RBAC profiles live in /etc/security/exec_attr (can you see a pattern in the filenames yet?) and - again - this file is pre-populated with command Solaris commands, grouped by profile.



# echo "Reboot:suser:cmd:::/usr/sbin/reboot:euid=0" >> /etc/security/exec_attr

Breaking the fields down again:



first field is the profile name

second field is the security policy - in this case, standard superuser

third field is the type - in this case, it’s a command

sixth field is the full path to the command

final field is the effective user ID the command is executed as

So far, it’s all pretty straightforward. Now we have a profile, and we have a command associated with that profile. Now we need to create a role.



RBAC roles are essentially normal user accounts, which have a restricted shell, and associated profile(s). The restricted shell is there to apply all the execution privilege and audit trail RBAC goodness.



Adding a role is nice and easy:



# roleadd -m -d /export/home/reboot reboot

64 blocks

Note the command line options to roleadd are the same as used when adding a normal Solaris user with useradd.



We also need to give the role a password:



# passwd reboot

New Password:

Re-enter new Password:

passwd: password successfully changed for reboot

And now we can see the role has been added to /etc/passwd:



# grep reboot /etc/passwd

reboot:x:1001:1::/export/home/reboot:/bin/pfsh

So it looks almost exactly the same as a normal Solaris user. Now all we need to do is add a profile to the role. We do this with the rolemod command, which - again - is very similar to the normal usermod command:



# rolemod -P Reboot reboot

Details of which profiles are assigned to roles - and which roles are assigned to users - live in /etc/user_attr - so we can look in there to see the changes we’ve made:



# grep reboot /etc/user_attr

reboot::::type=role;profiles=Reboot

Finally we’ll add the role to our user account:



# usermod -R reboot tomk

UX: usermod: tomk is currently logged in, some changes may not take effect until next login.

And just look in /etc/user_attr to make sure the changes have been made:



# grep reboot /etc/user_attr

reboot::::type=role;profiles=Reboot

tomk::::type=normal;roles=reboot

We can use the roles command to see what roles we have available to us:



$ roles

reboot

However, logged in as myself I still can’t reboot the machine:



$ /usr/sbin/reboot

reboot: permission denied

And that’s because the profile is assigned to the role, not to my user account:



$ profiles

All

Basic Solaris User

The clue on how to use roles was in how they are created and stored - they’re just like normal users. So to access a role, we su to it:



$ su reboot

Password:

The moment we su to a role, the whole RBAC audit trail kicks in. Everything, from that initial su onwards, is logged and tracked. Unlike sudo, this logging continues, even if we change shells or become another user (if the role allows us to). It’s this unbreakable audit trail that makes RBAC so powerful.



Now that we’ve assumed a role, we can check out the profiles available to us:



$ profiles

Reboot

So we can now execute the reboot command and bounce the box:



$ /usr/sbin/reboot

Connection to 192.168.13.101 closed by remote host.

Connection to 192.168.13.101 closed.

Have a look at the configuration files and see all of the roles and profiles that come pre-configured with Solaris. Play about with them and get familiar with the terminology. RBAC isn’t difficult or complex - it’s just very different. Get comfortable with it and you’ll soon be able to leverage it’s power to really secure your Solaris machines without denying users any functionality

Solaris Boot troubleshooting

Booting problems in Solaris


Booting problems poses serious challenge to the system administrators as system is down and no one can use it . This article tries to cover some of the general booting problems and their possible solutions to enable understand the problem cause and bring the system up very quickly.



Following are some of the booting issues ,error messages their meaning and possible solutions discussed in this article.



1) Booting in single user mode and mounting root disk

2) Making boot device alias

3) Timeout waiting for ARP/RARP packet”? error message

4) The file just loaded does not appear to be executable – error message

5) bootblk: can’t find the boot program – error message

6) boot: cannot open kernel/unix – error message

7) Error reading ELF header? – error message Cannot open /etc/path_to_inst error message

9) Can’t stat /dev/rdsk/c0t3d0s0 error message



1. Booting in single user mode and mounting root hard disk

Most important step in diagnosing the booting problems is booting the system in single user mode and examining the hard disk for possible errors & work out the corrective measure. Single user mode can be achieved by any of the following methods :-

ok> boot -s ;from root disk

ok> boot net -s ;from network



ok>boot cdrom -s ;from cdrom

Rebooting with command: cdrom -s

Configuring the /devices directory

Configuring the /dev directory


INIT: SINGLE USER MODE

#

# fsck /dev/rdsk/c0t3d0s0

# mount /dev/dsk/c0t3d0s0 /mnt



Perform the required operation on mounted disk , now accessible through /mnt ,& unmount the hard disk after you are done ;

# umount /mnt

# reboot



2.Making boot device alias

In case system can not boot from primary disk and it is needed to make another boot disk to access the data , nvalias command is used .

nvalias command makes the device alias and assigns an alternate name to a physical disk. Physical address of target disk is required which can be had by show-disk command on ok>.



ok> nvalias disk7 /iommu@f,e0000000/sbus@f,e0001000/dma@3,81000/esp@3,80000/sd2,0

The new aliased disk can be named as boot disk or can be used for booting by refering its name .

ok> setenv boot-device disk7

ok>reset

or

ok> boot disk7



3. Timeout waiting for ARP/RARP packet ?

At ok> type printenv and look for these parameters .

boot-device disk

mfg-switch? false

diag-switch? false

if you see “boot-device net ” or true value for the other two parameter change it to the values above.

In case you wants to boot from network make sure your client is properly configured in boot server and network connections & configuration are proper.



4. The file just loaded does not appear to be executable

Boot block on the hard disk is corrupted .Boot the system in single user mode with cdrom and reinstall boot block .



#installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t3d0s0



5. bootblk: can’t find the boot program

boot block can not find the boot programe – ufsboot in Solaris .Either ufsboot is missing or corrupted . In such cases it can be restored from the cdrom after booting from cdrom & mounting the hard disk

# cp /platform/`uname -i`/ufsboot /mnt/platform/`uname -i`



6. boot: cannot open kernel/unix

Kernel directory or unix kernel file in this directory is not found .Probably deleted during fsck or deleted by mistake. Copy it from the cdrom or restore from the backup tape.

# cp /platform/`uname -i`/kernel/unix /mnt/platform/`uname -i`/kernel



7. Error reading ELF header ?

Kernel directory or unix kernel file in this directory is corrupted.Copy it from the cdrom or restore from the backup tape.

# cp /platform/`uname -i`/kernel/unix /mnt/platform/`uname -i`/kernel



8. Cannot open /etc/path_to_inst

System can not find the /etc/path_to_install file .It might be missing or corrupted and needs to be rebuild.

To rebuild this file boot the system with -ar option :

ok>boot -ar

Press enter to select default values for the questions asked during booting and select yes to rebuild /etc/path_to_install

The /etc/path_to_inst on your system does not exist or is empty. Do you want to rebuild this file [n]? y

system will continue booting after rebuilding the file.



9. Can’t stat /dev/rdsk/c0t3d0s0

When booted from cdrom and done fsck the root partition comes out to be fine but on booting from root disk this error occurs. The device name for / is missing from /dev/dsk directory and to resolve the issue /dev & /devices directories has to be restored from root backup tapes .



SVM mirroring root disk

How to Mirror root With Solaris Volume Manager in the Solaris 9 and 10 OS



Prerequisites

First, you need to identify the disks that you want to create mirrors with. You can do this by using the format command to find the disks in question.

Run the format command; below is an example of the output:

AVAILABLE DISK SELECTIONS:

0. c3t2d0

/pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@2,0

1. c3t3d0

/pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@3,0

In my example, I'm mirroring the root partitions along with the other partitions from the disk drive.

My drives are c3t2d0 and c3t3d0.

Procedure for Mirroring root

First, partition your primary drive, typically the one that the Solaris OS is currently running on. (In my case, this is drive 0, c3t2d0.) I traditionally do this during the installation of the Solaris OS to prevent data loss.

You will need one partition that is about 10 Mbyte for the meta database.

Once you are satisfied with the partition that you have created, ensure that you label the disk, and then perform the following steps to transfer the same partitioning table.

Transfer the partition table from one drive to another.

prtvtoc /dev/rdsk/c3t2d0s2
fmthard -s - /dev/rdsk/c3t3d0s2

Note: Notice the use of s2, which is typically the overlap partition; if you changed this on the disk, please substitute the proper slice in its place.

Now that you have the two disks looking the same, execute the following:

metadb -a -c 3 -f c3t2d0s7 c3t3d0s7

The -c 3 creates three copies of the metastat database in this space, just in case a single copy gets corrupted (which is never good).

We will initialize the disk that makes up the root partition by doing the following. I'm using s0 because this is my root partition; you can substitute where appropriate.

metainit -f d11 1 1 c3t2d0s0

metainit -f d12 1 1 c3t3d0s0

Now we will create the actual mirror:

metainit d10 -m d11

After you have completed the preceding steps, you need to run the following command, which will automatically update /etc/system and /etc/vfstab to let it know that you are using a metadevice as your root disk.

metaroot d10

After you have executed the commands above, you need to reboot the machine before attaching the other half of the mirror to the root device. You can't attach a currently mounted device, or the machine will go crazy. In order to attach the device you will need to do the following:

metattach d10 d12

To check on the status of the mirror, you can do the following:

metastat d10

You will want to update the Openboot with the prior alias for the boot devices. You can do this by doing the following:

ls -l /dev/dsk/c0t0d0s0

You output will look similar to the following

lrwxrwxrwx 1 root root 42 Jul 12 2007 /dev/dsk/c0t0d0s0 -> ../../devices/pci@1e,600000/ide@d/sd@0,0:a

You will need to update the bold part above, with your output. You will then run the following command from the OS

eeprom "nvramrc=devalias mirror /pci@1e,600000/ide@d/disk@0,0:a devalias mirror-a /pci@1e,600000/ide@d/disk@1,0:a"

eeprom boot-device="mirror mirror-a"

eeprom "use-nvramrc?=true"

The below commands for doing this are from the OK prompt, don't do this else wise.

"nvalias mirror /pci@1e,600000/ide@d/disk@0,0:a mirror-a /pci@1e,600000/ide@d/disk@1,0:a"

"setenv boot-device mirror mirror-a"

If you are mirroring just the two internal drives, you will want to add the following line to /etc/system to allow it to boot from a single drive. This will bypass the SVM Quorum rule

set md:mirrored_root_flag = 1

Please note that if you are running a Sparc platform you can use the installboot command in order to install the boot blocks onto the head of the drive.

For a UFS based File system you will use the below command.

installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t0d0s0

And on a ZFS based File System you will use the below command.

installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t1d0s0

If you are on a X86 Platform you will want to use installgrub to do similar functions, below you will find references for that

/sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0d0s0