The scope of this article not covers about the basic things about file system, but its about troubleshooting of currupt file system.
So, when you create file system on hard drive it will sub devided into multiple file system blocks.
Blocks are used for -
1. To store user data
2. Some blocks used for to store file system's metadata.
(Metadata is kind of structure of your file system and it contents superblock, indoes and directories)
Superblock - Each of your filesystem has a superblock. File system like ext2. ext3 etc. Superblock contents the information about file system like -
* File system type
* Size
* Status
* Information about other metadata
Now you will guess that how important is superblock for your filesystem, if that is that currupt then you may not able to use that partition or may you will error while tring to mount that filesystem.
Following are some errors when superblock get currupts or some bad sectors
- You cant able to mount the filesystem, it will refuse to mount
- Filesystem gets hang
- Sometimes though you are able to mount that filesystem, but strange behavior occures.
These kind of errors occures because of bunch reasons. Most of the time fsck works fine for these errors -
$e2fsck -f /dev/hda3
(-f option for forcefully checking even filesystem seems clean)
Now fsck doesnt work because of lost of superblock, what you will do??
Note that Linux maintains multiple redundant copies of the superblock in every filesystem. You can find out this information with this following command -
$dumpe2fs /dev/hda6 grep -i superblock
dumpe2fs 1.32 (09-Nov-2002)
Primary superblock at 1, Group descriptors at 2-2
Backup superblock at 8193, Group descriptors at 8194-8194
Backup superblock at 24577, Group descriptors at 24578-24578
Backup superblock at 40961, Group descriptors at 40962-40962
Backup superblock at 57345, Group descriptors at 57346-57346
Backup superblock at 73729, Group descriptors at 73730-73730
To repair file system by alternative superblock
$e2fsck -f -b 8193 /dev/hda6
(Take backup using dd before doing running commands)
If you are using Sun Solaris, as My experience frequent power failure can get you hell :-( . I am using old sparc and one time in month I have run fsck using commands as per my last blog. So if your Sun Solaris lost the superblock then boot from cdrom or network, to retrive information about your filesystem's superblock give following command -
$newfs -N /dev/rdsk/devicename
Now use alternative superblock
$fsck -F ufs -o b=block-number /dev/rdsk/devicename
okie guys, hope this information helps somebody.
Monday, August 2, 2010
Replacing failed disk devices with the Solaris Volume Manager
The following example will show how to replace a disk named c0t0d0 using the cfgadm(1m) and metareplace(1m) utilities. The first step is to remove (if they exist) any meta state databases on the disk that needs to be replaced. To locate the locations of all meta state databases, the metadb(1m) command can be run with the “-i” option:
$ metadb -i
If meta devices exist, you can run metadb(1m) with the “-d” option to remove the databases. The following example deletes all meta state databases on slice 7 of the disk (c0t0d0) that we are going to replace:
$ metdb -d c0t0d0s7
Once the meta state databases are removed, you can use cfgadm(1m)’s “-c unconfigure” option to remove an occupant (an entity that lives in a receptacle) from Solaris:
$ cfgadm -c unconfigure c0::dsk/c0t0d0
Once Solaris unconfigures the device, you can physically replace the disk. Once the drive is replaced, you can run cfgadm(1m) with the “-c configure” option to let Solaris know the occupant is available for use:
$ cfgadm -c configure c0::dsk/c0t0d0
Once Solaris know the drive is available, you will need to VTOC the new drive with fmthard(1m) or format(1m). This will add a disk label to the drive, which defines the partions types and sizes. Once a valid VTOC is installed, you can invoke the trusty old metareplace(1m) utility to replace the faulted meta devices. The following example will replace the device associated with meta device d10, and cause the meta device to start synchronizing data from the other half of the mirror ( if RAID level 1 is used):
$ metareplace -e d10 c0t0d0s0
$ metadb -i
If meta devices exist, you can run metadb(1m) with the “-d” option to remove the databases. The following example deletes all meta state databases on slice 7 of the disk (c0t0d0) that we are going to replace:
$ metdb -d c0t0d0s7
Once the meta state databases are removed, you can use cfgadm(1m)’s “-c unconfigure” option to remove an occupant (an entity that lives in a receptacle) from Solaris:
$ cfgadm -c unconfigure c0::dsk/c0t0d0
Once Solaris unconfigures the device, you can physically replace the disk. Once the drive is replaced, you can run cfgadm(1m) with the “-c configure” option to let Solaris know the occupant is available for use:
$ cfgadm -c configure c0::dsk/c0t0d0
Once Solaris know the drive is available, you will need to VTOC the new drive with fmthard(1m) or format(1m). This will add a disk label to the drive, which defines the partions types and sizes. Once a valid VTOC is installed, you can invoke the trusty old metareplace(1m) utility to replace the faulted meta devices. The following example will replace the device associated with meta device d10, and cause the meta device to start synchronizing data from the other half of the mirror ( if RAID level 1 is used):
$ metareplace -e d10 c0t0d0s0
Solaris Performance Monitoring & Tuning - iostat , vmstat & netstat
Solaris Performance Monitoring & Tuning - iostat , vmstat & netstat
Introduction to iostat , vmstat and netstat
This document is primarily written with reference to solaris performance monitoring and tuning but these tools are available in other unix variants also with slight syntax difference.
iostat , vmstat and netstat are three most commonly used tools for performance monitoring . These comes built in with the operating system and are easy to use .iostat stands for input output statistics and reports statistics for i/o devices such as disk drives . vmstat gives the statistics for virtual Memory and netstat gives the network statstics .
Following paragraphs describes these tools and their usage for performance monitoring and if you need more information there are some very good solaris performance monitoring books available at www.besttechbooks.com.
Table of content :
1. Iostat
• Syntax
• example
• Result and Solutions
2. vmstat
• syntax
• example
• Result and Solutions
3. netstat
• syntax
• example
• Result and Solutions
4. Next Steps
________________________________________
Input Output statistics ( iostat )
iostat reports terminal and disk I/O activity and CPU utilization. The first line of output is for the time period since boot & each subsequent line is for the prior interval . Kernel maintains a number of counters to keep track of the values.
iostat's activity class options default to tdc (terminal, disk, and CPU). If any other option/s are specified, this default is completely overridden i.e. iostat -d will report only statistics about the disks.
syntax:
Basic synctax is iostatinterval count
option - let you specify the device for which information is needed like disk , cpu or terminal. (-d , -c , -t or -tdc ) . x options gives the extended statistics .
interval - is time period in seconds between two samples . iostat 4 will give data at each 4 seconds interval.
count - is the number of times the data is needed . iostat 4 5 will give data at 4 seconds interval 5 times
Example
$ iostat -xtc 5 2
extended disk statistics tty cpu
disk r/s w/s Kr/s Kw/s wait actv svc_t %w %b tin tout us sy wt id
sd0 2.6 3.0 20.7 22.7 0.1 0.2 59.2 6 19 0 84 3 85 11 0
sd1 4.2 1.0 33.5 8.0 0.0 0.2 47.2 2 23
sd2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd3 10.2 1.6 51.4 12.8 0.1 0.3 31.2 3 31
The fields have the following meanings:
disk name of the disk
r/s reads per second
w/s writes per second
Kr/s kilobytes read per second
Kw/s kilobytes written per second
wait average number of transactions waiting for service (Q length)
actv average number of transactions actively
being serviced (removed from the
queue but not yet
completed)
%w percent of time there are transactions waiting
for service (queue non-empty)
%b percent of time the disk is busy (transactions
in progress)
Results and Solutions:
The values to look from the iostat output are:
• Reads/writes per second (r/s , w/s)
• Percentage busy (%b)
• Service time (svc_t)
If a disk shows consistently high reads/writes along with , the percentage busy (%b) of the disks is greater than 5 percent, and the average service time (svc_t) is greater than 30 milliseconds, then one of the following action needs to be taken
1.)Tune the application to use disk i/o more efficiently by modifying the disk queries and using available cache facilities of application servers .
2.) Spread the file system of the disk on to two or more disk using disk striping feature of volume manager /disksuite etc.
3.) Increase the system parameter values for inode cache , ufs_ninode , which is Number of inodes to be held in memory. Inodes are cached globally (for UFS), not on a per-file system basis
4.) Move the file system to another faster disk /controller or replace existing disk/controller to a faster
one.
Virtual Memory Statistics ( vmstat )
vmstat - vmstat reports virtual memory statistics of process, virtual memory, disk, trap, and CPU activity.
On multicpu systems , vmstat averages the number of CPUs into the output. For per-process statistics .Without options, vmstat displays a one-line summary of the virtual memory activity since the system was booted.
syntax:
Basic synctax is vmstatinterval count
option - let you specify the type of information needed such as paging -p , cache -c ,.interrupt -i etc.
if no option is specified information about process , memory , paging , disk ,interrupts & cpu is displayed .
interval - is time period in seconds between two samples . vmstat 4 will give data at each 4 seconds interval.
count - is the number of times the data is needed . vmstat 4 5 will give data at 4 seconds interval 5
times.
Example
The following command displays a summary of what the system
is doing every five seconds.
example% vmstat 5
procs memory page disk faults cpu
r b w swap free re mf pi p fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 0 11456 4120 1 41 19 1 3 0 2 0 4 0 0 48 112 130 4 14 82
0 0 1 10132 4280 0 4 44 0 0 0 0 0 23 0 0 211 230 144 3 35 62
0 0 1 10132 4616 0 0 20 0 0 0 0 0 19 0 0 150 172 146 3 33 64
0 0 1 10132 5292 0 0 9 0 0 0 0 0 21 0 0 165 105 130 1 21 78
The fields of vmstat's display are
procs
r in run queue
b blocked for resources I/O, paging etc.
w swapped
memory (in Kbytes)
swap - amount of swap space currently available
free - size of the free list
page ( in units per second).
re page reclaims - see -S option for how this field is modified.
mf minor faults - see -S option for how this field is modified.
pi kilobytes paged in
po kilobytes paged out
fr kilobytes freed
de anticipated short-term memory shortfall (Kbytes)
sr pages scanned by clock algorithm
disk ( operations per second )
There are slots for up to four disks, labeled with a single letter and number.
The letter indicates the type of disk (s = SCSI, i = IPI, etc) . The number is
the logical unit number.
faults
in (non clock) device interrupts
sy system calls
cs CPU context switches
cpu - breakdown of percentage usage of CPU time. On multiprocessors this is an a
verage across all processors.
us user time
sy system time
id idle time
Results and Solutions:
A. CPU issues:
Following columns has to be watched to determine if there is any cpu issue
1. Processes in the run queue (procs r)
2. User time (cpu us)
3. System time (cpu sy)
4. Idle time (cpu id)
procs cpu
r b w us sy id
0 0 0 4 14 82
0 0 1 3 35 62
0 0 1 3 33 64
0 0 1 1 21 78
Problem symptoms:
1.) If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs
2.) if this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.
3.) If the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us) system is facing shortage of CPU resources.
Resolution :
Resolution to these kind of issues involves tuning of application procedures to make efficient use of cpu and as a last resort increasing the cpu power or adding more cpu to the system.
B. Memory Issues:
Memory bottlenecks are determined by the scan rate (sr) . The scan rate is the pages scanned by the clock algorithm per second. If the scan rate (sr) is continuously over 200 pages per second then there is a memory shortage.
Resolution :
1. Tune the applications & servers to make efficient use of memory and cache.
2. Increase system memory .
3. Implement priority paging in s in pre solaris 8 versions by adding line "set priority paging=1" in
/etc/system. Remove this line if upgrading from Solaris 7 to 8 & retaining old /etc/system file.
________________________________________
Network Statistics (netstat)
netstat displays the contents of various network-related data structures in depending on the options selected.
Syntax :
netstat
Introduction to iostat , vmstat and netstat
This document is primarily written with reference to solaris performance monitoring and tuning but these tools are available in other unix variants also with slight syntax difference.
iostat , vmstat and netstat are three most commonly used tools for performance monitoring . These comes built in with the operating system and are easy to use .iostat stands for input output statistics and reports statistics for i/o devices such as disk drives . vmstat gives the statistics for virtual Memory and netstat gives the network statstics .
Following paragraphs describes these tools and their usage for performance monitoring and if you need more information there are some very good solaris performance monitoring books available at www.besttechbooks.com.
Table of content :
1. Iostat
• Syntax
• example
• Result and Solutions
2. vmstat
• syntax
• example
• Result and Solutions
3. netstat
• syntax
• example
• Result and Solutions
4. Next Steps
________________________________________
Input Output statistics ( iostat )
iostat reports terminal and disk I/O activity and CPU utilization. The first line of output is for the time period since boot & each subsequent line is for the prior interval . Kernel maintains a number of counters to keep track of the values.
iostat's activity class options default to tdc (terminal, disk, and CPU). If any other option/s are specified, this default is completely overridden i.e. iostat -d will report only statistics about the disks.
syntax:
Basic synctax is iostat
option - let you specify the device for which information is needed like disk , cpu or terminal. (-d , -c , -t or -tdc ) . x options gives the extended statistics .
interval - is time period in seconds between two samples . iostat 4 will give data at each 4 seconds interval.
count - is the number of times the data is needed . iostat 4 5 will give data at 4 seconds interval 5 times
Example
$ iostat -xtc 5 2
extended disk statistics tty cpu
disk r/s w/s Kr/s Kw/s wait actv svc_t %w %b tin tout us sy wt id
sd0 2.6 3.0 20.7 22.7 0.1 0.2 59.2 6 19 0 84 3 85 11 0
sd1 4.2 1.0 33.5 8.0 0.0 0.2 47.2 2 23
sd2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd3 10.2 1.6 51.4 12.8 0.1 0.3 31.2 3 31
The fields have the following meanings:
disk name of the disk
r/s reads per second
w/s writes per second
Kr/s kilobytes read per second
Kw/s kilobytes written per second
wait average number of transactions waiting for service (Q length)
actv average number of transactions actively
being serviced (removed from the
queue but not yet
completed)
%w percent of time there are transactions waiting
for service (queue non-empty)
%b percent of time the disk is busy (transactions
in progress)
Results and Solutions:
The values to look from the iostat output are:
• Reads/writes per second (r/s , w/s)
• Percentage busy (%b)
• Service time (svc_t)
If a disk shows consistently high reads/writes along with , the percentage busy (%b) of the disks is greater than 5 percent, and the average service time (svc_t) is greater than 30 milliseconds, then one of the following action needs to be taken
1.)Tune the application to use disk i/o more efficiently by modifying the disk queries and using available cache facilities of application servers .
2.) Spread the file system of the disk on to two or more disk using disk striping feature of volume manager /disksuite etc.
3.) Increase the system parameter values for inode cache , ufs_ninode , which is Number of inodes to be held in memory. Inodes are cached globally (for UFS), not on a per-file system basis
4.) Move the file system to another faster disk /controller or replace existing disk/controller to a faster
one.
Virtual Memory Statistics ( vmstat )
vmstat - vmstat reports virtual memory statistics of process, virtual memory, disk, trap, and CPU activity.
On multicpu systems , vmstat averages the number of CPUs into the output. For per-process statistics .Without options, vmstat displays a one-line summary of the virtual memory activity since the system was booted.
syntax:
Basic synctax is vmstat
option - let you specify the type of information needed such as paging -p , cache -c ,.interrupt -i etc.
if no option is specified information about process , memory , paging , disk ,interrupts & cpu is displayed .
interval - is time period in seconds between two samples . vmstat 4 will give data at each 4 seconds interval.
count - is the number of times the data is needed . vmstat 4 5 will give data at 4 seconds interval 5
times.
Example
The following command displays a summary of what the system
is doing every five seconds.
example% vmstat 5
procs memory page disk faults cpu
r b w swap free re mf pi p fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 0 11456 4120 1 41 19 1 3 0 2 0 4 0 0 48 112 130 4 14 82
0 0 1 10132 4280 0 4 44 0 0 0 0 0 23 0 0 211 230 144 3 35 62
0 0 1 10132 4616 0 0 20 0 0 0 0 0 19 0 0 150 172 146 3 33 64
0 0 1 10132 5292 0 0 9 0 0 0 0 0 21 0 0 165 105 130 1 21 78
The fields of vmstat's display are
procs
r in run queue
b blocked for resources I/O, paging etc.
w swapped
memory (in Kbytes)
swap - amount of swap space currently available
free - size of the free list
page ( in units per second).
re page reclaims - see -S option for how this field is modified.
mf minor faults - see -S option for how this field is modified.
pi kilobytes paged in
po kilobytes paged out
fr kilobytes freed
de anticipated short-term memory shortfall (Kbytes)
sr pages scanned by clock algorithm
disk ( operations per second )
There are slots for up to four disks, labeled with a single letter and number.
The letter indicates the type of disk (s = SCSI, i = IPI, etc) . The number is
the logical unit number.
faults
in (non clock) device interrupts
sy system calls
cs CPU context switches
cpu - breakdown of percentage usage of CPU time. On multiprocessors this is an a
verage across all processors.
us user time
sy system time
id idle time
Results and Solutions:
A. CPU issues:
Following columns has to be watched to determine if there is any cpu issue
1. Processes in the run queue (procs r)
2. User time (cpu us)
3. System time (cpu sy)
4. Idle time (cpu id)
procs cpu
r b w us sy id
0 0 0 4 14 82
0 0 1 3 35 62
0 0 1 3 33 64
0 0 1 1 21 78
Problem symptoms:
1.) If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs
2.) if this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.
3.) If the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us) system is facing shortage of CPU resources.
Resolution :
Resolution to these kind of issues involves tuning of application procedures to make efficient use of cpu and as a last resort increasing the cpu power or adding more cpu to the system.
B. Memory Issues:
Memory bottlenecks are determined by the scan rate (sr) . The scan rate is the pages scanned by the clock algorithm per second. If the scan rate (sr) is continuously over 200 pages per second then there is a memory shortage.
Resolution :
1. Tune the applications & servers to make efficient use of memory and cache.
2. Increase system memory .
3. Implement priority paging in s in pre solaris 8 versions by adding line "set priority paging=1" in
/etc/system. Remove this line if upgrading from Solaris 7 to 8 & retaining old /etc/system file.
________________________________________
Network Statistics (netstat)
netstat displays the contents of various network-related data structures in depending on the options selected.
Syntax :
netstat
Subscribe to:
Posts (Atom)