http://www.sportsgrid.com/nfl/were-sorry-yahoo-just-emailed-all-of-their-fantasy-football-players-this-apology/
"So what happened? At Yahoo!, we have giant machines called “filers” that
process a lot of the real-time data and stats for us and for you. We do
millions of calculations every hour for our games, and normally our
machines can handle this with no problem. Recently, we discovered a
hardware issue in one of the filers that caused the other one to
overload. We replaced some hardware, re-configured the setup, and did
some testing. However this Sunday – at approximately 12:15 p.m. Eastern –
the new configuration failed. This created an overload on storage
capacity and took the Fantasy part of our site down."
Wednesday, November 14, 2012
Thursday, October 18, 2012
Tuesday, November 8, 2011
ESXi Storage Add Issue..Call fails for “HostDatastoreSystem.QueryVmfsDatastore- CreateOptions” for object “ha-datastoresystem”
Solution
Source VMWARE.
Warning: This process will destroy data on the target device. The steps outlined here are potentially hazardous for your environment if they are not followed exactly. If you are not comfortable performing these steps, contact VMware Technical Support and work with them to resolve the issue.
http://www.eversity.nl/blog/2010/08/call-fails-for-hostdatastoresystem-queryvmfsdatastore-createoptions-for-object-ha-datastoresystem/
http://www.digital52.com/help/gptremoval.html
Source VMWARE.
Clearing partitioning information in ESXi using the DD utility
Due to differences between ESX classic and ESXi, the parted utility is not available in ESXi. These steps describe how to clear partitioning information for a LUN under ESXi.Warning: This process will destroy data on the target device. The steps outlined here are potentially hazardous for your environment if they are not followed exactly. If you are not comfortable performing these steps, contact VMware Technical Support and work with them to resolve the issue.
- Open a console to the ESX or ESXi host.
- Identify the disk device in question from the log messages. For example:
/vmfs/devices/disks/vml.0200030000600508b30093fcf0a05b5b8cc739002f4d5341313531
- Use the
fdiskcommand to obtain the exact size of the target disk device in bytes:fdisk -l "/vmfs/devices/disks/DeviceName"
The output appears similar to:Disk /vmfs/devices/disks/DeviceName: 536 MB, 429491220480 bytes
255 heads, 63 sectors/track, 52216 cylinders, total 838850040 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System/vmfs/devices/disks/DeviceName1 128838850039 419424956 ee EFI GPT
- Use the dd command to erase the first 34 sectors (34x512 bytes) of the disk device with zeros:
dd if=/dev/zero of="/vmfs/devices/disks/DeviceName"bs=512 count=34 conv=notrunc
Caution: This operation deletes the partition table and master boot record on the disk device. The changes take effect immediately. This results in data loss and cannot be reverted.
- The GPT partition scheme stores a backup of the partition table
at the end of the disk device. Erase the last 34 sectors of the device
as well:
- Determine the offset at which the last 34 sectors begins:
(SizeInBytes / 512) - 34 = SeekOffset
For example, using the values in step 3:(429491220480 / 512) - 34 = 838850006
- Use the dd command to erase the last 34 sectors of the disk, starting at the offset found in step 5a:
dd if=/dev/zero of="/vmfs/devices/disks/DeviceName" bs=512 count=34 seek=SeekOffset conv=notrunc
Caution: This operation deletes the partition table and master boot record on the disk device. The changes take effect immediately. This results in data loss and cannot be reverted.
- Determine the offset at which the last 34 sectors begins:
- Depending on the original contents of the disk device, it may be necessary to erase a larger amount of data on the disk.
- Retry the storage operation.
http://www.eversity.nl/blog/2010/08/call-fails-for-hostdatastoresystem-queryvmfsdatastore-createoptions-for-object-ha-datastoresystem/
http://www.digital52.com/help/gptremoval.html
ESXi 5.0 Issue with ISP2432 HBA
So if you are upgrading to ESXi 5.0 and have these HBAs, they are not
supported. Doing a Rescan will not pick up the newly provided LUNs.
We have to issue a LIP to get the LUNs seen which is disruptive to the IO.
~ # echo "scsi-qlalip" > /proc/scsi/qla2xxx/5
~ # echo "scsi-qlalip" > /proc/scsi/qla2xxx/6
~ #
We have to issue a LIP to get the LUNs seen which is disruptive to the IO.
~ # echo "scsi-qlalip" > /proc/scsi/qla2xxx/5
~ # echo "scsi-qlalip" > /proc/scsi/qla2xxx/6
~ #
Saturday, October 8, 2011
ext3/ext4-Performance-RHEL 6
EMC Backend AX4-5F. RAID-10 12 600G 1k Drives in 4G Qlogic Fibre.
So what does dd does on the ext3 filesystems and ext4 ones. with 8MB block size
dd if=/dev/zero of=aman2 bs=8192 count=1000000
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB) copied, 8.50084 s, 964 MB/s
4MB block
dd if=/dev/zero of=aman2 bs=4192 count=1000000
1000000+0 records in
1000000+0 records out
4192000000 bytes (4.2 GB) copied, 6.19781 s, 676 MB/s
Now with ext3
dd if=/dev/zero of=aman3 bs=8192 count=1000000
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB) copied, 13.7891 s, 594 MB/s
iostat is still active after these 13 secs flushing data to the disk
10/08/2011 02:23:49 PM
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 2.56 10.57 0.00 86.88
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 0.00 0.00 0.00 0 0
sdb 4790.00 0.00 101200.00 0 101200
sdd 5272.00 0.00 103056.00 0 103056
sde 0.00 0.00 0.00 0 0
sdc 0.00 0.00 0.00 0 0
emcpowera 10063.00 0.00 204256.00 0 204256
dm-0 43984.00 0.00 351872.00 0 351872
Seems like ext4 is beating the hell out of ext3.Althoigh dd is not a pure performance testing but it gives a little pinch as to what to expect.
So what does dd does on the ext3 filesystems and ext4 ones. with 8MB block size
dd if=/dev/zero of=aman2 bs=8192 count=1000000
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB) copied, 8.50084 s, 964 MB/s
4MB block
dd if=/dev/zero of=aman2 bs=4192 count=1000000
1000000+0 records in
1000000+0 records out
4192000000 bytes (4.2 GB) copied, 6.19781 s, 676 MB/s
Now with ext3
dd if=/dev/zero of=aman3 bs=8192 count=1000000
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB) copied, 13.7891 s, 594 MB/s
iostat is still active after these 13 secs flushing data to the disk
10/08/2011 02:23:49 PM
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 2.56 10.57 0.00 86.88
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 0.00 0.00 0.00 0 0
sdb 4790.00 0.00 101200.00 0 101200
sdd 5272.00 0.00 103056.00 0 103056
sde 0.00 0.00 0.00 0 0
sdc 0.00 0.00 0.00 0 0
emcpowera 10063.00 0.00 204256.00 0 204256
dm-0 43984.00 0.00 351872.00 0 351872
Seems like ext4 is beating the hell out of ext3.Althoigh dd is not a pure performance testing but it gives a little pinch as to what to expect.
Saturday, October 9, 2010
Event Based Model-Ubuntu Boot Up Explained
http://www.linuxplanet.com/linuxplanet/tutorials/7033/1/
Ubuntu's Success Story: the Upstart Startup Manager (Linux Boot Camp p.2)
Controlling Boot Sequence, What is Event-Based
Akkana Peck
Thursday, April 8, 2010 11:32:09 AM
Boot Camp Part I explained how Linux boots, using the old "SysV init" system of scripts. But some modern Linux distros have been gradually migrating to a newer model, called Upstart.
Upstart has been around since 2006, but it's only in the last year or so that it's taken a major role in booting distributions like Ubuntu and Fedora. Debian and OpenSuSE are reportedly joining in soon, while it's available as an optional component on most other distros. No distro uses it as the sole boot method yet: even Fedora 12 and the upcoming Ubuntu 10.04 keep a lot of functionality in SysV scripts.
Upstart, in contrast, is event based. An "event" can be something like "booting" ... or it can be a lot more specific, like "the network is ready to use now". You can specify which scripts depend on which events. Anything that isn't waiting for an event can run whenever there's CPU available.
This event-based system has another advantage: you can theoretically use it even after the system is up and running. Upstart is eventually slated to take over tasks such as or plugging in external devices like thumb drives (currently handled by udev and hal), or running programs at specific times (currently handled by cron).
Currently, however, Upstart is used primarily for booting and shutdown, and it's not clear when we'll see it used for other purposes.
On most systems, including Fedora and released Ubuntu systems, Upstart uses /etc/event.d. But in Ubuntu's upcoming release that changes to /etc/init. Yes, that means Ubuntu 10.04 systems have both /etc/init, for Upstart, and /etc/init.d, for the old SysV files. Not only that, but if you upgrade from an earlier version of Ubuntu and you've put anything nonstandard in /etc/event.d, you'll have to migrate it by hand -- the upgrader does not do any migration (bug 402759.)
I mentioned that Upstart doesn't use /etc/init.d. But all Upstart-using distros still include that directory. Some of the files in it are regular SysV Init scripts that haven't been migrated yet. But some services that have migrated maintain a link from /etc/init.d to /lib/init/upstart-job. If you run one of those, it works, but it prints a warning first:
The beginning of the boot sequence is still the same. The computer's BIOS boots grub (or another boot loader) from the disk; grub loads the kernel and then passes control to the init process.
But on an Upstart machine, init comes from upstart. Instead of running a master rc script that calls the scripts for a specific runlevel, Upstart's init takes jobs from its job directory.
Events can be more detailed than that, For instance, Upstart knows about runlevels. The file rc2.conf, whose job is to run all those old SysV files in /etc/rc2.d, includes this:
Ubuntu's Success Story: the Upstart Startup Manager (Linux Boot Camp p.2)
Controlling Boot Sequence, What is Event-Based
Akkana Peck
Thursday, April 8, 2010 11:32:09 AM
Boot Camp Part I explained how Linux boots, using the old "SysV init" system of scripts. But some modern Linux distros have been gradually migrating to a newer model, called Upstart.
Upstart has been around since 2006, but it's only in the last year or so that it's taken a major role in booting distributions like Ubuntu and Fedora. Debian and OpenSuSE are reportedly joining in soon, while it's available as an optional component on most other distros. No distro uses it as the sole boot method yet: even Fedora 12 and the upcoming Ubuntu 10.04 keep a lot of functionality in SysV scripts.
An event-based model
The normal SysV boot process is synchronous -- meaning things happen one at a time, one after the other. First you run S10sysklogd, and only when that's finished you can start running S11klogd. If anything in the boot process takes a long time, everything else has to wait.Upstart, in contrast, is event based. An "event" can be something like "booting" ... or it can be a lot more specific, like "the network is ready to use now". You can specify which scripts depend on which events. Anything that isn't waiting for an event can run whenever there's CPU available.
This event-based system has another advantage: you can theoretically use it even after the system is up and running. Upstart is eventually slated to take over tasks such as or plugging in external devices like thumb drives (currently handled by udev and hal), or running programs at specific times (currently handled by cron).
Currently, however, Upstart is used primarily for booting and shutdown, and it's not clear when we'll see it used for other purposes.
New directories
Upstart eschews the old /etc/init.d and /etc/rcN.d in favor of a new directory containing "job definition files". And here's a point of confusion: it doesn't have a standard name.On most systems, including Fedora and released Ubuntu systems, Upstart uses /etc/event.d. But in Ubuntu's upcoming release that changes to /etc/init. Yes, that means Ubuntu 10.04 systems have both /etc/init, for Upstart, and /etc/init.d, for the old SysV files. Not only that, but if you upgrade from an earlier version of Ubuntu and you've put anything nonstandard in /etc/event.d, you'll have to migrate it by hand -- the upgrader does not do any migration (bug 402759.)
I mentioned that Upstart doesn't use /etc/init.d. But all Upstart-using distros still include that directory. Some of the files in it are regular SysV Init scripts that haven't been migrated yet. But some services that have migrated maintain a link from /etc/init.d to /lib/init/upstart-job. If you run one of those, it works, but it prints a warning first:
$ sudo /etc/init.d/dbus restart
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service dbus restart
Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the restart(8) utility, e.g. restart dbus
Boot sequence
How does the boot sequence change with Upstart?The beginning of the boot sequence is still the same. The computer's BIOS boots grub (or another boot loader) from the disk; grub loads the kernel and then passes control to the init process.
But on an Upstart machine, init comes from upstart. Instead of running a master rc script that calls the scripts for a specific runlevel, Upstart's init takes jobs from its job directory.
Job definition files
A typical Upstart job definition file looks like this:$ cat /etc/event.d/hostname.conf
# hostname - set system hostname
#
# This task is run on startup to set the system hostname from /etc/hostname,
# falling back to "localhost" if that file is not readable or is empty and
# no hostname has yet been set.
description "set system hostname"
start on startup
task
exec hostname -b -F /etc/hostname
In this simple case, the hostname job runs when Upstart sees a startup event. Events can be more detailed than that, For instance, Upstart knows about runlevels. The file rc2.conf, whose job is to run all those old SysV files in /etc/rc2.d, includes this:
start on runlevel 2
stop on runlevel [!2]
udev.conf includes: start on virtual-filesystems
stop on runlevel [06]
virtual-filesystems is an event indicating that filesystems such as /dev and /proc have been mounted. It's sent by another Upstart job: $ cat mountall.conf
# mountall - Mount filesystems on boot
#
# This helper mounts filesystems in the correct order as the devices
# and mountpoints become available.
description "Mount filesystems on boot"
start on startup
stop on starting rcS
expect daemon
task
emits virtual-filesystems
emits local-filesystems
emits remote-filesystems
emits all-swaps
emits filesystem
emits mounting
emits mounted
[ ... rest of script ]
All these signals will be emitted when the mountall job completes successfully. Then Upstart can start jobs like udev that are waiting for one of those events. Jobs can depend on more than one event:
Subscribe to:
Posts (Atom)