Steve Kamerman's Blog 2010-03-11T05:38:45Z Steve Kamerman Copyright 2010, Steve Kamerman SPHPBLOG Tera-WURFL 2.1.0

Here's what's new:



1. Experimental support for Microsoft SQL Server 2005/2008: with the MSSQL2005 Database Connector, you can now use MS SQL Server as a Tera-WURFL backend! The support is experimental for now, although I've put it through my barrage of over 45,000 user agents and it works properly. The Reduction in String stored procedure still needs to be optimized since I just ported it from MySQL 5 to T-SQL. The MSSQL backend is considerably slower than MySQL, and I beleive it's that procedure slowing it down.

2. SimpleDesktop Matching Engine: Tera-WURFL wasn't originally designed to differnetiate between desktop and mobile browsers (either was WURFL for that matter), but with this release I've introduced the SimpleDesktop Matching Engine which, when enabled, uses keywords and regular expressions to detect 90% of desktop browsers without having to resort to searching the database for a matching WURFL entry. In my tests performance increased by 176% for detection of 45,000 actual unique user agents (both mobile and non-mobile). This feature also dramatically decreases the number of items in your cache by using a single cache item for all desktop browsers.

3. Capabilities Filter: I've been meaning to implement the Capability Filter for a long time, but two very high traffic clients of mine convinced me to sit down and get it finished. This adds a new setting in TeraWurflConfig.php called CAPABILITY_FILTER. If you set it to false it will be disabled and all the capabilities in the WURFL will be stored in the database and available to your scripts (this is the pre-2.1.0 behaviour). Here's where the magic starts, you can set it to an array of the capabilities and groups of capabilities that you want to store and use, for example, if you just want the know what kind of device is visiting your site and whether or not it's wireless, you can use this filter:

public static $CAPABILITY_FILTER = array(
"brand_name",
"model_name",
"is_wireless_device"
);

This will shrink your device database by more than a factor of 10. I tested the filter with 20 capabilities against 45,000 unique user agents and it reduced the size of the cache from 645MB (without filtering) to 92MB (with filtering), then down to 24MB with both filtering and SimpleDesktop.

You can look forward to the Tera-WURFL 2.1.0 release around February 10, 2010.

Tera-WURFL is available at it's usual location, http://www.Tera-WURFL.com/]]> http://www.teratechnologies.net/stevekamerman/index.php?entry=entry100205-054200 2010-02-05T00:00:00Z 2010-02-05T00:00:00Z Script to update iTunes play count and ratings from any iPod or iPhone
Download UpdatePlayCount.zip (2KB)

Source Code (JScript/JavaScript)



/**
* UpdatePlayCount.js
*
* Description
* ---------------------------------------------------------------------------------------------------------
* This script will update your iTunes library with the play counts and ratings from any iPod or iPhone.
* It uses the iTunes COM Interface to communicate with your device. In order to determine which tracks
* match, the script will search through the iTunes Library looking for a matching Artist, Album and Title;
* as a result, this process can take a long time. Updates all the music and movies in the "Music" and
* "Movies" playlists on your device. If you need to update more folders you can modify the code.
*
* Usage
* ---------------------------------------------------------------------------------------------------------
* Double click on UpdatePlayCount.bat
*
*
* @package UpdatePlayCount
* @author Steve Kamerman, stevekamerman AT gmail.com
* @version Alpha 1.0 $Date: 2009/12/22 13:19
* @license http://www.mozilla.org/MPL/ MPL Vesion 1.1
* @language Microsoft JScript / JavaScript
*
*/

var logObject, logFile;
var logfileName = "UpdateItunes.log";
logObject = new ActiveXObject("Scripting.FileSystemObject");
logFile = logObject.CreateTextFile(logfileName, true);

var ITTrackKindFile = 1;
var iTunesApp = WScript.CreateObject("iTunes.Application");
var sources = iTunesApp.Sources;
var i;
var updateTracksCount = 0;
var missingTracksCount = 0;
var ipod;
var itunes;
var playlists;
var itunesPlaylists;
var iTunesXML;
iTunesXML = "";

for(i=1;i<sources.Count;i++){
if(sources.Item(i).Kind == 2){
ipod = sources.Item(i);
playlists = ipod.Playlists;
}
if(sources.Item(i).Kind == 1){
itunes = sources.Item(i);
itunesPlaylists = itunes.Playlists;
}
}

if(ipod == undefined){
WScript.Echo("Error: No iPod or iPhone found. Please make sure your device is listed in iTunes, then click on it's name and check \"Manually manage music and videos\". Restart iTunes and rerun this script.");
WScript.Quit(0);
}
WScript.Echo("iPod found, press OK to update iTunes with the play counts and ratings from your iPod.");
WScript.Echo("This process can take a long time to complete. To monitor the progress,\nopen iTunes and click on your iTunes Music folder\nthen sort by play count or rating and watch them change.\nPress OK to continue");
var currentPlaylist;
var currentTracks;
var currentTrack;
for(i=1;i<playlists.Count;i++){
currentPlaylist = playlists.Item(i);
if(currentPlaylist.Name != "Music" && currentPlaylist.Name != "Movies")continue;
currentTracks = currentPlaylist.Tracks;
for(a=1;a<currentTracks.Count;a++){
currentTrack = currentTracks.Item(a);
if(currentTrack.PlayedCount > 0){
updateITunesEntry(currentPlaylist.Name,currentTrack.Artist,currentTrack.Album,currentTrack.Name,currentTrack.PlayedCount,currentTrack.Rating);
}
}
}

WScript.Echo("Finished processing "+(updateTracksCount+missingTracksCount)+" tracks.\nUpdated Tracks: "+updateTracksCount+"\nMissing Tracks: "+missingTracksCount+"\nSee the logfile ("+logfileName+") for more details.");

function updateITunesEntry(playlist, artist, album, song, playCount, rating){
var itunesPlaylist;
var itunesTracks;
var itunesTrack;
var i;
var a;
for(i=1;i<itunesPlaylists.Count;i++){
itunesPlaylist = itunesPlaylists.Item(i);
if(itunesPlaylist.Name != playlist)continue;
itunesTracks = itunesPlaylist.Tracks;
for(a=1;a<itunesTracks.Count;a++){
itunesTrack = itunesTracks.Item(a);
if(itunesTrack.Artist == artist && itunesTrack.Album == album && itunesTrack.Name == song){
logFile.WriteLine(itunesTrack.Name+": count: "+itunesTrack.PlayedCount+"->"+playCount+", rating: "+itunesTrack.Rating+"->"+rating);
itunesTrack.PlayedCount = playCount;
itunesTrack.Rating = rating;
updateTracksCount++;
return(1);
}
}
}
missingTracksCount++;
logFile.WriteLine("WARNING: Track not found in iTunes Library: "+artist+" - "+song);
}
]]> http://www.teratechnologies.net/stevekamerman/index.php?entry=entry091222-053304 2009-12-22T00:00:00Z 2009-12-22T00:00:00Z I'm finally in Iraq http://www.teratechnologies.net/stevekamerman/index.php?entry=entry091222-053055 2009-12-22T00:00:00Z 2009-12-22T00:00:00Z Tera-WURFL vs. New WURFL PHP API 23,902 unique user agents through both Tera-WURFL 2.0.0 RC4 and the new WURFL PHP API (1.0.1-rc2). Here are the results:

Tera-WURFL
Total Time: 12.081017971039
Devices Processed: 23902
Total Queries: 23912

WURFL PHP API
Total Time: 217.57795381546
Devices Processed: 23902
Total Queries: N/A

Both results were obtained after repeated testing to verify that the user agents were cached.]]> http://www.teratechnologies.net/stevekamerman/index.php?entry=entry091030-225328 2009-10-31T00:00:00Z 2009-10-31T00:00:00Z Tera-WURFL 2 pre-release is available for testing! Here's a breakdown of the features of Tera-WURFL 2:

Rewrote some of the UserAgentMatchers and deleted others to bring Tera-WURFL on par with the Java WURFL API. With the introduction of desktop browser UserAgentMatchers, we no longer need to use the large web patch; instead, you can just use the 8KB one from wurfl.sourceforge.net (included). Also, I fixed some typos and bugs here and there. NOTE: if you are upgrading from version 2.0.0 RC1-RC3 you should delete all your database tables before you update. You can leave the terawurflcache table if you want to retain your cache. Although it will still technically work even if you don't delete the tables, you will be orphaning some unecessary tables in your database.

Complete code-rewrite from the ground up. The 2.x version of Tera-WURFL is loosely based on a pre-release of the Java WURFL Evolution Library, but the API is taken from Tera-WURFL 1.5.2. The following is a list of features found in Tera-WURFL 2.0:

  • User Agent Matchers have been created for each of the major manufacturers. These allow for specific matching methods to be applied to the user agent like string searching, RIS (Reduction in String) and LD (Levenshtein Distance).
  • Multiple patch files are now supported. Tera-WURFL ships with the current wurfl.xml, web_browsers_patch.xml and custom_web_patch.xml. Patch files can be added to TeraWurflConfig.php by separating them with semicolons in the TeraWurflConfig::PATCH_FILE directive. Patch files are loaded in order from left to right on top of the WURFL file, so if you want to override every other patch file, specify it last.
  • The custom_web_patch.xml file can be edited from the Web Administration page, and allows you to easily add non-mobile user agents to the patch file. The devices with these user agents will be detected as generic_web_browser (non-mobile).
  • Persistent Caching means that your cached devices stay cached. When you update the WURFL file or your patches, your device cache is also updated via the new database.
  • Cache Browser allows you to see what devices are hitting your site and what their capabilities were detected as.
  • Installation Script is better than 1.5.2. Once you download Tera-WURFL and extract it, edit TeraWurflConfig.php then go to /admin/install.php and follow the directions to finish installation.
  • PHP short_open_tags are no longer required to run Tera-WURFL. PHP has this feature disabled by default now.
  • Conclusive vs. Inconclusive Matching. If a device is matched with the UserAgentMatcher's primary matching method it is considered a conclusive match, if it is detected via a recovery matcher or by the CatchAllMatcher it is an inconclusive match. This information is available via the tera_wurfl capability group.
  • tera_wurfl Capability Group. The TeraWurfl->Capabilities array now contains a group called "tera_wurfl". This group contains the following Tera-WURFL related information:
    • num_queries - the number of database queries required to lookup the device.
    • actual_root_device - the WURFL ID of the actual device (not subrevision or generic), this can be null.
    • match_type - either conclusive or inconclusive.
    • matcher - the name of the UserAgentMatcher that detected the device.
    • match - whether or not there was an actual match. If there was no match, Tera-WURFL guessed which generic device is most similar to the device.
    • lookup_time - the time in seconds that it took to detect the device.
    • fall_back_tree - the complete fallback tree that built the capabilities of the device. This is a list of all the WURFL IDs from the detected device down to the base generic device.


Example Script



<?php
// Include the Tera-WURFL file
require_once("TeraWurfl.php");
// Instantiate the Tera-WURFL object
$wurflObj = new TeraWurfl();
// Get the capabilities from the object
$matched = $wurflObj->GetDeviceCapabilitiesFromAgent(); //optionally pass the UA and HTTP_ACCEPT here
// Show whether there was a conclusive match
if($matched){echo "Match found";}else{echo "Match NOT found";}
// Print the capabilities array
echo "<pre>".htmlspecialchars(var_export($wurflObj->capabilities,true))."</pre>";
?>
]]> http://www.teratechnologies.net/stevekamerman/index.php?entry=entry091030-223856 2009-10-31T00:00:00Z 2009-10-31T00:00:00Z Back to Iraq!

I'm heading back to the desert for another deployment with the Army National Guard


I've been in the Army National Guard since 2003 and have been lucky enough to serve only one tour in Iraq (2005). In 2008 I moved from Michigan to Texas and in doing so I was forced to switch Army units. The unit that grabbed me up was already deployed and I was happily holding down the fort in the rear detachment. Unfortunately, a quick 3 months later I was scooped up by a different unit 300 miles away for a deployment. I am now in the final stages of my preparation for this deployment and am scrambling to wrap up all my stuff before I leave. Hopefully I will have a lot of time to work on Tera-WURFL when I'm gone; I plan to further abstract the database layer and introduce a MSSQL Database Connector in the coming months. I will let you know once I'm in the sandbox!
]]> http://www.teratechnologies.net/stevekamerman/index.php?entry=entry091030-215958 2009-10-31T00:00:00Z 2009-10-31T00:00:00Z Recovering a RAID 0 array with Linux and Python! Background of my problem
A few days ago my Dell XPS 720 decided to die on me. When I push the power button it turns on for about 1/2 second and then turns off. After much debate, I bought a new power supply for it (non-standard of course [24pin AND 20pin power connectors]). Today I received the power supply and low-and-behold that wasn't the problem. Now I need to buy either a motherboard or a new CPU (the existing one is a Core 2 Quad Extreme 3.0). I decided that buying from Dell was a bad choice and I'll just build myself a Phenom II system. Meanwhile, the XPS 720 shipped with 2x 120GB 10k Raptors in a RAID-0 using the sub-par onboard SATA RAID controller. I would just love to get my data off the system, but I can't get it up and running and I'm not going to sink anymore money into it. I decided to figure out exactly how the data was stored on the array members and try to deinterlace it directly into a raw image that I could then mount via loopback device in Linux and copy all my data back out :) It sounds easy, huh? I found this GREAT little python code snippet that does the heavy lifting for me - thanks Simón A. Ruiz!!!

How I recovered the array


First, I noted which SATA ports my drives were plugged into on my XPS and labeled them SATA0 and SATA1 so I would remember (if you already unplugged them, fdisk -l the drives, the one with a valid partition table is SATA0). Once they were labeled, I went to another desktop and popped in an ubuntu 8.10 x64 Desktop CD that I had laying around and booted into the GUI (just so I could get a bigger terminal ;) ). Once in the GUI, I loaded up a couple of terminals and got root with sudo su -. Now I used fdisk to find my drives:


root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# fdisk -l /dev/sda

Disk /dev/sda: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x2b242b24

Device Boot Start End Blocks Id System
/dev/sda1 * 1 18935 152095356 7 HPFS/NTFS
/dev/sda2 18936 24036 40973782+ 83 Linux
/dev/sda3 24037 24792 6072570 5 Extended
/dev/sda5 24037 24792 6072538+ 82 Linux swap / Solaris
root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# fdisk -l /dev/sdb

Disk /dev/sdb: 160.0 GB, 160000000000 bytes
255 heads, 63 sectors/track, 19452 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x20000000

Device Boot Start End Blocks Id System
/dev/sdb1 1 6 48163+ de Dell Utility
/dev/sdb2 7 1312 10485760 7 HPFS/NTFS
/dev/sdb3 * 1312 38905 301963264 7 HPFS/NTFS
root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# fdisk -l /dev/sdc

Disk /dev/sdc: 160.0 GB, 160000000000 bytes
255 heads, 63 sectors/track, 19452 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table
root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# fdisk -l /dev/sdd

Disk /dev/sdd: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x8d399bc0

Device Boot Start End Blocks Id System
/dev/sdd1 1 91201 732572001 7 HPFS/NTFS
root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# fdisk -l /dev/sde

Disk /dev/sde: 16.0 GB, 16013852672 bytes
78 heads, 14 sectors/track, 28641 cylinders
Units = cylinders of 1092 * 512 = 559104 bytes
Disk identifier: 0x2c26e771

Device Boot Start End Blocks Id System
/dev/sde1 8 28642 15634496 c W95 FAT32 (LBA)


In my case, sda is my recovery computer's hard drive, sdb is my SATA0 array member, sdc is my SATA1 array member, sdd is my external firewire drive (I'll use this to store the data) and sde is my thumbdrive.

Note how /dev/sdb has a valid partition table and /dev/sdc does not - this is because of how RAID-0 works, it uses striping to put stripes (pieces) of data on each drive, alternating back and forth - the first drive in a RAID-0 should always have the partition table on it since that data is in the first stripe. The stripe size can vary depending on your setup and RAID controller - mine is 64k, so the RAID controller writes 64K of data to SATA0, then the next 64K to SATA1 and so forth.

Next I took the python script from Simón A. Ruiz' blog and modified it to show me a status indication:


#!/usr/bin/python
#
# raid0_deinterlace.py
#
# INTENT = This is a script for deinterlacing two raw dd images
# or drives from a RAID-0 and combine them into a single
# image file
#
# This is strictly experimental.
#
# Original Script By: Simon A. Ruiz; Thursday, May 1, 2008
#
# Modified By: Steve Kamerman; Thursday, May 28, 2009
# Changes: Added status indicator so you can estimate ETA

import datetime

inputFiles = [open("/dev/sdb","rb"),open("/dev/sdc","rb")]
outputFile = open("output","wb")
chunkSize = 65536 # change this to your stripe size in bytes

# And, so as not to have to figure this out every time through the loop...
numFiles = len(inputFiles)

i = 0
a = 0
gb = 0
while True:
if a == 16384:
gb += 1
print 'Copied', gb, 'GB of data', datetime.datetime.now()
a = 0
nextChunk = inputFiles[i%numFiles].read(chunkSize)
if not nextChunk:
print 'Done! No more data.'
break
outputFile.write(nextChunk)
i += 1
a += 1

outputFile.close()
for file in inputFiles:
file.close()


Since I'm using /dev/sdb and /dev/sdc, this script is setup correctly for me. If you are using different devices, change those names in script. You can also point the script at dd images instead of physical drives if that's what you're trying to recover from. This process will create a single file that contains the entire array in raw image form with the name output in the current working directory.

When you run this script you will see a notification for every GB it processes, something like this:


root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# ./raid0_deinterlace.py
Copied 1 GB of data 2009-05-27 23:14:30.742032
Copied 2 GB of data 2009-05-27 23:15:01.828132
Copied 3 GB of data 2009-05-27 23:15:33.043268
Copied 4 GB of data 2009-05-27 23:16:04.179799
Copied 5 GB of data 2009-05-27 23:16:35.243319
Copied 6 GB of data 2009-05-27 23:17:06.471911
Copied 7 GB of data 2009-05-27 23:17:37.526823
Copied 8 GB of data 2009-05-27 23:18:09.352929
Copied 9 GB of data 2009-05-27 23:18:40.196855
Copied 10 GB of data 2009-05-27 23:19:11.309716
Copied 11 GB of data 2009-05-27 23:19:42.529761
Copied 12 GB of data 2009-05-27 23:20:13.553971
Copied 13 GB of data 2009-05-27 23:20:44.419778
Copied 14 GB of data 2009-05-27 23:21:15.461924
Copied 15 GB of data 2009-05-27 23:21:46.650109
Copied 16 GB of data 2009-05-27 23:22:17.699218
Copied 17 GB of data 2009-05-27 23:22:48.770149
Copied 18 GB of data 2009-05-27 23:23:19.780599
Copied 19 GB of data 2009-05-27 23:23:51.019963
Copied 20 GB of data 2009-05-27 23:24:22.066913
Copied 21 GB of data 2009-05-27 23:24:59.482170
Copied 22 GB of data 2009-05-27 23:25:33.433475
Copied 23 GB of data 2009-05-27 23:26:04.662601
Copied 24 GB of data 2009-05-27 23:26:36.083603
Copied 25 GB of data 2009-05-27 23:27:09.586955
Copied 26 GB of data 2009-05-27 23:27:39.184255
Copied 27 GB of data 2009-05-27 23:28:19.708569
Copied 28 GB of data 2009-05-27 23:28:51.849345
Copied 29 GB of data 2009-05-27 23:29:23.161374
Copied 30 GB of data 2009-05-27 23:29:54.155994
Copied 31 GB of data 2009-05-27 23:30:25.264944
Copied 32 GB of data 2009-05-27 23:30:56.101558


Here's how to verify that this process is going to work:
Wait for a couple GBs to finish copying, then you can fdisk the new image and even mount it if the entire partition has already been copied! Here's how to check the partition table:


root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# fdisk -lu output
You must set cylinders.
You can do this from the extra functions menu.

Disk output: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x20000000

Device Boot Start End Blocks Id System
output1 63 96389 48163+ de Dell Utility
output2 98304 21069823 10485760 7 HPFS/NTFS
Partition 2 has different physical/logical endings:
phys=(1023, 254, 63) logical=(1311, 136, 41)
output3 * 21069824 624996351 301963264 7 HPFS/NTFS
Partition 3 has different physical/logical beginnings (non-Linux?):
phys=(1023, 254, 63) logical=(1311, 136, 42)
Partition 3 has different physical/logical endings:
phys=(1023, 254, 63) logical=(38904, 57, 1)



Don't worry about the errors for now - fdisk isn't happy that the file isn't written completely yet. Meanwhile, I was able to mount my two first partitions since they we very small. Note the starting sector of the partition you want to mount by looking at the fdisk -lu output (the -u displays sectors so it is mandatory). For my first partition, the starting sector is 63 (very very common). Now we are ready to mount the image. The mount command loop option takes a starting offset in bytes, but we see it as a starting sector (63 for me). Multiplying the sectors by 512 gets us the byte offset that mount is looking for. I put this multiplication in the code, so you just need to replace the 63 with your starting sector.


root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# mkdir /media/dell
root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# mount -o loop,ro,offset=$((63*512)) -t auto output /media/dell
root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# ls -al /media/dell/
total 168
drwxr-xr-x 3 root root 16384 1969-12-31 18:00 .
drwxr-xr-x 6 root root 160 2009-05-27 23:57 ..
-rwxr-xr-x 1 root root 726 2007-10-19 12:51 autoexec.bat
-rwxr-xr-x 1 root root 726 2007-08-13 10:45 autoexec.up
-r-xr-xr-x 1 root root 53569 2005-06-21 11:54 command.com
-rwxr-xr-x 1 root root 137 2007-10-19 12:51 config.sys
-rwxr-xr-x 1 root root 137 2007-08-13 10:45 config.up
-rwxr-xr-x 1 root root 85 2007-08-13 10:45 copyup.bat
-r-xr-xr-x 1 root root 29690 2005-07-25 11:48 dellbio.bin
-rwxr-xr-x 1 root root 320 2007-08-13 10:45 delldiag.ini
-r-xr-xr-x 1 root root 33352 2005-06-21 11:54 dellrmk.bin
drwxr-xr-x 2 root root 4096 2007-10-19 12:51 diags
-rwxr-xr-x 1 root root 15299 2007-08-13 10:46 himem.sys
-rwxr-xr-x 1 root root 7 2007-10-29 03:12 oobedone.flg


As you can see, I am able to access the data on that partition just fine! Using this same method, I can access my big partition:

root@ubuntu:/media/750GB EXT/RAID_RECONSTRUCT# mount -o loop,ro,offset=$((21069824*512)) -t auto output /media/xps_c_drive


I'm getting really tired now and my screwdrivers are starting to kick in, so I'm going to call this one finished!

-------------------------------
UPDATE July 27, 2009
-------------------------------

I finally have the time and space to recover my data, so I sat down and did it.

Here's what I've got mounted:


/dev/sdb1 on /tmp/750gb type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096)
/dev/sdc1 on /tmp/1tb type vfat (rw)
/dev/loop1 on /tmp/RAID type fuseblk (ro,nosuid,nodev,allow_other,blksize=4096)


I tried a number of ways to copy the data, but it kept failing because I was copying from an NTFS image to a FAT32 drive - their were files over the 4GB-1Byte limit, filenames over the limit and directories nested too deeply. I don't really care about these files anyway. A quick search revealed that the offending large files were just junk anyway (and a game I don't play):


find /tmp/RAID -size +4G -print


Then I used rsync to copy the data from the loop-mounted array to the new storage location:


ubuntu@ubuntu:/tmp/RAID$ sudo rsync --size-only --max-size=4g-1 -rtv /tmp/RAID/ /tmp/1tb/RAID_DATA/ 2>&1 > /tmp/1tb/RAID_DATA.log


The --max-size=4g-1 directive tells rsync to skip files that are larger than FAT32 supports (4GB minus 1 Byte). This is ONLY necessary when copying to FAT32. The rest of the options are easy to understand from the rsync MAN page.

Horray! I've got my data back!]]> http://www.teratechnologies.net/stevekamerman/index.php?entry=entry090528-010324 2009-05-28T00:00:00Z 2009-05-28T00:00:00Z Linux LVM Snapshot Backups dump command to make stateful filesystem backups in the past, so I recently wanted to take a similar approach to backup some Linux servers. Most GNU/Linux distributions use ext3 as their root filesystem by default these days which does not natively support snapshots, so I needed to find another way to do it. As it turns out, I used LVM on my main Linux servers to add a layer of virtualization to the storage subsystem. LVM does natively support snapshots, and here's how to do it.

1. Make room for the snapshot


I ran into a major problem right off the bat - the server I was testing LVM snapshots on had one hard drive with 2 partitions, 1 for /boot and one for LVM. The LVM partition contained one Volume Group called "TERA-VIRTUAL" and this VG had 2 Logical Volumes - "root" (mounted on /) and swap_1 (swap). The root LV had all the PE allocated to it and the filesystem beneath it was using all of the available space. The problem is that since there were no available Physical Extents in the VG (or in my PVs), I couldn't create a snapshot volume. This is important: when you create a snapshot on LVM you are really creating another Logical Volume that is used to hold the changes to the volume. This volume needs to me in the same Volume Group as the volume you are taking a snapshot of - so you need some extra space in that VG! In my case, I needed to resize the root fs, which was on a Logical Volume - this to me an hour plus to achieve since most LiveCDs either have no support for LVM (gparted-live) or the support is broken (backtrack 3 final usb). I booted ubuntu server 7.10 64bit into rescue mode and started a shell in the installer environment and once at a prompt typed vgmknodes to make the /dev/ entries for my LVs, then used resize2fs to shrink my FS and lvreduce to shrink my LV, leaving me with 23G of available Physical Extents in my VG (seriously, if you don't know what I'm talking about, don't try it - you will trash your hard drive!).

2. Create the snapshot and run backup


To create a snapshot you first need to figure out how much space you need - it is critical that you have enough space to hold all the changes to your volume until the snapshot is removed!. For me 10% of my 200G is fine - 20G.
The process is to create a snapshot LV, mount it, do your backup, unmount it, and remove the snapshot:


root@TERA-VIRTUAL:~# lvcreate --size 20G --snapshot --name snap /dev/TERA-VIRTUAL/root
Logical volume "snap" created
root@TERA-VIRTUAL:~# lvs
LV VG Attr LSize Origin Snap% Move Log Copy%
root TERA-VIRTUAL owi-ao 200.00G
snap TERA-VIRTUAL swi-a- 20.00G root 0.00
swap_1 TERA-VIRTUAL -wi-ao 9.46G
root@TERA-VIRTUAL:~# lvdisplay /dev/TERA-VIRTUAL/snap
--- Logical volume ---
LV Name /dev/TERA-VIRTUAL/snap
VG Name TERA-VIRTUAL
LV UUID fFbaH4-22Hq-s7a2-e1mo-DPov-8wff-dvuAiI
LV Write Access read/write
LV snapshot status active destination for /dev/TERA-VIRTUAL/root
LV Status available
# open 0
LV Size 200.00 GB
Current LE 51200
COW-table size 20.00 GB
COW-table LE 5120
Allocated to snapshot 0.01%
Snapshot chunk size 8.00 KB
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 254:2

root@TERA-VIRTUAL:~# mount /dev/TERA-VIRTUAL/snap /backup_snapshot/
root@TERA-VIRTUAL:~# cp -Rf /backup_snapshot/VirtualMachines/TERA-WEBSERVER /backup/
root@TERA-VIRTUAL:~# lvdisplay /dev/TERA-VIRTUAL/snap
--- Logical volume ---
LV Name /dev/TERA-VIRTUAL/snap
VG Name TERA-VIRTUAL
LV UUID fFbaH4-22Hq-s7a2-e1mo-DPov-8wff-dvuAiI
LV Write Access read/write
LV snapshot status active destination for /dev/TERA-VIRTUAL/root
LV Status available
# open 1
LV Size 200.00 GB
Current LE 51200
COW-table size 20.00 GB
COW-table LE 5120
Allocated to snapshot 1.96%
Snapshot chunk size 8.00 KB
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 254:2

root@TERA-VIRTUAL:~# umount /backup_snapshot/
root@TERA-VIRTUAL:~# lvremove /dev/TERA-VIRTUAL/snap
Do you really want to remove active logical volume "snap"? [y/n]: y
Logical volume "snap" successfully removed


As you can see from the transcript above, you can keep an eye on your snapshot size by using lvdisplay and looking for "Allocated to snapshot". As seen above, before the backup the snapshot volume was 0.01% full, but by the time my backup finished, it was up to 1.96%.

Now I've got a sane copy of my data without having to dismount my filesystem!
]]> http://www.teratechnologies.net/stevekamerman/index.php?entry=entry080803-221718 2008-08-04T00:00:00Z 2008-08-04T00:00:00Z FreeBSD watch command is not GNU/Linux watch command

# watch ls -lah


This example will keep showing the output of ls -lah every 2 seconds.

About a year ago I switched a couple servers over to FreeBSD 6.1 because of it's security features and more stable development model. For the last year I've been occasionally using the watch command on FreeBSD and getting an annoying error like this:


[root@devel /backup]# watch ls -lah
watch: fatal: bad device name


Today it finally occured to me to look at the manpage:


WATCH(8) FreeBSD System Manager's Manual WATCH(8)

NAME
watch -- snoop on another tty line

SYNOPSIS
watch -cinotW -f snpdev tty

DESCRIPTION
The watch utility allows the user to examine all data coming through a
specified tty using the snp(4) device. If the snp(4) device is not
available, watch will attempt to load the module (snp). The watch util-
ity writes to standard output.


It hardly looks like the watch I know! It turns out that FreeBSD has a different use for watch - snooping on other consoles. On FreeBSD, the GNU watch (like on Linux) is called gnu-watch. If you don't want to install it, you can easily whip up a poor-man's replacement:


#!/usr/local/bin/bash
while [ 1 -lt 2 ]
do
clear
date
echo ------------------------------------
eval $@
sleep 2
done


TIP: To replace the default 'watch' with this script, save the above script as /root/watch, then make it executable, chmod +x /root/watch and alias it in your shell, alias watch='/root/watch'. You can save this alias permenantly with the bash shell by adding it to ~/.bash_profile .

Here's a sample output from the poor-man's watch:

Thu Jul 10 22:09:27 EDT 2008
------------------------------------
total 17113832
drwxr-xr-x 3 root wheel 512B Jul 10 21:03 .
drwxr-xr-x 21 root wheel 512B Jul 10 18:25 ..
-rw-r--r-- 1 root wheel 97M Jul 10 15:41 archive1.tar
-rw-r--r-- 1 root wheel 40M Jul 10 15:41 archive1.tar.bz2
-rw-r--r-- 1 root wheel 308M Jul 10 17:19 archive2.tar
-rw-r--r-- 1 root wheel 93M Jul 10 17:19 archive2.tar.bz2
-rw-r--r-- 1 root wheel 600M Jul 10 17:13 archive2a.tar
-rw-r--r-- 1 root wheel 192M Jul 10 17:13 archive2a.tar.bz2
-rw-r--r-- 1 root wheel 3.3G Jul 10 19:09 archive2b.tar
-rw-r--r-- 1 root wheel 1.4G Jul 10 19:09 archive2b.tar.bz2
-rw-r--r-- 1 root wheel 5.4G Jul 10 19:33 archive2c.tar
-rw------- 1 root wheel 1.9G Jul 10 22:09 archive2c.tar.bz2
-rw-r--r-- 1 root wheel 2.1G Jul 10 17:44 archive2d.tar
-rw-r--r-- 1 root wheel 886M Jul 10 16:08 archive3.tar
drwxr-xr-x 2 root wheel 512B Jul 10 19:44 scripts
]]> http://www.teratechnologies.net/stevekamerman/index.php?entry=entry080710-203456 2008-07-11T00:00:00Z 2008-07-11T00:00:00Z Dreamweaver crashes every day Runtime Error! This application has requested the Runtime to terminate it in an unusual way. Here's what it looks like:



This error makes my blood boil! I've done everything to try it including, but not limited to:

Delete the configuration Files


This seems to give me the most success. Dreamweaver stores a file cache and some non-site related settings in C:\Users\[your username]\AppData\Roaming\Adobe\Dreamweaver 9\Configuration. If you delete or rename this directory then Dreamweaver will recreate it when it starts up.

Delete the Dreamweaver Windows Prefetch file


When a program starts up, Windows trys to optimize the startup process by creating a "cache" type file in C:\windows\Prefetch. Next time the program opens it uses this file to speed up the process. You can safely delete everything in the Prefetch folder. Doing this helped me get Dreamweaver working yesterday.

Get rid of Logitech SetPoint


Someone on an Adobe forum said this worked for them - I haven't had much success

Restart computer with only essential processes enabled


Also from Adobe, this seems to have no impact on my computer - it still crashes.

Maybe it's the 8KB file bug???


According to many people on Adobe's forum, if you a working on a file that is exactly 8,192 bytes (8KB) Dreamweaver will crash. Well, my files are taking up 8,192K because the block size on my disk pushes 7+K files up to 8,192 (look at "size on disk"), but I put a huge HTML comment at the end of the file to push it past 10K and it didn't help.

Doing the exact same thing every time will make Dreamweaver crash every time


Here's my most effective solution as of this morning: once Dreamweaver crashes, you can't do the exact same thing when you reopen it or it will just keep crashing. I was trying to add a 4x3 table to a document and everytime I did Insert->Table->OK it immediately crashed (I did this 10+ times while changing other Windows stuff). Then I went to show my business partner and instead of adding a table, I went to a different Site and changed some code around, then when back to the original site and BAM! it works fine now! What the HECK is going on here?

Final thoughts


You know, I've been working with Linux and open source software for over 10 years, and I finally break down and by a $2,000+ software package - I expect it to work! If it doesn't work I expect that I can get support to make it work (and not that "please restart your computer then uninstall/reinstall" BS)! Come on Adobe!!! If I charge my clients >$100/hr and I have to fight with Dreamweaver for 1 hr per day, where's my $36,500 from Adobe for the last year?

P.S. I think the problem is related to MS Visual Studio 2005's runtime update and Vista or something.]]> http://www.teratechnologies.net/stevekamerman/index.php?entry=entry080618-112602 2008-06-18T00:00:00Z 2008-06-18T00:00:00Z