I'm just about the release Tera-WURFL 2.1.0 (Stable) and I thought I'd give you an idea of what to expect. I was originally positioning this as a revisional update (i.e. 2.0.1), but decided to make it a minor version upgrade because it adds a lot of functionality and includes a few more settings than version 2.0.0.

Here's what's new:

1. Experimental support for Microsoft SQL Server 2005/2008: with the MSSQL2005 Database Connector, you can now use MS SQL Server as a Tera-WURFL backend! The support is experimental for now, although I've put it through my barrage of over 45,000 user agents and it works properly. The Reduction in String stored procedure still needs to be optimized since I just ported it from MySQL 5 to T-SQL. The MSSQL backend is considerably slower than MySQL, and I beleive it's that procedure slowing it down.

2. SimpleDesktop Matching Engine: Tera-WURFL wasn't originally designed to differnetiate between desktop and mobile browsers (either was WURFL for that matter), but with this release I've introduced the SimpleDesktop Matching Engine which, when enabled, uses keywords and regular expressions to detect 90% of desktop browsers without having to resort to searching the database for a matching WURFL entry. In my tests performance increased by 176% for detection of 45,000 actual unique user agents (both mobile and non-mobile). This feature also dramatically decreases the number of items in your cache by using a single cache item for all desktop browsers.

3. Capabilities Filter: I've been meaning to implement the Capability Filter for a long time, but two very high traffic clients of mine convinced me to sit down and get it finished. This adds a new setting in TeraWurflConfig.php called CAPABILITY_FILTER. If you set it to false it will be disabled and all the capabilities in the WURFL will be stored in the database and available to your scripts (this is the pre-2.1.0 behaviour). Here's where the magic starts, you can set it to an array of the capabilities and groups of capabilities that you want to store and use, for example, if you just want the know what kind of device is visiting your site and whether or not it's wireless, you can use this filter:

public static $CAPABILITY_FILTER = array(
"brand_name",
"model_name",
"is_wireless_device"
);

This will shrink your device database by more than a factor of 10. I tested the filter with 20 capabilities against 45,000 unique user agents and it reduced the size of the cache from 645MB (without filtering) to 92MB (with filtering), then down to 24MB with both filtering and SimpleDesktop.

You can look forward to the Tera-WURFL 2.1.0 release around February 10, 2010.

Tera-WURFL is available at it's usual location, http://www.Tera-WURFL.com/

I happen to have an iPhone and an iPod Touch sync'd to my iTunes back in the US, but now I'm in Iraq and don't have access to my computer with the iTunes library on it. I do have the music, but I don't want to sync my iPod to my iTunes because all the play counts and ratings will be gone. I had a little free time last night so I wrote a script to update these attributes for me! It is written in JScript/JavaScript and uses the iTunes COM Interface to communicate with your device and the iTunes Library. It is not optimized very well so it will iterate over the entire iTunes Library for each track it encounters on your device, looking for a matching Artist, Album and Track Name to update. It also maintains a log file that lists every track that was updated including it's old and new values, as well as any tracks on the iPod/iPhone that it could not find a match for in the iTunes Library. I created this script for my own purposes and decided to share it, so I don't intend to provide much support for it. The zip file contains two files: UpdatePlayCount.bat and UpdatePlayCount.js. Extract the files somewhere, plug in your iPod/iPhone, then open iTunes and click on your device, then check "Manually manage music and movies" and restart iTunes, then double click on UpdatePlayCount.bat to update your library. You can see that progress is being made in iTunes by clicking on your Library's Music folder then sorting the music by Play Count or Rating. This list is updated in real time.

Download UpdatePlayCount.zip (2KB)

Source Code (JScript/JavaScript)


/**
* UpdatePlayCount.js
*
* Description
* ---------------------------------------------------------------------------------------------------------
* This script will update your iTunes library with the play counts and ratings from any iPod or iPhone.
* It uses the iTunes COM Interface to communicate with your device. In order to determine which tracks
* match, the script will search through the iTunes Library looking for a matching Artist, Album and Title;
* as a result, this process can take a long time. Updates all the music and movies in the "Music" and
* "Movies" playlists on your device. If you need to update more folders you can modify the code.
*
* Usage
* ---------------------------------------------------------------------------------------------------------
* Double click on UpdatePlayCount.bat
*
*
* @package UpdatePlayCount
* @author Steve Kamerman, stevekamerman AT gmail.com
* @version Alpha 1.0 $Date: 2009/12/22 13:19
* @license http://www.mozilla.org/MPL/ MPL Vesion 1.1
* @language Microsoft JScript / JavaScript
*
*/

var logObject, logFile;
var logfileName = "UpdateItunes.log";
logObject = new ActiveXObject("Scripting.FileSystemObject");
logFile = logObject.CreateTextFile(logfileName, true);

var ITTrackKindFile = 1;
var iTunesApp = WScript.CreateObject("iTunes.Application");
var sources = iTunesApp.Sources;
var i;
var updateTracksCount = 0;
var missingTracksCount = 0;
var ipod;
var itunes;
var playlists;
var itunesPlaylists;
var iTunesXML;
iTunesXML = "";

for(i=1;i<sources.Count;i++){
if(sources.Item(i).Kind == 2){
ipod = sources.Item(i);
playlists = ipod.Playlists;
}
if(sources.Item(i).Kind == 1){
itunes = sources.Item(i);
itunesPlaylists = itunes.Playlists;
}
}

if(ipod == undefined){
WScript.Echo("Error: No iPod or iPhone found. Please make sure your device is listed in iTunes, then click on it's name and check \"Manually manage music and videos\". Restart iTunes and rerun this script.");
WScript.Quit(0);
}
WScript.Echo("iPod found, press OK to update iTunes with the play counts and ratings from your iPod.");
WScript.Echo("This process can take a long time to complete. To monitor the progress,\nopen iTunes and click on your iTunes Music folder\nthen sort by play count or rating and watch them change.\nPress OK to continue");
var currentPlaylist;
var currentTracks;
var currentTrack;
for(i=1;i<playlists.Count;i++){
currentPlaylist = playlists.Item(i);
if(currentPlaylist.Name != "Music" && currentPlaylist.Name != "Movies")continue;
currentTracks = currentPlaylist.Tracks;
for(a=1;a<currentTracks.Count;a++){
currentTrack = currentTracks.Item(a);
if(currentTrack.PlayedCount > 0){
updateITunesEntry(currentPlaylist.Name,currentTrack.Artist,currentTrack.Album,currentTrack.Name,currentTrack.PlayedCount,currentTrack.Rating);
}
}
}

WScript.Echo("Finished processing "+(updateTracksCount+missingTracksCount)+" tracks.\nUpdated Tracks: "+updateTracksCount+"\nMissing Tracks: "+missingTracksCount+"\nSee the logfile ("+logfileName+") for more details.");

function updateITunesEntry(playlist, artist, album, song, playCount, rating){
var itunesPlaylist;
var itunesTracks;
var itunesTrack;
var i;
var a;
for(i=1;i<itunesPlaylists.Count;i++){
itunesPlaylist = itunesPlaylists.Item(i);
if(itunesPlaylist.Name != playlist)continue;
itunesTracks = itunesPlaylist.Tracks;
for(a=1;a<itunesTracks.Count;a++){
itunesTrack = itunesTracks.Item(a);
if(itunesTrack.Artist == artist && itunesTrack.Album == album && itunesTrack.Name == song){
logFile.WriteLine(itunesTrack.Name+": count: "+itunesTrack.PlayedCount+"->"+playCount+", rating: "+itunesTrack.Rating+"->"+rating);
itunesTrack.PlayedCount = playCount;
itunesTrack.Rating = rating;
updateTracksCount++;
return(1);
}
}
}
missingTracksCount++;
logFile.WriteLine("WARNING: Track not found in iTunes Library: "+artist+" - "+song);
}


Just wanted to let you know that I'm here in Iraq and beginning my mission. I may provide more details about where I am and what I'm doing here in the future. It looks like I won't have as much time as I thought, but I still plan to get some work done on Tera-WURFL while I'm here!

I ran a database of 23,902 unique user agents through both Tera-WURFL 2.0.0 RC4 and the new WURFL PHP API (1.0.1-rc2). Here are the results:

Tera-WURFL
Total Time: 12.081017971039
Devices Processed: 23902
Total Queries: 23912

WURFL PHP API
Total Time: 217.57795381546
Devices Processed: 23902
Total Queries: N/A

Both results were obtained after repeated testing to verify that the user agents were cached.

Tera-WURFL started in late 2006 as a personal project to make the WURFL PHP Tools faster by storing the device capabilities in a MySQL Database instead of flat files. It's now late 2009 and I've finally got Tera-WURFL 2.0 up and running. As of now I've released RC4 and I'm working on RC5. Hopefully I can get version 2.0.0 Stable out before I leave for Iraq. Here's a breakdown of the features of Tera-WURFL 2:

Rewrote some of the UserAgentMatchers and deleted others to bring Tera-WURFL on par with the Java WURFL API. With the introduction of desktop browser UserAgentMatchers, we no longer need to use the large web patch; instead, you can just use the 8KB one from wurfl.sourceforge.net (included). Also, I fixed some typos and bugs here and there. NOTE: if you are upgrading from version 2.0.0 RC1-RC3 you should delete all your database tables before you update. You can leave the terawurflcache table if you want to retain your cache. Although it will still technically work even if you don't delete the tables, you will be orphaning some unecessary tables in your database.

Complete code-rewrite from the ground up. The 2.x version of Tera-WURFL is loosely based on a pre-release of the Java WURFL Evolution Library, but the API is taken from Tera-WURFL 1.5.2. The following is a list of features found in Tera-WURFL 2.0:

• User Agent Matchers have been created for each of the major manufacturers. These allow for specific matching methods to be applied to the user agent like string searching, RIS (Reduction in String) and LD (Levenshtein Distance).
• Multiple patch files are now supported. Tera-WURFL ships with the current wurfl.xml, web_browsers_patch.xml and custom_web_patch.xml. Patch files can be added to TeraWurflConfig.php by separating them with semicolons in the TeraWurflConfig::PATCH_FILE directive. Patch files are loaded in order from left to right on top of the WURFL file, so if you want to override every other patch file, specify it last.
• The custom_web_patch.xml file can be edited from the Web Administration page, and allows you to easily add non-mobile user agents to the patch file. The devices with these user agents will be detected as generic_web_browser (non-mobile).
• Persistent Caching means that your cached devices stay cached. When you update the WURFL file or your patches, your device cache is also updated via the new database.
• Cache Browser allows you to see what devices are hitting your site and what their capabilities were detected as.
• Installation Script is better than 1.5.2. Once you download Tera-WURFL and extract it, edit TeraWurflConfig.php then go to /admin/install.php and follow the directions to finish installation.
• PHP short_open_tags are no longer required to run Tera-WURFL. PHP has this feature disabled by default now.
• Conclusive vs. Inconclusive Matching. If a device is matched with the UserAgentMatcher's primary matching method it is considered a conclusive match, if it is detected via a recovery matcher or by the CatchAllMatcher it is an inconclusive match. This information is available via the tera_wurfl capability group.
• tera_wurfl Capability Group. The TeraWurfl->Capabilities array now contains a group called "tera_wurfl". This group contains the following Tera-WURFL related information:
  • num_queries - the number of database queries required to lookup the device.
  • actual_root_device - the WURFL ID of the actual device (not subrevision or generic), this can be null.
  • match_type - either conclusive or inconclusive.
  • matcher - the name of the UserAgentMatcher that detected the device.
  • match - whether or not there was an actual match. If there was no match, Tera-WURFL guessed which generic device is most similar to the device.
  • lookup_time - the time in seconds that it took to detect the device.
  • fall_back_tree - the complete fallback tree that built the capabilities of the device. This is a list of all the WURFL IDs from the detected device down to the base generic device.

Example Script


<?php
// Include the Tera-WURFL file
require_once("TeraWurfl.php");
// Instantiate the Tera-WURFL object
$wurflObj = new TeraWurfl();
// Get the capabilities from the object
$matched = $wurflObj->GetDeviceCapabilitiesFromAgent(); //optionally pass the UA and HTTP_ACCEPT here
// Show whether there was a conclusive match
if($matched){echo "Match found";}else{echo "Match NOT found";}
// Print the capabilities array
echo "<pre>".htmlspecialchars(var_export($wurflObj->capabilities,true))."</pre>";
?>