So, what’s a blade server?

I’ll get to exactly what a blade server is shortly, but first an analogy serves well for visualization: think of your local telephone company and the phone line they provide you. If you want an extra telephone in your house and you’ve already got the connectors wired, you buy a new phone, connect it to the jack, then you’ve instantly got a dial tone with no fuss and no pain. This is exactly how blade servers are supposed to work for the computing industry. Put simply, a blade server (or simply “blade” for short) is a modular computer; a self contained motherboard, processor, RAM, and storage module that’s easily plugged or unplugged from a rack specifically designed to accept them. It typically has a proprietary connector containing power, network, and health management functionaliy that’s automatically connected and configured when the blade is plugged into the backplane (which is the “jack”.)

So, when a company or university needs more compuational power because the web-server is bogged down generating pages or because a scientific simulation would be too slow otherwise, they buy an extra blade and plug it in to instantly get another computing resource. Of course they probably need to configure software on the new computer for it to be useful, but that’s not important for this discussion. The take-away point is that a blade gives a no-fuss method for additing additional computers to your infrastructure, and as a bonus, when a computer inevitably fails blades give a great way to swap out the failed module with a new one in a matter of seconds.

The SSI Project

A major problem hindering blade adoption has always been the lack of any standard (blades have been around en-masse for at least a decade, and yet most haven’t ever heard of them!) Concretely, Appro will sell you their own design for a backplane and blade, which is different then the one IBM sells, which is different than the one Dell sells. Of course, this is a headache for numerous reasons. First there is inherit risk in the future cost of any blade you buy; if a vendor decides there isn’t enough margin in their blade product line and doubles their prices, in general you can’t seek a third party blade as a second source to combat the margin surfing. Next, the same vendor may decide to end-of-life the very blades that your backplane accepts, leaving you searching eBay for used parts should you ever need more blades or replacement components. And worst case, what happends if the vendor goes out of business right when you buy your first backplane and single node? It’s potentially the Edsel car of the computing industry!

Intel has done something to change all of this. Their goal is of course to sell more CPUs, and blades are a perfect way for them to do so since the marginal per-blade upgrade cost is typically much less than that of a full computer, people buy more CPUs because they can afford more blades. So, they’ve pulled together a consortium of juggernauts in the blade industry, to design a standard architecture for blade servers and their backplanes to ensure that most of the drawbacks to blade infrastructure are washed away. If and when vendors adopt the standard, you’ll be able to cross company lines for sourcing blade servers and backplanes, just like you can cross company lines for hard disks, RAM, workstations, etc. today. If they pull such a feat off, it will be a landmark event in the computing industry to say the least, an event as significant as Compaq’s upheaval of the PC market with the reverse engineering and re-implementation of IBM’s BIOS, or with AMD’s implementation of the x86 processor line in the i386 and i486 processor days. Let’s hope they do.

The Gaping Hole

Unfortunately, the SSI picture isn’t all roses today; the standards committee has inadvertently created a security and debuggability nightmare.

I’ve glossed over the networking aspects of blade computing, but further discussion is warranted, because this is the cheif problem with the current SSI implementation. The backplanes for blade servers usually have an integrated network switch of some sort, with ethernet and infiniband incarnations being the most common. The utility here is clear; by including a network switch in the backplane a single network cable can be connected to the backplane and provide outside-world connectivity for every blade in the rack. There’s not a thing wrong with the idea behind this method, but the SSI implementation lacks a way to monitor inter-node traffic which probably makes the security administrator and MPI application developer readers groan.

What’ exactly is the problem, in case you didn’t catch it? (And if you didn’t catch it don’t worry — it can be a subtle point even if you’re on the periphery of one of the above categories.) The problem is that you can’t see anything that the nodes in an SSI backplane say to one another. In effect, you can only monitor the connections from the backplane to the outside world.

Consider the following illustrative case for why the current SSI specification is currently broken: A very common architecture for an internet website running an online store is to run a single or a few web-server blades that talk directly to internet shoppers serving them images, shopping carts, and other pages, with two or three times as many database blades connected to the web-servers with current information about item availability, stock, prices, outstanding orders, etc. A typical attack vector is the following: a malicious user breaks into the web-server through a known or newly developed vulnerability over an encrypted (https) link. The user then directs the web-server to fetch credit card numbers, names, and addresses from the database server, typically through the unencrypted link between the web-server and the database server, then tunnels the information through the encrypted link back to their PC. With the SSI blade system, a network forensics or capture device would have no way of seeing the unencrypted data-leakage, since it would happen exclusively on the blade backplane. In fact, unless the database query statements are audited and/or an SSL decryptor in used to feed the forensics systems, the company under attack will probably never know. In practice, most corporations have neither an SSL decryptor nor query auditing, since both are an expensive and detail-oriented tasks and their need is normally mitigated by forensics devices snooping the un-encrypted traffic.

Another, concrete example is in order. Super-computers are typically strung together from many single computers of the same makeup, obviously a prime market for blade servers. The developers of the applications that run on super-computers typically use the Message Passing Interface (MPI) framework for making the single computers act in parallel and in lock-step as one large super-computer. MPI programming is unfortunately error prone and hard, however, which is why super-computer programmers command big salaries. To debug MPI programs the quintessential method is to capture the messages that individual computers pass one-another, and examine them for errors or other incorrect behaviour. With a super-computer made of SSI blades, however, this debugging paradigm is completely unavailable. A packet capture appliance has no single point of entry, and thus doesn’t see the messages that nodes pass one-another. Instead, developers need to debug through other means, like developing a framework for dumping messages to a log on each machine, collecting them, ordering them, and analyizing them, hoping that the framework didn’t miss a critical component of the message; or perhaps they could run tcpdump on each node, and hope that the traffic is slow enough for that tool to keep up (which may sound trivial but is in fact a major problem,) though in that case they still need a way to collect coalesce the resulting PCAP files.

Of course, there are many other examples of what is lost without the ability to snoop backplane network traffic, but the idea behind the problem should at least be clear with the scenarios already presented. What’s needed then, is a fix. The SSI specification can be augmented to support a network TAP port and all of these issues vanish in a blink. I’ve personally told the SSI developers about this issue, and its now on their radar. More feedback, of course, will always help.

Conclusion

The SSI platform represents a giant leap forward for the computing industry as a whole but it introduces a major security and the debugging nightmare into environments that already have too many of those things. A simple change can make the collective lives of every SSI blade user simpler, so they can worry about everything else.

After it was installed, I went searching the internet far and wide for what proved to be a nearly unreachable prize — the 2.0 patch to Flight Unlimited 3. It took several hours of crawling, and ultimately registering for a Finnish website (using google translate to help along the way) to find the patch. To save everyone else the trouble, consider this a silly gift from me to you:

FU3PATCH 2.0, US Edition (Download)

If you’ve got a copy of the UK edition or other world editions, please send them to me and I’ll include them here!

One of the great mysteries of High Energy physics is the origin of the highest energy cosmic rays; the Pierre Auger experiment has given weak evidence that active galactic nuclei are a potential source, but this analysis is far from conclusive. One area of interest which I’m pursuing (and an area which will probably make its way into my thesis) involves confirming these results through improved modeling of the data processing for the Telescope Array experiment and its predecessor, HiRes, by correcting what is potentially a serious analysis error that eliminates the ability to confirm Auger’s result.

The essence of this error lies in the method used to find and remove events triggered from lasers from the Telescope Array and HiRes data sets. Without publicly revealing too many details until the analysis is complete, it is plausible that correcting this error will lead to a reevaluation of the origins of Cosmic Rays which could be compatible with the Auger collaboration’s results.

The first major overhaul of targeted advertising happened with the inception of keyword based internet advertising. Early pioneers like DoubleClick (1996) and Yahoo shaped the second generation of ad delivery. Now instead of simply placing a Financial Services ad in the printed Dow Jones Industrial charts, a Stock Broker could place their ad in-line with an internet article that talks about selecting a Stock Broker. Not only was this more efficient in terms of targeting the buying consumer, but ad-space became much cheaper — a print advertising campaign that cost several thousand dollars and hit many unrelated parties became an online campaign that cost a few hundred dollars.

DoubleClick in particular tried to bridge the gap between the keyword based online advertising and the new, more intelligent history based advertising. What does that mean, exactly? DoubleClick advertises in many places and they collect data each time they serve an advertisement, regardless of the site. They take this data, and try and correlate many events to a single user; they try and track you through each site you visit in your day-to-day life in order to build an advertising profile, then they use this profile to serve you more targeted ads.

As a concrete example, a consumer (let’s call him Joe) visits Expedia, Delta Airlines, and Kayak, presumably looking for airline tickets. Assuming DoubleClick advertises on both Expedia and Kayak, they can try and put two-and-two together to determine that Joe is planning a trip, and they can serve travel related advertising to Joe on any site that he visits whether or not the content on the site is related to travel.

By making correlations between Joe’s site visits and the detected patterns, DoubleClick has been able to increase the effectiveness of their advertising campaigns. There are, however, some serious limitations to the DoubleClick model. 3rd generation advertising attempts to overcome these limitations.

Present Limitations

• Present advertising software lacks a comprehensive view of a user’s actions. They see only what they see, and that’s a very small picture.

When DoubleClick’s software realized that Joe was going to take a trip, they used every opportunity available to try and present him with ads for travel. Unfortunately, DoubleClick’s internet coverage is quite small; fewer than 1 in 5,000 sites are reachable by DoubleClick. This means that while DoubleClick may have properly guessed that Joe intended to take a trip, they can’t see when Joe visits Delta’s website directly (which doesn’t have DoubleClick ads) and purchases his tickets. Not only is DoubleClick’s ad wasted since Joe doesn’t need travel arrangements, they’ve also wasted their opportunity to present him with another more appropriate ad. No one benefits from this situation.

• Keyword driven advertising misses the actual content.

Perhaps the most well known keyword based advertising system on the internet is Google’s AdSense. AdSense has built an advertising network spanning many sites (incidently this network is intimately related to DoubleClick’s network as Google purchased DoubleClick in 2007 and has begun to integrate its products with DoubleClick). These sites are corporate and personal alike, with all types of content and all motivations. When a content provider signs up for AdSense, they integrate Google advertising into their own content and are paid based on readers’ clicks on the advertising.

When Google delivers advertising based on keywords found on a content providers’ site, they are doing so with blinders on. For instance, a web page with investment advice may mention the term “Real Estate” tens of times, but not fundamentally talk about buying or selling a home. Nonetheless, AdSense will spot the many uses of the term “Real Estate”, and probably serve an ad for a local REALTOR. This ad doesn’t really benefit anyone involved, and again the opportunity to serve a better, more targeted ad is lost. This is the reason that the clicks-per-view for Google AdSense are quite low (0.05 clicks per impression).

• Conversions aren’t used to target other advertising.

When a user is served an online advertisement and then clicks on the ad to buy a product, everyone benefits. The advertiser has made a sale, the advertisement content provider (like Google’s AdSense) has generated revenue from the click, and the consumer gets something they’re after. There is unrealized potential for more benefit from this transaction for every party, but the data regarding the transaction ends at a sale. In the next, or 3rd, generation of targeted advertising, data regarding a sale is sacred and potentially much more valuable than the keywords on the website in which the ad was served.

To explain how this data is valuable, consider a poker tournament. A good poker player will find a “tell” in all of her opponents. A tell is a twitch, an involuntary twitter, a change in breathing patterns, or something else discernible when a player is bluffing, or playing as if their cards were better than they actually are. By finding and learning to watch for a tell like this, a good player has given herself an advantage in the game by being able to recognize when the odds for a win are in their favor.  Compare this with the data collected in a sale — the sale is a consumer tell. Advertisers should be able to categorize, classify, and manage the events and content leading up to the click and sale, perhaps even going back weeks or months in browsing history.  Once this tell is discovered, the advertiser can look for the same tell in another consumer’s patterns to try and predict another purchase, or apply this tell to other consumers that behave similarly.

The 3rd Generation of Targeted Advertising

Advertisers and marketers alike are advancing the advertising landscape to overcome these limitations and soon will take advantage of the tells ready to be found.  Deep packet capture, or the practice of capturing and storing all network traffic for a period of time, is becoming a standard component in the IT management community and has already hit mainstream in the intelligence community.  The idea is simple:  Store everything that happens on a network because you probably won’t know until long after the traffic has passed which parts are interesting.

The benefits of this data retention to government intelligence agencies should be obvious.  If the FBI arrests a suspected criminal, they could potentially obtain a warrant and see all of the suspect’s past network traffic which includes website visits, IM conversations, email, and other sensitive data.  The CIA and the NSA may use the same data as a basis for analysis; more on that later.

For IT managers, the benefit of data retention probably isn’t as immediately obvious, though the benefits are still quite important and ultimately valuable.  Foremost, a company involved in a lawsuit with an employee over wrongful termination would no-doubt like to have a complete history of all network traffic the litigious employee originated to use as evidence in a courtroom.  Similarly, the same company might be a defendant in another lawsuit and could produce network traffic during discovery or as evidence while defending itself in court.

From the technology development and maintenance standpoint, data retention through deep packet capture can provide an IT staff with a forensics toolkit for finding slow points in network infrastructure.  The staff may observe that packet transit times on a particular network are 5 times slower than other networks and use this data to find a faulty switch.  At the application level, the IT staff might develop a model for what standard network traffic looks like and apply the model to new traffic to quickly find problems and security breaches.

But back to the main question, how does this apply to targeted advertising?  The answer is that deep packet capture provides a history and dataset upon which “tell tracking” can be built — that is, it gives enough information to advertisers to fundamentally change the advertising landscape. By installing a capture box at the internet service provider (ISP) level, a 3rd generation advertising company would see all network traffic originating from or destined to that internet provider.  Every single instant message a consumer sends and every web page they visit can be used to build a profile which goes leaps beyond what DoubleClick has done.  Because the advertising company could see all traffic, with and without advertising alike, the amount of data available to find a consumer tell grows exponentially, as does the ability to convert adspace into sales.

Thus it becomes clear that packet capture is a required basis for 3rd generation advertising.  This basis isn’t sufficient to propel advertising into new efficiencies and conversion ratios though.  There is another critical component to be built on top of the packet capture, in the same way that flour is a critically important component in the final product when baking bread: statistics.

Statistics are the “how” for tell tracking. By building statistical profiles for all consumer traffic at once, advertisers can search for (and find) consumer behavior patterns which lead to sales.  Every iota of every data packet will be analyzed by a statistics engine, and each of these iotas will be analyzed in aggregate.  The statistical information generated from this process will be applied to other consumers data, and a spider web of interconnected behavior and patterns will be built.

At this point it’s easy to see why the government is interested in installing packet capture devices wherever possible and how they might analyze that collected information.  Law enforcement will be made more efficient as will traditional intelligence gathering and anti-terrorist monitoring.  Cyber-warfare attacks will be more easily recognized, predicted, and prepared for.  The same style software will also be a boon to advertisers.

This spider web will allow advertisers to assign probabilities to all types of behavior, the interesting behavors include being able to tell with a 78.3% confidence that Bob Richards is going to buy a new car in the next 3 days and selling Bob’s information and statistical profile to Ford or GM for hundreds of dollars.  The best part is that as time progresses and more data is collected, the spider web will grow bigger and become more accurate, possibly even to the point of realizing that Bob Richards is going to buy a new car before Bob himself knows it.  And because the advertisers aren’t looking at keywords to target their advertising, they could give Bob a car ad on a furniture restoration website and have higher conversion rates than placing an ad on Edmunds today, hoping to find a car buyer.  Markets will take the next step toward ultimate efficiency, advertising will become cheaper on a cost-per-lead basis, and consumers will see more ads that are relevent to them.

So why hasn’t this happened before now?  Simply put, computers weren’t fast enough, storage wasn’t cheap enough, and the industry tends to move in small baby steps without making the giant leap to the next revolution.  The gap is still wide, and an enterprising company stands to create and own a new market which uses statistics to push targeted advertising into its third generation.

Utah law requires real estate licensees to use this form.  Buyer and Seller, however, may agree to alter or delete its provisions or to use a different form.

Unfortunately, this leaves licensees like me stuck in a conundrum:  There are many contracts for the purchase (or eventual purchase) of real estate in which I am a principal (that is, a Buyer or Seller), and which aren’t appropriate for the REPC.   I’d like to use some other contract.  Am I allowed to?  The verbage is unclear and leaves a logical loophole.  I contacted the Utah Association of REALTORs and asked their lawyer this question.  His answer?  In his opinion, acting as a principal gives you the same recognitions that a non-licensed Buyer or Seller has.  He followed this opinion with the phrase, “just to be safe, you may want to contact the Division of Real Estate”.

I followed up with the Division of Real Estate’s principle lawyer.  His answer:  The law is vague, but you must always use the Utah approved REPC, regardless of your status in the transaction.  So — two lawyers that specialize in Real Estate Law, and two different results.  Can’t we just clear up the language?  I hate wasting the paper to print a REPC, only to write an addendum that says all language replaces and superceeds the REPC.  Can’t we do a bit better?

There are three basic solutions for a problem like this:

• Use a web storage and document platform, like Apple’s MobileMe or Amazon’s S3 service, or through a home-rolled storage platform running on my co-located server.
• Manually copy files from one computer to the others as required.
• Use synchronization software.

Because of the way I choose to use computers, there are additional constraints imposed.  First, I use Linux, which means the software I’m interested in should work in Linux.  Second, and even more important, my laptops only have periodic internet connectivity, though I want access to my documents and files all of the time, which means the documents must be cached locally.  Thirdly, I develop applications for many platforms like the GP2X, the iPhone, and the Nintendo DS — I’d like to build the development environments for each of these targets once, and let the synchronization software distribute the toolchains and source code as appropriate.  This requirement demands that synchronization software be able to handle file attributes (like the executable bit), and be able to work with many, many files.

Unfortunately for me, none of the software I’ve found fits these needs very well.  The primary contender’s that I’ve studied are: iFolder, PowerFolder, and Unision.

iFolder

Among each of the products that I tried, iFolder seemed like an immediate slam dunk.  Not only does iFolder support Linux, it supports Mac OS X and Windows 2000+.  It also stores files on the server with encryption (which is optionally not viewable by even the administrator), it operates in peer-to-peer or centralized server mode, it supports disconnected operations, and it’s a mature, Novell backed product, or so it would seem.  Unfortunately, there are some serious flaws which I encountered just during the setup process:

• iFolder is amazingly hard to compile, and the directions to do so are quite out-dated.
• iFolder setup is designed for an old version of SuSE.  None of the file-paths exist on most distributions, including newer versions of openSuSE.
• iFolder requires mod-mono and other odd-ball software extensions.
• Peer-to-peer client mode doesn’t actually work.
• Clients need big chunks of the server to compile and run.

After many hours of trying to overcome these issues and wiping my home file server to install openSuSE, I was able to get an iFolder client and server running, using the server package from the openSuSE build service and home-rolled clients based on the packaging of simias and ifolder-client in the Ubuntu Launchpad.  At least now I’d have a working solution, even if it meant I had to run openSuSE on my file server (it’s not all bad, just slow due to it’s debug kernel and access controls, and it’s non-uniform, which presents other problems).

It’s too bad, however, that things weren’t this easy.  iFolder did work properly and easily on my simple “school/” folder, which contains all of my recent school work.  After this folder, I tried to synchronize my “cabinet/” folder, which is my digital equivalent of a file cabinet.  iFolder got slower and used more memory, but was able to deal with this folder properly as well.

Finally, I tried to synchronize my “Projects/” directory, which contains a few gigabytes worth of toolchains and coding projects.  The result?  iFolder pushed my load average above 10, used all of my available memory and swap, and made the OOM killer go wild on my workstations.  Oy-vei!

And thus, we keep looking for tools.

Unison

Unison is cross platform just like iFolder, which is a plus but not a requirement.  Unlike iFolder, unison doesn’t have a persistent client which runs continuously watching for a server connection, or watching for changes.  Instead, you must run it manually whenever you want to synchronize, or use cron, scripting, or some other task to make it run periodically and automatically, and to make it support offline operations.  I wrote a simple shell script which does some of this:

#!/bin/bash
#
# Simple script to check for server connectivity and try and synchronize
# files with unison
# 

# The time to wait in-between synchronizations
SYNC_TIME=10m

# The unison profiles to synchronize
PROFILES=school cabinet Projects

# The server to check for connectivity.
SERVER=cessnaii.local

while [ /bin/true ]; do
      for project in $PROFILES; do
      	  ping -c 1 $SERVER >/dev/null 2>&1 && \
            	  unison $project -batch
      done
      sleep $SYNC_TIME
done

There are a few problems with this script, but it serves well with some caveats.  The problems:

• You must have pre-configured the unison profiles.  You can do this with unison-gtk.
• You should have ssh key exchange pre-configured.

The other problems are general problems with unison — you must store to a VFS file system, and there is a master copy.  (Alternatives might be storing to davfs, storing to a database, storing to sshfs, and similar.)

PowerFolder

PowerFolder is a commercial file synchronization software with an open source component branch.  Because PowerFolder is a java application, it works on any platform that supports java (in theory).  I did all of my testing with the open source version of PowerFolder, because I didn’t want to pay a fair amount of money for their commercial offering which has some limitations (to be discussed shortly):

PowerFolder worked somewhat well, but it suffered from similar problems to iFolder.  I won’t go into the details because I believe the free software alternatives are better than PowerFolder, but the overview is:

• PowerFolder didn’t properly handle Unix file attributes.
• PowerFolder’s memory usage grew and grew on big folders.
• PowerFolder didn’t support offline operation.
• Java requires a JVM, which typically requires a 32-bit machine in Intel land (though IcedTea and OpenJDK are correcting that, and GCJ is always a viable alternative.)

The Brave, New World

In the end, I’ve decided to write a better synchronizer, one that is smarter and easier than the current offerings, and one that doesn’t require 64GB of memory to operate on large folders! (I’ll consider 640k  as about the right amount of memory usage.)  I’m planning to implement this tool as a QT based systray applet, with the following properties:

• Automatically detect server presence, synchronize as required.
• Exteremely low CPU and memory usage.
• Simple GUI configuration tool.
• Allow encrypted storage.
• Use storage plugins, specifically a plugin for generic filesystems and a plugin for KIO slaves (which will automatically support DAV, SSH, NFS, CIFS, Cameras, etc).
• Portable between Windows, Linux/UNIX, and Mac OS X.
• Use Kde wallet and similar services for password storage as required.

Stay tuned for more updates!

Dr. Efros hypothesized about how these LHMs would affect the matter distribution in the Universe.  He showed how divergent electromagnetic waves (including light) could be re-focused while travelling through the LHM, which would skew the matter distribution seen in the Universe (things would appear closer than they actually are).  He also suggested that he couldn’t think of a way in which we would know that this is happening.

I believe that we could determine if this is happening, in principle, using standard candles as a yardstick to compare with galactic redshift.  Although light from celestial objects would appear refocused, they would still be subject to redshift effects.  By comparing the two, we could in principle determine if we’re peering through LHMs.