Atomic Data makes search engine dominance possible

This is the first installment in a series that blends website architecture, data structures, and SEO marketing into a collaborative design pattern.

Designing a product catalog is one of those "better get it right" projects that any e-commerce firm faces.  When you discuss lifespans of projects, this one has the longest lifespan of them all.  Since I've been through this a couple of times, I thought I would share my thoughts and designs as I delve into yet another one.

There are a lot of political and technical pressures put on a product catalog from many departments within an organization including IT, Marketing, Executive, Operations, and particularly the "Industry Expert" within any company.  It is important to not only recognize them, but to appreciate them.  At the end of the day, almost everyone is "right" in their desires to have the catalog data serve them in a certain way.  As you put yourself in their shoes by doing a proper discovery before you start designing you should try to not only understand what they want, but why they want it.

Atomic Data

Your marketing team will call this "flexibile product information", your IT team may call this "dynamic product data", but at the end of the day, it's product data that is smashed into all of its discrete component pieces.

This is one of the first pressures that will be placed on you and you need to be prepared to deal with it properly.  It is important to understand that there is a competing struggle in any database design... Flexible vs. Fast.  If you think of a product as a construction made from legos, then the properties of those products are the individual lego pieces.  The concept of "atomicity" means that you can assemble your lego construction with Red, Blue and Green legos to make a space ship... and then you can rearrange those same Red, Blue and Green legos and build a house.

Now you've all seen the non-atomic way of building a product.  It's a row in a product table and it tends to look like this:

You are limited however when you decide to stock a product that has a "Sub Sub Type", or a product that only has one color, or a product that has two vendor brands on it.

You also have a design flaw where you are "numbering instances" of properties.  In this case "Color1" and "Color2" are going to cause problems for you when you want to search by "Color".

There is also a failure to properly "atomize" the data with things like "SubDept" being equal to "Ladies Apparel".

Let's compare this model to one that is fully "fourth normal" or highly "atomic".

Lets analyze this model.  The product is statically registered in a much abbreviated product table.  It serves now primarily as a hook that you can hang things from.  We've decided to establish all of our atomic types as "Type", "Gender", "Vendor", "Brand", and "Color".  You can see how this can be reused.  For the "Live Strong Velocity Ladies Sport Top" it makes sense that Color (to this product) "means" White and Yellow... but to other products the same property of "Color" could "mean" other colors.

You can also see the intrinsic hierarchy here that establishes "Apparel" as a "top category" over "Top" and likewise, "Top" as a parent category over "Tank Top".  This enables you to still utilize hierarchies in your product data representations while granting you also the ability to search ad-hoc through your product data in a non hierarchical manner by using the raw properties.

 I have taken an apparel data model and created a good sample of how the property to product mappings for a decent catalog could be structured:

This model describes the relationship between products and properties but also illustrates some of the intrinsic relationships between the properties themselves.  For example, if you mapped a City to a product, you could "infer" what State and Country relationship existed by recursing through the Property-to-Property relationships.

So... which data model is right?  The answer could likely be ... Both!  It really depends on your requirements which we will discuss in Part 2 - Best Business Practices for Product Catalog Data Structures - Speed versus Flexibility.

This is the fourth installment in a series that blends website architecture, data structures, and SEO marketing into a collaborative design pattern continuing from Part 3 - Best Business Practices for Product Catalog Data Structures - Customer Paths.

It may seem counterintuitive to discuss search engine optimization (SEO) techniques in the midst of a conversation about data structures, architecture diagrams and in-store plan-o-grams, but it can directly relate to your choice of data models.  As we discussed in the previous article, it is important to structure your website to conform with the needs of entering customers in a way that segments them properly so they find the things that they were searching for.  Part of this is anticipating what a customer is going to want before they enter your store. 

When dealing with search engines, there are two customers to contend with... the "Natural" search engine... and the "Paid" search engine.  These two customers are very important to understand and to distinguish and need to be treated with a deference and distinction from the "real" customers that frequent your online store.  The complexity arises to some degree because these two "customers" happen to be "ghost shoppers".  You never know when they are going to arrive and they generally float through your store much like a customer would, but they are searching for every product on every shelf in every aisle and in every department... all at the same time.  The complications continue because you want to manage what the ghost shoppers can and cannot see so they don't memorize portions of the store that you don't want reported on the search engines.  This may come across as elemental theory to an SEO expert, but in the context of blending SEO concepts, architecture and data structure modeling, it illustrates one aspect of the equation.

Imagine now that you are a search engine, whose job is to find, identify and classify billions of e-commerce pages throughout the Internet with the primary objective of finding pages that are considered "relevant."  I quote the term "relevant" because what that precisely means changes with the breeze and the whim of arcane departments of voodoo at the various search engine optimization firms.  With that said, you want to look at a natural search engine as a stream of water pouring into your website.  This stream is going to remember whatever it touches, so you want to ensure that it finds the things that you want it to see.  You also need to consider the diffusion of the stream of water as well.  Don't let the natural search engine stumble across pages like "Privacy Policy" or "Terms & Conditions" as that won't deliver any tangible benefit for you.  In similar fashion, on your landing pages you should try to structure your site so the links that are the most compelling draws for the majority of natural searching customers should be setup to receive the largest stream of natural search "attention." 

You also need to anticipate every possible combination of keywords that would be used to "land" on any given destination.  Lets take a look at the SEO Path Aliasing diagram to illustrate that:

We have already covered Customer Paths but sometimes the proper "path name" doesn't match an actual English phrase.  This means that the combinations of words that make sense for categorizing a mix of products may not make linear sense for a keyword search.  Our diagram above illustrates this with the green path of "Ladies / Nike".  There may not be many customers that would enter that phrase in a search, but it may be a logical progression as they navigate through a website.  This is where Aliased Paths come in.  In our example, the Aliased Path for "Ladies / Nike" could be "Ladies Nike Apparel"... sure this one is a bit of a stretch...  I'm not sure how many actually type in the word "apparel" but you'll need to work with me on this one.

You will note that this path is identified as "overridden".  In smaller e-Commerce websites, it may be a simple matter to manually go through each Customer Path and identify the possible Aliases but in far larger catalogs this quickly becomes a daunting task.  It doesn't mean that overridden Path Aliases aren't an important part of configuring your catalog categorization scheme, but you can, for the most part, rely on the auto-generated Path Aliases for many of the Customer Paths in your catalog.  Take the path "UCI Pro Tour / Tank Tops" which easily converts to an English text keyword search of "UCI Pro Tour Tank Tops". 

Note also our attempt to focus the "stream" of the natural search flow throughout the various Customer Paths.  Many search engines respond to a setting within the hyperlinks of a "NOFOLLOW".  This mechanism gives you some measure of control over which links you allow the natural search "probing" to find.  You will note how the various Customer Paths are identified as NOFOLLOW for those paths that we want the search engines to pass on as they traipse through our pages.  This poses another logistical issue in a large-scale e-Commerce website which we will address in the next segment, Part 5 - Best Business Practices for Product Catalog Data Structures - SEO Weighted Auto Mapping

This is the seventh installment in a series that blends website architecture, data structures, and SEO marketing into a collaborative design pattern continuing from SEO Weighted Auto Mapping - Best Practices for Web Product Catalogs - Part 6.

I initially intended to stop at Part 6 but as I continued to think about the usefulness of highly atomic product catalog data for e-Commerce catalogs, I began to think about how to use that information in more ways.  Dr. Flint McLaughlin commonly teaches a theory he calls the "Marketing Experiments Optimization Methodology":

He obviously can explain it far better than I can, but the general rule is that your first priority is to optimize your product (and it's value proposition), followed by the presentation of that product, followed by optimizing the channels through which the product is sold.  I will certainly investigate the first two, but if you focus solely on the "Channels" aspect of this, let's think about how our Product Property data can help us optimize our Marketing Channels.  I'll revisit one of my earlier diagrams that shows the various paths that can be traveled to finally land on a product.  This concept that the customer can "come from anywhere" and is going to want complete flexibility in how they choose to travel through your website is just as valid for other websites that show your products as well.

Consider a model where we have completed our website, we have optimized it for the search engines, and we are generating significant traffic.  Jared's First Law of e-Commerce states (yes, I just made that up...) that it is illegal to make money on the Internet.  The United States Treasury is the only institution entitled to "make money"... so our real objective is not to "make" money... but to "divert" money from other places so we get that money.  If you think of your e-Commerce project in that light, it refocuses some of your goals.  Now that we know that our job is to divert money that people would have otherwise spent somewhere else... what are the best tactics to accomplish this?

One of the primary mechanisms for this is called Product Feed Syndication.  Syndication reminds me of Gary Larsen penning a Far Side comic strip.  He spent his time creating a quality comic strip, but he didn't call up every newspaper and ask them individually to print them in their papers.  He used a Syndicator or an Agent whose job was to do that for him.  In similar fashion, you've created your entire e-Commerce catalog with rich product information and you've published your own website containing them... but you're not the Wall Street Journal quite yet...are you?  That means that the vast majority of eyeballs that may be interested in your products are cruising through other websites than your own and you need to divert them to you.

 Syndication Product Feeds are taps into your database that compose the products, their details, and their categorizations in a format that is consumable by the various feed agents (or syndicators).  These can take several forms which I like to classify as:

1. Comparison Search Engines
2. Online Marketplaces
3. Product Review Sites

This article will primarily address the Comparison Search Engines but some of the principles can apply to the others.

Comparison Search Engines serve an important role in helping you expand your Channel Footprint.  Basically, if you're the mom-and-pop shop on the corner, the Comparison Search Engines are the mall in downtown (some larger and more useful than others) and it's useful to have a presence in both places.

The challenge you have with Comparison Search Engines is they too require navigation paths.  They may call them "categorizations" or "paths" or "trees" but they map directly to the Paths described in the diagram above (Aliased Path Automapping) just like they do on your own website.... with the exception that their paths are going to differ from yours in many cases.  One good example is the path that many Comparison Search Engines use for Sports Apparel.  They start with "Apparel", then continue to "Sports" and then to "Cycling" and so on.  However, they also provide paths that start with "Sports", continue with "Cycling", and then "Apparel" which really contains the same product mix as the one before.  Many feeds have a very flat structure to them, meaning that for each product, you may be able to specify four separate "Paths" that will be submitted with that product to the Comparison Search Engines.

Because our Product Property data is atomic and weighted, we can simply query out the top four weighted paths from our database for each product and layer that into our feeds to the Comparison Search Engines.

We can also tightly couple our Property mappings to their Paths or Categorizations by dissecting their Paths into their component parts.  If a Comparison Search Engine has a root level of "Apparel", we can match that to our Property Type of "Apparel".  That can form the basis of our query then that only pulls the top four weighted paths for that product that originates with the Apparel Property Mapping.  If you really want to get aggressive on each Comparison Search Engine, then you could layer in a direct mapping from the Product Properties to each and every path variation provided by each Comparison Search Engine categorization.

As I have time, I'll try to revisit this article and distill some more useful information about the various Comparison Search Engines and how to feed each one.

Once you have mastered the Comparison Search Engine syndication feed, we can continue to leverage our Atomic Product Data in Online Marketplaces and Auction Sites - Best Practices for Web Product Catalogs - Part 8.