As a proof of concept, we built the logcollector application that combines the Logs Reader API and Fantasm to send email alerts in the event of user specified log level (eg. CRITICAL) messages.

The sample code can easily be extended for your own purposes.

Downloading and Installing

Python code for logcollector can be downloaded from Fantasm’s SVN Google code repository:

> svn export https://fantasm.googlecode.com/svn/trunk fantasm
> cd fantasm/application/logcollector

To use logcollector, simply deploy the application as a separate version into your App Engine application.

> head -n 2 app.yaml
application: myapp
version: logcollector
> appcfg.py update .

Running

logcollector can be used to monitor the logs of any deployed version by hitting the following URL:

http://logcollector.myapp.appspot.com/logcollector/

…and filling out the form shown in Figure 1. Version must be a valid deployed version of myapp, Sender must be a valid admin email address, and To can be any email address you want to send the alert email to. Choosing a Minimum Log Level that is too low may result in a very full inbox!

Figure 1. Interface to start a LogCollector.

Currently the Log Reader API can only query logs from one version at a time. You can monitor several versions simultaneously by submitting the form for each version.

Over on his personal blog, VendAsta CEO Brendan King has just concluded a one-year experiment in non-blogging. He spent 2011 learning the ropes on Twitter, and after twelve months seems to have concluded that pith, concision, and haphazard abbreviation aren’t really his style. (Mine either.) Brendan told me he’s going to try and get back to blogging regularly in 2012. He even tasked Liz with updating his blog banner for the first time since the MyFrontSteps days.

What's MyFrontSteps again?

Meanwhile, enjoy Brendan’s recap of VendAsta’s most successful year yet.

Job openings!

I’ve been with this company almost from the start, and as far as I can recall, our website has never advertised fewer than two job openings. We’re always looking for qualified people. Lack of man- and womanpower is our main obstacle to unfettered growth. We’re forever just a couple programmers away from world conquest.

Over the holidays I helped Allan update the Careers page. We’ve got five positions open right now. I know you’re out there somewhere…

• A Front-End Developer who knows Ajax, JavaScript, and CSS, and has experience in interface design.
• A Software Engineer in Testing with excellent coding skills and a familiarity with internet technologies, who’ll help us improve the performance of our products.
• A Web Designer who combines Photoshop and Illustrator expertise with an understanding of web usability principles.
• An Account Manager with marketing and client-relations acumen to help keep our partners happy.
• And – as always – qualified Software Developers.

You can apply for all the above jobs on the VendAsta Careers page.

As a VendAsta Platform Partner, AD Ideas will provide our software to their client list of independent directory publishers, who will in turn offer monthly subscriptions to their advertisers.

AD Ideas president and founder Debbie Johnson says the partnership will put small directory publishers on an equal footing with their larger competitors: “There’s a race on right now in the yellow pages industry to lock down a Reputation Management supplier. Independent YPs are at a disadvantage because they don’t have the bulk buying power that the big publishers have. With this agreement, independent publishers and local utility companies can offer a nationally recognized Reputation Management solution at a price that’s competitive with the big companies.”

On our side, VendAsta co-founder Ches Hagen sees the partnership as a blueprint for future growth in the reputation management industry: “We’re in the rapid expansion phase right now – we’re contacted by another big publishing chain every couple weeks. Every media company in North America wants a reputation management option for their advertisers. This deal with AD Ideas demonstrates that you don’t have to be in thirty cities to compete in the rep man space. There are opportunities for players in small markets too.”

At VendAsta we needed to migrate a number of applications from the Master/Slave datastore to the High-Replication datastore. This process is straightforward, and Google provides tools to facilitate it.  However, there is no Google tool to migrate blobstore data between applications.

This posting describes the approach developed and used by VendAsta.

Overview

The approach developed at VendAsta uses the open-source Fantasm workflow engine to run state machines (App Engine Tasks) on the source (Master/Slave) and target (High-Replication) applications that pass blobstore data to each other via the urlfetch API.

Figure 1 shows an overview of the messages passed during the operation of the blobstore migration.

A. The process is started by calling /start_migration/ on the source application.

B. /start_migration/ starts a named state machine Send to continue (in parallel) over source blobs on the source application.

C. The Send machine calls /start_pull/ on the target application.

D. /start_pull/ starts a named state machine Pull on the target application to perform the copy.

E. The Pull machine fetches blob data from the source application via the /pull_endpoint/ and saves it on the target application.

Figure 1: Messages passed during blobstore migration.

Named Tasks are used throughout to ensure idempotency. For instance, a blobstore migration can only be started once, and a specific blob will only be copied across a single time.

Downloading

Python code for this migration tool can be downloaded from Fantasm’s SVN Google code repository:

> svn export https://fantasm.googlecode.com/svn/trunk fantasm
> cd fantasm/applications/blobstore_migration

Installing

These instructions assume you want to copy blobstore data from the Master/Slave application sourceapp.appspot.com to the HRD application targetapp-hrd.appspot.com.

Deploy the source code into both the source and target applications by editing app.yaml and running appcfg.py update for both.

> head -n 2 app.yaml
application: sourceapp
version: blobstore-migration
> appcfg.py update .

> head -n 2 app.yaml
application: targetapp-hrd
version: blobstore-migration
> appcfg.py update .

Running

The blobstore migration code will be available as a separate version (on its own unique URL) on each App Engine application. You can kick-off a migration by calling the following URL on the source application:

http://blobstore-migration.sourceapp.appspot.com/blobstore_migration/start_migration/

Figure 2: User interface to start a blobstore migration.

1. Edit the “FIXME” in the Target URL to point to your targetapp-hrd application.
2. If you need to re-start a migration for any reason, supply a new Task Name.
3. Click “Submit”.

You will receive a “Success!!” message if the migration was successfully started.

Migrating Data Models

The Pull state machine provides a hook method migrate() that runs on the target application allowing modifications to data models that reference the source blob keys.

def migrate(self, sourceBlobKey, targetBlobKey):
     """ Contains user code to migrate data

     @param sourceBlobKey: a string blob key in the source app
     @param targetBlobKey: a string blob key in the target app
     """
     pass

This method is available in the file blobstore_migration_fsm.py. You can add any user code you would like. If this code results in a recoverable error (e.g. DeadlineExceeded) then the infrastructure will ensure that subsequent retries of the Pull machine will pass the same blob keys to the method, ensuring you can upgrade your references in an idempotent manner.

At VendAsta we chose to not use this hook, and simply refactored our BlobInfo querying code as described in the following section.

Querying for blobstore.BlobInfo Instances

VendAsta’s customers have existing URLs that reference blob keys on source (Master/Slave) applications. These URLs still need to be supported, and not return error response codes. This can be accomplished with a minor bit of refactoring of blobstore.BlobInfo querying code.

This refactoring is facilitated by the BlobKeyMap models that get created on the target application when running the blobstore migration application. You must copy this db.Model definition into your own source code in order to use it. These models have key names identical to the values in the unindexed property sourceBlobKey.

class BlobKeyMap( db.Model ):
  """ Maps Master/Slave blobs to High-Replication blobs """
  sourceBlobKey = db.StringProperty(required=True, indexed=False)
  targetBlobKey = db.StringProperty(required=True)

So, instead of looking up a single (non-list) BlobInfo like this:

blobInfo = blobstore.BlobInfo.get(blobKey)

…use something like the following instead:

def getBlobInfo(blobKey):
  # first lookup the blob in the target application
  blobInfo = blobstore.BlobInfo.get(blobKey)
  if not blobInfo:
    # if not found, look for the key in the source application
    blobKeyMap = BlobKeyMap.get_by_key_name(str(blobKey))
    if blobKeyMap:
      blobKey = blobKeyMap.targetBlobKey
      # finally, lookup the source blob by target key
      blobInfo = blobstore.BlobInfo.get(blobKey)
  return blobInfo

blobInfo = getBlobInfo(blobKey)

You can do similar things with lists of blob keys.

Existing migration examples enforced the use of blobstore.BlobReferenceProperty to automatically migrate references. VendAsta’s data model included some stringified/serialized blob keys so it was not practical to convert the data model prior to migration.

Conclusion

VendAsta has written a self-contained application that you can use to copy blobstore data between App Engine applications. It is straight-forward to deploy this code as a separate version on your applications. It provides a robust and fault-tolerant mechanism to copy blobstore data between App Engine applications.

–

Related: In September, VendAsta’s Jason A. Collins wrote about using Fantasm for incremental data backups on Google App Engine.

There’s a reason we’ve chosen to build these particular products. They go together – not just in the sense that they all deal directly or indirectly with reputation, but in the sense that their functions are closely integrated:

• Our solution for businesses with multiple locations, National Brand Reports, organizes and parses data gathered by Reputation Management.
• Our Mobile-Optimized Microsites help businesses publish and promote positive content, including content found through, you guessed it, Reputation Management.
• And our as-yet-unreleased Assisted Reputation Management service, as you might have deduced from the name, will augment the functionality of our Reputation Management solution.

Each of these products stands on its own, and we’re happy whenever a partner chooses to offer any one of them. But obviously we’re happier when they’re all offered together. To make that option more attractive, starting this week we’re integrating the admin panels for all the above products into a single interface known as Partner Central.

Henceforth, when our partners log in to administer their Reputation Management, Microsites, and National Brands accounts, they’ll be redirected to Partner Central^*. Their login credentials will remain the same, and the interface they see will be identical to what they’re used to. But along the top of the screen they’ll see a menu of all VendAsta’s products – including the ones they don’t yet subscribe to.

The most deviously brilliant part is, these products will be enabled by default, so that our one-product partners can try our other offerings for free without having to sign up. Want to see how VendAsta can help your SMB customers get on the mobile web? Now there’s nothing stopping you from testing out our Microsites platform.

You can get to Partner Central from the login link on the top right corner of vendasta.com.

–

* Some of our partners have their own user interfaces which integrate VendAsta data via our APIs. These partners will be unaffected by the switchover to Partner Central.

(Note the many black VendAsta t-shirts in attendance.)

This past Saturday, the fourth floor of the Avenue Building was unusually crowded with smart people.

Why the mob? Because VendAsta stepped up to host BarCamp 2011, Saskatoon’s premier celebration of all things smart and geeky.

Last year, when BarCamp was at Louis’ Pub, we nonplussed attendees with our generous offer of free chicken strips. This year the only way we could outdo ourselves was by holding BarCamp right here in our office, where we could put up as many confusing posters as we wanted.

I should explain to our non-geek readers that BarCamp is the annual technology-themed social gathering that originated in Palo Alto in 2005 and has since spread around the world. BarCamp (the name references an obscure Silicon Valley in-joke) is an informal, semi-unstructured event combining the best aspects of computer day camp (jargon-filled presentations, whiteboard diagrams, geek-on-geek bonding) with the best aspects of a bar (proximity to alcohol). Participants are encouraged to speak on any aspect of technology and a schedule is put together on the fly.

The schedule (subject to revision).

Although VendAsta supplied a sizable contingent as usual, only two of us were brave enough to give a presentation this year. In his charming German-accented English, Andreas discussed Fighting Fragmentation With Fragmentation: Optimizing Android apps for Honeycomb and beyond. And in his Saskatchewan-accented English (no less charming in its way), Jeff Tomlin discussed the lessons learned from our company’s rapid growth over the last couple years. Besides Andreas and Jeff, many VendAsta associates, alumni, and long-lost friends spoke, like this guy, this guy, and this guy…

I wasn’t there – I was at home frantically raking the yard in preparation for Sunday’s snowfall – but by all reports, events went swimmingly, due in no small part to Allan’s organizational efforts. There were fears of catastrophe early on when the keg resisted all attempts to coax beer out of it, but a solution was improvised, and the 150 or so thirsty attendees relaxed into a day of drinking, scarfing pizza, socializing, and learnin’ stuff.

So props to @allanwolinski, to @gingerk and her team of BarCamp volunteers, to @RhodesNathan who got the sound system running, to @blairkelsie who helped out behind the bar, and to our graphic design team of Liz and Marie-Louise, who in a week created all our signs and posters and this cool whiteboard illustration in the lounge.

VendAsta past, present, and future (illustration by Liz Syrnick).

Let’s do it again next year!

Last month our partners at LocalBizNOW rolled out VendAsta’s newest product to one of their biggest clients – a major brand with over 1400 locations in the United States (whom I unfortunately can’t name).

This lucky client is getting a first look at our National Brand Reports, which aggregate reputation data from locations nationwide into a single report that can be broken down by region, city, and individual branches.

This is our venture into brand-level reputation management. Heretofore our audience has been local small and medium businesses, which means our data is organized around locality. As Brendan likes to say, there’s a difference between what people think about Nike and what they think about the Nike Store in Santa Clara, California.

Our local focus gives us an edge that the competition can’t offer. Other rep man products monitor conversations about the brand: What are people tweeting about Target today? You glean some important insights this way, but their usefulness is usually restricted to the folks in the marketing department down at corporate HQ.

By contrast, National Brand Reports offer instantly useful feedback on what’s working and what’s failing in the day-to-day operations of the branches. This is done by collecting online reviews, mentions, and social engagement from all the locations into a single report, which can then be zoomed in to examine data of increasing granularity.

Are your midwestern branches consistently trailing the nation in social audience? Why is Metro Detroit bucking the trend? Look at this franchise in Dearborn – it’s bringing up the average for the whole metro area. What are they doing right?

It’s a whole new way of monitoring conversations about your brand, and we’re betting there’s a great future in it.

Take a look…

We'll start with the report zoomed out to the national level. By clicking on a region...

...We zoom into the regional level. Now we click a location...

...To see the visibility results for just one location.

L-R: Darren, Guy, Emil, and Michael reflecting after a hard day's work in the mine.

This carefully-lit photo was taken a kilometre underground in the potash mine at Allan, Saskatchewan, this past Friday. Guy and I are with Emil and Darren, safety trainers at Allan, whom we helped to film two new instructional videos for their online training system.

The training software we originally developed for PotashCorp’s Patience Lake mine is now being rolled out to Allan, and part of our job is to update their 15-year-old lockout procedures^* instructional video and create a second video for sounding & scaling^† techniques.

So Guy and I spent the day tagging along while Emil, Darren, and their star Lonny locked and unlocked various electrical panels, tapped the roof with a scaling bar, and in between, drove us considerable distances through unlit subterranean tunnels. At one point in our travels we were directly underneath the village of Elstow, which is 7 km north of the entrance to the mine – and the tunnels extended well north from there. We must have done about 20 km of underground travelling by the end of the day.

With our province’s economy being powered in large part by the potash industry, it’s instructive to witness the process of actually scraping the stuff out of the ground. I’m happy to report that things seem to be running pretty smoothly down there.

–

* Lockout is the procedure by which electrical equipment is secured so workers can perform maintenance on it – essentially, each worker puts a lock on the electrical panel so it can’t be started accidentally.

† Sounding & scaling are done to ensure that the roof and walls of a work area are solid. Sounding involves tapping the roof with a long aluminum bar, listening for hollow spots, and scaling means prying down loose bits of rock that might otherwise fall and injure someone.

We’ve moved to the heart of downtown Saskatoon, to the historic Avenue Building, which dates back to 1912. There’s a sandwich shop, a men’s clothing store, and an old-timey barbershop (currently unoccupied, sadly) on the ground level. The elevator pipes in Doris Day and Ella Fitzgerald tunes while you ride up to our office on the top floor. The only thing missing is a guy in white gloves handing out towels in the washroom.

So farewell, old VendAsta office! Farewell floorboards held in place with duct tape! Farewell mysterious thermostats that controlled the temperature in anyone’s office but your own! Farewell boardroom carpet spattered with three years of food stains from Friday lunches! If you’re looking for us, you can find us here:

There’s a ton of photos of our new office (mostly courtesy of Allan) on the VendAsta Facebook page.

Mobile Business Pages are specifically designed to display key business information on mobile devices, giving consumers the mobile-friendly content they’re looking for, and giving businesses the opportunity to target and reach mobile consumers. These smartphone-friendly “microsites” showcase business logos, email addresses, products and services, brands, menus, hours of operation, methods of payment, high definition videos, photos and online coupons.

“Changes in technology are impacting the way consumers and businesses connect,” said Georgina Syrgiannis, Vice-President of Corporate Marketing at DirectWest. “Whether they’re connecting through print, online or mobile, we’re constantly innovating to provide consumers with rich content and businesses with creative advertising solutions.”

With Google indicating that roughly one in seven web searches are now conducted from smartphones, the importance of mobile users is becoming ever more apparent. “People get frustrated trying to navigate sites that were designed for a 17-inch monitor on a screen that’s one-tenth that size,” said Syrgiannis. “But most companies still don’t have mobile-optimized websites. We’re making it easy for Saskatchewan advertisers to get ahead of the curve.”

Mobile Business Pages will be linked from DirectWest’s online search portal mysask411.com, and will also appear prominently in searches on Google and other search engines. VendAsta CEO Brendan King explained that the pages were designed to be easy for search engines to discover and index. “There are simple things you can do to improve a website’s Google ranking. Unfortunately, small business owners often don’t have the technical know-how to do make sure those things get done. So we’re handling that stuff for them.”

Syrgiannis said that ease-of-use was the main priority in developing Mobile Business Pages. “We’ve found the ideal spot between making the sites customizable and making them simple to set up. Advertisers will choose from a few basic style and colour options and we’ll do the rest. Simple. Easy.”

About DirectWest

DirectWest, a wholly-owned subsidiary of SaskTel, is the exclusive provider of the print, online and mobile SaskTel Phonebooks. For over 50 years, DirectWest has been bringing buyers and sellers together with the most complete, used and preferred phonebooks in Saskatchewan. Fueled by the success of the SaskTel Phonebook in print, the power of the online phonebook at www.mysask411.com and the popularity of Saskatchewan’s leading internet portal, DirectWest has recently launched their very own blackberry app, iPhone app, desktop gadget for Windows and desktop widget for Macs.

About VendAsta

VendAsta Technologies Inc. is a Saskatoon based company that builds white label software platforms for publishers. VendAsta’s offerings include platforms for reputation management and group buying. The company was founded in January of 2008 and employs a team of 30 software professionals.

-30-

For more information, please contact:

Michelle Englot, SaskTel Corporate Communications
Phone: (306) 777-4476
E-mail: michelle.englot@sasktel.sk.ca