Twitter Relevancy and the Fall of Censorship

As we all now know, the aftermath of the Iranian elections is upon us, resulting in violent clashes between protestors and police due to presumed fraudulent voting practices by the incumbent political party. Yesterday, Marshall Kirkpatrick from ReadWriteWeb wrote about the absence of CNN from the coverage, and the message has resonated, as CNN has since escalated the election riots to become their top story.

Selective coverage of major international events by mainstream American media outlets has long been thought to be influenced by the Federal Government, particularly in wartime. Conspiracy theories aside, this latest development validates the power and relevancy of Twitter as a major media platform, and that thousands of tiny voices can generate enough noise to influence the media, instead of the other way around.

Twitter's relevancy has become too large to be ignored, despite recent reports that there aren't as many voices on Twitter as originally believed. Media propaganda is dead, if it even still existed, as we know that our voice can still be heard. As long as you can tweet about something and still see your content show up on an application powered by Twitter Search, you know your speech is still free.

Despite it's ability to influence major channels, Twitter has also been providing more relevant, raw and uncensored coverage of the events than traditional sources. We are all able to see the real story, often through powerful images and video, that we wouldn't see through FCC regulated outlets, and we see all of it right as it happens.

If you aren't watching this, you're missing everything. And I don't just mean with the Iran Election story, but the story of how we perceive the world. Everything is changing right before our eyes, and we're all writing the story.

Mission Accomplished

Well, my blog went stale again. I apologize to the 100 people or so out there that read it. It's funny how my mindset becomes full-speed ahead for a project or idea, and I forget to stop and think about what might happen if things go well. Well, things certainly went well with the launch of twendz.

According to Compete, we increased traffic to our domain by 469% for the month of March. I'd imagine twendz had something to do with that.

In case you were wondering, the launch of twendz was quite a success. Frighteningly so. I was asked by our Marketing Manager for some figures to baseline against, basically metrics for us to set a goal to beat. I used Happytweets as a measuring stick, as I felt like that did pretty well in terms of how well it spread virally. I figured if we could beat the growth Happytweets has had, I would be pretty satisfied. What a joke.

Hold On Tight

You've heard of the Slashdot effect, right? The idea behind the saying is that if Slashdot picks you up on your homepage, your server is basically doomed based on the amount of traffic you receive. Another common analogy is when Fark.com picks up a story, you get "farked." As in, your web server is "farked." My buddy almost got kicked out of college because he had content on school servers that got farked.

Well, we didn't get the Slashdot effect, and we didn't get farked. We got CNETted. CNET gets just slightly more traffic than Slashdot and Fark. Here is a good visual representation of how these sites stack up together in terms of traffic. Last month, they were one of the top 100 websites in the world in terms of the amount of traffic they receive. Let's just say that after the CNET story broke on their homepage, we had to act very quickly to make sure we didn't get "farked" times 10.

We survived the traffic spree, and this spawned hundreds of other stories about my little pet project that started with a weekend stuck at home in a snowstorm. That next week, I flew to Vegas for the Microsoft's MIX09 Conference where they showed my demo video of twendz for their Showoff competition (see below).

Things have been incredibly busy and exciting since launch, not only since so many people are talking about Waggener Edstrom now (we were even on the local Portland news!), but since then there have been many opportunities to continue to grow the sentiment analysis technology, and package it in a way that is going to give our clients a competitive advantage in how they measure their brand.

Okay, So Now What?

I've received a ton of feedback about twendz. Most of it has been very positive, but there have been some opportunities to improve on what we've done (translation: I'm not perfect). I'm still dreaming up some crazy ideas and working on them in some ungodly hours of my days and weekends (with Jolt cola), but this time I'm listening more. I'm trying to read between the lines of what the users want, and what kinds of things people in my organization are saying. This is leading to taking more risks, and trying things that I'm not completely certain are going to be successful, so that's all I can really say for now. Hopefully I'll be able to continue to be innovative as well as have fun creating cool new things. I owe it to myself and my peers to keep trying.

twendz Helps Make Sense of the Conversation With Real Time Sentiment Analysis

Since last December, I've been exploring possibilities of ways to make sense out of massive amounts of data, particularly conversations in social media. What started as a personal side-project in creating an automated sentiment analysis algorithm using natural language processing, and my own secret sauce, grew into a fun service prototype called Happytweets.

Happytweets was interesting, and gained some momentum, but I realized that I needed to think bigger. I needed to create something with this technology I now had that solved a real business problem. So one weekend I did, and my employer took notice. Together, we have vastly improved my intial prototype by enhancing the user experience, improving and testing the heck out of my sentiment algorithm, and today we are releasing it together as a free service offering from Waggener Edstrom Worldwide.

It's called twendz, and it helps you generalize the conversation

twendz leverages the power of Twitter search to quickly make a generalization about a conversation using real-time, automated sentiment analysis. With twendz, you can search for any topic, product, brand -- even yourself, and immediately generalize the attitudes and feelings expressed in Twitter conversations for your query. This all happens in real-time, so as the conversations changes and evolves, so does twendz. There's only been one other company that has ever done something like this in real-time effectively, and after their acquisition by Twitter last year, they seem to have lost interest (see: it's broken).

Get a quick pulse on your product

The void left by Summize is filled in a whole new way with twendz, as twendz was designed as a Twitter monitoring tool with sentiment. It changes and evolves with the conversation, showing you the most frequently discussed subtopics for your query, and analyzes the sentiment for those. If what's happening now isn't valuable or relevant for what you searched for, you can view a brief history up to this point for an added perspective.

Monitor your personal brand on Twitter

In just a minute or two, you can discover how people feel about that new pepsi logo right now, and asess the impact based on how much it has been talked about lately. Or, if you want to dive deeper and see just the negative things people are saying about you (assuming you are talked about a lot, like say, Steve Jobs), then you can filter on just negative sentiment to draw attention to only the negative tweets you need to respond to.

Get real-time news that is relevant, or just have fun

twendz pre-populates the latest top Twitter trends in the header so it's easy to jump-in to what's popular now, but it works great for discovering different perspectives on hot current events. I often find myself monitoring fun topics for my own personal amusement. Do people really like bacon as much as it seems like? Now you can find out the answer.

If you have questions or comments about twendz, I encourage you to visit our page on Get Satisfaction where I respond to most questions and comments directly, but I'm also really interested in your direct feedback, as I'm constantly looking for ways to improve the twendz as an offering, as well as the sentiment algorithm.

Patterns of Human Interaction

Ryan Sharp recently posted on his blog a wave of thought about how language is a symbol and a common currency. He states that we naturally develop categories to help us process thoughts and ideas in a more efficient way. This is much how we identify mental landmarks to determine our bearings in a familiar location, but within the context of natural language.

I started exploring this thought further in regards to a lot of research I've been doing recently with natural language processing from an automated perspective, and I started thinking of examples of mental patterns we create regarding specific words and phrases given a certain context.

For example, whenever we are ready to end a conversation on the phone, we say, "Goodbye." This holds true in any case except for when the person has made us angry, or in the movies. With other forms of medium, saying goodbye is optional. Whether that's a digital instant message, or sometimes even in person, if no one says, "Goodbye," it's irrelevant. People keep moving and no feelings are hurt.

What's strange is that we have a mental pattern; an expectation that both parties are required to say, "Goodbye," before ending a phone call, and even if a call is dropped from a mobile connection, we are still left finding ourselved feeling uncomfortable, like the conversation can't be truly ended and resolved until we at least call the person back and finalize the discussion.

These types of patterns are everywhere in our language and interaction, and eventually it may even be possible to predict the medium content is derived from by simply plotting the right data points to represent these patterns.

Happytweets Retrospective

During the Christmas holiday season, Portland found itself blanketed in one of the biggest snowstorms in the last century. Since the city doesn't generally anticipate that kind of accumulation, and that Portlanders generally don't fare well when faced with driving in such conditions, I had quite a bit of time indoors parked in front of my PC. More time than usual, anyway.

I'm not sure how I came about the idea that eventually turned into HappyTweets, but I had spent my first weekend of the snowstorm researching and applying a methodology that would eventually become the algorithm for the site. I can't say much just yet about the inner-workings of the site, as I've brought the concept to my employer and it has some potential. I anticipate to be sharing more on this in the coming months, but one small step at a time.

Word of Mouth, FTW

Happytweets took off in a way that far exceeded my expectations, especially for a quick, fun project I threw together in under ten hours. It truly speaks to the viral potential of Twitter and how all anyone really needs is an original idea, and a clever marketing angle. I don't consider myself to be all that clever, probably just lucky in this case, but I have definetely walked away with lessons learned as well as immediate action items to improve my algorithm.

The site has lost it's peak appeal, as I expected it would quickly, though it lasted much longer than I expected. It received a considerable amount of traffic in it's first month at about 708 visits per day. Traffic started slow, but really took off after I added a "Retweet This" feature, which made retweeting a user's score possible with just two clicks, and didn't require them to authenticate with Twitter. I released this shortly following the TwitterRank authentication scare, so I didn't want to send the wrong message.

Postmortem

My biggest takeaway is that I learned more about the Twitter API. Because I was relying on the account API (different from search), I was restricted to just 70 requests an hour. There were several times that the traffic load caused me to violate the request limit, which caused a negative user experience for some Tweeps. Some of them let me hear it, too.

Another is that I relied heavily on Twitter Search to get a sense of how the public was reacting to the tool. As to be expected, there were some skeptics, but in general, the reaction was pretty positive. And for the record, no, the algorithm is not based purely on emoticons. :)

The last big takeaway was that the agorithm did not perform well for users that have not posted very many tweets. I also wanted to create a graph that showed how your happiness changed over time, but found that there were not very many returning visitors, so this wouldn't be all that valuable.

Finally, I owe a huge shout-out to @semaphoria and @divamatrix for helping spread the word. The two of you truly reinforce the argument that Twitter influence can have a snowball effect.

Liquid Courage

Don't get me wrong, I'm no designer. I try, but all developers try at one point or another to design websites. Few of us try and are successful. I've heard there are even some of us that can do both really, really well. There should be a category in the Webbys for these people and their talent, because it makes others look at them with such awe in their freakish rarity.

Inevitably, even when there is a true designer presence in the mix for the countless rounds of website planning discussions, the engineer responsible for constructing the site is faced with the question of going with a liquid layout versus a fixed-width layout. It's never an easy discussion.

With the browser perspective constantly changing with new browsers being introduced and upgrades always on the horizon, good design feels like a moving target. Whether they will admit or not, designers naturally lock-in to the mindset of a fixed-width layout because it is one variable they can control. Display resolutions might vary dramatically, but as long as you account for the lowest resolution that a significant share of the global user base is using, then nobody can complain. You're meeting that 98.5% accessibility threshold.

Wisdom of Crowds

As technology rapidly improves, our expectation bar moves up as well. For a long time, the minimum resolution best-practice was to accomodate for an 800 x 600 resolution. Now, it's more like 1024 x 768, and we're beginning to see websites flirt with 1280 x 1024.

Are we really moving in the right direction, though?  Aren't we just letting design popularity dictate the presentation of the medium? Here is a really ugly graph I made in two minutes representing a study done on this topic with a small subset of popular websites.

Mobile Changes Everything

The biggest problem with the fixed-width approach is the mobile web. Sure, mobile resolutions are gaining just as rapidly as standard resolutions, but as our minimum acceptability bar increases, we continue to alienate our small screen friends. One can argue that many high-end phones today have browsers based on the mobile Webkit, but this still isn't an optimal experience. You have to admit, double-tapping just to read the content takes away from the visual experience because you can no longer see the imagery. The idea of keeping content "above the fold" is also completely lost.

As for the devices that don't have the mobile Webkit, it's a huge challenge, as their rendering engines react very differently and a browser-targeted CSS hack isn't necessarily going to fix the issue.

Websites not optimized for mobile devices also take much longer to load. Engadget's primary website took me 21 seconds to load on an iPod Touch over wifi, whereas their mobile-optimized site took less than two seconds.

Somewhat Ironic

I don't get any brownie points for doing this well myself. If you look at my portfolio, every website is fixed-width. Some may look like they are liquid layouts, but these are just design tricks using repeated images.

It would be great to see a mobile design challenge like those that were issued back when pure CSS design started to gain momentum. That, or a clever contest to see who could make the best liquid layout that is compatible with mobile devices. Maybe there could be potential for sponsorship?