Brides More Important Than Grooms - a Mathematical Proof

Every once in a while we run into an interesting statistical linguistic tidbit. Such was the case with Newsodrome's weddings news page.

It is said that weddings are for the brides. That little girls dream about weddings since kindergarten, plan them thoroughly from floral arrangement to detailed guest list and play it over and over in their heads. The only accessory missing from the plan is the groom, the least important, yet to be filled detail of the wedding... sounds a little sexist, only... that apparently, we've stumbled on a mathematical proof that this is indeed the truth!

Newsodrome has a lot to do with language analysis. When we start a news topic, such as Newsodrome's "Wedding related news", we create a wedding-corpus and break it down to its basic components - phrases and their meaning.

So... what are the most significant words appearing on wedding related blogs? other than "wedding" itself, the word "bride" is, predictably, the most significant one. What surprised us is the word "groom". Ahmmm... we looked and looked but couldn't find it in our list of significant words!

We passed words such as "dress", "cake", "reception" and "planning". We sifted through other words such as "guests", "gown", "ceremony" and "bridesmaids", but it still wasn't there. We scrolled down past "hair", "flowers", "invitations" and "shoes". Still not there. Scroll down quite a bit further, and.. there is was! "groom", right between "hotel" and "chocolate". You see, "bride" was the number #1 most indicative word for articles discussing weddings. "Groom"? about the 100th!

So here you have it. A statistical proof that Grooms are simply not that important for weddings :)

Don't forget to check out our weddings news page. On the way, check out our fashion, shopping and celebrity news!

Newsodrome as a Blog Directory

Newsodrome's news service attempts to bring the most interesting, up-to-date news on a huge variery of topics, from Fashion to Cricket. The news comprising each topical newspage is collected from hundreds of online news sources. Blogs, primarily.

Today, we added a new feature to Newsodrome - a view into the blog directory Newsodrome uses as its news sources, the building blocks of Newsodrome's newspages.

Check out our Social Media Blog Directory as an example, or go to your favorite topic and check if your blog is already listed. It's not there? Just click the "Suggest a Blog" link and tell us about it. We'll have it up in no-time!

New on Newsodrome - Suggest a Blog!

Newsodrome's topic news pages are created by an automatic editor.
This automatic editor goes over all the blogs it recognizes in a specific topic (yoga, physics, social media, ...), reads the latest posts, classifies and ranks them. The top ranked posts are shown on the respective news page.
Until now, the blogs for each topic were gathered internally.

Today, we added a new feature to Newsodrome, allowing you to suggest a new blog for the automatic editor to consider for each topic newspage!



For now, we manually process these suggestions, but we promise to be swift :)
So go to newsodrome.com and add your blog!

- Tal

Bring Back the Wisdom of Crowds to Social News!

...or "Why Social Media and Wisdom of Crowds Don't Mix"

Prelude

I'm taking a detour from Newsodrome and niche news to discuss social media... -
A couple of months ago I was watching a short lecture by James Surowiecki, the man who coined the term "Wisdom of Crowds". Two concepts caught my attention: "Under the right conditions, groups can be remarkably intelligent" and "Groups are only smart when the people in them are as independent as possible.".
I heard a coin drop... what I realized is that "Social voting" and "Wisdom of Crowds" just don't naturally mix!
In a way, it makes perfect sense. If we all vote as a group, we actually vote as a single person and the wisdom is gone with the crowd.



How to Become a Social News Power User

Yesterday, I ran across a "how to become a digg power user" post, the latest in a repeating theme of stories about how to become a digg/StumbleUpon/reddit/Mixx/propeller power user.
They all have the same basic idea: add as many friends as you can, vote for everything your friends are submitting and IM your friends to vote for your own submitted stories.
But what I now realized, is that following the "power user" advices can ruin the best aspects of the wisdom of crowds!

Deterioration of Quality?

There is a lot of talk about the deterioration of the quality of digg's front page stories and I can attest to it in a way. While I read digg daily, I was much happier with its quality at the very early days of digg when it was a small technology news-site, fighting head-to-head with Slashdot (I confess that I might be just a little nostalgic here...). A possible reason for this decline is that as digg got larger and as more social engagement tools were added, the mediocrity of the groups overpowered the wisdom of crowds.
The elements required for a crowd to be wise include diversity of opinion, independence and decentralization. Factors that can break the wisdom of the crowd include being too homogeneous, too centralized and too imitative... Surely we have a problem here.
Kevin Rose and the rest of the digg folks are trying to solve this problem by penalizing group votes. However, this hasn't significantly changed the top diggers chart as it was before the changes.

Enough With the Rant and Off to a Solution (?)

Our principles of a social news website that revives the "Wisdom of Crowds" -

1. Social and voting aspects of the news service are decoupled.


2. Users can vote only on a selected portion of the submitted stories, selected by the service.


3. Submission and voting is anonymous.

Here is our suggested solution:
Everyone can submit new stories BUT you can't vote on just any story, but rather on a subset of the stories, randomly chosen by the system.
Let's look at a scenario: Blonde58 wants a story to reach the front page. She sends an IM to all her friends, but when they reach the upcoming stories section, the only place where you can vote, they only have a slim chance of seeing blonde58's particular story and vote for it.
Add some standard anti-group-voting algorithms and this may just be the solution we've been looking for.

What Now?

We described what we believe is a service that can bring the wisdom of crowds back into social news.
Improving the social news is quite a divergence from "Newsodrome", our niche news website and our main focus (tune in next week to read why social news and niche news don't mix...). But for the interest of the community, we are more than willing to make this idea a reality and test it out in the wild. Please follow this link [Updated: voting ended] if you are interested and want us to implement this solution.

Love to hear your thoughts.

@itail

How many blogs should we read?

Introduction

How many blogs should we read? Seems trivial - as long as we find it interesting and care to afford the time, we might as well read as many blogs as we can, right? Not quite. For most of us, there are far more interesting blog posts out there than we have time to read.
So, to rephrase the question - if I want to get a good coverage for a topic I care about - how many blogs should I read?

Background

According to Technorati there are ~4600 blogs discussing knitting. Many of them are interesting enough to attract a large group of readers. Personally, knitting is not one of my topics of interest. But between social media, scifi, software development, parenting, enterprenourship, program analysis, taichi, ..., my choice of blogs is quite staggering.

I know I'll have a great time reading the latest in these topics, I know that there is excellent content in the blogosphere about them and I know how to find the most popular blogs. The question remains - how many blogs should I read in each topic to get a good coverage of the top stories? Will picking just the top blog in each topic be sufficient? How much of the top content will I miss this way?

Research

Given my academic background, I want solid numbers to guide my decision ;-)

I looked for statistical evidence on the number of interesting blog posts in a specific topic. I found two great resources - Techmeme and Digg.

Techmeme employs AI algorithms to find the best technology news and is considered by many in the field as the definitive resource for breaking tech news.

Digg is a social news website that relies on its readers and the "Wisdom of the Crowd" to find the the best stories in a variaty of topics. With these two resources, all that we need is to check the distribution of blogs that made the front page - will we find a single dominant blog?
How many blogs are needed to get an 80% coverage of the top stories in my areas of interest according to Digg and Techmeme?

Results

The results were quite surprising, both by their conclusiveness and their implications.

As you can see below, the top technology blog, TechCrunch is responsible for less than 6% of the headlines on Techmeme! that's quote a low over-all tech coverage... still, maybe TechCrunch cites the rest of the 94% of the interesting stories? looking further into the data, I found that TechCrunch appears in the discussion threads of about 6% of the stories. These means that according the Techmeme, if I read only the top technological blog, TechCrunch, I would miss about 88% of the most interesting tech stories! Not a very good coverage.

In fact, looking at the chart below, I'll need to read more than 100 blogs (!) to get coverage of 80% of the top technology stories. That means sifting through 20,000 blog posts per month(!!).

Lucky for us tech guys that Techmeme is around to bring some order into this torrent of information...

Is this fragmentation a problem of the technological blogs or is it common to other topics as well?
I turned to digg and checked the top "Gaming", "Science" and "Health" stories in the last month (unfortunatly digg doesn't have much coverage for my more niche topics of interest).
The results, below, were amazingly consistent with the Techmeme numbers.


Conclusions and Final Thoughts

How many blogs should we read in each of our areas of interest? One is certainly not enough! Fifty seems to be the magic number, but is quite absurd.
So how can we reach the stories that are most interesting to us? Techmeme and Digg do a great job but only for a few selected topics.
Hopefully, in time, we will be able to tackle this problem with our own news service - Newsodrome.

How many blogs do you read on each of your areas of interest? what other resources do you follow for the best relevant stories? do you feel it's a problem worth solving?

What is Newsodrome?

In my previous post, I wrote about why we started Newsodrome. In this post, I wanted to tell you what Newsodrome can do for you and share some of our excitement.

Newsodrome shows you the most interesting news in your areas of interest. News that matter to you. And it will do that in your familiar, uncluttered newspaper format.
Interested in Cars? Fashion? Video games in general or specifically in Wii? is you favorite sport Baseball or is it Yoga? want the latest in Python development or the latest buzz about Knitting? you can get it all at Newsodrome, up-to-date and from the best online sources.

Powering Newsodrome is a computerized editor. An artificial intelligence that sifts through huge amount of content, classifies it and ranks it. But the automaton's work doesn't end there... behind Newsodrome is our vision of an AI that evolves to offer you better, more interesting news over time. After showing you your newspaper, our computerized editor continues to listen to your feedbacks - articles you liked and disliked, topics you read more than others, news sources that attracted your attention - and it evolves. It learns from your experience and from the experiences of others and over time it will pick on subtle hints that even you might have missed - a writer you tend to read more than others, words and expressions that put you off, specialized topics within your topics of interest that better capture your attention. The end result? your personalized newspaper with the best news the Internet has to offer on your areas of interest.

But... Newsodrome is at a very early stage. We chose to start small - expose only a few selected news topics with a basic AI, keeping things to a minimum and build it from there, basing our expansion decisions on your comments and feedback.

So - stop Google-ing for news. Forget about registering to dozens of blogs. Leave the days of sifting through tons of articles just to find the one article that is of interest to you. Come visit us at Newsodrome instead!

Waiting for your thoughts and ideas.

Welcome to Newsodrome!

News.

For a long time it's been my habit to keep up-to-date with the news. Not the generic top-of-the-world, one-size-fits-all news you find in the large online newspapers, but the news that really matter to me.

At first it was only work-related news. I read news about my company and our product, the latest on my line of work, on people management, on the technological platform that we use, news about my CEO and so much more. Lately, in an "A-HA" moment, I started applying the same news gathering processes on my hobbies, my favorite musicians, authors, celebrities. Joy.

It's really great to see that more and more people contribute their opinions online. They put them in blogs, newsgroups and dedicated websites around the web for everyone to read. Some of these guys truly know what they're talking about and are also a pleasure to read! This means that if you have the time, and know how, you can find top-notch news on any topic on the Internet. Really everything.

I find reading these news both important and fun, even though my news gathering process is quite time consuming and the quality-to-noise ratio online can sometimes be abysmal, requiring a lot of sifting to find the really good stuff.

This also been a constant itch. With so many micro areas-of-interest each and every one of us has, and with so many potential news sources online, how does one find the most interesting articles to fit her unique taste?

When I discuss this issue with colleagues I find one of three common responses.

1. The "Unaware" - he doesn't realize that you can find great news on his specific, personal areas of interest. He settles instead on reading just the really generic news you find in the large online newspapers.
2. The "Casual Reader" - She visits a forum, blog or online magazine discussing her favorite topics, but misses out on interesting stories that appear in other online news-sources. She knows about different websites that discuss her other areas of interest, but can't find the time for the "footwork" required to visit all these different websites.
3. The "Newsoholic" - He reads dozens of blogs, several on each topic of interest. He gets good coverage, but constantly complains that he invests most of his reading-time sifting through less interesting stories in search of the really interesting stuff. When he returns from the weekend, he gets depressed by the amount of news that piles up in his news reader. (Myself, I am a "Newsoholic", but I preach on the amazing world of online journalism to the "Unaware", e.g. most of my acquaintances.)
   

So what might be the solution to these issues? how can the "Unaware" become aware of great pin-pointed content without being too tech-savvy? how can the "Casual reader" get a better coverage without much footwork? how can the "Newsoholic" get the same coverage with less clutter, focusing only on the most interesting news?

I believe we just might have an answer. We call it "Newsodrome" and I'll tell you more about it in our next blog post.