网络资源的拷贝粘贴 备份参考之用


29 December 2010

How to hide Thinking at Work so that the Non-Programmers don't suspect Slacking?

Better programmers than me can write in essays about walking around with a coffee mug and call it programming. And it's perfectly accepted at a place that knows the business. Or see what Gregory House (TV show "House M.D.") does when he is thinking.

But what about the other places where you are the only programmer?

If you don't stare at boring stuff on the monitor for 8 hours straight, co-workers suspect you being a slacker. Yes, not the managers who see the output. Only the co-workers who see the process and can't relate to this kind of work.

Yesterday I had to explain to a trainee of some other profession that software development is like flying. The explanation from the Hitchhiker's Guide to the Galaxy. I don't think she bought it.

link|flag
88  
Seriously? Why the 3 close requests? At the moment there are 9 answers, with most of them upvoted. People are interested in this question/topic. – stesch May 1 '09 at 13:39
4  
This might be better as a community wiki question? – James McMahon May 1 '09 at 15:25
17  
If it gets closed it's an automatic reopen vote from me. – cletus May 1 '09 at 23:50
5  
Note: according to Jeff this is programming related blog.stackoverflow.com/2009/04/… "What is a programming related question? Here's a solid set of guidelines generated by the SO community itself: ... Questions about social engineering, management, or career building, ergonomics, or other "soft" topics related to development work." – cletus May 2 '09 at 17:28
28  
rofl, to paraphrase the question "how do I hide thinking at work, because thinking is discouraged by my employer" – STW May 2 '09 at 19:24
show 14 more comments

locked by Michael Myers 12 hours ago

35 Answers

1 2
up vote 145 down vote accepted

It's funny you ask this question because just yesterday I had a client blow up at a colleague and I who were discussing Google Analytics and Macbooks. Now this client gets charged for our time and we both exclude such time from our timesheets but such breaks are actually important: you can mentally recharge and it can help to work through problems and so on.

Now I put people into two categories:

  • Process-oriented: these people are concerned with how you do things. Do you turn up for work on time? Is your desk neat? Have you filled out your weekly status report? And so on; and
  • Results-oriented: within reason, these people don't care how you do something, they just care that you do it.

I am firmly in the second category and I have a real problem with people in the first. In my experience, people in the first category seem to suffer from insecurity. It's a fundamental psychological principle that controlling your environment makes you happy and such people, who are faced with something where they feel out of control or don't know how to achieve the best results, resort to controlling their environment by dictating how you do things.

You're right in that this is particularly problematic for non-programmers when dealing with programmers. They often just don't understand what we do and how we do it, like how things can take longer than estimated and so on.

Yesterday's case was even worse because this same guy would sit with a colleague and discuss the house she's buying and other things in the middle of doing his own work (and he's charging his clients for his time and I'm sure he's not charging for such downtime), which I found particularly galling as a real double standard on downtime.

So how do you deal with it? Preventively. Try and associate with results-oriented people by choosing the right job, staying in a good job, leaving a bad one, hiring the right people and so on. There will be a point where you'll simply have to put up with such nitpicking however. You can try and explain how the process doesn't work where you just sit there and stuff happens in some linear fashion but in my experience more often than not non-programmers just don't get it and you're wasting your breath and it'll sound like you're just trying to make excuses for slacking off.

In my case, as a consultant, I'm simply going to finish the current project (which is soon thankfully) and then respectfully decline any future work as I've already got more in the pipeline than I can already do.

link|flag
48  
I completely agree. I used to work for one client that would complain about the amount of times I showed up an hour or two after everyone else, but wouldn't consider the fact that I leave four or five hours after everyone else and that I produced more results in less time than the permanent staff. Results-Oriented people tend to be a joy to work with, process-oriented people [in my experience] tend to nit pick and micro-manage and drive me crazy. If you want the job done exactly your way, by all means go ahead and do it, otherwise go away and let me do what you hired me for. – BenAlabaster May 1 '09 at 13:30
6  
I'm with you both. Also had a month ago a quarrel about me arriving an hour or so after everyone else. When I said I stay correspondingly longer in the evenings they told me that nobody actually sees me do that. Between the lines I got they don't trust me. Suggestions to check out Windows logs or introduce any other kind of observation were declined however. And yes, I'm also basically the only developer there. – User May 1 '09 at 15:38
18  
Fair points, but it smells of a bit of a rant to me. You also exhibit the classic programmer need to divide everything into black/white, 1/0... when talking about Process/Results oriented. If you're dealing with anyone of average or better intelligence, it often becomes very obvious to them that they've been "pigeon holed" and is a sure fire way of degrading a working relationship. In fact, the act of pigoen holing is itself a way of reducing the complexity of human interaction, ie "controlling your environment". Hmmm... Does that make you Process or Results oriented? ;) – Ash May 2 '09 at 6:36
6  
I think it's fine if results-oriented boss leans toward being process-oriented one if results are dissapointing. – Arnis L. Jul 21 '09 at 18:22
2  
while i would lean more towards the second category, i cannot see this as an either-or, mutually-exclusive grouping. yes, granting folks the autonomy and freedom they need to achieve the results is very important as it liberates people. But at the same time, caring on about results alone can lead to compromises to mandatory standards, ethics, security, or safety requirements, etc. Those items belong to the process-oriented category. And i have seen such compromises happen. There has to be a balance for these work practices. – icelava Jul 28 '09 at 6:35
show 9 more comments

One solution could be this:

During their lunch break get them to do a simple task like as a crossword. Ask them to be conscious of the amount of time they spend reading/writing vs. the amount of time they spend thinking [or looking up the answer]. Tell them that this is exactly the same as the time you spend reading/typing vs. the amount of time you spend thinking and looking up answers.

Better still Sudoku because it can involve hours of trying to figure out where you went wrong and why none of your numbers add up. So while the task itself seems relatively easy, after all, it's just numbers in boxes it can take time. Get them to predict how long it'll take them before they start and see if their prediction is right - guaranteed it won't be. You can hit them with a double whammy for why your time expectations never quite add up either.

This is the only way I can really think of to get them to understand the way engineers work. I see lots of "those types just don't get us types" - people are adaptable, people can understand things if presented in a fashion they can understand, it's just that nobody's ever taken the time to present them with an argument they get.

link|flag
23  
I really like this explanation. I'm going to use it when needed. – Sandor Davidhazi May 6 '09 at 22:35
3  
Wicked analogy! +1 – John MacIntyre May 7 '09 at 18:23
1  
That is a fantastic analogy. I'm going to have to remember that. – Colin Cochrane Jul 21 '09 at 17:41
5  
The real question should be: How should non programmers in the office hide gossipping about American Idol and other such crap so that us programmers don't suspect slacking! – BenAlabaster Aug 10 '09 at 16:33
show 6 more comments

alt text

Just let your build scripts run, see lots of command lines spewing thousands of useless trace messages, and you're all good...

link|flag
38  
Must... resist... urge... to... upvote... – Michael Myers May 1 '09 at 15:09
7  
@mmyers - don't worry, I've got ya covered +1 :P – BenAlabaster May 1 '09 at 15:15
11  
BRB- building screensaver of never-ending compiler output – Graphics Noob Aug 26 '09 at 20:53
show 7 more comments

Years ago I had a gig writing some "Business Basic" stuffs. I was teamed up with a rather "brilliant" programmer with a strange work flow..

He would sit at his computer.. for half to 3/4's of the day.. and play doom / quake / wolfenstine 3D (he was really into the FPS). when he had enough, He'd fire up the editor and just type. At the end of the day, whatever he wrote.. was gold.

We had a new account manager start with the company that walked in on him playing one morning and in dealing with her, was not very polite.

She complained to the CEO that "one of those programmers isn't doing any work at all but playing those violent video games". To which the CEO replied to her "YOU DIDN'T DISTURB HIM DID YOU!?!?".

The games helped him think. He designed, wrote, tested and debugged all of the code in his head while he was playing. Once he was done, it was just a matter of brain dumping and getting it all out.

link|flag
44  
That's ridiculous and unprofessional. – Tim May 1 '09 at 14:18
55  
Upvote not because I agree with the attitude, but just because I love the story. :-) – T.E.D. May 1 '09 at 14:19
25  
@Tim - Maybe he gets paid to produce good code (rather than to strictly 'think and code'). If he produces enough golden material, then he's doing his job. – StuffMaster May 1 '09 at 21:40
14  
Thinking isn't something you can do for eight hours straight and still be productive. Downtime is a vital part of the cognitive process -- it's when your brain gets a chance to rearrange what it already knows and spot correlations it couldn't while under load. How many of us have come up with some of our best ideas in the shower? If this programmer knows how and when to take their downtime, more power to them. Sounds like it works. – Joe White May 2 '09 at 19:38
21  
Was this programmer's surname Carmack? – Arec Barrwin Oct 16 '09 at 15:40
show 18 more comments

Why, leave something verbose running on the terminal, of course.

link|flag
33  
I've actually had my boss walk by my desk while doing some SVN dumps and he said "That's what a programmers screen is supposed to look like" It hurt a bit, but now I have an easy "I'm busy!" sign. – Wally Lawless May 1 '09 at 13:41
5  
@Power-coder: You mean a programmer's secondary screen isn't supposed to be permanently etched with a browser window displaying the StackOverflow website? – BenAlabaster May 1 '09 at 14:01
8  
One well written line of code can replace a thousand bad ones. – Sneakyness Jul 26 '09 at 16:16
show 1 more comment

Take a walk

I've found that it's more the appearance of not "working" than anything else. If I leave my desk for a few minutes and walk around outside no one thinks anything of it. Much different than sitting at your desk looking idle.

link|flag
2  
I agree. It's a perception thing. Just sitting is always lazy. But moving either fingers or legs convinces people you're /really/ working. – Matthew Flaschen May 2 '09 at 0:46
3  
Out of sight, out of mind. – Walt Gordon Jones May 2 '09 at 20:22
15  
ps walking around with a piece of paper in your hand = working in most places. – Martin Beckett Jul 8 '09 at 20:30
3  
MGB is right. I got away with wandering the halls in High School for a very large amount of time by simply walking around with paper. – Sneakyness Jul 26 '09 at 16:17
show 1 more comment

Here's a solution from commandlinefu.com:

cat /dev/urandom | hexdump -C | grep "ca fe"

It will make some data scrolling off the terminal.

alt text

link|flag
10  
That's what I appreciate, a command-line solution to any programmer's problem! – thrashr888 23 hours ago
4  
Did you mean hexdump -C < /dev/urandom | grep "ca fe" – Sam 18 hours ago
3  
Add | pv -q --rate-limit 75 for some extra slowness. – Cristian Ciupitu 18 hours ago
show 1 more comment

How about thinking on the lav (the john)?! Of course it limits you to one thought session a day unless you don't mind people thinking you have diarrhea

link|flag
4  
lol +1, I do some of my best thinking on the loo. It's the only time I'm not constantly interrupted by coworkers asking retarded questions I don't have time to answer. – BenAlabaster May 1 '09 at 13:37
10  
Works at home, too. Unless you have toddlers constantly banging on the door. – Even Mien May 1 '09 at 13:40
3  
My office is noisy. I use this technique surprisingly frequently during the daily, shouted political argument going on one aisle over. The important thing is finding the lavatory in the basement or up a floor that's less-frequented. (Fewer distractions.) – Greg D May 1 '09 at 14:21
11  
I had one place where I'd do that multiple times a day, because the handicapped stall was bigger than my cubicle. :-) – T.E.D. May 1 '09 at 14:21
7  
The paper in the lavatory isn't so good for taking notes ... and let's not talk about the pen. – JeeBee May 1 '09 at 15:30
show 6 more comments

Think on paper, i.e., make a note of your mental thoughts. I find diagrams make things a lot clearer for me when thinking anyway, so it's a win-win.

link|flag
15  
Paper is so passé, get a whiteboard... – BenAlabaster May 1 '09 at 13:52
4  
@balabaster, ii actually prefer paper for it is much more portable and scaleable (On a whiteboard you have to erase everything when it's full, on paper you just get a new paper) – Pim Jager May 1 '09 at 14:30
2  
@Pim Jager - Understandably - I think in this day and age, I think we need to come up with electronic white boards that just dump the contents to file and then sync with the computer when you connect it up. Something lightweight enough to carry to the board room if necessary. – BenAlabaster May 1 '09 at 14:39
2  
That's it, now I want one of these: e-beam.com/products/whiteboard.html – BenAlabaster May 1 '09 at 14:43
show 10 more comments

Pull out some paper and scribble technical-looking stuff on it. Then when you're thinking, hold your pen and have one hand on the paper. Occasionally add something to the paper.

If you're working on hard problems, you'll probably need a pen and paper to assist your train of thought anyway.

link|flag
3  
This "pen and paper" you speak of -- can you explain what it is and how it works? – Gabe yesterday
show 1 more comment

Pretending you're working on something that's physical, not mental, is a short term solution, I think.

You'd better teach your co-workers that thinking takes most part of the process of programming. Of course, it's a long term solution that may show no value on the first few weeks - or months - but it will surely pay off.

They'll even leave you alone when they see you're quiet, with a mug of coffee in your hand.

Leave him there, they'll say, he's doing his best.

link|flag
4  
+1 Exactly, why try to pretend you're working. That's generally the stance someone who isn't working takes. Just do what you do! – Robin Day May 1 '09 at 13:24

I honestly don't think you can. Sure, you can "pretend" you're working by scribbling, having technical websites open. But if you're doing that, you're not doing the actual "thinking" bit that gets the work done.

I'd stick to worrying about how you get your work done best and not trying to appear to be working to others. If it matters to them they will know the truth.

link|flag

Why do you even think you need to hide the fundamental part of your job as a programmer, ie thinking, from co-workers?

If a co-worker says "Wow! You really don't seem to do much each day, do you?", just explain, in a good natured way, what you are actually doing when they notice you staring off into the distance.

If they continue to have a problem, WHO CARES! Just let the responsible manager deal with it. Of course this relies on the fact that your output is actually acceptable.

This is exactly how I have dealt with this sitaution myself in the past. If someone just doesn't like you, you are not change their views, but most people are simply curious.

A programmer that believes hiding and being deceptive to colleagues about what it they do, will always have difficulty dealing with non-programmers.

link|flag

Have your eyes closed and when someone walks up to you, say "Amen".

link|flag
2  
and put a crucifix on your wall and get a prayer mat too... or whatever your religion dictates. A sure fire way never to be pestered by anyone ever again. Especially if your religion is of a non-mainstream nature :P – BenAlabaster May 1 '09 at 14:05
4  
May the sauce of the FSM lead you to enlightenment. – nilamo Aug 27 '09 at 5:32
show 1 more comment

I recall the movie The Firm when Gene Hackman tells Tom Cruise that if he is even thinking about a client, it's billable time. If it's good enough for a lawyer...

link|flag
2  
My employer couldn't pay me enough if we would handle it the same way. Programmers are constantly trying to learn and improve themselves. I've ideas in the shower before work, but can't put shower time on the time sheet. – stesch Oct 31 '09 at 13:18
show 1 more comment

Do you really have to do this (I mean explanations)? I know it's tough times, etc. but think of finding some place else when you can.

According to your question several things are wrong with your place of work:

  1. Other people (including peers) should have implicit respect to what you do;
  2. Other people should not be able to observe you all the time;
  3. You being concerned with what other people think of you is counter-productive;
  4. "Flow" is a critical process for software development and you are clearly in environment that doesn't embrace this process.

Pick up Peopleware by DeMarco and Lister to read about these and other things that define and explain right teams and places to work.

link|flag

I worked for a couple years for a small company as the sole developer. I know for a fact that they thought I was just slacking off when they'd pass by my door and see me on the internet looking something up, or reading a book like Pragmatic Programmer; and even when I was actually writing diagrams related to the current workload.

In short, I don't think it is possible for someone unfamiliar with programming practices to understand what we do. Anymore than it is possible for someone unfamiliar with engines to see a mechanic just standing there staring under the hood, and understand that the guy is actually working.

link|flag
8  
Yes, I had negative comments in a job once because coworkers saw me sitting around reading "magazines". Like Dr. Dobbs, Programmer's Journal, Proceedings of the ACM. – Cyberherbalist May 1 '09 at 18:51
2  
@Cyberherbalist: The kind of coworkers who can't learn anything by themselves? – stesch May 2 '09 at 0:30
3  
About reading Pragmatic Programmer at work: it's like an automechanic reading "Basic car care" at work instead of fixing cars. – simon Oct 7 '09 at 17:17
1  
The OP didn't mention reading Pragmatic Programmer, so your comment is a straw man argument. A more appropriate analogy is that of an automechanic reading a reference manual about car engines at work when stumped, and I think it's a very good thing. – Seun Osewa 22 hours ago
show 2 more comments

What I hate is when people see you browsing the internet and think you're slacking off. A key part of expectations or perception management is to not have your monitor facing a busy thoroughfare. I hate people walking behind me as a general rule anyway.

link|flag
show 1 more comment

Program in Java. Then you'll have to do a lot more typing. Or, if you're already programming in Java, well, hmmm...COBOL, maybe?

link|flag
show 2 more comments

Get a whiteboard and scribble some ideas on it while you're thinking. I find it helps me structure my ideas better than just walking around while thinking.

link|flag
1  
I don't have a whiteboard in my cubicle at this client, but I've been pestering them for one for ages. Having a whiteboard is great for diagramming random ideas - a great thought processor. – BenAlabaster May 1 '09 at 13:38
2  
Absolutely. The same managers who demand you work also get very excited when they see people writing things on whiteboards viewing it as a sign of creativity, planning and communication. I once had a manager who when we had clients in demanded that the team sit around a whiteboard pretending to discuss stuff so the environment would seem dynamic... Terrifying but true. Still, in her defence she was generally a pretty good manager other than that. – Jon Hopkins May 1 '09 at 13:45
show 2 more comments

I always come running back from the toilet, and say I got a great idea while I was thinking there.

The worst part: it is even true, I just make a show out of it. Do the same with coffee break, just break of mid-phrase if you get an idea etc.

link|flag

I've seen this one. We have an internal IM system where I work. Last year, word came to us (from our boss, as a friendly warning) that people in other departments were looking at the IM client to see our status (if we were "away" or not) and using that to determine that we were slacking off.

So we set our away timeouts to be very high, so it looks like we're always busy, which we usually are.

Between watching programs run that may take time, reading stuff on screen, talking to other programmers next to you, etc... it can look that way.

I don't worry about, really. That's how I work (and the others around me too). That just the way things work in our department. That metric can work with some jobs (i.e. % of time in a call for a call center employee), but it doesn't work for everyone.

What can you do about it? I wouldn't do much. I like the crossword suggestion above. If you are worried about others opinions, I wouldn't be. If they are complaining/making remarks, you can try being nice (the crossword thing) but if it continues you could simply talk to you manager asking for people to basically be informed "he's doing his job, let him be". It's not the job of the clerk 3 desks over to monitor how hard you're working.

The people who watch this kind of thing are usually the same people who are slacking themselves (or at least worried they are, such as a workaholic).

As long as your boss knows you're doing a good job and doesn't think you're wasting the company's money, you should be fine.

link|flag

Write your thoughts on a paper, so they see that you are working !

link|flag

Usually what I do is I'll leave something on my computer screen (query in progress, report, data table, etc.) and i'll play with my rubik's cube while periodaclly looking at the screen.

link|flag
4  
yes...gotta have screen full of windows with things so complex it always looks like your working. Until they notice web browser and you're typing a comment on SO...ohhh shit. – dotjoe May 1 '09 at 15:26
show 1 more comment

Sometimes when I have a difficult problem, like everything I've tried just isn't working, I lean back in my chair and close my eyes. Guess what that looks like.

link|flag
3  
Some times I do that and wake up slightly alarmed when a co-worker wakes me up with some random question. Closing your eyes is dangerous business. – kch May 1 '09 at 15:01
1  
The last job I worked at, I had the same thing - I was leaned back, eyes closed, thinking. Boss walks into my cube, quietly. She says my name and I jump a little - wasn't expecting the interruption. Later that day, my supervisor calls me over and tells me the boss is going to have her "eyes on me" because of the situation. And she was a programmer in the past. Apparently, she's able to think just fine while coding - different strokes for different folks, I guess. – Knobloch May 1 '09 at 18:30
show 1 more comment

I think it's important to distinguish between accepted forms of "thinking" and less conventional methods like some of the ones above (playing FPS games etc). Most workplaces expect some degree of professionalism, and while playing games might be the most effective method for you, society doesn't generally accept that as professional behaviour (yet ;) ).

Usually I just leave the CLI for a linux server open on one screen - most people see that and run.

link|flag
2  
If you can't do even that, take a few screenshots of the command line on your home computer and bring them in to work. :) – Barry Brown May 2 '09 at 1:08
show 2 more comments

If they don't affect your getting a paycheck, what does it matter what they think?

link|flag
9  
If enough people think you are slacking off, eventually management would be forced to do something, which could affect you paycheck. – Rob Z May 1 '09 at 17:14
4  
Even if management knows you're not slacking, it's often easier for them to force you to change your behavior than force everyone else to mind their own business. – patros May 28 '09 at 16:04
show 4 more comments

If you're the only programmer in your division, it's a pity. Otherwise you could write mini-documents like a model overview - anything high-level - to help you with your implementation thougts.

  • revise your own thoughts
  • provide a written document as basis for possible "stand-up" reviews with fellow developers
  • aggregate artefacts for later detailed documentation purposes (which mostly are required in not-so agile companies)
link|flag

You can run your test cases in selenium IDE. x 100

link|flag

My strategies:

  • Bring up some gnarly code in Emacs, and stare in the general direction of the screen.
  • Take a walk.

The problem with the second approach is that if a thought strikes that requires looking something up, you are often a minute or more away from your workstation. Also, I often bump into someone on the walk I need to talk to. That isn't nessecarily a bad thing, but it doesn't help with the task at hand. Still, you end up learning something that you needed to learn. Consider it a holistic development technique. :-)

link|flag
show 2 more comments

11 May 2009

不合时尚的追求- Freeman J.Dyson (转载自水木)

发信人: happyzhe (good boy), 信区: AI
标  题: 转文:不时兴的人和不时兴的思想,常常对科学的进步有决定性的意义
发信站: 水木社区 (Sun Apr  5 16:50:47 2009), 站内



不合时尚的追求

Freeman J.Dyson

   一、引言 今天,我很高兴以高等研究所代表的身份,向Humboldt基金会的会友讲话,因为高等研究所和Humboldt基金会都在努力支持国际范围的科学研究,又都面临着同样的困境和难处。我们正试图坚持150年前von Humboldt所开创的传统。为了对von Humboldt有所了解,我查阅了1910年版的不列颠百科全书,看到科学史家Agnes Clerke写的极漂亮的文章,如果你查以后的版本,则只能读到Clerke文章的片断。Clerke在文中描述了von Humboldt建立第一个国际气象和磁力观测网的工作,结尾铿锵有力:"国际间的科学协作,乃是现代文明最富丽的硕果,而正是Humboldt的努力,成功地促成了第一次合作。"高等研究所和Humboldt基金会正以von Humboldt为榜样,尽力在我们自己的时代,加深和扩大国际间的科学协作。

    二、科学研究的时尚 我决定谈谈科学中的时尚问题,因为对于科学,特别是对于高等研究所和Humboldt基金会,这是个严肃而日趋重要的问题,我首先谈在高等研究所里看到的时尚;然后讲我们能从跨越漫长时期的科学史中吸取的教训;最后,就今后如何更明智地对待时尚说几句话。

    有种说法总是对的,而且在今天比以往任何时候都更加真切,即:对于能力一般的年轻科学家,最聪明的办法是追随占优势的时尚。任何一名青年科学家,要是没有杰出的才华,也没交上难得的好运,他首先关心的是找到一项工作并保有它。为此,他必须涉足于某个科学领域--它是控制着职业市场的、占据高位和有影响的权威们感兴趣的,并从事一项自己能胜任的工作。这些权威认为重要的科学问题,几乎就可以定为合时尚的问题。当然,给予工作的权力,在今天一般不由单个权威掌握,而由一个权威组成的委员会控制。但是,跟个人相比,委员会更难从一个时代的潮流中解脱出来。所以,关心自己生存的的青年科学家倾向于顺着踏就的路前进,这是毫不奇怪的。那些第一流的高级学术机构,向能轻车熟路跟随时尚的人提供保证,给予升职晋级,对不追随时尚者则只提供极少的机会。

    我们研究所也不例外,三十四年前,我首次来这里作访问成员。当时主要的权威是Robert Oppenheimer。他决定物理学中的哪些领域值得搞,他的口味总是跟当时最时兴的方向吻合。我那时年纪轻轻、雄心勃勃,拿着一篇讨论时髦问题的急就篇找到他,很快得到了一个永久的职位。这是那时的状况,今天也依旧如故。有些了解研究所历史的人可能反对上述看法,他们会说,研究所毕竟也给了Kurt G&ouml;del 一个永久职位,情况确是如此。G&ouml;del乃是本世纪少有的几个名不虚传的天才之一。在我们的同事中,他是唯一能跟Einstein以平等地位一起散步、聊天的人。G&ouml;del从事非常深刻但不合时尚的一个数学领域的研究。随着年龄的增加,他显得更加赶不上潮流。我们研究所有理由为给他在教授会提供席位而骄傲。只有一个事实使这和光荣减色:G&ouml;del自来研究所生活和工作,从普通成员升到教授竟花了十四年时间。G&ouml;del有如此的独立和潮流精神,才使得我觉得,研究所在经历十四年的踌躇之后,终于使他成为一名教授,总算也是值得一提的一点功劳吧。晚做总比不做强!

   今天来研究所工作的青年物理学家,比起三十年前的我,受到更大的压力。 首先,他们多半是靠跟政府的合同得到钱的,合同约束他们在确定的时间内从事指定 科学领域的研究。当然,我们不必过份从字面上理解“合同”这个措词。 管理合同的国家科学基金会和能源部的官员都是理智的, 允许我们对所承担的义务作带点伸缩性的解释。 如果研究所内某些靠合同挣钱的成员,打算搞跟合同无关的课题, 那时没有人会强迫我们把这些人赶到大街上去。 有些人的兴趣所在不适于签入合同,他们的工作一般就由研究所基金会支持。 但不管怎么说,合同仍是严肃的,具有约束力, 它从总体上规定了研究所物理部的访问成员应积极从事的工作领域。 合同确定了物理学的主流应该是什么。我们对邀请来工作的成员, 必然要求他们的工作能容易地纳入这一项或那一项合同之中。

    三十年后的今天,我也成了权威中的一份子。我努力鼓励年轻的物理学家在非时尚的领域搞研究,但只能以一种既不道明又十分无力的方式进行。我试图让很少几个没有合同支持的研究领域保持生气。我力图让研究所的大门向具有独立思想和逆潮流的人物敞开。我要始终开着一扇门,以待另一个Kurt G&ouml;del找上门来。不过我不得不承认,我企图阻挡时兴潮流的努力,其效果跟杰出的前辈Canute王[注1]阻止大西洋潮汐的结果差不多。今天的年轻人,被一种比合同或权威更强的力所驱赶,追求着时兴的玩意儿。这股驾驭年轻人时髦的力量就是同辈的压力,就是追赶时髦本身的刺激。他们知道舞台在那儿,并想登台表演。他们知道只有短暂的时间来证明自己是个科学家。他们知道在配给他们使用的短时间内做出有价值成就的最好办法是随大流,尽快地在已成熟的领域摘取科学果实。

    年轻的科学家们力争尽快获得成功、力争尽快取得报偿, 这本身并非坏事。他们的努力集中在一些时兴的专门领域,也不一定有害。 毕竟,时尚问题之所以成为时尚,并非由于象某些时装设计师那样的灵机一动所致, 而是大部分科学家认为它们重要。有一条普遍法则,大多数人的判断总是有根据的。 时兴的领域常常都是些在其中获得了极其重要发现的领域。 年轻科学家拥向这些领域,以期作出轰动世间的发现,这是无可非议的。 确实,在时兴领域中许多人同时研究一些课题, 大大增加了研究所日常生活中乐趣和激情。 对于你在时兴领域开发宝藏进的每一次小的成功、每一回短暂的凯旋, 朋友们都会在饭桌上或讨论班里谈起它。如果失去对时兴问题的共同旨趣, 如果没有这种对新鲜消息和传闻的关心,我们研究所的生活将变得十分乏味。

    那么,我为什么还不满足呢? 我为什么要为那些年轻人--他们正做着我自己在那种年龄时做的事--鸣不平呢? 我之所以有牢骚,因为我认为我们的工作不应该百分之百都是合潮流的。 时尚的研究是有用的、重要的和激动人心的。 我们可以为年轻一代搞时髦课题并有所建树而骄傲。 出于我能理解并尊重的理由,我们将看到他们中的大多数人会永远乐于搞时尚课题。 我的意思只是说,必须为少数不搞时尚研究的人留下位置。 我们应该发现那些不适于纳入流行款式的少数人,并对他们加以鼓励。 在为研究所选人时,必须稍微偏向一点非正统和不从习俗的人。 如果连我们这里都不给搞非时尚科学家的从业者一席工作之地,那么还有谁会给呢?!

    三、以往的历史 由于存在许多非时尚的科学, 支持它们的主要困难之一是选择问题。非时尚的科学千姿百态,没有任何统一的结构。 上星期,我穿过批Princeton 大学的Forrestsl Cumpus (一处校园)时, 遇到两个研究生静静地坐在草地中间,起初, 我以为他们正在享受阳光和八月午后的静谧。 可是走近时,我看到他们正全神贯注地做什么精巧的操作,手一点不能颤动, 精神不得丝毫分散。当走到跟前时,我才弄清他们正忙着把一小块铅粘在蜜蜂背上。 我静静地在旁观看,等他们做完全部工作,便跟着来到他们的实验蜂箱, 箱上装备有照相机和录相机。这两个人正在更精确地做Karl von Frisch 的经典实验, 并进一步扩充实验内容:原实验用于研究蜜蜂用舞蹈传递信息的系统。 他们已发现,当蜜蜂发现蜜源离巢相当远时,它们的舞蹈更明显、更有力、了更精确。 不幸,大多数蜜蜂只在蜂巢附近找蜜,返巢时只是简单地、马马虎虎地跳一阵。这两个学生想观察高精度的舞态,便设计一套办法让蜂表演更明显的舞姿。当一只蜜蜂负重45毫克铅时,只要飞一小段距离它就以为飞了很长时间。蜜蜂以所费的气力来感知飞行的距离。所以,负重的蜜蜂每次采蜜回巢后,都跳出精细的舞蹈。

    上面说的是典型的非时兴科学的例子。事情发生的地点就在我们Princeton大学的门口,我并非提议高等研究所应支持某个昆虫学学派。但蜜蜂实验说明,一切这类非时兴研究的特点,使得支持它们变得困难。它们的规模很小,研究对象各式各样,风格特异,看起来缺乏严肃性。

    为了说清非时尚科学具有真正的和持久的重要性,我来谈我擅长的领域:数学物理。数学物理是这样的人从事的学科,他们力图用严格的方式和纯数学的方法,达到对物理现象的深刻理解。这门学科处于物理和数学的交界处。数学物理学家的目标不是对现象进行数量方面的计算,而是从质的方面去理解。他们提出定理,加以证明,但不依赖数学和计算机。他们的目标在于用数学的精确性,阐明物理理论赖以确立的概念的含义。

    数学物理有三个性质使它跟眼下的讨论有特殊的关系。 第一,它为更实际的物理领域提供基本思想和专门语汇,它从大的范围阐明事物的性质,因而很重要; 第二,它的进展缓慢,一个新概念从创生到能有效的使用, 基本上经历五十到一百年之久; 第三,它几乎总是非时兴的,因为它的周期比科学浪潮的周期大约慢车10倍。 由于它不时髦,所以在欧洲对它的关注与支持,总比在美国强得多。

    有一位伟大的数学物理家的工作,对今日的物理学仍然无比重要,我指的是Sophus Lie,他已去世八十年了。他的伟大工作完成于十九世纪七十至八十年代,但只是在刚过去的二十年间,才支配了研究粒子的物理学家的思想。 Lie第一个理解并清晰地陈述了群理论可作为物理原理的起点。他几乎靠单枪匹马构造了浩大而漂亮的连续群理论,并预见到有朝一日它将成为物理学的一个基础。一百年后的今天,每个按照破缺或无破缺对称性研究粒子分类的物理学家,都自觉或不自觉地使用Sophus Lie的语言。可是当Lie在世时,他的思想并不合时尚,几乎没几个数学家理解它,更不用说物理学家了。Felix Klein 是为数很少的能理解和支持他的大数学家之一。

    Lie属于这样一种人,他们似乎承受着不公平的厄运, 1870年普法战争爆发时,年轻的Lie正在法国漂泊。他是挪威人操着带普鲁士口音的法语。枫丹白露的爱国者认定他是普鲁士奸细,把他投入监狱,由于法国战败,形势一片混乱,当Lie的法国朋友最终找到关他的牢房并成功地使他获释时,他正静居囚笼,搞出了新的数学发现(Lie, 1877)。在世纪交替之际,Rouse Ball出版的数学史中,以悲怆的语调结束对Lie工作的评述(Rouse Ball, 1908): “看来,Lie一直很失望,因他的工作价值没得到普遍承认,他为此而苦恼……。在他生命的最后十年,他常陷入沉思,想着他被过份忽视了的过去,使他心情不快。”

    另一位伟大的数学物理天才是Hermann Grassmann, 他在世时比Sophus Lie更不合时尚。 1844年在Stettin当预科学校的教员时,他发表了题为Die Lineale Ausdehnungslehre(扩张演算)的著作,首次引入了向量、向量空间和反交换代数的基本概念。它们在二十世纪的物理学中极其重要,但在十九世纪时却不然。在他生活的世纪,Grassmann一直在那所不知名的预科学校当教员,科学院的权威对他不闻不问。不过,他比Sophus Lie 有更强的适应性。他不是老想着得不到数学家们的承认这件事,而是开辟第二战场,去学习梵文。他把Rig-Veda(印度古经典四吠陀之一)译成了德文,因而有了不小的名气。也许,如果命运安排你成了不被承认的数学天才,为了健康起见,去当个预科学校的老师比当大学教授要好一些。

    为准备这次讲演,我到研究所图书馆查过资料, 我高兴地发现了一本1878年版的Ausdehnungslehre(维数理论), 标题页上用铅笔写的Minkowski的名字 --他是Einstein的老师, 第一个理解相对论的数学家。1878年出的这本书中有Grassmann写的序言(仍是在Stettin写的),他兴奋的表达了如下希望:新版本将比三十四年前的头一版受到学术界更多注意。序言之后有一行脚注:"Der Verfasser ist w&auml;hrend des Druckes gesorben"(本书付印时作者已去世)。 只是到了十九世纪九十年代,Felix Klein--一位在为非时尚的事业战斗时总是毫不吝啬气力的人,才促成了对Grassmann的正式承认,并出版了他的全集(Grassmann, 1844, 1878, 1894)。

    数学物理在更近期的一个伟大发现,是Hermann Weyl于1918年提出的规范场的思想。这一思想仅过了五十年就在现代基本粒子物理学中获得了地位。量子色动力学是1981年粒子物理学家最时髦的理论,从概念上看,它就是Lie的群论代数和Weyl的规范的综合。Weyl提出规范场时的情况,跟Lie群和Grassmann 代数发现时遭遇完全不同。Weyl既有名气,工作也得到了承认。 他在1918年搞的正是物理学中最时兴的领域:新诞生的广义相对论。 他创立规范场是为了解决将重力和电磁力统一起来的时尚课题。 几个月内,他的规范场变成最时髦的玩意儿。 然后Weyl和其他人发现,规范场的提出并没达到预期的效果, 即他们在事实上并不适于Weyl原来创立它们的目标。 它们很快又不时兴了,甚至几乎被人忘却。 又经过五十年漫长岁月之后,规范场在一个完全不同的方向上--量子电动力学及其在近期引出的量子色动力学方面的推广, 清楚地显示出它的重要性。为规范恢复名誉的关键一步,是由我们Princeton的同事Frank Yang和他的学生Bob Mills于1954年迈出的, 那是在Hermann Weyl 去世前一年的事(Yang和Mills,1954)。 没有证据说明Weyl知道或注意到Yang和Mills使用从他脑袋里蹦出来的娃娃所从事的研究。

    规范场的故事充满了讽刺意味。一个时髦的思想,本想用来解决某个问题,但这个问题本身是短命的。经受长期的冷落之后,规范场最终以物理学里程碑的雄姿屹立于世。在漫长的数学物理发展史上,不乏这种反复的例证。Hamilton发明的四元数,曾被欢呼为解决十九世纪物理问题的灵丹妙药。可是在世纪转折之际,因无用而被弃置。到本世纪二十年代,它有以量子力学中自旋矩阵的形式恢复了青春,现在,它又光荣地跃入了夸克场理论。 Gauss发明的微分几何,起初只是他从事测地学和绘制地图等实际工作的副产品,经天才的Riemamn之手,它被改造成一个具有抽象一般性的新天地;五十年后,又作为Einstein重力理论的基础立于世人面前。这些历史有一个共同点,它们都经经历一个漫长的时期,从发生到结束通常超过单个人的生命期,而最终的结果完全无法预知,发明具有决定性意义的概念的人中,没有一个能对最终使用这种概念的物理领域有些微的感知。

    往事讲得不少了,我想,我已经给各位充分的历史见证以证明我的论点: 不时兴的人和不时兴的思想,常常对科学的进步有决定性的意义。 现在该讲讲现实和未来我没有理由期待今后科学思想的发展格式跟过去不同。 我们能够期待,在未来的岁月,非时兴的思想显示其重要性的机会跟过去一样频繁, 当然,这要经过漫长的孕育期,并在人们所不熟悉的领域崭露头角。 作为科学进步的卫士,我们面临着如何识别有前途但不合时尚的思想以及 如何支持它们的问题。

    四、魔怪和教训 首先,让我们环顾数学世界, 看看能否鉴别出在二十一世纪可能成为物理学基本构件的非时髦的思想。 要是走运,我们说不定能挑出未来杰作的侯选者。 当然,不能奢望在我们的有生之年,就弄清这种挑选是否正确。

    粗略地讲,非时尚的数学就是Bourbaki的权威们宣布为不属于数学的那部分数学,许多非常漂亮的数学发现属于这一范畴。据Bourbaki的观点,一种思想要称得上是数学,应该是一般的、抽象的、统一的,并跟数学的其余部分有清晰的逻辑关系。被排除在数学之外的是特殊的事实和具体的对象,它们的存在缺乏相应的理由,数学农称之为偶然或散在的事物。非时尚的数学主要跟具有意想不到的妙处的对象有关,如特殊函数、特殊的数域、异常的代数、散在有限群。我劝诸君到数学中这些尚未系统化、尚未形成学科的部分,去寻找物理学下一次革命的火种。它们具备奇异性和意外性的品质。它们不容易纳入漂亮的Bourbaki的逻辑结构。正是基于上述理由,我们应该珍爱它们,去开发它们。请记住两年前我们的所长Harry Woolf在一次讲演中的基调,那是他在研究所纪念Einstein诞辰白周年纪念会上引用的Francis Bacon的一句话: "没有奇特的奇异性,也就不存在于不同的美丽"(Woolf,1980)。

    我将简要地谈谈散在有限群(Conway, 1980)的特殊的奇异性。 散在有限群的历史始于法国数学家Emile Mathieu,他在1861年发现了第一个这种群, 1873年又发现第二个。 跟通常获得这类发现的情形一样,Mathieu并不知道自己发明了散在群, 事实上,他的文章的标题中没有"群"这个词(Mathieu, 1861, 1873)。但是,他清楚的知道已找到某种非常漂亮和重要的东西。 用几何语言讲,我们可以说他已经发现在12和24维空间中, 存在一种具有奇特对称性的结构,但在任何维数不是12或24的空间中, 不存在这种结构,他的工作发表了,但在其后的一百年里并不时兴。 正如被公认的数学家喜欢说的那样,这是珍奇的孤品,没有开辟任何前进的道路。

    大约七十五年后,Mathieu群在编码业务中表现出某种实用的重要性。每个Mathieu群都给一种特别有效的纠错码提供了基础。当然,Mathieu群在实际中的应用并未招致数学家的青眯,他们的口味让Bourbaki给限制住了。

    接着的二十年间,风云突变,各方面的数学家用各种方法发现了新的散在群组成的宏大的"动物园"。他们之中有的是按照Mathieu的思想找到的;另一些是通过研究一个非时尚的问题引出的:把24维的台球尽可能紧地装进24维欧几里得空间(Leech, 1967);还有的是在大计算机上试算排列组合问题时创造的。

这些发现有一些共同点:具体性、经验性、实验性和偶然性。这跟Bourbaki的精神正好相违。包括Mathieu的结果在内,总共发现了25个散在群。与此同时,群论专家的团体,用更一般和抽象的方法,成功地证明了散在群的总数不能大于26,所以,两年前的形势是还剩下一个散在群可寻。当时知道,如果着最后一个群存在,它将是所有散在群中最大和最漂亮的(Conway和Norton, 1979)。正在猎取它的人给它起了绰号叫"魔怪"或"亲密的巨兽"。

    去年,当Bob Griess从Michigan大学来高等研究所访问时,上面的故事终于有了结尾,他找到了构造这个魔怪的方法(Griess, 1981)。正巧在昨天,我收到从Michigan几来的长篇论文的最后部分,其中包括对他的研究的完全和肯定的评价(Griess, 1982)。对于那些不辞辛劳地从细节上弄清Bob Griess构造的人来说,魔怪的面目已暴露无遗,他们可以感到满足和快意。现在,这最后的也是最大的散在群已无懈可击地独立于世,成为一座不朽的纪念碑。

    这一切对物理学有什么意义呢?也许,一无所有。也许,散在群只是数学史上一弯可爱的滞水,远离浩荡主流的奇妙的插曲。我们绝没发现一点儿迹象,说明物理世界中的对称以任何方式跟散在群的对称发生联系。迄今,我们所知道的是,不管有无散在群存在,物理世界的面貌和功能依然如故。但是,我们不应过分地肯定它们之间无关。缺乏证据跟不存在证据到底不是一回事。在物理学史上,有过比意想不到的散在群的出现更奇怪的事。我们应该永远准备好迎接意外的事情。我必须坦白地招认,我内心存着希望,没有任何事实和证据支持的希望:有朝一日,在二十一世纪,物理学家将与魔群邂逅,以某种出人意料的方式将其纳入物理世界的结构。这只是一种莽撞的推测,几乎肯定是错的。有利于这一推断的唯一证据来自神学。这个强有力的证据是:宇宙的创造者喜爱对称。如果他喜爱对称,那么还有什么别的对称?quot;魔怪"的对称更可爱呢?

    散在群只是不合潮流的数学家创造的怪异而奇妙的思想宝库中的一例。我还能举出许多例证。你能想象一个正面体,--由完美的对称元构成的物体,--排列成完美的对称结构,总共有11个面吗?去年,我的朋友Donald Coxeter(在Toronto)找到了这个多面体(Coxeter, 1981)。 有朝一日,会不会发现Zeta函数的零点(Riemann在120年前猜测它们具有某些性质, 现今仍是数学中重要的秘密之一)跟物理世界有隐秘的联系呢? 去年,Andrew Odlyzko(Bell实验室使用Cray计算机的数学家) 发现了Zeta函数零点的某些新的和出人意料的性质。 Kurt G&ouml;del的不完全性定理(证明纯数学中存在这类问题,任意给定一组有限个公理和推理规则,都无法解答它),是否有一天会使我们对物理知识的限度有更深入的理解?不管你在哪个思想的王国游历,总会发现各种奥秘的暗示,听到有关藏匿着的各种事物间联系的传闻。
时间不多了。 我必须践约讲讲对支持科学研究的具体意见。我是针对高等研究所和Humboldt基金会讲的。这既是我们的义务,也是我们这两个比政府更具慧眼的独立组织的殊荣。我们应能采取一种比政治家和博士后的学生看的更远的科学观。目光远大的科学观教我们怎么做呢?从上面讲过的许多故事应引出什么教训呢?教训只有一条,很简单:应该更多地注意、更有力地支持非时尚的研究。在科学史上任一特定时期,最重要和最重要和最富成果的思想往往潜伏着不被利用,原因仅仅是它不合时尚。具体到数学物理领域,从新思想的孕育到它成为科学思想的主流,通常要磨蹭五十或一百年。如果这是衡量基础性进展的尺度, 那么结论必然是:在数学物理领域从事基本研究的任何人几乎肯定是不合时尚的。

    当然,我们不应该停止支持使大多数年轻科学家忙碌和高兴的时尚研究。 但我们应拨出一部分经费,也许是十分之一或四分之一, 以支持从事非时尚工作的不合潮流的人。我们不应该害怕看到做傻事, 或是看到一堆破烂;我们不应该害怕支持可能完全失败的冒险事业。 因为我们是独立的、我们有权利冒险和犯错误。 那些仅仅支持搞无危险、无犯错误机会的研究的机构,实际上只是支持了平凡的人。 如果我们靠良知和勇气,支持不时兴的人,去做正统观念认为是不对题和冒险的事, 这就提供一种好的机会,为科学拯救很难得到的Sophus Lie或是Hermann Grassmann。 当我们时代的所有时髦动人的成果早被人遗忘之后,他们的思想仍将驰名于世。

参考文献


Conway, J. H. (1980) Monsters and Moonshine, Then Mathematical Intelligencer, 2, No. 4, 165-171.
Conway, J. H. and S. P. Norton (1979) Monstrous Moonshine, Bull. London Math. Soc. 11, 308-339.
Coxeter, H. S. M. (1981) "A Symmetric Arrangement of Eleven Hemi-Icosahedron", to be published.
Grassmann, H. (1844, 1878, 1894) Die Lineale Ausdehnugslehre, 1st ed. (ott Wigand, Leipzig) 1844, 2nd ed. (Otto Wigand, Leipzig) 1878, 3rd ed. in Grassmann's collected works edited by F. Engel (Teubner, Leipzig) 1894.
Griess, R. L. (1981) A Construction of F_1 as Automorphisms of a 196883-dimensional Algebra, Proc. Nat. Acad. Sci USA, 78, 689-691.
Griess, R. L. (1982) The Friendly Giant, Invent. Math. 69, 1-102.
Leech, J. (1967) Notes on Sphere Packings, Can. J. Math. 19, 251-267.
Lie, S. (1877) letter to A. Meyer, published in Sophus Lie, Gesammelte Abhandlunger, ed. F. Engel (Leipzig, Teubner, 1922), Vol. 3, Anmerkungern, p. 691.
Mathieu E. L. (861, 1873) Mémoire sur l' éstude des functions de plusieurs quantities, J. de Math. Pures et Appliquées, 6, 241-323, "Sur la foncion cinq fois transitive de 24 qunatités", J. de Math. Pures et Appliquées, 18, 25-46.
Rouse, Ball, W. W. (108) A Short Account of the History of Mathematics, 4th ed. (MacMillan, London), p. 478.
Woolf, H., ed. (1980) Some Strangeness in the Proportion: A centennial Symposium to Celebrate the Achievments of Albert Einstein (Addison-Wesley, Reading, Mass.)
Yang, C. N. and R. L. Mills (1954) Conservation of Isotopic Spin and Isotopic Gauge Invariance, Phys. Rev. 96, 191-195.


译注1: Canute王:英格兰及丹麦的王(995-1035)。

原题:Unfashionable Pursuits。译自The Mathematical Intelligencer 5:3 (1983), 此报告是1981年8月24日在Princeton高等研究所做的。
中文翻译:袁向东译,吴允增较。原载于《数学译林》。

8 May 2009

The Business of Mining the Twitter Stream

The Business of Mining the Twitter Stream

(http://datamining.typepad.com/data_mining/2009/02/the-business-of-mining-the-twitter-stream.html)

February 19, 2009

While mining Twitter data for business and marketing intelligence (trend/buzz analysis, sentiment/opinion mining, authority/influence analysis) looks like a compelling path to explore for a business model, it is important to consider the proposition from the point of view of the customer. Enterprises have been working with vendors in this space (mining social media content for BI) for well over 5 years and already have expectations regarding the features and quality of reports that these analytics needs to deliver to be useful (actionable).

  • Domain coverage: how broad is the topical space available in the solution? Crawling all data sources is the way to win here.
  • Demographic coverage: the broader the demographic coverage (and the accuracy with which the demographic features of the content authors can be determined) the better.
  • Content Analysis/Text Mining: how well does the solution take all the unstructured content and deliver structured interpretations that can then act as the input for further data mining. This is generally a matter of applied research (taking the current state of the art in text mining and making it work with the greater variety and complexity of social media content).
  • Timeliness: how timely is the analysis. This is generally a function of how timely the data is collected. Blog data, for example, can be gathered in a very timely manner thanks to the ping/feed  mechanism. However, the reality of real time mining is that the consumer of the data is the real calibrator - real time may mean 4 hourly, not second by second.

If the business model for Twitter is going to be mining the Twitter stream for BI/MI, then they will be competing with companies that gather very large data sets (weblogs, usenet, message boards, reviews, groups, mailing lists, etc.). Seth Grimes suggested that the short texts of the Twitter stream may make hard problems like sentiment mining simpler as the limited space requires the author to be concise. However, this is a double edges sword as it means that the depth of analysis will be far shallower.

I believe that mining Twitter data will be a very exciting experiment, but I think that if Twitter goes down this path, it will have to either provide analytics over the other data sets, or partner with an existing company (say Visible Technologies). In fact, such a partnership would take the burden of building out an analytics engine away from the small Twitter team allowing them to continue to focus on infrastructure and ensuring the flow of this valuable data stream.

Add to del.icio.usEmail this

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c994053ef011278fcc1a428a4

Listed below are links to weblogs that reference The Business of Mining the Twitter Stream:

Comments

Matthew, thanks for the mention. I'd venture that tweet mineability is also easier because short messages cover a single topic.

Short messages are easy to post so they folks can post more frequently. So maybe the more interesting thing to mine from twitter is message propagation. Then from propagation threads and connectedness patterns, one could infer influence networks and knowledge about the types & topics & forms of messages that travel farthest and fastest.

I don't get it, anyone can mine twitter for sentiment (using the search API)... why would twitter reinvent the wheel?

Nice analysis, Matthew - I'd also say that as part of the way that people use Twitter is to share links to interesting content/conversations elsewhere, the need to be analysing the networks around the Twitter streams is very important indeed.

There's a reality of the value of the raw data to the marketplace, which I'll get to in a minute. Regardless, short messages may very well be harder to search, not easier. Here's some reasons why:

* For indexing purposes, it's not only the corpus of the text that matters, it's the number of objects. So a search architecture has to take that into account. It's a non-trivial problem; especially with the kind of volumes involved here. Not to mention that servers are going to be thrashed with reading/writing if anything is meant to be done real time. (Perhaps less so for batch analysis of course.)

* Next we have the nature of the messages themselves. Due to the 140 character nature, there's an increase in odd acronyms even beyond the brb, lol, etc. Perhaps synonym dictionaries could be produced, but the variability here seems extreme just based on anecdotal experience.

* Regarding sentiment mining, that's difficult enough in larger text, but may be harder in small text. Not for raw sentiment where the phrases are obvious. But sentiment analysis lags with regards to humor and sarcasm, which may need more markers to divine actual meaning.

These are solvable problems. And in the latter case, it might not matter that terribly much if some stuff gets missed as general trends can still be spotted easily enough. Personally, I feel confident someone(s) will work this out to some reasonable degree of satisfaction.

Next, as to the dollars. I can tell you from experience the industry does not value the raw data terribly highly for specific social media data streams. The value added analysis? Yes. The actual data? Not so much. This is because it's easy enough for a variety of people to crawl blogs, forums and so forth. And several do, though in some cases there's really only a couple of providers feeding data to the 60+ reputation monitoring companies.

Unless Twitter made itself the sole availability for the full data stream, they wouldn't be able to command that great a price. I'm just guesstimating based on past experience with other data types here, but MAYBE 1M / month if they sold to every rep services company out there. (Who would in turn add analysis and re-sell for much more.) That's decent money, but it's not 'to the moon' money. I could be wrong here. People are valuing this stuff more highly. But to really capitalize on it, there's no way they could just let anyone suck down all they could eat off the stream. Which means less open. Which is fine. They're entitled to do so.

We'll see!

Nice post. I have included this blog into my rss subscriptions. Very nicely put on data mining using social media. I honestly have not thought about it in this much detail but it makes sense and could be used as a great competitor intelligence tool!

I'm still working on it but Twitter data sure is tasty.. Lots of goodies !

Thanks for the run down'

Mike

www.wannadevelop.com

There is a lot of potential in analyzing Tweets : Segmentation of users, Sentiment Analysis to name a few. In my experience, the fact that tweets are maximum 140 characters makes things easier in catching emerging trends but also in Text analysis.

Combining Information Extraction and Ontologies (using IE to mark Text and using NLP to insert information to an Ontological Setting) is the way to go although it requires considerable effort.

Google