• Calender

    July 2015
    M T W T F S S
    « Jun   Aug »
  • Contact

    Send us press releases, events and new product information for publication..

    Email: nchuppala@ outlook.com

Life as a WebFOCUS Specialist- Interesting Read

From the sidelines, I have been watching an online forum conversation started by a gentleman named Larry about the seemingly low hourly rates offered for WebFOCUS BI developers. Today, I decided to respond with, I hope, minimal backlash. Here are my personal opinions: 

Against my better judgment, I will add my thoughts on the current status of the WebFOCUS services market.

We all realize that while Information Builders was the leader in the 4GL market in the 1980s, they are one of the smaller players in today’s big BI market. Some customers like how IBI is privately held and innovative but most want the security of buying a big name like Microsoft, Oracle, SAP, and IBM.

Information Builders seems to be happy where it is and is organized to handle its $300 million in annual revenue where it has been for the past decade. It seems happy not becoming a multi-billion dollar BI giant. It is like the corner pizza shop with a cult following with no desire of becoming the next big Pizza Hut in the mass market. If this is a hobby and a lifestyle for the owner, there is nothing wrong with the approach.

The job market has lower demands for a small software vendor’s products; there is less respect which translates to lower wages for practitioners. To Larry’s point, that can mean lost opportunity costs for WebFOCUS people. You might personally be losing money specializing in WebFOCUS instead of in a product from one of the market leaders. It could be a bad career choice.

IBI is not just a software vendor; it has its own quasi-independent professional services firm. Most new customers will go the safe route and choose the vendor to do initial and difficult projects despite the high cost. After that, customers want to build their own internal expertise so as to not be held captive by vendors. Outside firms and independents competing against IBI services will only get the table scraps. If they really want to get in on the game, competitors must offer quality services at a much lower price.

The WebFOCUS marketing message is that it is very easy. Customers get the idea that anybody can sit down at the screen and do it. Here enters the “McDonald’s principle” of wages. If you can take any average Joe off the street and train him quickly to do a job, there is no reason to pay him more than minimum wage. You can always replace him with another when he complains about wages. Only when more experience and training is needed and it is harder to replace the person does the wage need to increase.

WebFOCUS is an enterprise BI tool which means a practitioner needs to know lots of different things: platforms, databases, security, web application development, and so forth. I’ve heard this called being a “T-professional.” Represented by the horizontal top of the T, you have a broad understanding of lots of things but no real in-depth knowledge. For the vertical part of the T, you have a deep understanding of one thing, which is WebFOCUS and (as we have seen) not highly respected in the marketplace. That is Larry’s point of being paid less to do WebFOCUS and Oracle together than just Oracle work by itself; to make more money, you have to flip the T to be respected.

The idea of 4GLs as viable development tools died in the early 1990s. Customers do not want to have to know a 4GL to build a web dashboard. They do not want to have 4GL coders on staff.

WebFOCUS must hide the 4GL under the covers and IBI sales people only mention the code when it can be used in a positive difference from other products that don’t have an underlying, accessible language. This general anti-4GL sentiment means WebFOCUS developers must be proficient with the GUI and only touch the 4GL code in rare instances behind a closed door. So even if you are a really great 4GL coder, you are forced to keep your mouth shut or be perceived as anachronistic trying to use technologies from decades ago. The market has no respect and no demand for 4GL coders.

For all these reasons and more, there is a dearth of WebFOCUS professionals. Seize the day!

To be truly successful, however, you should deepen your knowledge of enterprise business intelligence and web application development. Flip your T and go deep in technologies that are widely respected and in demand.

The Hidden Traps in Decision Making


by John S. Hammond
, Ralph L. Keeney
, and Howard Raiffa
From the January 2006 Issue

Before deciding on a course of action, prudent managers evaluate the situation confronting them. Unfortunately, some managers are cautious to a fault—taking costly steps to defend against unlikely outcomes. Others are overconfident—underestimating the range of potential outcomes. And still others are highly impressionable—allowing memorable events in the past to dictate their view of what might be possible now.

These are just three of the well-documented psychological traps that afflict most managers at some point, assert authors John S. Hammond, Ralph L. Keeney, and Howard Raiffa in their 1998 article. Still more pitfalls distort reasoning ability or cater to our own biases. Examples of the latter include the tendencies to stick with the status quo, to look for evidence confirming one’s preferences, and to throw good money after bad because it’s hard to admit making a mistake.

Techniques exist to overcome each one of these problems. For instance, since the way a problem is posed can influence how you think about it, try to reframe the question in various ways and ask yourself how your thinking might change for each version. Even if we can’t eradicate the distortions ingrained in the way our minds work, we can build tests like this into our decision-making processes to improve the quality of the choices we make.

Making decisions is the most important job of any executive. It’s also the toughest and the riskiest. Bad decisions can damage a business and a career, sometimes irreparably. So where do bad decisions come from? In many cases, they can be traced back to the way the decisions were made—the alternatives were not clearly defined, the right information was not collected, the costs and benefits were not accurately weighed. But sometimes the fault lies not in the decision-making process but rather in the mind of the decision maker. The way the human brain works can sabotage our decisions.

Researchers have been studying the way our minds function in making decisions for half a century. This research, in the laboratory and in the field, has revealed that we use unconscious routines to cope with the complexity inherent in most decisions. These routines, known as heuristics, serve us well in most situations. In judging distance, for example, our minds frequently rely on a heuristic that equates clarity with proximity. The clearer an object appears, the closer we judge it to be. The fuzzier it appears, the farther away we assume it must be. This simple mental shortcut helps us to make the continuous stream of distance judgments required to navigate the world.

Yet, like most heuristics, it is not foolproof. On days that are hazier than normal, our eyes will tend to trick our minds into thinking that things are more distant than they actually are. Because the resulting distortion poses few dangers for most of us, we can safely ignore it. For airline pilots, though, the distortion can be catastrophic. That’s why pilots are trained to use objective measures of distance in addition to their vision.

Researchers have identified a whole series of such flaws in the way we think in making decisions. Some, like the heuristic for clarity, are sensory misperceptions. Others take the form of biases. Others appear simply as irrational anomalies in our thinking. What makes all these traps so dangerous is their invisibility. Because they are hardwired into our thinking process, we fail to recognize them—even as we fall right into them.

For executives, whose success hinges on the many day-to-day decisions they make or approve, the psychological traps are especially dangerous. They can undermine everything from new-product development to acquisition and divestiture strategy to succession planning. While no one can rid his or her mind of these ingrained flaws, anyone can follow the lead of airline pilots and learn to understand the traps and compensate for them.

In this article, we examine a number of well-documented psychological traps that are particularly likely to undermine business decisions. In addition to reviewing the causes and manifestations of these traps, we offer some specific ways managers can guard against them. It’s important to remember, though, that the best defense is always awareness. Executives who attempt to familiarize themselves with these traps and the diverse forms they take will be better able to ensure that the decisions they make are sound and that the recommendations proposed by subordinates or associates are reliable.

The Anchoring Trap

How would you answer these two questions?
Is the population of Turkey greater than 35 million?

What’s your best estimate of Turkey’s population?

If you’re like most people, the figure of 35 million cited in the first question (a figure we chose arbitrarily) influenced your answer to the second question. Over the years, we’ve posed those questions to many groups of people. In half the cases, we used 35 million in the first question; in the other half, we used 100 million. Without fail, the answers to the second question increase by many millions when the larger figure is used in the first question. This simple test illustrates the common and often pernicious mental phenomenon known as anchoring. When considering a decision, the mind gives disproportionate weight to the first information it receives. Initial impressions, estimates, or data anchor subsequent thoughts and judgments.

Anchors take many guises. They can be as simple and seemingly innocuous as a comment offered by a colleague or a statistic appearing in the morning newspaper. They can be as insidious as a stereotype about a person’s skin color, accent, or dress. In business, one of the most common types of anchors is a past event or trend. A marketer attempting to project the sales of a product for the coming year often begins by looking at the sales volumes for past years. The old numbers become anchors, which the forecaster then adjusts based on other factors. This approach, while it may lead to a reasonably accurate estimate, tends to give too much weight to past events and not enough weight to other factors. In situations characterized by rapid changes in the marketplace, historical anchors can lead to poor forecasts and, in turn, misguided choices.

Because anchors can establish the terms on which a decision will be made, they are often used as a bargaining tactic by savvy negotiators. Consider the experience of a large consulting firm that was searching for new office space in San Francisco. Working with a commercial real-estate broker, the firm’s partners identified a building that met all their criteria, and they set up a meeting with the building’s owners. The owners opened the meeting by laying out the terms of a proposed contract: a ten-year lease; an initial monthly price of $2.50 per square foot; annual price increases at the prevailing inflation rate; all interior improvements to be the tenant’s responsibility; an option for the tenant to extend the lease for ten additional years under the same terms. Although the price was at the high end of current market rates, the consultants made a relatively modest counteroffer. They proposed an initial price in the midrange of market rates and asked the owners to share in the renovation expenses, but they accepted all the other terms. The consultants could have been much more aggressive and creative in their counterproposal—reducing the initial price to the low end of market rates, adjusting rates biennially rather than annually, putting a cap on the increases, defining different terms for extending the lease, and so forth—but their thinking was guided by the owners’ initial proposal. The consultants had fallen into the anchoring trap, and as a result, they ended up paying a lot more for the space than they had to.

>> What can you do about it?

The effect of anchors in decision making has been documented in thousands of experiments. Anchors influence the decisions not only of managers, but also of accountants and engineers, bankers and lawyers, consultants and stock analysts. No one can avoid their influence; they’re just too widespread. But managers who are aware of the dangers of anchors can reduce their impact by using the following techniques: 

Always view a problem from different perspectives. Try using alternative starting points and approaches rather than sticking with the first line of thought that occurs to you.
•Think about the problem on your own before consulting others to avoid becoming anchored by their ideas.
•Be open-minded. Seek information and opinions from a variety of people to widen your frame of reference and to push your mind in fresh directions.
Be careful to avoid anchoring your advisers, consultants, and others from whom you solicit information and counsel. Tell them as little as possible about your own ideas, estimates, and tentative decisions. If you reveal too much, your own preconceptions may simply come back to you.
•Be particularly wary of anchors in negotiations. Think through your position before any negotiation begins in order to avoid being anchored by the other party’s initial proposal. At the same time, look for opportunities to use anchors to your own advantage—if you’re the seller, for example, suggest a high, but defensible, price as an opening gambit.

The Status-Quo Trap

We all like to believe that we make decisions rationally and objectively. But the fact is, we all carry biases, and those biases influence the choices we make. Decision makers display, for example, a strong bias toward alternatives that perpetuate the status quo. On a broad scale, we can see this tendency whenever a radically new product is introduced. The first automobiles, revealingly called “horseless carriages,” looked very much like the buggies they replaced. The first “electronic newspapers” appearing on the World Wide Web looked very much like their print precursors.

Decision makers display a strong bias toward alternatives that perpetuate the status quo.

On a more familiar level, you may have succumbed to this bias in your personal financial decisions. People sometimes, for example, inherit shares of stock that they would never have bought themselves. Although it would be a straightforward, inexpensive proposition to sell those shares and put the money into a different investment, a surprising number of people don’t sell. They find the status quo comfortable, and they avoid taking action that would upset it. “Maybe I’ll rethink it later,” they say. But “later” is usually never.

The source of the status-quo trap lies deep within our psyches, in our desire to protect our egos from damage. Breaking from the status quo means taking action, and when we take action, we take responsibility, thus opening ourselves to criticism and to regret. Not surprisingly, we naturally look for reasons to do nothing. Sticking with the status quo represents, in most cases, the safer course because it puts us at less psychological risk.

Many experiments have shown the magnetic attraction of the status quo. In one, a group of people were randomly given one of two gifts of approximately the same value—half received a mug, the other half a Swiss chocolate bar. They were then told that they could easily exchange the gift they received for the other gift. While you might expect that about half would have wanted to make the exchange, only one in ten actually did. The status quo exerted its power even though it had been arbitrarily established only minutes before.

Other experiments have shown that the more choices you are given, the more pull the status quo has. More people will, for instance, choose the status quo when there are two alternatives to it rather than one: A and B instead of just A. Why? Choosing between A and B requires additional effort; selecting the status quo avoids that effort.

In business, where sins of commission (doing something) tend to be punished much more severely than sins of omission (doing nothing), the status quo holds a particularly strong attraction. Many mergers, for example, founder because the acquiring company avoids taking swift action to impose a new, more appropriate management structure on the acquired company. “Let’s not rock the boat right now,” the typical reasoning goes. “Let’s wait until the situation stabilizes.” But as time passes, the existing structure becomes more entrenched, and altering it becomes harder, not easier. Having failed to seize the occasion when change would have been expected, management finds itself stuck with the status quo.
>> What can you do about it?

First of all, remember that in any given decision, maintaining the status quo may indeed be the best choice, but you don’t want to choose it just because it is comfortable. Once you become aware of the status-quo trap, you can use these techniques to lessen its pull:
•Always remind yourself of your objectives and examine how they would be served by the status quo. You may find that elements of the current situation act as barriers to your goals.
•Never think of the status quo as your only alternative. Identify other options and use them as counterbalances, carefully evaluating all the pluses and minuses.
•Ask yourself whether you would choose the status-quo alternative if, in fact, it weren’t the status quo.
•Avoid exaggerating the effort or cost involved in switching from the status quo.
•Remember that the desirability of the status quo will change over time. When comparing alternatives, always evaluate them in terms of the future as well as the present.
•If you have several alternatives that are superior to the status quo, don’t default to the status quo just because you’re having a hard time picking the best alternative. Force yourself to choose.

The Sunk-Cost Trap

Another of our deep-seated biases is to make choices in a way that justifies past choices, even when the past choices no longer seem valid. Most of us have fallen into this trap. We may have refused, for example, to sell a stock or a mutual fund at a loss, forgoing other, more attractive investments. Or we may have poured enormous effort into improving the performance of an employee whom we knew we shouldn’t have hired in the first place. Our past decisions become what economists term sunk costs—old investments of time or money that are now irrecoverable. We know, rationally, that sunk costs are irrelevant to the present decision, but nevertheless they prey on our minds, leading us to make inappropriate decisions.

Why can’t people free themselves from past decisions? Frequently, it’s because they are unwilling, consciously or not, to admit to a mistake. Acknowledging a poor decision in one’s personal life may be purely a private matter, involving only one’s self-esteem, but in business, a bad decision is often a very public matter, inviting critical comments from colleagues or bosses. If you fire a poor performer whom you hired, you’re making a public admission of poor judgment. It seems psychologically safer to let him or her stay on, even though that choice only compounds the error.

The sunk-cost bias shows up with disturbing regularity in banking, where it can have particularly dire consequences. When a borrower’s business runs into trouble, a lender will often advance additional funds in hopes of providing the business with some breathing room to recover. If the business does have a good chance of coming back, that’s a wise investment. Otherwise, it’s just throwing good money after bad.

One of us helped a major U.S. bank recover after it made many bad loans to foreign businesses. We found that the bankers responsible for originating the problem loans were far more likely to advance additional funds—repeatedly, in many cases—than were bankers who took over the accounts after the original loans were made. Too often, the original bankers’ strategy—and loans—ended in failure. Having been trapped by an escalation of commitment, they had tried, consciously or unconsciously, to protect their earlier, flawed decisions. They had fallen victim to the sunk-cost bias. The bank finally solved the problem by instituting a policy requiring that a loan be immediately reassigned to another banker as soon as any problem arose. The new banker was able to take a fresh, unbiased look at the merit of offering more funds.

Sometimes a corporate culture reinforces the sunk-cost trap. If the penalties for making a decision that leads to an unfavorable outcome are overly severe, managers will be motivated to let failed projects drag on endlessly—in the vain hope that they’ll somehow be able to transform them into successes. Executives should recognize that, in an uncertain world where unforeseeable events are common, good decisions can sometimes lead to bad outcomes. By acknowledging that some good ideas will end in failure, executives will encourage people to cut their losses rather than let them mount.

>> What can you do about it?

For all decisions with a history, you will need to make a conscious effort to set aside any sunk costs—whether psychological or economic—that will muddy your thinking about the choice at hand. Try these techniques:
•Seek out and listen carefully to the views of people who were uninvolved with the earlier decisions and who are hence unlikely to be committed to them.
•Examine why admitting to an earlier mistake distresses you. If the problem lies in your own wounded self-esteem, deal with it head-on. Remind yourself that even smart choices can have bad consequences, through no fault of the original decision maker, and that even the best and most experienced managers are not immune to errors in judgment. Remember the wise words of Warren Buffett: “When you find yourself in a hole, the best thing you can do is stop digging.”
•Be on the lookout for the influence of sunk-cost biases in the decisions and recommendations made by your subordinates. Reassign responsibilities when necessary.
•Don’t cultivate a failure-fearing culture that leads employees to perpetuate their mistakes. In rewarding people, look at the quality of their decision making (taking into account what was known at the time their decisions were made), not just the quality of the outcomes.

The Confirming-Evidence Trap

Imagine that you’re the president of a successful midsize U.S. manufacturer considering whether to call off a planned plant expansion. For a while you’ve been concerned that your company won’t be able to sustain the rapid pace of growth of its exports. You fear that the value of the U.S. dollar will strengthen in coming months, making your goods more costly for overseas consumers and dampening demand. But before you put the brakes on the plant expansion, you decide to call up an acquaintance, the chief executive of a similar company that recently mothballed a new factory, to check her reasoning. She presents a strong case that other currencies are about to weaken significantly against the dollar. What do you do?

You’d better not let that conversation be the clincher, because you’ve probably just fallen victim to the confirming-evidence bias. This bias leads us to seek out information that supports our existing instinct or point of view while avoiding information that contradicts it. What, after all, did you expect your acquaintance to give, other than a strong argument in favor of her own decision? The confirming-evidence bias not only affects where we go to collect evidence but also how we interpret the evidence we do receive, leading us to give too much weight to supporting information and too little to conflicting information.

In one psychological study of this phenomenon, two groups—one opposed to and one supporting capital punishment—each read two reports of carefully conducted research on the effectiveness of the death penalty as a deterrent to crime. One report concluded that the death penalty was effective; the other concluded it was not. Despite being exposed to solid scientific information supporting counterarguments, the members of both groups became even more convinced of the validity of their own position after reading both reports. They automatically accepted the supporting information and dismissed the conflicting information.

There are two fundamental psychological forces at work here. The first is our tendency to subconsciously decide what we want to do before we figure out why we want to do it. The second is our inclination to be more engaged by things we like than by things we dislike—a tendency well documented even in babies. Naturally, then, we are drawn to information that supports our subconscious leanings.

We tend to subconsciously decide what to do before figuring out why we want to do it.

>> What can you do about it?

It’s not that you shouldn’t make the choice you’re subconsciously drawn to. It’s just that you want to be sure it’s the smart choice. You need to put it to the test. Here’s how:
•Always check to see whether you are examining all the evidence with equal rigor. Avoid the tendency to accept confirming evidence without question.
•Get someone you respect to play devil’s advocate, to argue against the decision you’re contemplating. Better yet, build the counterarguments yourself. What’s the strongest reason to do something else? The second strongest reason? The third? Consider the position with an open mind.
•Be honest with yourself about your motives. Are you really gathering information to help you make a smart choice, or are you just looking for evidence confirming what you think you’d like to do?
•In seeking the advice of others, don’t ask leading questions that invite confirming evidence. And if you find that an adviser always seems to support your point of view, find a new adviser. Don’t surround yourself with yes-men.

The Framing Trap

The first step in making a decision is to frame the question. It’s also one of the most dangerous steps. The way a problem is framed can profoundly influence the choices you make. In a case involving automobile insurance, for example, framing made a $200 million difference. To reduce insurance costs, two neighboring states, New Jersey and Pennsylvania, made similar changes in their laws. Each state gave drivers a new option: By accepting a limited right to sue, they could lower their premiums. But the two states framed the choice in very different ways: In New Jersey, you automatically got the limited right to sue unless you specified otherwise; in Pennsylvania, you got the full right to sue unless you specified otherwise. The different frames established different status quos, and, not surprisingly, most consumers defaulted to the status quo. As a result, in New Jersey about 80% of drivers chose the limited right to sue, but in Pennsylvania only 25% chose it. Because of the way it framed the choice, Pennsylvania failed to gain approximately $200 million in expected insurance and litigation savings.

The framing trap can take many forms, and as the insurance example shows, it is often closely related to other psychological traps. A frame can establish the status quo or introduce an anchor. It can highlight sunk costs or lead you toward confirming evidence. Decision researchers have documented two types of frames that distort decision making with particular frequency:

Frames as gains versus losses.

In a study patterned after a classic experiment by decision researchers Daniel Kahneman and Amos Tversky, one of us posed the following problem to a group of insurance professionals:
You are a marine property adjuster charged with minimizing the loss of cargo on three insured barges that sank yesterday off the coast of Alaska. Each barge holds $200,000 worth of cargo, which will be lost if not salvaged within 72 hours. The owner of a local marine-salvage company gives you two options, both of which will cost the same:

Plan A: This plan will save the cargo of one of the three barges, worth $200,000.

Plan B: This plan has a one-third probability of saving the cargo on all three barges, worth $600,000, but has a two-thirds probability of saving nothing.

Which plan would you choose?

If you are like 71% of the respondents in the study, you chose the “less risky” Plan A, which will save one barge for sure. Another group in the study, however, was asked to choose between alternatives C and D:
Plan C: This plan will result in the loss of two of the three cargoes, worth $400,000.

Plan D: This plan has a two-thirds probability of resulting in the loss of all three cargoes and the entire $600,000 but has a one-third probability of losing no cargo.

Faced with this choice, 80% of these respondents preferred Plan D.

The pairs of alternatives are, of course, precisely equivalent—Plan A is the same as Plan C, and Plan B is the same as Plan D—they’ve just been framed in different ways. The strikingly different responses reveal that people are risk averse when a problem is posed in terms of gains (barges saved) but risk seeking when a problem is posed in terms of avoiding losses (barges lost). Furthermore, they tend to adopt the frame as it is presented to them rather than restating the problem in their own way.

Framing with different reference points.

The same problem can also elicit very different responses when frames use different reference points. Let’s say you have $2,000 in your checking account and you are asked the following question:
Would you accept a fifty-fifty chance of either losing $300 or winning $500?

Would you accept the chance? What if you were asked this question:
Would you prefer to keep your checking account balance of $2,000 or to accept a fifty-fifty chance of having either $1,700 or $2,500 in your account?

Once again, the two questions pose the same problem. While your answers to both questions should, rationally speaking, be the same, studies have shown that many people would refuse the fifty-fifty chance in the first question but accept it in the second. Their different reactions result from the different reference points presented in the two frames. The first frame, with its reference point of zero, emphasizes incremental gains and losses, and the thought of losing triggers a conservative response in many people’s minds. The second frame, with its reference point of $2,000, puts things into perspective by emphasizing the real financial impact of the decision.

>>What can you do about it?

A poorly framed problem can undermine even the best-considered decision. But any adverse effect of framing can be limited by taking the following precautions:
•Don’t automatically accept the initial frame, whether it was formulated by you or by someone else. Always try to reframe the problem in various ways. Look for distortions caused by the frames.
•Try posing problems in a neutral, redundant way that combines gains and losses or embraces different reference points. For example: Would you accept a fifty-fifty chance of either losing $300, resulting in a bank balance of $1,700, or winning $500, resulting in a bank balance of $2,500?
•Think hard throughout your decision-making process about the framing of the problem. At points throughout the process, particularly near the end, ask yourself how your thinking might change if the framing changed.
•When others recommend decisions, examine the way they framed the problem. Challenge them with different frames.

The Estimating and Forecasting Traps

Most of us are adept at making estimates about time, distance, weight, and volume. That’s because we’re constantly making judgments about these variables and getting quick feedback about the accuracy of those judgments. Through daily practice, our minds become finely calibrated.

Making estimates or forecasts about uncertain events, however, is a different matter. While managers continually make such estimates and forecasts, they rarely get clear feedback about their accuracy. If you judge, for example, that the likelihood of the price of oil falling to less than $15 a barrel one year hence is about 40% and the price does indeed fall to that level, you can’t tell whether you were right or wrong about the probability you estimated. The only way to gauge your accuracy would be to keep track of many, many similar judgments to see if, after the fact, the events you thought had a 40% chance of occurring actually did occur 40% of the time. That would require a great deal of data, carefully tracked over a long period of time. Weather forecasters and bookmakers have the opportunities and incentives to maintain such records, but the rest of us don’t. As a result, our minds never become calibrated for making estimates in the face of uncertainty.

All of the traps we’ve discussed so far can influence the way we make decisions when confronted with uncertainty. But there’s another set of traps that can have a particularly distorting effect in uncertain situations because they cloud our ability to assess probabilities. Let’s look at three of the most common of these uncertainty traps:

The overconfidence trap.

Even though most of us are not very good at making estimates or forecasts, we actually tend to be overconfident about our accuracy. That can lead to errors in judgment and, in turn, bad decisions. In one series of tests, people were asked to forecast the next week’s closing value for the Dow Jones Industrial Average. To account for uncertainty, they were then asked to estimate a range within which the closing value would likely fall. In picking the top number of the range, they were asked to choose a high estimate they thought had only a 1% chance of being exceeded by the closing value. Similarly, for the bottom end, they were told to pick a low estimate for which they thought there would be only a 1% chance of the closing value falling below it. If they were good at judging their forecasting accuracy, you’d expect the participants to be wrong only about 2% of the time. But hundreds of tests have shown that the actual Dow Jones averages fell outside the forecast ranges 20% to 30% of the time. Overly confident about the accuracy of their predictions, most people set too narrow a range of possibilities.

Think of the implications for business decisions, in which major initiatives and investments often hinge on ranges of estimates. If managers underestimate the high end or overestimate the low end of a crucial variable, they may miss attractive opportunities or expose themselves to far greater risk than they realize. Much money has been wasted on ill-fated product-development projects because managers did not accurately account for the possibility of market failure.

The prudence trap.

Another trap for forecasters takes the form of overcautiousness, or prudence. When faced with high-stakes decisions, we tend to adjust our estimates or forecasts “just to be on the safe side.” Many years ago, for example, one of the Big Three U.S. automakers was deciding how many of a new-model car to produce in anticipation of its busiest sales season. The market-planning department, responsible for the decision, asked other departments to supply forecasts of key variables such as anticipated sales, dealer inventories, competitor actions, and costs. Knowing the purpose of the estimates, each department slanted its forecast to favor building more cars—“just to be safe.” But the market planners took the numbers at face value and then made their own “just to be safe” adjustments. Not surprisingly, the number of cars produced far exceeded demand, and the company took six months to sell off the surplus, resorting in the end to promotional pricing.

Policy makers have gone so far as to codify overcautiousness in formal decision procedures. An extreme example is the methodology of “worst-case analysis,” which was once popular in the design of weapons systems and is still used in certain engineering and regulatory settings. Using this approach, engineers designed weapons to operate under the worst possible combination of circumstances, even though the odds of those circumstances actually coming to pass were infinitesimal. Worst-case analysis added enormous costs with no practical benefit (in fact, it often backfired by touching off an arms race), proving that too much prudence can sometimes be as dangerous as too little.

The recallability trap.

Even if we are neither overly confident nor unduly prudent, we can still fall into a trap when making estimates or forecasts. Because we frequently base our predictions about future events on our memory of past events, we can be overly influenced by dramatic events—those that leave a strong impression on our memory. We all, for example, exaggerate the probability of rare but catastrophic occurrences such as plane crashes because they get disproportionate attention in the media. A dramatic or traumatic event in your own life can also distort your thinking. You will assign a higher probability to traffic accidents if you have passed one on the way to work, and you will assign a higher chance of someday dying of cancer yourself if a close friend has died of the disease.

A dramatic or traumatic event in your own life can also distort your thinking.

In fact, anything that distorts your ability to recall events in a balanced way will distort your probability assessments. In one experiment, lists of well-known men and women were read to different groups of people. Unbeknownst to the subjects, each list had an equal number of men and women, but on some lists the men were more famous than the women while on others the women were more famous. Afterward, the participants were asked to estimate the percentages of men and women on each list. Those who had heard the list with the more famous men thought there were more men on the list, while those who had heard the one with the more famous women thought there were more women.

Corporate lawyers often get caught in the recallability trap when defending liability suits. Their decisions about whether to settle a claim or take it to court usually hinge on their assessments of the possible outcomes of a trial. Because the media tend to aggressively publicize massive damage awards (while ignoring other, far more common trial outcomes), lawyers can overestimate the probability of a large award for the plaintiff. As a result, they offer larger settlements than are actually warranted.

>>What can you do about it?

The best way to avoid the estimating and forecasting traps is to take a very disciplined approach to making forecasts and judging probabilities. For each of the three traps, some additional precautions can be taken:
•To reduce the effects of overconfidence in making estimates, always start by considering the extremes, the low and high ends of the possible range of values. This will help you avoid being anchored by an initial estimate. Then challenge your estimates of the extremes. Try to imagine circumstances where the actual figure would fall below your low or above your high, and adjust your range accordingly. Challenge the estimates of your subordinates and advisers in a similar fashion. They’re also susceptible to overconfidence.
•To avoid the prudence trap, always state your estimates honestly and explain to anyone who will be using them that they have not been adjusted. Emphasize the need for honest input to anyone who will be supplying you with estimates. Test estimates over a reasonable range to assess their impact. Take a second look at the more sensitive estimates.
•To minimize the distortion caused by variations in recallability, carefully examine all your assumptions to ensure they’re not unduly influenced by your memory. Get actual statistics whenever possible. Try not to be guided by impressions.

Forewarned Is Forearmed

When it comes to business decisions, there’s rarely such a thing as a no-brainer. Our brains are always at work, sometimes, unfortunately, in ways that hinder rather than help us. At every stage of the decision-making process, misperceptions, biases, and other tricks of the mind can influence the choices we make. Highly complex and important decisions are the most prone to distortion because they tend to involve the most assumptions, the most estimates, and the most inputs from the most people. The higher the stakes, the higher the risk of being caught in a psychological trap.

The traps we’ve reviewed can all work in isolation. But, even more dangerous, they can work in concert, amplifying one another. A dramatic first impression might anchor our thinking, and then we might selectively seek out confirming evidence to justify our initial inclination. We make a hasty decision, and that decision establishes a new status quo. As our sunk costs mount, we become trapped, unable to find a propitious time to seek out a new and possibly better course. The psychological miscues cascade, making it harder and harder to choose wisely.

As we said at the outset, the best protection against all psychological traps—in isolation or in combination—is awareness. Forewarned is forearmed. Even if you can’t eradicate the distortions ingrained into the way your mind works, you can build tests and disciplines into your decision-making process that can uncover errors in thinking before they become errors in judgment. And taking action to understand and avoid psychological traps can have the added benefit of increasing your confidence in the choices you make.

A version of this article appeared in the January 2006 issue of Harvard Business Review.

John S. Hammond is a consultant on decision making and a former professor of Harvard Business School in Boston.
Ralph L. Keeney is a professor at Duke University’s Fuqua School of Business in Durham, North Carolina.
Howard Raiffa is the Frank Plumpton Ramsey Professor of Managerial Economics (Emeritus) at Harvard Business School. They are the authors of Smart Choices: A Practical Guide to Making Better Decisions (Harvard Business School Press, 1998).

How to Choose a Statistical Model

Aureus Analytics Blog 

PMML – A good citizen of the Analytics Ecosystem


As a data science professional how would you respond to the question, “How familiar are you with PMML?”

Your choices are:

–        “I have no idea what PMML is”

–        “I have heard and read about PMML”

–        “I have played around with PMML”

–        “I have used PMML in my projects”

I have seen many responses to the question and the most popular one was “I have no idea what PMML is”.

Is this surprising? I guess, if we look at the distribution of people who responded this way, most of them were somewhat removed from the task of model deployment. One might say that the response is normal since they are only ones responsible for the model-building process. However, it is quite common for a model development team to also assume the task of model deployment into production. Even if they are not directly involved in the deployment, they will most likely have to guide the IT team. Herein lies the unfortunate aspect. Not knowing what PMML is will lead the deployment team into an awkward, inefficient and difficult production process. It is time-consuming and usually costly. This is the reason why people in the data science field should be aware of what PMML is and how it can be useful.

PMML stands for Predictive Model Markup Language. It was developed by the Data Mining Group, an independent, vendor-led committee. PMML provides an open standard for representing data mining models. In this way, models can easily be shared between different applications by avoiding proprietary issues and incompatibilities. Currently, all major commercial and open-source data mining tools already support PMML. Check out R’s PMML package.

PMML is an XML-based language which follows a very intuitive structure to describe data pre- and post-processing as well as predictive algorithms. Not only does PMML represent a wide range of statistical techniques, but it can also be used to represent input data as well as the data transformations necessary to turn raw data into meaningful features.

Consider now that if there were an application that can process a PMML model and execute it in any production environment—common these days are Hadoop, Spark, Storm, Teradata, Greenplum, Oracle, IBM Pure Data, AWS, Azure, etc.—we would have a very clean and simple model deployment environment that “connects” many different modeling tools to different production systems. I think this is one way to get every component of the analytics ecosystem to work together without obstruction. This is the reason why I titled this blog “PMML – A Good Citizen of the Analytics Ecosystem.”

Actually, this process doesn’t need to be restricted to just model deployment into production. It applies equally well to model deployment into test environment. For example, one could use open-source R to develop a model on a reasonable sample size and apply the model in its PMML form against very large data/Big Data in the test system.

The simplicity of the PMML approach can mean making your analytical model available to your business users in a day or less, versus in months.

What was your answer to the question?

*** Note: the diagram is a resource shared by Zementis, Inc., a leader and specialist in PMML model deployment



The world is generating more than 2.5 billion gigabytes of data every day and 80 percent of it is “unstructured”— everything from images, video, and audio to social media and a blizzard of impulses from embedded sensors and distributed devices. This massive volume underscores why many organizations are racing to find experts who can bring strategic vision to big data analysis and help the business derive insights that form the basis for competitive advantage.


1: Understand your business objectives

2: Have a clear strategy for Hadoop

3: Plan to building a data-driven culture

4: Choose the right platform

5: Start small




ProtonMail: Why Switzerland?

  • No advertisers.
  • No governments.
  • No third parties.
  • Only you have access to your inbox – this means no advertisers, governments, or third parties gaining access to your data. With ProtonMail, you can ensure private communications stay private.
  • Designed for Productivity
    Our next generation inbox is designed to help you work faster while also being a pleasure to use. Every detail from toolbar layout to button placement, is optimized to help you better read, organize, and send email.
  • Expiring Emails
    Make your emails disappear when you want them to. You control who has permission to read your emails.
  • Anonymity
    No email or phone number required to create your account. No IP logs which can be tied to your username. We respect your right to privacy.
  • Swiss Security
    In addition to automatic end-to-end encryption, we provide maximum physical security for your data. ProtonMail is hosted in a former military command center deep inside the Swiss alps.
  • The Future of Email
    Encrypted communications is the future. Get a sneak peak at what the next generation of email looks like today.

Sign Up for ProtonMail  https://protonmail.ch


A question we often get asked is, Why is ProtonMail based in Switzerland and are there any real advantages?

We believe there are and in this article, we will share why. The first thing that comes to mind is that Switzerland is outside of US and EU jurisdiction. Unless you host your servers on a boat in international waters, you will need to be under some legal jurisdiction and in the post-Lavabit environment, this choice is particularly important. A common misconception is that the EU offers more legal protection than the US, but many of the same surveillance directives that exist in US law also have EU counterparts, in particular, German law may actually offer less legal protection than American law.

Switzerland however, is NOT part of the EU (a fact they like to remind the EU of every once in a while), and Switzerland applies a very different set of privacy laws.  In the US and EU, gag orders can be issued to prevent an individual from knowing they are being investigated or under surveillance. While these type of orders also exist in Switzerland, the prosecutors have an obligation to notify the target of surveillance as soon as possible, and the target has an opportunity to appeal in court. There are no such things as National Security Letters and all surveillance requests MUST go through the courts (this is not the case in Germany). Furthermore, while Switzerland is party to international assistance treaties, such requests for information must hold up under Swiss law which has much stricter privacy provisions.

Nearly every country in the world has laws governing lawful interception of electronic communications. In Switzerland, these regulations are set out in the Swiss Federal Act on the Surveillance of Postal and Telecommunications Traffic (SPTT) last revised in 2012. In the SPTT, the obligation to provide the technical means for lawful interception is imposed only on Internet access providers, so ProtonMail, as a mere Internet application provider, is completely exempt from the SPTT’s scope of application. This means that under Swiss law, ProtonMail CANNOT be compelled to backdoor our secure email system. Furthermore, any attempt to extend the SPTT will inevitably fail because the Swiss public is strongly opposed to any extension and an extension could be subject to a public referendum.

This combination of factors means that a Lavabit like situation cannot occur with ProtonMail. However, ProtonMail has taken the Lavabit concept one step further and actually does not even possess the keys required to decrypt user data. As a result, even if ProtonMail was forced to turn over all our computer systems, user data is still safe.

We believe that comprehensive security can only be achieved through a combination of technology and legal protections and Switzerland provides the optimal combination of both. By coupling Switzerland advanced IT infrastructure with its unique legal environment, ProtonMail can deliver a service that is both reliable and secure.

For more information about internet surveillance in Switzerland and requests for information made to ProtonMail, please view our Transparency Report.

MLcomp is a free website for objectively comparing machine learning programs across various datasets

MLcomp is a free website for objectively comparing machine learning programs across various datasets for multiple problem domains.

Do a comprehensive evaluation of your new algorithm.
Upload your program and run it on existing datasets. Compare the results with those obtained by other programs.
Find the best algorithm (program) for your dataset.
Upload your dataset and run existing programs on it to see which one works best.
How do I learn more? Quick Links:
Sign up for a free account!