Sunday, June 24, 2007

Art and Science in Trading and Psychotherapy

Note: The following is my post to an online discussion group that had taken up the issue of art vs. science in trading:

The art/science debate/dichotomy has dogged the field of psychotherapy as well. There we've seen two stances taken toward reconciliation. The first is to identify effective treatments from controlled outcome studies and then create treatment manuals around these. Practitioners are then expected to follow the manuals with fidelity, so that treatment remains "evidence based".

The second approach is to assess outcomes among practitioners who employ subjective methods in talk therapy, relying on clinical judgment and experience. Those studies find that some therapists are consistently more effective than others, that much of their efficacy is independent of the specific treatment approaches they take, and that much of their efficacy is wrapped up in their ability to forge positive working relationships with patients.

In the first instance, insurers attempt to steer patients toward evidence-based approaches to maximize outcomes. In the second instance, insurers attempt to steer patients toward practitioners with the best outcomes. Both represent attempts to bring science to bear in a field that is highly discretionary and subjective.

I believe we see the same thing in the trading world. At hedge funds that I work for as a psychologist, we see a first approach, which is the development of automated trading systems based upon backtested historical patterns in the markets. These address the need for scientific money management by eliminating as much of the human element of trading as possible. Their mechanical trading systems are the equivalent of the therapist's evidence based treatment manual.

The second approach taken by the funds is the collection of copious metrics on traders who employ discretionary judgment. From these metrics, it is possible to determine which traders are achieving results beyond their peers and beyond what would be expectable by luck. Just as insurers track the risk-adjusted outcomes of practitioners and determine, say, who are the best heart surgeons, funds similarly trace the outcomes of their discretionary traders and allocate more capital to their all-stars.

Both of these modes strike me as legitimate applications of science. I don't think all psychologists need to work from treatment manuals, and I don't think all traders need to trade backtested systems. If, however, those psychologists and traders make claims of efficacy, the burden of proof is upon them (and the assessment of their objective outcomes).

Note that even once discretionary practitioners are found to be efficacious, that doesn't necessarily mean that they are successful because of their chosen orientations (Freudian, behavioral, etc.). Indeed, studies have found that specific orientations account for only about 10-20% of outcome variance. Specific practitioner skills and characteristics of patients account for far more variance in outcomes.

Similarly, I suspect that whether a successful trader adheres to one form of TA or another or some other analytical method accounts for surprisingly little variance in P/L outcomes. Specific trader competencies and characteristics of markets being traded may well account for the lion's share of profits. But these are meaningful questions and raising them can only further the cause of science in trading, even as it refines discretionary practice.

Added Note From Dr. Brett: Once we treat traders as trading systems and objectively measure their performance, the dichotomy between what is scientific and subjective falls away. Subjective methods in trading, as in psychotherapy, are objectively valid if they can be shown to produce results beyond those expectable by chance.

9 comments:

Lew said...

Isn't the difference between art and science one of verifiable (or not) method? A black box system's rules for trading decisions are clear for all to see, whereas the discretionary trader's decision factors on each trade may not even be known to him/her. In trying to answer the question "How do they do it?" by listening to seminars and reading, it seems to me that successful traders are not telling me all that they consider -- because they can't. It's a very difficult task which is best done one-on-one, day-in-day-out, on the line in action. A lot more communication goes on in that context than in a seminar, book, or article, which are more limited communication venues.

Brett Steenbarger, Ph.D. said...

Hi Lew,

If we consider the trader to *be* the black box, we can then treat his or her output objectively. In other words, we can objectively determine whether subjective judgment confers an edge. As you point out, the task of deconstructing the expertise of the discretionary trader is tougher, but not at all impossible. I've seen good trading systems built from studying the patterns of successful discretionary traders.

Brett

Alex said...

Brett - How would I know if my trading produces results beyond those expectable by chance. Is there a specific statistical test you suggest?

bwilhite said...

This post makes me think of "The Art of Motorcycle Maintenance"...great book.

A study of the philosophy of science (I'm talking about scholarly work here...not the popular treatment in "Art of..." although it gets you going) will show that there is much art in science also. The dichotomy art/science is not as clear-cut as many perceive, imo. Yes, semantically the terms have very different ranges of meaning...but the things is that there is much art in science, and much science in art. Science is built on metaphor and model...and where do those come from? In fact, the underpinning metaphor/model is a very strong factor in how the scientist conceptualizes and goes about his work. You could even say it defines his work for him. I wish this was more recognized on the popular level. Heck, I wish even the scientists would more openly and honestly acknowledge these issues.

...at least that's my .02 :D

Anyway, great post.

BW

Brett Steenbarger, Ph.D. said...

Hi Alex,

Great question. Sophisticated researchers would use Monte Carlo simulations to see how portfolios behave with random price changes plugged in. After many runs, you get a distribution of outcomes, and then you can see how a portfolio manager's results compare to this distribution.

At a much simpler level for the average trader, non-parametric tests (such as chi-square) could look at, say, ratios of winning to losing trades (or sizes of winners and losers) and detect differences from average.

Brett

Brett Steenbarger, Ph.D. said...

Hi BW,

I totally, totally agree: it takes much creativity to be a good scientist. It's how scientists assemble observations (the models they draw upon) that often generate the breakthroughs--

Brett

matw said...

Alex, thanks for asking so I didn't have to. I was wondering the same.

Brett, I have an issue with the Monte Carlo method you describe. By "random price changes plugged in" I assume you mean a random price

y = x + r,

where r is the random price change, and x is the historic price. If history was kind, and x yielded an above average positive result, then that result biases the Monte Carlo run if r is unbiased. In other words, the randomness is added to a particular historic event that may be far from the mean.

That is not a good test. I need to go off and ponder.

Mat

matw said...

BW,

Many scientists I know are artists, and they love it (myself included). Creativity is the key in both endeavors. IMO, what differentiates art from science is the measure. Art is far more difficult to quantify (for better and worse).

Mat

bwilhite said...

Hey MatW,

I apologize for lumping all scientists together :) I know a good few myself.

In my experience there are those who acknowledge their assumptions and underlying models, and those who do not (in any endeavor). Unfortunately, the popular media tends to portray the latter type (again in any endeavor), and consequently they *seem* to be the ones that are the majority. But undoubtedly this can lead to skewed perspectives.

I agree with you on the art...also adding 1) Science seeks to have predictive power, 2) Be repeatable (generally...it works best when it is anyway), and 3) Science works best when it deals with physical reality (because of the measurement thing...although some fields fall outside of this).

An interesting contrast in all of this...is that generally (a few exceptions) art is singular (i.e. NOT #2), and it usually doesn't care about predictive power.

I suppose it's this contrast between singularity and repeatability that is probably a lot of what people might mean when talking about trading as an art vs. as a science.

BW