post-autistic economics review
Issue no. 31, 16 May 2005
article 4

 

 

      issue 31 contents                            PAE Review index                               home page

 

 

 

Symposium on Reorienting Economics (Part IV)

 

 

Reorienting Economics Through Triangulation of Methods

Paul Downward  and Andrew Mearman
(Loughborough University and University of the West of England, UK)


© Copyright: Paul Downward and Andrew Mearman 2005

 

Introduction

 

Tony Lawson’s widely acclaimed book Economics and Reality (1997) provided a systematic ontological critique of mainstream economic analysis. The critique focussed upon the inappropriateness of the ‘deductivist method’ of neoclassical economics, which implicitly assumes a closed-system ontology, and the open-system nature of reality. Deductivism is an approach that invokes covering laws of explanation, whether or not these are derived from formally deductive or inductive premises. This method in turn assumes a closed-system ontology, in which strict regularities of the type ‘if event X, then event Y’ will occur. For Lawson, the emphasis on mathematical modelling and econometric testing are evidence of this closed-system approach. Further, the use of econometric testing, so defined, is an example of empirical realism – reflecting a belief in a flat ontology comprising only observations of events.

 

While essentially a critique, Economics and Reality also advocated retroduction – the movement in thought from a phenomenon of interest to the mechanism(s) which (at least in part) caused it – as the appropriate logic of inference for an open system. The reasons are broadly two fold. First, cause cannot be associated with the constant conjunction of events in a non-experimental open-system, but rather the emergence of events out of causal mechanisms which draw upon human agency and structures. Using this method to gain an  understanding of events, the researcher then needs to establish the mechanisms that gave rise to them. Second, our empirical observation of actual events and the mechanisms that produce them will be related transfactually; i.e., the causal mechanisms exist and may operate irrespective of the observed events. Consequently our understanding of them will be fallible, a fact that follows not only from the complex codetermination of events, but also because of the hermeneutic issues associated with interpreting and communicating our understanding of causes.

 

For Lawson, retroduction is likely to be achieved most successfully through contrast explanation, a method based on the premise that research should be directed towards explaining ‘surprising’ breakdowns in patterns of events, because here changes in causal mechanisms may be more apparent. Contrast explanation thus shares the same structure as experimental reasoning but, of course, reflects an open-system context in which causal mechanisms are not isolated and stable.

 

One criticism of Economics and Reality is that its practical guidance to economists remains limited, beyond informing them what they ought not do. Reorienting Economics (2003) also emphasises the ontological critique of mainstream method, although in cautious terms. However, there is an expanded account of ‘contrast explanation’ and a discussion of the merits of borrowing biological metaphors in producing social explanation. Lawson demonstrates that heterodoxy’s appeal to evolutionary biological metaphors as opposed to mathematical systems has promise, but should be tempered by concern with the ontological relevance of mathematics to social systems. Lawson then focuses upon a discussion of the broad, often implicit, ontological similarities between heterodox approaches to economics and critical realism. As such they offer possibilities for redefining economics in a much more pluralistic way.

 

 We believe that there is a need for practical guidance in conducting research projects informed by critical realism, and in this paper we argue, drawing upon ongoing work, for the adoption of triangulation as a principle of research design. Triangulation is commitment in research design to investigation and inference via multiple methods which are not placed in any a priori hierarchy. We argue that triangulation allows retroduction to be made operational, facilitates pluralism – rather than rejecting outright entire methods – and allows economics to be reorientated towards other social sciences, as Lawson recommends (Downward and Mearman (2004a, 2004b, forthcoming). Thus, triangulation can be an important step towards fulfilling Lawson’s project.

 

 

Triangulation

 

In social research in its broadest sense, and moving away from the spatial origins of the metaphor, triangulation implies combining together more than one set of insights in an investigation. Denzin (1970) offers a taxonomy of triangulation: 1) data triangulation – the combination of different data types; 2) investigator triangulation – the combination of insights from different investigators; 3) theoretical triangulation – the combination of different theoretical perspectives; and 4) methodological triangulation – the combination of different methods. Denzin further distinguishes between within-method triangulation, i.e., combining different cases of the same method; and between-method triangulation, i.e. combining studies of different methodologies. Clearly, between-method triangulation is more radical, because it could involve, for example, an econometric methodology in combination with an ethnographic method. We have argued elsewhere that Critical Realism can create a philosophical basis for triangulation.

 

Triangulation is common in other applied social sciences (see below) but not in economics. There is some use of triangulation by professional (including government) economists, such as those at the Bank of England, but its use by academic economists is rarer; and when used it tends to be for pragmatic reasons. A common reason for combination of data types is that the main source of data is incomplete. For example, the Bank of England uses survey and anecdotal evidence from its Agents as the most up to date picture of economic conditions. This compensates for the inevitable lag in official quantitative data. Economists cite epistemological reasons for triangulation, such as the fallibility of knowledge, relatively rarely.

 

In the literature of social science research methods there is discussion of the basis for triangulation. It is common to reject triangulation for ontological reasons. Positivist social scientists (or those influenced by positivism) tend to hold that methods should only be used which conform to positivist principles, such as objectivity, observability, and precision. All of these principles are met by quantitative data. For interpretivists, the opposite position is taken. Interpretivism includes hermeneutic concerns that social phenomena are intrinsically meaningful; that meanings must be understood; and that the interpretation of an object or event is affected by its context. Moreover, for the above reasons, meanings cannot be measured, counted or understood. Unsurprisingly, interpretivist approaches tend to focus on the limitations of quantitative analysis in the social arena. Silverman (1993) offers a typical example of the interpretivist approach. Silverman argues that quantitative methods retain a positivist perspective in which data collectors basically follow established protocols and data providers simply reveal aspects required of the protocol as ‘objective’ entities. In contrast, qualitative methods are ‘interactionist’ and reflect the interviewer creating the interview context and the interviewee engaging in a dialectic with the definition of the situation, so that the research reflects social relationships which are inherently subjective and not objective. On this basis Silverman rejects quantitative methods as inappropriate to social research.

 

It remains the case that the justification for triangulation in social science tends to emphasise pragmatism. This is clearly unsatisfactory for a Critical Realist, whose concern is for ontological consistency between method and material. This paper argues that such consistency can be established because triangulation is essential to retroduction. The argument proceeds by considering alternative research methods.

 

 

Triangulation Rather than Rejection of Methods

 

Lawson (1997) spends considerable time critically examining the use of econometrics in an open-system reality, in which event regularities are unlikely to occur. It is easy to form the impression from Lawson’s work that econometrics is being rejected as inappropriate practice. Lawson (1999) clarifies this position, arguing that econometrics is perfectly permissible providing that it is used in closed systems; however, given that these are held to be unlikely to occur, econometrics is likely to have extremely limited use. Downward and Mearman (2002) examine such Critical-Realist objections and find them unsustainable. If methods are to be rejected because they impose some degree of closure on an open reality, then in fact, all methods so far proposed must also be rejected. Any method which supposes that an entity remains fixed for long enough for it to be identified as an object of study imposes closure on an open reality. Thus quantitative analysis involves closure, because the act of quantification involves the assumption of qualitative invariance across subjects. Likewise, if qualitative investigation is concerned with collating insights and offering stylised interpretations and narratives, this assumes qualitative invariance – or, in Critical-Realist terminology, intrinsic closure. Contrast explanation is thus challenged because, first, it tends to involve quantitative contrasts; second, because it makes assumptions about what are surprising, abnormal or significant instances, which in turn presupposes a notion of a normal deviation from a trend or fixed pattern, and finally because qualitative endeavours to explain the contrasts are not fundamentally distinct in terms of maintaining a degree of closure. However, this is not necessarily problematic per se. As Mearman (forthcoming) notes, a Critical-Realist abstraction necessarily involves a focus on what is real and essential to the temporary exclusion of other factors, regarded as transient or insignificant. Moreover, by abstracting from other factors, one assumes that those factors are behaving in particular, consistent ways.

 

Therefore, we argue that for practising economists, Lawson’s arguments are potentially too destructive, leaving no techniques available for concrete analysis. In contrast, as reality comprises parts which have different degrees of openness, these should be explored by methods which also exhibit different degrees of openness: for example, statistical inference requires more closure than statistical estimation.

 

The above discussion can be viewed as reflecting (an aspect of) the fallibilism of all knowledge claims. More generally, fallibilism of theory tends to have two geneses: ontological and epistemological. Epistemologically, humans have limited computational capacity to deal with the world, even one of a simple structure, particularly when faced with multiple meanings and incommensurability. Thus, no one theory is likely to be able to capture adequately these aspects. Furthermore, if the object of enquiry (and by extension, the world) is highly complex, it is extremely difficult for any theory to capture its aspects adequately. This concern applies when the social world is held to comprise open systems, in which, for example, the future is open, because current agency affects future structures in unpredictable ways. Even for a super computer, society is complex and unpredictable. In this light, no one model should be relied upon to give an accurate picture of the object of enquiry.1 Rather, methodological triangulation requires that in any act of inference to causal claims about an object of enquiry,  two (or more) methods with different degrees of openness, are combined.. Simultaneously,  triangulation provides the researcher with a means of inference which addresses the concern over open systems and avoids the rejection of methods. Thus, triangulation offers a way out of the  practical impasse found in Reorienting Economics and in Economics and Reality..

 

 

Triangulation and Retroduction

 

As we have argued elsewhere, a Critical-Realist position can provide an ontological justification for triangulation; and triangulation is a necessary element of the logic of retroduction and hence is crucial for operationalising Critical Realism (Downward and Mearman, 2002, forthcoming).

 

A central element of the Critical-Realist programme is that every paradigm (and its associated methods and theories), has an underlying ontology. Thus, the link between a method or theory and its ontology cannot be avoided (Danermark, et al (2002: 152-3). How far do other philosophical positions meet this criterion? Above it was argued that both positivist and interpretivist approaches make a strong pre-commitment to a particular ontology that explicitly rules out the need for (methodological) triangulation. Specifically, positivism and interpretivism both reject methods which do not meet their ontologies. Consequently, one might imagine that both meet the Critical-Realist criterion.Nevertheless, Critical Realists criticise both perspectives because each has empirical realist foundations: the empirical is the basis for, and in fact constitutes, reality. For example, positivism embraces an inductive view of explanation to which value-free or observation of objective reality is crucial. Furthermore, either informally or formally, through statements of initial conditions and assumptions, deduced consequences or predictions are assessed empirically. Additionally, as noted earlier, for Critical Realists, there is a factor common to deduction and induction and which characterises their essential logic: explanations are presented in the form of ‘covering laws’ which commits an epistemic fallacy, i.e., to conflate what is experienced with what exists. The conception or knowledge of phenomena manifest in the theorist’s ideas and arguments is treated as logically equivalent to the phenomena under review. In this sense, knowledge is presented as being effectively complete. This is either in a literal or positivist sense in which theories directly represent an external world, or in the ‘idealist’ or interactionist sense that the world is merely the proposed collection of ideas, whether these are deductive constructs or subjectively specific concepts.

 

Thus, from the Critical-Realist perspective, both quantitative and qualitative analysis (either based on phenomenology or on hermeneutics) commit an epistemic fallacy; that is, they conflate the subject and object of analysis through the invocation of covering laws. Therefore, the traditional quantitative/qualitative choice can be viewed as unnecessary and, moreover, reflects fallacious thinking from a critical realist perspective. This argument opens up the possibility of triangulation.

 

Critical Realists also espouse an ontology of depth realism, which has two prongs: realism and depth. Realism holds that the nature of the object drives the process of research. Depth implies that specific types of methods are necessary for (social) scientific investigation. Critical Realism implies that, because the objects of social science are inherently complex and have many important aspects which belong in various categories, it is not possible to capture some of the broader aspects of the objects with narrow methods (for instance counting the frequency of the entity). Furthermore, because, according to Critical Realists, agency is absolutely crucial, so people’s motives cannot be ignored; and because reasons are causes, it is essential to explore these different concepts in analysis. Moreover, because reasons are causes, Critical Realists reject the traditional equation of, on the one hand, qualitative equals exploratory or descriptive; and on the other, quantitative equals explanatory methods (Danermark, et al, 2002: 163). Indeed, Critical Realists share with pragmatists the view that to associate qualitative methods with hermeneutics (or whoever) and quantitative analysis with positivism (or whoever) is limiting – instead they are looking to transcend the dichotomy. Also, research is governed by the need to get to the bottom of a question – i.e., to uncover generative mechanisms – and whichever are the best methods to use should be used (Danermark, et al: 162-4). However, unlike pragmatism, realism has an ontological grounding to that position.

 

The connection between the ontology and the method – added to the belief that reality is stratified – can be argued to lead some Critical Realists to abandon the strict dual of quantitative/qualitative in favour of one in terms of intensive versus extensive research design (Sayer, 1992: ch.9; Danermark, et al: 2002, ch. 6). The former design is what is typically thought of as a characteristic of social science. Research explores the contextual relations surrounding a particular unit of analysis (i.e., qualitative research). This is as opposed to the latter research design which emphasises the formal relations of similarity between units of analysis, that is, it produces taxonomic descriptions of variables (i.e., quantitative research).

 

However, this seems to reinstate the old dual despite the two types of method being treated as complementary empirical procedures with complimentary strengths and weaknesses. Thus it is typically argued that the causal insights from extensive research will be less. Moreover, one is reminded that the validity of the (qualitative) analysis of cases does not rely upon broad quantitative evidence. In this sense the traditional view put forward for triangulation as validating qualitative insights is not necessarily applicable.

 

The focus on intensive/extensive research designs in combination seems necessary because of the nature of reality. A number of authors (for example, Olsen, 2003) have talked in terms of a zoom-lens approach, in which the investigator necessarily zooms in to focus on critical detail of a case (intensive research) but, for various reasons (including the often necessary relations between an object and other objects), the investigator must then zoom out, to get a wider sense of the object’s context. Yet, the ontological basis for making such claims seems unclear.

 

In a number of papers we address these issues and argue that combining methods is central to retroductive activity. The following discussion briefly restates these arguments.

 

One of the consequences of the above discussion is that ‘quantitative’ and ‘qualitative’ approaches are not a dual: they overlap to a degree in underlying logic and can also refer to the same objects of analysis. They can share, or be conditioned upon, the same ontological perspective and are not of necessity wedded to particular, and different, ontological presumptions. Rather, the choice of method is not paradigmatic or one of ontology, because that ontology is shared by the methods, but simply reflects the specifics of the question being asked. If the questions probe different features of a phenomenon then different methods might be needed. It remains that they focus on the same phenomenon.

 

Two conclusions follow from this discussion. The first is that different research methods can be logically employed in a triangulating strategy to reveal different features of the same reality without the presumption of being exhaustive. The second is that the Critical-Realist perspective renders a need for the triangulation to have an explicitly ontological dimension to capture related but different layers of this reality. Figure 1, adapted from Downward (2003: 298) illustrates potential options. 


 

Figure 1 Applied Critical Realism

Philosophical               Focus of                                               Applied Method

Position                       Analysis                                                                                  

Critical Realism           Cause                                                  'Qualitative'

 

Grounded Theory/Discourse Analysis etc

                                                                                    (Meaning/Categories/

Contextual Relations)

 

Retroduction                                                    Quasi-closure                                                                                                                                       Triangulation

 

                                                                                                'Quantitative'

                                                                                               

Sample Specific

                                                Univariate/Bivariate/

                                                                                                Multivariate

Parametric/Nonparametric

e.g.

Descriptive statistics

Frequencies, cross-tabulations

                                                                                                Correlations

                                                                                                Regression

Factor analysis

                                                                                                ANOVA

                                                                                                Inferential

                                                                                                Univariate/bivariate/

                                                                                                Multivariate

                                                                                                Parametric/Nonparametric

                                                                                                e.g.

Empirical Realism       Events  - - - - - - - - - - - - - - - - -  

(as causes)                                           Tests of mean/variance differences

                                                                                                Tests of association

Tests of overall and specific parameter significance

Canonical regression     

Discriminant Analysis

Cluster analysis            

 

In the first column are the two opposing positions identified in the Critical-Realist literature; Critical Realism and empirical realism. The latter is, of course, the empirical counterpart to deduction, induction and the hypothetico-deductivist model of explanation. The next column describes the focus of analysis framed within these alternative philosophical positions, which are, respectively, real causes and empirical events. In the case of empirical realism, relationships between events are purported to reveal causes in the covering law sense as indicated by the horizontal broken line linked to, say, typical statistical testing methods.

 

In contrast, Critical Realism maintains that investigating causes involves moving below the level of events through retroduction. The third column thus reveals that corresponding to what is typically identified as ‘qualitative’ research methods, the context-laden meaning of concepts, categories and relationships can be established and causal narratives constructed. However, this process of defining shared meanings and categories in essence breaks down the qualitative orientation of the research and, of necessity, begins to invoke aspects of closure in seeking a degree of generality and purporting to refer to the same object of analysis. Hence the direction of ‘quasi closure’ indicated in the column. Thus, categorising phenomena implies assuming invariant (or a degree of invariance in) qualities. In essence, defining a variable requires at least intrinsic closure.

 

Increasing the degree to which closure is invoked thus begins to ‘legitimise’ various statistical procedures. These can refer to one, two or many variables but be either sample specific or inferential in orientation, which means that a probability distribution is referred to in a parametric or non-parametric manner to make claims about the generality of the purported relationships between variables. It is, of course, clear that the degree of closure assumed increases as one moves down the column. For example referring to sample specific descriptive methods of analysis such as averages, correlations or regression implies that values of variables have consistent meanings and that these variables can be combined in a relatively constant or enduring manner as indications of, say, outcomes of causal links.

 

Probabilistic inference assumes, much more strongly, that the results carry over, in a measurable sense, to contexts beyond the sample. In this sense one is increasingly invoking the extrinsic condition of closure, and one can clearly see the strength of the assumptions underpinning the methods which appear in this part of the column. What the diagram does reveal, in entirety however, is that in general movements towards statistical methods naturally shifts attention towards ‘events’ as opposed to ‘causes’ which are in essence qualitative. On this basis one can argue that rather than revealing covering laws, in contrast statistical methods can reveal phenomena from which causal research can begin and combinations of which contribute to our understanding of the phenomena under investigation (Downward & Mearman, 2004c). Thus the final column reveals that triangulation between the methods - that is linking the insights gained from these different research methods captures the retroductive logic of critical realism. Importantly, any quantitative analysis becomes merely a scenario whose legitimacy will rest upon the robustness of the qualitative invariance invoked in causal mechanisms and, of course, the lack of influence of countervailing causes. The discovery and robustness of such causal claims, along with their implications, will of necessity always be open to revision. In this respect, probabilistic inferences are also conditional upon and should be assessed in connection with analysis of the nature of the object under investigation.

 

 

Conclusions

 

This paper has argued that Lawson’s Reorienting Economics still leaves significant lacunae in its implications for what economists, if they take Lawson seriously, are to do in their concrete research. Moreover, we interpret Lawson’s new contribution as reinforcing his previous ones on the proper role of econometrics and other ‘traditional’ methods in economics: that such methods should not be used unless in highly specific narrow circumstances. It has been argued that a commitment in one’s research design to methodological triangulation, i.e. inference via the combination of methods (informed by different and perhaps competing philosophies), can contribute several significant benefits: 1) the rejection of methods apparently implied by Lawson is avoided; 2) the self-defeating implication of (1), viz., that all empirical analysis is invalid in open systems, is also avoided; 3) triangulation is consistent with depth realism; and 4) triangulation is essential to the operationalisation of the Critical-Realist logic of retroduction.

 

Moreover, triangulation allows economics to be reoriented toward the social sciences, as Lawson recommends in Reorienting Economics. In other social sciences, and implied by Denzin’s taxonomy, triangulation is much more widespread. For example, Danermark et al (2002: 152) claim that within the sociological community the view is widely supported that there is no universal method and that there is a need for multimethodological approaches. Thus, in the applied social sciences, triangulation is common in nursing, health and education, and tourism (see, for example, Downward and Mearman, forthcoming). Yet in economics,‘scientific status’ is sought, where scientific status is thought to mean systematic explanation, shaped by empirical evidence, and arrived at via a narrow set of methods. Hence, economics is typically perceived to be closer to the ‘hard’ sciences than other social sciences because of the axiomatisation of the discipline (Hausman, 1998). Economics stands alone from other branches of social enquiry and, indeed, disciplines.

 

Thus, triangulation raises interesting questions concerning the nature of social science. If one mixes methods of research and, in so doing, attempts to bring specific disciplinary tools to the analysis then such a ‘multi-disciplinary’ approach will entail the ontological clashes discussed earlier because, by construction the different disciplines embrace different methods and, as a result, different ontologies as expressed by traditional philosophy of science. In contrast, to ‘unite’ social science, what is required is an attempt to transcend the separate disciplines to produce an ‘inter-disciplinary approach’. Social science, so defined, naturally involves triangulation, because the methods qua disciplinary boundaries are removed. It is argued in this paper that Critical Realism provides the methodological apparatus within which such a view of social science can be constructed. Aspects of the subject matter of the disciplines, if not the currently expressed nature of the disciplines, thus become branches or fields of the same domain of investigation brought together by triangulation.

 

 

Note

 

1. Bhaskar (1978: 43) supports fallibilism as concomitant with realism. However, Bhaskar (1978: 197-9) rejects fallibilism as an overarching concept, because it can be confused with judgemental relativism, i.e. the notion that all beliefs or theories have equal merit. Bhaskar prefers his concept of epistemic relativism, which allows for theories to be incorrect, but more correct than other theories. However, Sayer (2000: 20) notes that in open systems, there is always the possibility of “misattributions of causality”; i.e., fallibilism. Danermark et al (2002: 152-3)’s caution that investigators must be very careful when making inferences. They also refuse to rule out a priori any type of methods; however, this might reflect more strongly their position that any methods must be informed by the nature of the object under study. Furthermore, though, Critical Realists argue that agents’ responses are corrigible, which means that replies to questionnaires, etc. cannot be taken completely at face value. Thus, the investigator’s evaluation of a subject’s responses is an important element in research; and given the corrigibility of responses, other data types or methods should be utilised. More generally, data might also be fallible, perhaps because it is incomplete or its collection was difficult.

 

 

References

 

Bhaskar, R. (1978). A Realist Theory of Science, London: Verso.

Danermark, B., Ekstrom, M., Jakobsen, L. and Karlsson, J. Ch., (2002). Explaining Society: Critical Realism in the Social
      Sciences
, London: Routledge.

Denzin, N.K. (1970). The Research Act in Sociology, Chicago: Aldine.

Downward, P. (2003). ‘Conclusion’, in P. Downward (Ed.) Applied Economics and the Critical Realist Critique, London:
      Routledge.

Downward, P. and Mearman, A. (2002). ‘Critical Realism and Econometrics: A Constructive Dialogue with Post Keynesian
      Economics’, Metroeconomica, 53 (4): 391-415.

__________________________(2004a). ‘Triangulation and Economic Methodology’, paper presented at the conference
      of the International Network of Economic Methodology, Amsterdam, August.

__________________________ (2004b). ‘Triangulation and Economics: Reorienting Economics into Social Science’,
      paper presented to the Cambridge Realist Workshop.

__________________________ (2004c). ‘Presenting “Demi-regularities” of Pricing Behavior: the Need for Triangulation,’
      in M. Forstater and L.R.Wray (Eds.). Contemporary Post Keynesian Analysis: A Compendium of Contributions to the
      Seventh International Post Keynesian Workshop
, New York: Elgar.

_________________________ (forthcoming). ‘On Tourism and Hospitality Management Research: A Critical Realist
      Proposal’, Tourism and Hospitality Planning and Development, forthcoming.

Hausman, D. (1998). Economics, Philosophy of, in Routledge Encyclopaedia of Philosophy edited by Edward Craig.
      Routledge: London.

Lawson, T. (1997). Economics and Reality, London: Routledge.

_________ (1999). ‘Connections and Distinctions: Post-Keynesianism and Critical Realism’, Journal of Post Keynesian
      Economics
, 22 (1): 3-14.

_________ (2003). Reorienting Economics, London: Routledge.

Mearman, A. (forthcoming). ‘Critical Realism in Economics and Open-Systems Ontology: A Critique’, Review of Social
      Economy
, forthcoming.

Olsen, W. (2003). ‘Triangulation, Time and the Social Objects of Econometrics’, in P. Downward (Ed.) Applied Economics
      and the Critical Realist Critique
, London: Routledge.

Sayer, A. (1992). Method in Social Science: A Realist Approach, London: Routledge.

_______ (2000). Realism and Social Science, London: Sage.

Silverman, D. (1993) Interpreting Qualitative Data: Methods for Analysing Talk, Text and Interaction, Sage: London.

 

___________________________
SUGGESTED CITATION:
Paul Downward and Andrew Mearman
, “Reorienting Economics Through Triangulation of Methods”,  post-autistic economics review, issue no. 31, 16 May 2005, article 4 http://www.btinternet.com/~pae_news/review/issue31.htm