Definitional Test of Complex Systems

Back in the spring and summer of 2010 I posted a series of discussions about the need for complexity scientists to do a better job of comprehensively testing the empirical utility of their definitions--see, for example, one of the posting by clicking here.  My main argument was that:

1. Most complexity science today explores only specific aspects of complex systems, such as emergence or network properties.
2. While only specific aspects are explored, these same scientists assume the full definition upon which they rely to be true in terms of their topic of study, but without empirical test.
3. The testing I recommend is not about determining if a topic is a complex system, which is useless as most things are complex systems.  Instead, testing should focus on the empirical and theoretical utility of the definition used.  In other words, does the definition yield new insights that could not otherwise have been obtained?
4.  The testing I recommend should also link complexity method with definition.  In other words, scientists need to explore how complexity methods (in particular, computational modeling, case-based modeling, qualitative method, etc) help to determine/demonstrate the empirical utility of defining a topic as a complex system.


At the end of my series of posts I argued that some sort of formal test was necessary that scholars could use to conduct such as test.   Well, a year and a half later, here is our Definitional Test of Complex Systems.


The Definitional Test of Complex Systems:

The DTCS is our attempt at an exhaustive tool for determining the extent to which a complex system's definition fits a topic.  The DTCS is not, however, a standardized instrument.  As such, we have not normed or validated it.  Instead, it is a conceptual tool meant to move scholars toward empirically-driven, synthetic definitions of complex systems.  To do so, the DTCS walks scholars through a nine-question, four-step process of review, method, analysis, and results---see Table 2 above. 

The DTCS does not seek to determine if a particular case fits a definition; instead, it seeks to determine if a definition fits a particular case.  The challenge in the current literature is not whether places are complex systems; as it would be hard to prove them otherwise.  Instead, the question is: how do we define the complexity of a topic?  And, does such a definition yield new insights?  Given this focus, Question 9 of the DTCS functions as its negative test, focusing on three related issues: the degree to which a definition (a) is being forced or incorrectly used; (b) is not a real empirical improvement over conventional theory or method; or (c) leads to incorrect results or to ideas already known by another name.  Scholars can modify or further validate the DTCS to examine its further utility.  Let us briefly review the steps of the DTCS:

STEP 1: To answer the DTCS's initial five questions, researchers must comb through their topic's literature to determine if and how it has been theorized as a complex system.  If such a literature does exist, the goal is to organize the chosen definition of a complex system into its set of key characteristics: self-organizing, path dependent, nonlinear, agent-based, etc.  For example, if our review of the community health science literature, we identified nine characteristics.  If no such literature exists, or if the researchers choose to examine a different definition, they must explain how and why they chose their particular definition and its set of characteristics, including addressing epistemological issues related to translating or transporting the definition from one field to another. 

STEP 2: Next, to answer the DTCS's sixth question, researchers must decide how they will define and measure a definition and its key characteristics.  For example, does the literature conceptualize nonlinearity in metaphorical or literal terms?  And, if measured literally, how will nonlinearity be operationalized?  Once these decisions are made, researchers must decide which methods to use.  As we have already highlighted, choosing a method is no easy task.  So, scientists (particularly those in the social sciences) are faced with a major challenge: the DTCS requires them to test the validity of their definitions of a complex system, but such testing necessitate them to use new methods, which many are not equipped to use.  It is because of this challenge that, for the current project, we employed the SACS Toolkit, which we discuss next.  First, however, we need to address the final two steps of the DTCS.

STEP 3: Once questions 1 through 6 have been answered, the next step is to actually conduct the test.  The goal here is to evaluate the empirical validity of each of a definition's characteristics, along with the definition as a whole.  In other words, along with determining the validity of each characteristic, it must be determined if the characteristics fit together.  Having made that point, we recognize that not all complexity theories (particularly metaphorical ones) seek to provide comprehensive definitions; opting instead to outline the conditions and challenges. Nonetheless, regardless of the definition used, its criteria need to be met.

STEP 4: Finally, with the analysis complete, researchers need to make their final assessment: in terms of the negative test found in question 9 and the null hypothesis of the DTCS, to what extent, and in what ways is (or is not) the chosen definition, along with its list of characteristics, empirically valid and theoretically valuable?