The Lectical Assessment System

Speaker A My real initiation into the work of Jax only occurred about six months ago when I began to start going through requisite organization. And as I turned the pages, I was going, well yes, of course, yes, absolutely. Oh yes, that totally makes sense. And started to think, wow, we've got some commonality here and this is something that's worth exploring. It's hard for me to know where to start because every group that I speak to is different in their knowledge, different in their interests. And so I just thought that I would by showing you a picture and in this picture, basically what I'm trying to do is convey what I can do. What we can do at DTS that we don't know of anyone else who's doing what we're doing yet. And this is in the fields of psychology and also in other fields, though. But when I listened to what you're doing, Glenn, it's getting so close to what we're doing, it blows my mind, because nobody in the psychological world is really doing this kind of work. But what we can do that nobody else can do right now is because we've developed an assessment system that is completely independent of particular content. In other words, the scoring criteria that we use look through the content of a performance down to what we call a deep structure or a core structure. We can take this same instrument, which we have actually shown behaves like a ruler, and we can put it up against performances in a variety of domains. And the ruler goes from birth through adulthood, and we use it from age five until about four or five all the way through adulthood. And because we can do that, we can do things like tracking the development of an individual over the course of many years in multiple domains. So we can actually provide a profile of an individual in their development. And that's just a training that's like a tip of the iceberg, but it gives you an idea of something that can be done with a completely content independent metric. The other thing that we can do with this instrument is we can use it because it's like a ruler. We can use it to assess the level of a bunch of performances and then we can go in and we can say, okay, well, what's the relation between this level of performance and the particular content within that particular area or domain? And we can also use it to analyze the task demands of particular positions in school, in a managerial situation, in any number of contexts. So we can line up the performances of individuals with the task demands of work, for example, which you guys already know about because you're already doing it. We could also do this for schooling, and we're doing it for clinicians who are working with clients who are trying to work through problems that they're having or just simply trying to grow so that just gives you an idea of some of the kind of general power. And the reason that we can do this is because we just decided we were going to measure one thing and that we call that thing hierarchical complexity. And the term comes out of a long hundred year old tradition in psychology and cognitive developmental psychology. So a lot of people don't really understand why we would want to measure one thing. I think you've seen by seeing the picture, you've seen a little bit of why we do that. But there are some very good reasons for doing this. The first of them from my perspective, is because science progresses. You could argue that science progresses because we develop measures. We develop really good measures. And each measure that we have developed to a very fine tuned level and calibrated well, like temperature and distance and time are measuring stable kind of universal dimensions of performance that we can all agree exist out there somehow. And then at some point in the past, all of them were measured in qualitative ways. So developing measures always starts with kind of qualitative knowledge. Hot, cold, hotter, colder, hot enough to, you know, hot enough to melt ice, hot enough to melt grass. Those are qualitative. And then you develop to having a ruler that's calibrated that everybody can use and agree upon. But the other reason that you want to use rulers is because when you use a ruler, it's devoid of content, it's devoid of any kind of normativeness. It doesn't say 100 degrees is better than ten degrees, it just says 100 degrees is hotter than ten degrees. So when you have a psychological ruler that measures hierarchical complexity, the developmental dimension argue, and Jax also argued, then you can maintain objectivity with respect to particular belief systems, theoretical perspectives, cultures and contexts. So the more devotive content we get, the better we are at taking this assessment to a bunch of different contexts and being able to say a five is a five no matter what we're talking about. And we know what a five means. It means that it's at this level of hierarchical complexity. So what we do this with is an instrument called the Lectical Assessment System. And here I've shown you how we think about our levels. Interestingly enough, I've adopted Kurt Fisher's model as the model that I use. And notice here that we have tiers and levels just like Jack's. And actually also even though it looks like there's only three levels per tier, we talk about the last level of a tier being the same thing as the first level of the next tier. So we have a four. Interestingly enough, we have actually a four level model, but the way we think about it is just a little bit different. So the instrument that we use is content and context independent and it's got the five tiers, three levels per tier, but they're really four. And it's got repeating structures just like Jack's ruler and we call them singles mappings and systems. And I think to some extent it's arbitrary what you call them. It's what actually means when you're actually looking at a text or something and trying to evaluate it. So I don't really care too much about the particular knowledge. What I care about is the meaning underlying them. And then we also can analyze four phases per level and that's something that's a new capability that we've developed over the last six years. And I'm going to argue that this is exactly the same dimension identified by Jax. Now, the way that we do this is we go straight to what we call core structure. Now, Grant, I think you're getting there. You're very close. What you don't have is some of the scoring criteria that we use that help us specify a little bit better. From my perspective, historically though, most developmental assessment systems, especially the early ones, started out by kind of looking at conceptual content because that's the first thing that you see. But then what people did was they started to do these longitudinal research projects where they would take a group of a cohort of people, like 40 to 60 people and they would follow their development over several years, sometimes up to 20 years. And then they would use the longitudinal evidence to try to build up definitions for the stages. And what they started to do was get to something that we call surface structure or domain structure and work, for example. And in Jack's system, work is the domain. And so the structures that most people are talking about are the structures at the level of talking about work, what we want of work, what we wanted to do. What I saw when I was doing my graduate work was that there are all these developmental assessments out there and all of the ones that had been developed on the basis of longitudinal evidence looked the same to me. That there was something about them that I knew was the same but I had no language for it. I couldn't talk about it, I couldn't prove it. I didn't have any way of showing it at that time. So I set out to become a psychometrician and a cognitive developmental psychologist and made my mission figuring out what this underlying dimension was. It turned out a bunch of other people had already started working on this. I was able to kind of stand on the shoulders of giants and just take the thing one step further. But basically what we do is we look at two aspects of structure. We look at what we call logical structure, which is what Glenn is looking at and we do it in a fairly similar way to the way that you're doing it though we have slightly different constructs but we also look and this is what's different about what we do. We look at the conceptual structure and that sounds a little funny when we're looking at concept. There's no structure there. It's just a single concept there. But what we do to get at the conceptual structure is we look at a performance and then we say, well, within this argument that this person's making, there are all of these conceptual elements, which ones look like they're maybe the most sophisticated or abstract elements. And so we pick out one or two and we say, okay, well, given the overarching meaning that this person's conveying in this argument, what is the simplest possible thing they could mean by this particular concept? And we take that definition and we analyze the structure of the definition. So we analyze the conceptual structure and we analyze the logical structure. We put them together and we can determine which level that performance is at with a high degree of accuracy. In addition to that, we also have done research that has shown us how individuals develop within levels over time. So the hierarchical structure is this, and then there's a horizontal complexification that happens as people spend more time in a level. And so we can do phase scoring where we can score a person based upon how elaborated their conceptions are within a level. So the fine tuning is four phases per level. So we've talked a little bit about the ruler. I don't know if I need to go over this again, but the properties of good rulers are that they're based on unimental constructs, that there's a system for assessing the amount of the thing, that there's a method for evaluating reliability and validity, and that the measure can be standardized so that everybody who uses it is measuring the same thing. So instead of having like a current kind of Tower of Babel that we've got, where we're kind of guessing about how different levels align with one another across systems, you have one basic system that underlies all the systems, so that a five is always a five. And there's a whole bunch of stuff published because I did that academic thing. So here's the way that I think that the skill levels align with jack's levels. And there are a lot of different ways that you can think about what each level looks like from a structural point of view. We use these little cubes. We find them useful. This particular cube here at the bottom, where we say single correlations represents the first kind of abstraction or abstract kind of concept or qualitative kind of concept or qualitative concept that emerges out of the previous level. So that's why it's a Cuban, not just a dot, because it's building upon earlier constructs. Then we have things that we call mapping. Two of those people are putting together two of those kinds of concepts and then we have elaborated mappings where people are chaining them and then people start to connect them with one another initially and kind of just iterative loops. And that's the emerging systems of correlations and then those become more complex and those are elaborated and then they begin to put the systems together and extract generalities from those systems that they can speak about as independent of the systems themselves. And we call those sensations and then they get more of those and then they start to connect those. And so you can see that it's a progression. And I've kind of lined those up with the Jacques levels here. So I think that's all. Let me just see one more thing. After we do scoring, then we look at the content. So we don't associate particular content with the levels as part of the definition of the levels, we can use this ruler to say well, what kind of content does get associated with this level in this particular domain, in Japan and China or in this particular context of work? So we keep that question of what content's associated with the level open and we start to just describe what that content is. And the reason this is important is because people functioning at the same level can really think differently from one another. So that it doesn't always mean exactly the same thing just because someone's at the same level. And individuals function at different levels in different knowledge domains, and this is very important. We kind of tend to think about level as something that a person has. It's a one thing, it's unitary. But in fact, all of the research that we've done shows that there's variability and the further apart the domains are in their content, the greater the variability. So a person could be really great at reasoning about reflective judgment, but you give them a workplace dilemma and they fall because they don't have the knowledge that they need to build the structures at the higher level that they can reason at in another area. Also, the reason that we do this is because looking at the content and the structures separately makes it possible for us to actually describe learning pathways. And this is another value added thing that using a ruler can provide. And we have demonstrated that this can be done in knowledge domains as diverse as physics and ethical reasoning, where we actually go out and interview people of a whole bunch of different ages from different backgrounds, and then we score them the performances for their developmental level. And then we analyze the content. And then we build up descriptions of how the particular conceptualizations develop over time within individuals. That makes it possible to say you're performing at level Y and you're already doing this and this, but you're not quite doing this and this. Here's the next thing you need to learn, which is something that people haven't been able to do with developmental assessments.