Issues in Cognitive Assessment in Organizational Setting - Questions and Answers

Speaker A BIOS obtained similar results to those described by Dr. J.

Speaker B That's a polite brand saying, I like to to finish what Louise was talking about before she was cut off. And the answer to that is yes. I think it's very important everybody here doesn't get left with the impression that BIOS, of which I'm a part of technology supported BIOS, just leaves it at capability assessment. We don't we follow very much the same model that Jacobs described. When we look at, we say that the capability is necessary, but of itself not sufficient.

Speaker C And the question or the integration that Owen Jacobs is showing from his military perspective in terms of what are they seeing in what surrounds the capability is very similar to what we have found. Can we go to the forward? So you see our factor analysis with civilian managers around the world is picking up very similar trends that Dr. Jacob showed. So we'll see that as we see mold increasing or the capability of potential increasing, we're picking up consulting, driving, creativity and catalyst in the team roles, challenging in the conflict handling activist in learning styles, dominance, affiliation, ingenuity, again in the personality, energy and managerial style in Zelli. The point it's not the detail of this I want to get across. It's the fact that we are looking at it in the whole. So we are totally congruent with the panel here because it is absolutely important that you give people the tools to make sense of how they see the world and the ability to then construct a path forward to do it better. In the final slide, we also look at behavioral competencies. Somebody mentioned Robert Keegan here. Extremely important work on maturity in adults. We test it with a 360. It includes inhibitors. That's just one example inaccessibility a self report saying that person is never inaccessible and everybody else finds that person totally inaccessible. How can they lead? They may have the highest capability in the group, but they can't lead. And if you can't present them with that sort of information in an easily understandable, authoritative and digestible manner, then you don't give people the wherewithal to build a part out of it. Sorry, I answered my own question, but it was a plant.

Speaker A Glenn, you referred to the need to assess new work during the interview process. What do you mean by that?

Speaker D Okay, the difference between parroting back something that I've learned versus being faced with trying to figure something out and explain it in terms of judgment and problem solving, is that the answer?

Speaker A If we don't measure capability but proctivity, why is development predictive or can it be changed?

Speaker E Okay, more heresy. Elliot, I think, was right in the sense that we develop at our own rate to increase our perception of complexity. So the progressions curves are good, all other things being equal, because unless somebody tells me what to do different, all other things will be equal. So I progress at my own rate if I have my own resources to work with. But I will tell you that we have done research that shows that structured development of thinking process skills increases the outcome on measures of capacity like the MCPA or the CPA. So yes, within some limits, I'm quite confident it's possible to increase quote capability. It requires an outside intervention because the individual is not likely on his own to stumble on what it is that increases it.

Speaker D Let me add a little bit to that. I think that that's an area that there should be some serious research done. Because the point that I made about the difference between the structural aspects of the material and the content aspects of the material, I think come to play here. So whether we would have to look differently to see if has the domain increased up to what the potential was or has the potential current potential capability changed. So I don't know that we're saying that yet.

Speaker E Second point, if I may. The issue in making and Mike, I wish you had done my slides, beautifully done. The issue is making meaning. So the question is, how do we make meaning? And we do so by reflective thinking and by structured feedback on the reflective thinking process. So this is how you go about doing it. And Mike, I think, hit on it very nicely. But those are techniques that require some kind of outside coaching and feedback in order to develop the skill of thinking reflectively to a higher level.

Speaker A Just make one comment on that because another question was.

Speaker C Is maturation a result.

Speaker F Of what you do or independent of it? The orthodoxy, certainly Elliot's belief was, as I understood it, was that you are born not into a level, but into a growth, a maturation curve. And it's not that what you do is irrelevant to it, but that unlike learning a skill which you need. You're not going to learn French, how to speak French if you don't have opportunities to interact with French people, to be trained, et cetera. But his sense was, it doesn't take any special experience for you to mature. The consistency of the growth of people across their careers which those lines were fitted to, is evidence of that. On the other hand, there certainly is a school of thought and we're know, and this is where the field, I think, is changing and things we need to look at. And then I think, theo you believe that people can be aided to move quicker along the lines.

Speaker G I don't like the word quicker.

Speaker C Well, but still.

Speaker G Quicker wouldn't be a term that I would use at that. I just felt that Jax had kind of overstated his case in the text that I read. I think it's very clear that each of us is born with limitations on our upward capacity. But I don't think it's quite as deterministic as it sounded like he was saying it was this is.

Speaker F Seemed to be yeah, this is what I mean.

Speaker A If anyone can prove anything else, that's nice.

Speaker F Yes.

Speaker H I don't think this is dogma know.

Speaker F If it turns out you can move people, this is Know Piaget, who looked at children and met much of the same way that Elliot was looking at adults and his work was grounded in it. Someone in the audience would ask Piaget, how do you move a child quicker from pre operation to operational stage? And he'd say, yes. The American question. I think we don't have to be dogmatic about this. I think we just need to look at the make a comment in terms.

Speaker I Of moving quicker, I want to caution against maybe wanting to move quicker and faster and up the curves because if you don't ground that capability in knowledge and skills and experience, you're setting yourself up for failure. Absolutely. And you and you often can't go back to get that. So you get these high flies in organizations that want to skip steps and move further and then they don't have the foundation upon which to fall back upon. And I've seen that in organizations. So just a cautionary note about moving people quicker.

Speaker F I want to move on several questions for Owen. I want to know more about how a detailed, fascinated person is negative affected in his or her capability.

Speaker E Could you say that one more time?

Speaker F I want to know more about how a detailed, fascinated person is negatively affected.

Speaker A In his or her capability.

Speaker E How does that a detailed, fascinated person what.

Speaker I I'll try and answer that. I think what I got out of Owen's slides is that somebody who's detail orientated wants to dive down into the detail and deal with the work on a concrete basis. Sorry, you can't hear me. So higher up in the levels of thinking, one has to take a broader perspective and then dive into the details that are appropriate for that particular situation. That might be a level four perspective. Level five.

Speaker G You don't want to delve in the.

Speaker I Detail, you want to have that helicopter view.

Speaker G So what we're saying is if there.

Speaker I Is a correlation between detail orientation, negative correlation between detail orientation and mode and propensity to move forward, if a person is too detailed, there's a possibility that they might want to micromanage and might want to delve too much into the.

Speaker G Detail and forget about that border perspective.

Speaker F Yeah.

Speaker E The metaphor is not seeing the forest, but rather seeing the individual trees. Someone who wants to see the individual trees doesn't get intrinsic reward from seeing the forest as a whole. It just appears to be that way. So if you go for preferences, then the individual who's more likely to score high on detail preference is more likely to have an S preference and a J preference. It's just preference. Okay.

Speaker F And maybe following up on that is the template itself an artifact of the complexity of mental process or just the proclivity to develop templates?

Speaker E No, we all develop templates. We all develop I'm a constructivist. So constructivist folks say we build internal representations of the external reality that we face. So we make sense of what those external realities are by looking for causal relationships among the elements. So if I can see the elements of the system that I'm looking at and I understand the causal relationships now I know how to pull the levers in that system, for example. So we all tend toward it's a universal tendency. I think we all tend toward making meaning of what we are seeing and hearing. Now those things are stored and they're used as an so called experience base for then understanding new situations. So I think it's not an artifact. I think it is a product in the making which helps us understand the world we live in and which we need to understand the world in which we live.

Speaker F Simple, quick one is this battery of assessment tools going to be commercially available and if so, when?

Speaker E Oh, that's a very hard question. No, maybe.

Speaker A That'S very clear.

Speaker E Of course BIOS has very similar instruments.

Speaker F Yes. Theo, are you ready for this? Okay, go ahead. How do you differentiate correlation from.

Speaker E Sensation.

Speaker F Without looking at content of information? How do you collect the data for.

Speaker G Oh, okay, well, that's a lot of questions. I don't make the claim that we don't look that the content doesn't have a role to play in the scoring system. But what I claim is that we don't care what the particular content is. That all we care about is the structure of the conceptual content. So we're analyzing the content in the same way that we analyze the explicit argument structure. Our data collection process can either take place through interview or it can take place through essay. Writing online I know that there are some differences in the interview styles that are used in the different methodologies and then the same is true in psychology. We have chosen to use probed style of interview or a probed style of essay. And the reason that we do that is that we think that because there are kind of personality differences in how forthright people are about their thinking how much they're going to tell you about their thinking processes that by probing everyone in a similar way, we give everybody kind of equal chance to show us how well they can think about a particular kind of problem. So we use a kind of open ended probed essay and we do several of these and each one is scored independently and the final score is based upon a mean of and we don't even weight the means. It's just a mean of the scores for the individual performances. The reason for doing that is because, psychometrically speaking, you increase the reliability and accuracy when you have more data points than if you have a single data point.

Speaker D I'd like to make a point on that. One of the differences with CIP is we're really attempting to find what the highest level is.

Speaker G We look for the highest. Yes, we do look for the highest, but it turns out that it doesn't make very much difference. Well, there's not that much variation across essays, usually. And we get a kind of a phase quality to the score. We get it the phase by actually taking the average. So it gives us a little bit more differentiation than we would get.

Speaker B Right.

Speaker A I want to address this to Mike. Although it starts with Theo. Theo says a five is a five. The Australians and Brits seem to see Stratum five as a higher conceptual level than North Americans.

Speaker F How can we reconcile this?

Speaker A Why do you want me to answer?

Speaker G I actually have a slide on this.

Speaker A Yeah, because we've talked about how people you find people coming out with different levels. Yeah, I mean, in large part what we feel is well, it's hard to say this to short because we have.

Speaker E Some disagreements with these things.

Speaker F We do things that we don't probe.

Speaker E So that we don't intervene.

Speaker A Surface structures, surface content. The noise that's created oftentimes in different cultures, causes us to see things as less or more complex based on our own perception of the way the world works. So when I'm working in Indonesia, it's a much different perception of structure because of my own noise versus their noise. And those have to be filtered out.

Speaker E Which is one of the reasons that.

Speaker A I like what both Glenn and these people are doing by looking at structure rather than seeing content as dependent. Not that you throw it out, you use content. But I think it's a difficult question, Herb, because that's something that we're struggling with because we're working in 27 different countries and we are seeing the perception of complexity and capability looking different in different cultures. And when we go to Russia, it's very different. When we go to Indonesia, Malaysia, Singapore, India, it's very different. So I think these things are something that we have to find a way around. And I think there's another difference that just occurs to me. Well, North America. We got requisite primarily from Elliot. And when we're looking at this stuff, we're looking at listening at right angles, primarily Brits and Australians. I think you're looking more at by us and CPA. And I think there's got to be something in the tools and how the tools might be that skills and values are different. There's no question they are. No question they are. But for me, a person that is contextual and the declarative comes to level five. And it's five wherever you find in the world.

Speaker I In our experience working around the world, sorry, the capability is the same wherever you are. It is more around that. With a culture that's built.

Speaker G I actually have done several years of research comparing different systems. And there are good psychometric models that we can use to actually address these questions and find out exactly what's going. And this is a slide that just shows some early work that I did comparing Kohlberg's moral judgment system with Armand's value of reasoning about good life with my work on education using electrical assessment. And this system just shows, this is the scale that this particular model aligns everything on based upon the performances of the same group of individuals that you see in the distribution on the left. And you can see that my system, it looks like people get the score earlier than they do on the other systems. So my system looks easier. And part of the reason my system looks easier is because of the way that I probe and also because I award the highest level score on each one. These other systems probe in different ways and they also are not scoring at the level of the core structure, they're scoring at the level of domain structure. There's a lot more noise in the specificity of the scores as well. So each one of these are like distributions, the confidence that you can have in the score that you give on that item, given that scoring system. So what we've been trying to do is tighten it up so that the levels we have more confidence in the and we see this kind of pattern across all the systems that I've looked at. So you can actually kind of analyze and figure out exactly why we're getting these different patterns.

Speaker H Glenn described. The same thing that we find is the first time you take a managerial group through this process, we end up with judgments about role size and potential that get modified about 10% over the course of the subsequent two years. What we find is that when we take the group first through role sizing assessment of current and future potential and then take them through a rigorous process of effectiveness appraisal and we ask people to calibrate people's effectiveness against others in the same size role. We find that we're able to stop those roles that have been incorrectly sized because they say, well, this person who's this effective in this role is adding much more value than the other people in the same role, but it's really value comparable to this other group in a higher size role. And it's just been a very funny serendipitous finding.

Speaker E Yes.

Speaker D I think what we tell people the first time you're going to do this, this process is probably inherently the talent pool evaluation process is probably inherently accurate to about a third of a stratum. So you go back and retest them and they're going to bounce around a third. And you throw all that data out a year later and you say, how many people have really moved? Okay? And I've seen 20%, 25% of the people which have really moved. They may have moved into a whole new stratum boundary. They may have moved by two thirds of a stratum. The assessment of them has moved. But what we attribute that to is it takes a while for the managers to get a handle on what work is. It's really over in this realm of what work is. And they really are under assigning is very common. You position the role here, but then you get in to say, well, what are the tasks associated with the role? And they're asking them to work down here. And by the time they've done this, about the second cycle, you see those adjustments being made, and you also see the impact of on the job because they've been dealing with now the question of, wait a minute, if the work is supposed to be here and this is what we're given, we need to make adjustments.

Speaker I Thank you. Where you place a person in the strata or in the mode, there are three data points that we use similar to your triangulation approach. So we look at the overall pattern of what we're hearing out of the CPA in terms of where the person currently is. We look at the career path and map the career path in terms of where are those transitions and when did they happen? And we check that out with the individual in front of us. Does this sound about right in terms of where those transitions are? And we pose a future related question to see stretching the person into a future type, more higher level thinking and see how they deal with that. So there are three data points at which you can then peg the person, whether they're low, medium, or high within that mode or that career path or that stratum.

Speaker A To Owen, do you tell the participants what level and mode they score on the MCPA? If so, how and why is this useful?

Speaker E In the feedback session, I personally try not to assign a number or a specific level. What I try to do is sense from the individual whether that person is comfortable with where he or she is in his or her position at the time. Then what I do is I speak to the individual's growth potential and what to do in order to cash in on that growth potential. What do you need to do and what experiences do you need to look for, and how can you deliberately grow? Conceptually? You Sam.