In this episode of Be EPIC, Matt is joined by Alan Ellstrand, associate dean for programs and research and professor in the Department of Strategy, Entrepreneurship and Venture Innovation at the Sam M. Walton College of Business. His research areas include corporate governance, top management teams, and executive leadership.
During their conversation, they discuss Assurance of Learning, or AoL for short, which is one of the AACSB’s standards in their Continuous Improvement Reviews. The AoL encourages business colleges to step back and ask, "Are we truly teaching the types of things that we say we are teaching?” and “Are students leaving college with the right skills, abilities, and understanding that is in alignment with the degree that they are receiving?" Alan and Matt discuss how AoL is implemented within the Walton College and how they’re closing the loop at all levels.
Edisode Transcript
[music]
0:00:08.3 Matt Waller: Hi, I'm Matt Waller, Dean of the Sam M. Walton College of Business. Welcome to BeEpic, the podcast where we explore excellence, professionalism, innovation, and collegiality, and what those values mean in business education, and your life today.
[music]
0:00:28.8 Matt Waller: I have with me today, Alan Ellstrand. He is the Associate Dean for Programs and Research in the Walton College. He's also a professor in management, and his research areas include corporate governance, top management teams, and executive leadership. And he's published his work in top-notch journals like the Academy of Management Journal, Strategic Management Journal, Organization Science, and many others. He's been in the Walton College for a long time. He has served as Department Chair for the Department of Management, Director of the MBA programs for the Walton College. He has won the Walton College Outstanding All-Around Faculty Award, and many other awards. He's very engaged, and he has been very involved in... We have a governing body called the AACSB. They do the accreditation of all of the business schools in the country and the world. They're really considered the gold standard for accreditation.
0:01:45.6 Matt Waller: And every five years we have a review. They're called the Continuous Improvement Review, CIR. And during these CIRs, there's a team, usually two deans, and then they've got an accounting department chair, sometimes a practitioner as well. And AACSB has put standards in place to make sure that business schools are delivering the value to students that we should be delivering, that's good for student, that's good for society, that's good for the academy. And these standards, there's many of them, but the one we're going to be talking about today, actually, we're talking about part of a standard. We're talking about something called Assurance of learning, AoL for short, Assurance of learning. And we put a lot of effort into it, and Alan has really headed this up. Alan, thanks for joining me today.
0:02:51.3 Alan Ellstrand: Thank you for that great introduction, Matt. That was really kind, and I am really excited to join you today to engage in a discussion about AoL. So thanks for having me.
0:03:03.2 Matt Waller: First of all, let's start out with what is AoL and why should people care about it?
0:03:11.9 Alan Ellstrand: That's a really good question, I'm glad you asked. So as you mentioned, AoL is Assurance of learning. Yeah, we have many courses in our curriculum throughout the Walton College at all levels. We've got undergrad classes, we've got master's classes, we have doctoral classes, and in those courses and programs, we teach our students certain material that we feel is very important to them, and we hope that our students will leave with certain skill sets and certain understanding when they leave the college, we confer their degree, and we feel that their degree represents that. What the AoL does it encourages or enables us to step back and say, "Are we truly teaching the types of things that we say we are in our curriculum? And do our students leave the Walton College with certain skills, abilities, and understanding that we believe is in alignment with that degree that we're conferring?"
0:04:25.7 Alan Ellstrand: Just because we say we do those things, we may not be. The Assurance of learning, it's really more of an internal look, it makes us look in the mirror and say, "You claim that you are teaching students X, Y, and Z. Are you really doing that?" We assess that really at the program level. We do use work that students have produced as part of that process, but it's not a matter of saying, "Well, we know we're teaching the students well because they graduated with a cumulative GPA of 4.0, so they must be learning it." This is really going deeper and saying, are the students actually learning what we feel they should when they get a bachelor's degree in finance, or a master's degree in accounting, or even a PhD degree in Business Administration?
0:05:25.8 Matt Waller: Within AoL, I know there's this concept of closing the loop and I know we've put a lot of effort into that. Would you mind talking about that a little bit?
0:05:35.3 Alan Ellstrand: Absolutely. Closing the loop is kind of the end game, it's what you really are going for. And as I mentioned, we make all these assessments, we collect data, determine whether or not the student work actually reflects what we hope it will, and in many cases, and in fact, in most cases, yes, the learning objectives, the competencies that were in place were actually being met by the curriculum that we have in place, and students, in turn, were producing at a high enough level that it all kind of going through that process, it all played out and it verified what we were doing. Where closing the loop really becomes important is if we have any areas where in fact we're not reaching the standards that we set, it's important that we step back, reflect on where our deficiencies might be, and take a look at the curriculum or at the assessment tool. I mean, it could be a matter of maybe the assessment tool was bad, maybe it wasn't really collecting data on what we were trying to measure, maybe it was a poor measure.
0:06:52.2 Matt Waller: So it can be a process issue like that, and we can correct that by changing the way that we collect data and assess for that learning objective or their competency, or we can say, "You know, that was a perfectly valid measure and we are just not effectively teaching that." In those cases, we can look at our curriculum and say, "You know, maybe we just need to work harder, maybe we need to include more content material in those areas, maybe we need to add a course to our curriculum in order to do this." I think a great example would be one of our learning objectives is oral communication. Now in the Walton College, all of our students take a communication class when they're freshmen, but that is kind of the last time that students have any direct curriculum in oral communication.
0:07:56.8 Matt Waller: Certainly, throughout the curriculum, they have many opportunities to present and to be critiqued, but frankly, those are in content classes, so I could be in a finance class, and at the end of class, I have to give a presentation on net price and value. The instructor may assess me or kind of take into account how effective I was as a speaker, but there was no content material on how to become a better speaker necessarily in that class. So if we feel at the end of the process that our students are not reflecting, effective oral communication skills, we may need to close the loop, say they need a second course, maybe a Business Communication course that would be targeted at building their oral communication skills. So that would be one way that we can close the loop on that.
0:08:55.2 Matt Waller: So I mentioned earlier that there's a team that comes from other universities that reviews us for the continuous improvement review. And so what happens when they come, they may say, "Here's some things you're really well," they may say, "Here are some things you're not doing so well, but you need to improve." In our case, last time, there was no criticism of our... Well, they thought we were doing a good job of it, but we still set out to improve it. I know you did in particular, you had a vision for improving it. Would you mind talking a little bit about how you've improved it since the last review that we have.
0:09:36.5 Alan Ellstrand: By not having baseline recommendations from last time, it really enabled us to take a look at it ourselves and say, "Where do we feel we're falling short" And of course, as you know, we adopted the 2020 standards, which are quite a bit different from the 2013 standards in several ways. One of those is, there's clearly a greater emphasis on indirect measures in the 2020 standards. And the college has always been really good about reaching out to external board members, members of the business community, external stakeholders of all kinds. This was not as important under the 2013 standards, it's become much more important. We were really positioned pretty well to be able to point to great examples of where we've used external feedback to inform our programs.
0:10:35.6 Alan Ellstrand: There is one really critical area that we took a look at and said, "Even though our reviewers back in 2016 didn't bring this up, it's an area that we really need to work on." And that is in getting broader faculty engagement in our AoL efforts. You may recall that in 2016 we had a staff AoL director. He was really good, he was an instructional designer, gave us a very firm platform in terms of the process and the documents. But I think this happens in a lot of schools, when you have a staff person who handles it, it's very easy to just let them handle it, and that became a deficiency in the college. And it became a particular deficiency because he left the college right before the pandemic in 2020, and in going back through what he had done, frankly, we found some areas that were not as well documented as they needed to be. We took that position and we brought a clinical faculty member into that role, Sarah Dobrzykowski.
0:11:48.8 Alan Ellstrand: I have a faculty member, they leave the classroom, they understand what's going on, and I think they had a lot more credibility with other faculty, we've made it very clear that Sarah is in a role of being a facilitator, but she does not control our AoL efforts. And we've gotten all of our curriculum committees directly involved, other key faculty members, for example, course coordinators, we're engaging, our department chairs. And one of the things we're gonna be doing working on this fall, it's really relevant to all faculty. All faculty need to know what are the competencies, what are the learning objectives that are in place. And everybody sell this, every course, because we even want to communicate that to our students.
0:12:40.2 Matt Waller: Does it seem to you like faculty are interested in engaging in this?
0:12:45.1 Alan Ellstrand: I think we need to explain a little bit more about why AoL is important. Certainly the committee chairs that we've talked to have all been great about putting time on their agenda to have regular AoL meetings and also making Sarah an ex officio member of all the committees so she will be there. But it's gonna take a little bit of work on our part, to build that culture of engagement in AoL so that ultimately, it would be wonderful if when the AoL reports come out, faculty say, "Great, I can't wait to see this, because I hope that my course is teaching our students what I thought it was. I hope that we're passing along these competencies and learning objectives, and if I'm not, I need to be changing what I'm doing in the classroom."
0:13:42.4 Matt Waller: So Alan, let's take a specific right now. You were talking earlier about some of our college-wide competencies that we want and the technical competency, which is discipline knowledge. Would you mind giving an example of how this actually works? How would we measure that?
0:14:07.2 Alan Ellstrand: Sure. So I'll give you a couple examples. One that I'm actually pretty excited about is how we measure our oral communication. Back in 2018, which was the previous time that we assessed oral communication, and we do that every other year, we asked students in the Strategic Management class who were given an assignment, they had a case assignment about Elon Musk, every student was requested to give a maybe 3-5 minute presentation on this case study and what they learned. We set up some process where they had to go to the Student Union and there was some place where they could do the recording. It did not work out. It was a disaster. Things weren't recorded that should have been. Basically, we didn't get the data. It was not good. In 2020 with the pandemic, we were unable to do it. We normally would have been collecting that data right around the time that we pivoted to remote instruction. And as you know, we had our hands full just keeping the ship afloat at the time. So we take another look at this and in 2021, we took a completely different approach. We still tied it to a case study, this time it was a case study of the Sugar Bowl football game, which was really an interesting case study and one that I think students were excited about.
0:15:38.1 Alan Ellstrand: We have a tool in the Walton College that we use for... Well, our Career Services Office use. It's called VMock and it helps students create better resumes, but it also has a module where they can do practice interviews. It uses some AI techniques to critique their interviews. And our colleague here in Boston came up with a great idea and recommended, "Why don't we use that to have the students make their oral communication presentations? All students have access to it, they can do it on their laptop and it automatically records it." And then we had staff in our communications lab do the assessments. They had a rubric, they applied the rubric, and what I really liked about that was A, we had communication experts. These were doctoral students in communication doing the assessments. And B, a relatively small number of people assessed a large number of student presentations, so there was a high level of reliability when they were making these assessments of students. One positive bonus that came out of this, the faculty who work in our Strategic Management who made that assignment, they liked it so well that they are incorporating that as a regular part of the Strategy Program going forward. So we will do those presentations on the VMock platform all the time, which I thought was really great. Anytime you can get a faculty to buy into something like that, I think it's a real plus.
0:17:23.9 Alan Ellstrand: Some of the other ones, for example, of critical thinking problem solving, we usually will identify a few courses in the curriculum and it could be a case study, it could be long form exam question, and we will identify specific deliverables that students have to do, identify them, have them assessed often. If it's critical thinking problem-solving, we will have maybe the course coordinator do the assessment, something so that we have some level of consistency. So it depends. Some of the discipline knowledge questions, they may even be multiple choice questions. We'll have a portfolio of 10 questions and see how students do in terms of their responses on those. So we try to find items that we think are really representative of the learning objectives and the competencies that we have in place so that we're basically assessing the same thing over and over.
0:18:27.9 Matt Waller: Alan, how do we implement AoL at the PhD level? I've taught doctoral level courses and of course, I've been around this business a long time, but when I was engaged in teaching the doctoral program, we really didn't have AoL. I'm really curious as to how we do that.
0:18:50.4 Alan Ellstrand: So let me back up. In our previous assessment, this was when we had our CIR visit in 2016. We did things like apply rubrics to research papers to presentations students made at their dissertation defenses and very honestly, I don't think we were getting the kinds of data that was very helpful. When you think of how idiosyncratic those things are... Every research paper is different. It would be very hard to have a rubric that would really apply across the board. We really got away from that this time and we tried to benchmark against some other schools to find out what they were doing and we found that some of the schools are using broader measures, things like pass rates on the comp exam, pass rates on dissertation defenses, even placements, how successful we are in graduating, our graduation rates. I think those are really important measures. And I have to tell you, when Sarah and I presented those to the Doctoral Programs Committee this spring, they were really interested to hear that data. Apparently, that was not data that we necessarily bring to that committee and share, but they wanted to know where are we placing our students? Anecdotally, everybody knows...
0:20:17.3 Matt Waller: And we've all been doctoral chairs of dissertation committees. We know where our students are, but we don't know that broadly at the program level. That was assigned me that, "Hey, you know, this is at least valuable information to our doctoral programs committee." And I would love it if our CIR team could give us some guidance on maybe what they've seen, either at their own schools, or if they've done on other CIR visits, 'cause we would love to be able to have some really solid measures that are helpful to us, that improve... Help us improve our doctoral programs, but are also very meaningful to our faculty in terms of being able to better prepare our students for their careers going forward.
0:21:09.1 Matt Waller: One last question, Alan. We have a number of boards... Just across my three boards I think I've got 200 people on them. And it makes a big difference. As you know, we leverage it big-time for just about everything we do, and it's been really helpful, but how have we used the board to help us with Assurance of learning?
0:21:36.6 Alan Ellstrand: Sure. And this goes to the same direct measures that I talked about earlier. We have done I think a really solid job of leveraging these external stakeholders. And a good example is I think three or four years ago we had a discussion about opportunities for specialized Master's programs in the college. At that point we were learning to adjust our MBA programs, we have professional MIS and our full-time MIS, and we had a MAC program at that time. But we saw lots of opportunities in other disciplines to roll out new Master's programs. One of the things we did, we presented our ideas about what these Master's programs would look like. And we presented them to your Dean's Executive Advisory Board, the Dean's Alumni Advisory Council. I presented, I remember, at the Diversity and Inclusion Board, and we got some great feedback from these sources. It also has the additional benefit of helping inform them what we're doing. I think their radars are always up on ways that they can help us. They may be able to share opportunities for us with those new programs, they can certainly, in some cases, be employers to hire those people. So for them to be aware of those programs coming, for them to help shape those programs, which gives them buy-in, and at the same time it helps us to craft programs that are really going to meet the needs of the students.
0:23:17.9 Alan Ellstrand: Though it's been a real win for us to have that kind of support.
0:23:21.8 Matt Waller: We'll, Alan, the Walton College is fortunate to have you and your vision and leadership for Assurance of learning. And I know you really want to make it as good as it can be, but also, as you said before, we were relying on a staff person, and when the staff person left we realized, oh, we really need to make this a part of our culture, part of our overall college processes, and you've really done a great job of doing that. So thank you for all you're doing.
0:23:55.6 Alan Ellstrand: Well, thank you, Matt. I appreciate the opportunity to work with you on that, and in everything we do. And thanks for the opportunity to just share a little bit about AoL. It's something that a lot of people have probably never heard of, and it's a pretty important part of what we do in terms of trying to ensure that our students are well prepared to face the chAlanges in the work environment.
[music]
0:24:25.4 Matt Waller: Thanks for listening to today's episode of The BeEpic Podcast from the Walton College. You can find us on Google, SoundCloud, iTunes, or look for us wherever you find your podcast. Be sure to subscribe and rate us. You can find current and past episodes by searching BeEpic Podcast, one word. That's B-E-E-P-I-C Podcast. And now, BeEpic.
[music]
