This page provides a variety of resources to support researchers, students, scholars, practitioners, theorists, and even metatheorists. Anyone who wants to improve their capacity for working with and improving theory for improved practical application. A special thanks to Dr. Bernadette Wright for developing so many of these wonderful resources to support researchers, evaluators, and practitioners across the country and around the world.
Here, you will find information about:
- The one book with (almost) everything you need to know about research
- Guest Blog Posts, White papers and shorter online articles
- Short case studies and tip sheets for research and evaluation
- Deeper academic papers and books
- Links to other organizations seeking to accelerate the social sciences
- Tools and techniques for presenting your knowledge maps
- Reading lists
The one book is: Practical Mapping for Applied Research and Program Evaluation
- Bernadette Wright – Research and Data Analytics Director, Meals on Wheels America
- Steven E. Wallis – Capella University; Foundation for the Advancement of Social Theory
This book is designed to support students, researchers, and practitioners by providing a more functional approach for improving the effectiveness and usefulness of research in the social sciences (including program evaluation, organizational development, and community development) with clear and direct “step by step” instructions. Importantly, the research results research are evaluated on three dimensions (quality and quantity of data, meaning or relevance to stakeholders, and logical structure). Using this innovative approach makes it possible to understand and resolve seemingly impossible problems.
Additional material (teacher’s guide, sample syllabus, handouts, etc.): https://practicalmapping.com/
Blog Posts, White Papers, and Shorter Articles
- Building Institutional Capacity, Without Breaking the Bank. Candid.org (pending)
- Gap Analysis for Literature Reviews and Advancing Useful Knowledge. https://www.researchtoaction.org/ (pending)
- Breaking Through the Disciplinary Barrier with Practical Mapping. Integration and Implementation Insights Blog. (pending)
- Synthetic Literature Reviews: An Introduction.
- How to Teach a Course on Evaluation. American Evaluation Association AEA365 Blog.
- Where to Teach a Course on Evaluation. American Evaluation Association AEA365 Blog.
- How to Promote Your Evaluation Course. American Evaluation Association AEA365 Blog. (pending)
- Accelerating the Social Sciences: An Innovative Structural Approach. Social Science Research Centre Blog.
- Back to the Start to Improve SCRUM: Big Lessons from a Tiny Virus. Strategic Planning for Nonprofits Linkedin Site.
- Accelerating Evolution in Your Knowledge Ecology. RealKM Weekly Wrap.
- Basics of Theory: A Brief, Plain Language Introduction. Methodspace (SAGE).
- Are You Preparing Yourself to Change the World? Fielding Graduate University. Presentation here.
- The Face of AEA: Featuring Bernadette Wright. AEA (American Evaluation Association) August 2019 Newsletter.
- Better Knowledge for Greater Success. CharityChannel.
- Program Evaluation That Works. Grantspace by Candid blog post.
- Measuring Nonprofit Outcomes and Progress: What Data Should You Collect? Charity Channel online article.
- Choosing and Using the Right Online Survey Tool, Charity Channel online article.
- 10 Tips for Writing Well-Worded Survey Questions. Charity Channel online article.
- Toastmasters Tips on Delivering Negative Evaluation Findings. American Evaluation Association AEA365 Blog. Bernadette Wright.
- Mapping Your Way to Successful Collaborations. National Center for Mobility Management blog. Bernadette Wright.
- Making Sense of the Knowledge Explosion with Knowledge Mapping. SAGE Methodspace. Steven E. Wallis & Bernadette Wright.
- Three Tips for Mastering the Evaluation Plan. Grant Professionals Association, Grant News Weekly.
- How Good is Your Evidence? Stanford Social Innovation Review (SSIR) online article by Bernadette Wright & Steven E. Wallis.
- How to Increase Survey Responses. CharityChannel Press online article by Bernadette Wright.
- Using KUMU for Visualizing Interview Data by Bernadette Wright. American Evaluation Association AEA365 blog.
- Six Steps to Effective Program Evaluation: Step 6 – Communicate Your Results. CharityChannel Press online article by Bernadette Wright.
- Six Steps to Effective Program Evaluation: Step 5 – Plan Your Methods, CharityChannel Press online article by Bernadette Wright.
- Six Steps to Effective Program Evaluation: Step 4 – Review Related Research, CharityChannel Press online article by Bernadette Wright.
- Evaluation Approaches: You Don’t Need to Choose Just One. Guest post by Bernadette Wright, Route One Evaluation.
- Six Steps to Effective Program Evaluation: Ask Useful Questions (Part 3). Charity Channel Press online article by Bernadette Wright.
- Six Steps to Effective Program Evaluation: Understanding Your Program (Part 2). Charity Channel Press online article by Bernadette Wright.
- Six Steps to Effective Program Evaluation: Collaborate with Stakeholders (Part 1). CharityChannel Press online article by Bernadette Wright.
- Getting to Evidence-Based Policy: Three Perspectives. Stanford Social Innovation Review (SSIR) online article by Bernadette Wright.
- Program Evaluation is Not a Dirty Word: Evaluative Thinking to Improve Your Results. CharityChannel Press online article by Bernadette Wright.
- Bernadette Wright on Little Is Known? Better Using Background Research. American Evaluation Association AEA365 Blog.
- Five Tips for Demonstrating Your Value without Breaking Your Bank, Guest post by Bernadette Wright, Foundation Center Washington, DC Blog.
- Strategic Knowledge Mapping: A New Tool for Visualizing and Using Evaluation Findings in STEM. Guest post by Bernadette Wright, EvaluATE Blog.
- Bernadette Wright on A New Method to Increase Success in Entrepreneurship.
American Evaluation Association AEA365 blog.
- Do It with Science! Demonstrate Your Value with the Scientific Method. Article by Bernadette Wright, Grassroots Institute for Fundraising Training (GIFT) August 2014 eNews, Consultant Corner.
- Ladel Lewis and Bernadette Wright on Members Wanted: Recruitment, Retention, and Reclamation. American Evaluation Association AEA365 blog.
- Strategic Knowledge Map Analysis, My Dear Watson. Guest post by Bernadette Wright and Ladel Lewis, Twenty LYS blog.
- Navigating to Success in Becoming Part of Complete Systems of Care. Guest post by Bernadette Wright, AgeBlog (American Society on Aging).
- Bernadette Wright and Ladel Lewis on Delivering the Goods. American Evaluation Association AEA365 blog.
- Bernadette Wright on Evaluating the Decision to Be an Independent Evaluator. American Evaluation Association AEA365 blog.
- The integral puzzle: Determining the integrality of integral theory. Integral World.
- An emerging innovation in policy evaluation: From theory to practice. Article in PA TIMES (newsletter of ASPA – the American Society for Public Administration). Wallis, S. E. (2019). Wallis, S. E. (2019).
- Finding Patterns of Knowing to Improve Communication and Understanding. Knowledge Bite, SIETAR Espania.
Some Key White Papers
Basics of IPA. Leibniz Institute of Agricultural Development in Transition Economies/Fulbright Specialist Program.
Strategic Knowledge Mapping for Improved Policy and Strategic Planning. Explains why previous approaches to strategic planning have failed and how to create a more effective map using innovative approaches.
The Science of Conceptual Systems: Its History and Usefulness for Improved Decision-Making and Organizational Success. A deeper look at the academic research on how to improve effective decision-making for organizational success.
Reuse, Recycle: Rethink Research. Reviewing related research can let you avoid wasting time and money re-testing things that other studies have already demonstrated. Here, we introduce tips and resources that you can use to review the evidence in a cost-effective and reliable way.
Strategic Planning: A New State of the Art. By Steve Wallis and Kent Frese
Strategic Planning 3.0. White paper. Describes four levels of strategic planning and how each supports effective decision making.
When to Use a Randomized Controlled Trial and When Not to. White paper. A randomized controlled trial (RCT) to measure your outcomes is expensive. You do not want to conduct one unless it is going to be useful. This resource guide introduces ways you can tell what approach might work for your situation.
Short Case Studies and Tip Sheets
You’re Worth the Investment: Now Prove It. Six Steps to Effective Evaluation Planning. Tip sheet. Your evaluation plan can make or break your grant funding proposals and chances for future funding and growth. This tip sheet by Bernadette Wright and Ladel Lewis presents six steps to planning an evaluation that gets you useful results.
Strategic Knowledge Maps in Action. Case study. Strategic Knowledge Mapping is a new way to assess and improve a map, or a set of interconnected ideas, such as a strategic plan, a program, or a law. We can also use Integrative Propositional Analysis (IPA) to link together related maps. This 3-page case study shows the new method in action to assess and find ways to improve a sample map for a coalition to reduce drug abuse.
When Is a Good Time for Evaluation? Tip sheet. Incorporating evaluation and strategic research from the start of a projects provides leaders with valuable unique insights that can enable them increase their competitiveness and positive impact, in addition to demonstrating their results to funders and stakeholders. This tip sheet provides a chart of example questions that are often relevant in each stage of a project and example uses of the evaluation findings.
Checklist for Choosing an Internal Evaluation or External Evaluator. Tip sheet. Will you evaluate your project in-house or work with an external evaluator? Whether you are seeking to identify effective ways to design a new project (formative evaluation), demonstrate what your activities accomplished (summative evaluation), or both, this checklist will help you decide.
Asking Useful Evaluation Questions. Tip sheet. “So what?” is a question that every evaluation must answer. This tip sheet provides considerations and a flow chart that can help you decide if a question would be useful to answer, to ensure you get information you can use to benefit your activities.
On the Road to Ethical Research. Tip sheet. As you write that winning proposal, it’s important to anticipate ethical issues that may arise during your project. These issues arise in specifying the research problem; identifying a purpose statement and research questions; and collecting, analyzing, and disseminating the results of the data. This resource provides examples.
Ethics Anyone? A Guide to Ethics in Social Research. Tip sheet. Whether you are evaluating your organization’s activities in-house, commissioning an external evaluation, or working as an evaluation consultant, ethical questions are likely to be an issue. This resource guide describes professional guidelines/publications and suggested further reading on ethics in evaluation and social research.
Case Studies in Action. Tip sheet. Case studies are a frequently misunderstood, valuable research method. This tip sheet describes how you can use case studies for:
- marketing and messaging,
- demonstrating impact, and
- identifying the best strategies for your specific circumstances.
- Also published on BetterEvaluation.
Academic Papers with Key Ideas for Improving Theories
Wallis, S. (2019). The Missing Piece of the Integrative Studies Puzzle. Interdisciplinary Science Reviews 44(3-4).
Wallis, S. E. (2016). The science of conceptual systems: A progress report. Foundations of Science, 21(4), 579–602.
Wallis, S. E. (2014). Evaluating Explanations through their Conceptual Structures. In M. Lissack, & A. Graber (Eds.), Modes of Explanation: Affordances for Action and Prediction New York: Palgrave MacMillan.
Key Insights into Structure and Abstraction
Wallis, S. E. (in press). Evaluating and improving theory using conceptual loops: A science of conceptual systems (SOCS) approach. Cybernetics and Human Knowing.
Wallis, S. E. (2020). Understanding and improving the usefulness of conceptual systems: An Integrative Propositional Analysis-based perspective on levels of structure and emergence. Systems Research and Behavioral Science (in press).
Wallis, S. E. (2020). Exceeding the Limits: Commentary on ‘The Limits of Transparency’. Systems Research and Behavioral Science, 37(1), 170-173.
Wallis, S. E. (2020). Commentary on Roth: Adding a conceptual systems perspective. Systems Research and Behavioral Science, 37(1), 178-181.
Wallis, S. (2020). Orthogonality: Developing a structural/perspectival approach for improving theoretical models. Systems Research and Behavioral Science, 37(2), 345-359.
Wallis, S. (2019). Understanding the Systemic Structure of Models: A Missing Piece of the Design Thinking Puzzle (commentary on Michael Lissack’s two-part article: “Understanding is a Design Problem: Cognizing from a Designerly Thinking Perspective.”). She Ji: The Journal of Design, Economics, and Innovation 5(4).
Wallis, S. E. (2016). Structures of logic in policy and theory: Identifying sub-systemic bricks for investigating, building, and understanding conceptual systems. Foundations of Science, 20(3), 213-231.
Wallis SE (2014) Abstraction and insight: Building better conceptual systems to support more effective social change Foundations of Science 19:353-362.
Wallis, S. E. (2009). Seeking the robust core of organisational learning theory. International Journal of Collaborative Enterprise, 1(2), 180-193.
Some Earlier Insights
Wallis, S. E. (2010). Toward a science of metatheory. Integral Review, 6 (Special Issue: “Emerging Perspectives of Metatheory and Theory”).
Wallis SE. (2008). Validation of theory: Exploring and reframing Popper’s worlds. Integral Review 4(2): 71-91.
Society and Sytems
Wallis, S. E., & Valentinov, V. (2017). What Is Sustainable Theory? A Luhmannian Perspective on the Science of Conceptual Systems. Foundations of Science, 22(4), 733-747.
Wallis, S. E., & Valentinov, V. (2016). The imperviance of conceptual systems: Cognitive and moral aspects. Kybernetes, 45(9).
Wallis, S. E., & Valentinov, V. (2016). A limit to our thinking and some unanticipated moral consequences: A science of conceptual systems perspective with some potential solutions. Systemic Practice and Action Research, 30(2), 103-116.
Wallis, S. E. (2010). The structure of theory and the structure of scientific revolutions: What constitutes an advance in theory? In S. E. Wallis (Ed.), Cybernetics and systems theory in management: Views, tools, and advancements (pp. 151-174). Hershey, PA: IGI Global.
Re-Structuring Evaluation Findings into Useful Knowledge. Danielle Houston, Bernadette Wright, & Steven E. Wallis. Journal of MultiDisciplinary Evaluation (JMDE), September, 2017.
Table-top mapping focus
Strategic Knowledge Mapping: The Co-Creation of Useful Knowledge. Steven E. Wallis & Bernadette Wright. Developments in Business Simulation and Experiential Learning, Vol 42, 2015. Winner of 2015 Best Paper Award, Innovations and Future Directions in Education Track, 42nd Annual ABSEL (Association for Business Simulation and Experiential Learning) Meeting, Las Vegas, March 2015.
Topic Foci: Sample / Example Papers
In these papers, IPA and related tools have been used to evaluate and/or synthesize/integrate multiple theoretical perspectives within and/or between disciplines.
Business / Entrepreneurship
Wright, B. & Wallis, S.E. (2015). Using Integrative Propositional Analysis for Evaluating Entrepreneurship Theories. SAGE Open, 5(3).
Panetti, E., Parmentola, A., Wallis, S. E., & Ferretti, M. (2018). What drives technology transitions? An integration of different approaches within transition studies. Technology Analysis & Strategic Management, (online), 1-22.
Wallis, S. E. (2012). The right tool for the job: Philosophy’s evolving role in advancing management theory. Philosophy of Management, 11(3), 67-99.
Goltz, S. M. (2017). Enhancing simulation learning with team mental model mapping. Management Teaching Review, 2(3), 211-224.
Russell D, Wallis SE (2015). Designing a Learning Analytic System for Assessing Immersive Virtual Learning Environments. In: Russell D, Laffey J (eds) Handbook of Research on Gaming Trends in P-12 Education. IGI Global, Hershey, PA.
Wallis, S. E. (2019). Integrative Propositional Analysis for Developing Capacity in an Academic Research Institution by Improving Strategic Planning. Systems Research and Behavioral Science 37 (1).
Wallis, S. (2010). Developing effective ethics for effective behavior. Social Responsibility Journal, 6 (4).
Wallis, S. E. (2018). Actionable knowledge mapping to accelerate interdisciplinary collaborations for research and practice. International Society for the Systems Sciences (ISSS). July 22-27. Corvallis, OR.
Wallis, S. E. (2012). Existing and Emerging Methods for Integrating Theories Within and Between Disciplines. 56th annual meeting of the International Society for Systems Sciences (ISSS). July 15-22, 2012. San Jose, California, p. 23.
Wallis SE (2014). Existing and emerging methods for integrating theories within and between disciplines. Organisational Transformation and Social Change 11:3-24.
Wallis, S. E. (2014). A systems approach to understanding theory: Finding the core, identifying opportunities for improvement, and integrating fragmented fields. Systems Research and Behavioral Science, 31(1), 23-31.
Wallis, S. E., & Johnson, L. (2018). Using Integrative Propositional Analysis to Understand and Integrate Four Theories of Social Power Systems. Journal on Policy and Complex Systems, 4(1), 169-194.
Legislation / Laws
Wallis S., Wright B. (2016). Integrative Propositional Analysis: The Missing Link for Creating More Effective Laws. The Science of Laws Journal Vol. 2, 10-15.
De Wee, Guswin (2020). An Evaluation of the Integrative Propositional Analysis Model for Complex Policy Environments: The Case of the Drakenstein Housing Policy, 2010-2017. Master’s Thesis, Nelson Mandela University.
Wallis, S., Wright, B. (2018). Integrative Propositional Analysis for Understanding and Reducing Poverty. Kybernetes 48 (6).
Steven E. Wallis, Bernadette Wright, & Franklin David Nash (2016).Using Integrative Propositional Analysis to Evaluate and Integrate Economic Policies of U.S. Presidential Candidates. SciPolicy Watchdog white paper.
Wallis S., Wright B. (2016). Integrative Propositional Analysis: The Missing Link for Creating More Effective Laws. The Science of Laws Journal, 2, 10-15.
Shackelford, C. (2014). Propositional Analysis, Policy Creation, and Complex Environments in the United States’ 2009 Afghanistan/Pakistan Policy. Doctoral Dissertation, Walden University.
Wallis, S. (2013). How to Choose Between Policy Proposals: A Simple Tool Based on Systems Thinking and Complexity Theory. E:CO – Emergence: Complexity and Organizations, 15(3).
Wallis, S. (2011). Avoiding Policy Failure: A Workable Approach. Litchfield Park, AZ: Emergent Publications.
Wallis, S. E. (2010). Towards the development of more robust policy models. Integral Review, 6(1), 153-160.
Research & Evaluation
Wallis, S. (2019). Improving our theory of evaluation through an African-made process: An ubuntu answer to the decolonising question. Administratio Publica, Vol. 27(4).
Houston, D., Wright, B., Wallis, S., (2017). Re-structuring evaluation findings into useful knowledge. Journal of Multi-Disciplinary Evaluation, Vol 30 (29).
Wallis, S. E. (2015). Integrative Propositional Analysis: A new quantitative method for evaluating theories in psychology. Review of General Psychology, 19(3), 365-380.
Wallis, S. (2015). Are theories of conflict improving? Using propositional analysis to determine the structure of conflict theories over the course of a century. E:CO – Emergence: Complexity and Organizations. 17(4).
Wallis, S. E. (2008). From reductive to robust: Seeking the core of complex adaptive systems theory. In A. Yang, & Y. Shan (Eds.), Intelligent Complex Adaptive Systems (pp. 1-25). Hershey, PA: IGI Publishing.
Wallis, S. E. (2008). Emerging order in CAS theory: Mapping some perspectives. Kybernetes, 38(7).
Wallis, S. E. (2009). The complexity of complexity theory: An innovative analysis. Emergence: Complexity and Organization, 11(4), 26-38.
Places to Go for Advancing Science
Videos to See
Wright, B. & Wallis, S.E. Washington Evaluators Virtual Brown Bag: Causal Knowledge Mapping for More Useful Evaluation. June 27, 2017.
Wallis, S.E. (2017). Hidden Logics: New Templates for Intercultural Understanding? SIETAR Europa Webinar.
Wallis, S.E. & Wright, B (2017). Knowledge Mapping for Greater Social Good. Data Analysts for Social Good.
Wallis, S.E. & Wright. B. (2016). Network Knowledge Mapping: Mapping the Known, Discovering the Unknown. School of Public Affairs, University of Colorado Denver.
Presenting the Results of your Research
A few key (free) sites to get you started:
Plectica – Easy to use, very intuitive.
Insight Maker – Great for turning diagrams into computer models.
KUMU – For making great presentations.
Some tips for working in Kumu
Using KUMU for Visualizing Interview Data by Bernadette Wright. American Evaluation Association AEA365 blog, 3/23/17.
A longer list of online mapping resources
Reading List: Integrative Propositional Analysis, Causal Knowledge Mapping, & Assessing Research Quality. Integrative Propositional Analysis (IPA) and Causal Knowledge Mapping are methods for improving understanding of and ability to address policies, programs, and systems.The materials in this reading list detail the research behind these techniques, examples of how they are applies, software/tools, tools for assessing the quality of evidence, and more. This is part of the handouts for participants at our workshop at the American Evaluation Association 2018 Summer Evaluation Institute.
Suggested Books on Research and Evaluation. Here, we summed up some suggested books on applied research and evaluation methods. These books correct common misconceptions about evaluation and provide good information on the basics of results-focused evaluation.