FAST Resources

This page provides a variety of resources to support researchers, students, scholars, practitioners, theorists, and even metatheorists. Anyone who wants to improve their capacity for working with and improving theory for improved practical application. A special thanks to Dr. Bernadette Wright for developing so many of these wonderful resources to support researchers, evaluators, and practitioners across the country and around the world.

Here, you will find information about:

  • The one book with (almost) everything you need to know about research
  • Guest Blog Posts, White papers and shorter online articles
  • Short case studies and tip sheets for research and evaluation
  • Deeper academic papers and books
  • Links to other organizations seeking to accelerate the social sciences
  • Videos
  • Tools and techniques for presenting your knowledge maps
  • Reading lists

The one book is: Practical Mapping for Applied Research and Program Evaluation

  • Bernadette Wright – Research and Data Analytics Director, Meals on Wheels America
  • Steven E. Wallis – Capella University; Foundation for the Advancement of Social Theory

This book is designed to support students, researchers, and practitioners by providing a more functional approach for improving the effectiveness and usefulness of research in the social sciences (including program evaluation, organizational development, and community development) with clear and direct “step by step” instructions. Importantly, the research results research are evaluated on three dimensions (quality and quantity of data, meaning or relevance to stakeholders, and logical structure). Using this innovative approach makes it possible to understand and resolve seemingly impossible problems.

Publisher’s site: https://us.sagepub.com/en-us/nam/practical-mapping-for-applied-research-and-program-evaluation/book261152

Additional material (teacher’s guide, sample syllabus, handouts, etc.): https://practicalmapping.com/

Blog Posts, White Papers, and Shorter Articles

Some Key White Papers

Basics of IPA. Leibniz Institute of Agricultural Development in Transition Economies/Fulbright Specialist Program.

Strategic Knowledge Mapping for Improved Policy and Strategic Planning. Explains why previous approaches to strategic planning have failed and how to create a more effective map using innovative approaches.

The Science of Conceptual Systems: Its History and Usefulness for Improved Decision-Making and Organizational Success. A deeper look at the academic research on how to improve effective decision-making for organizational success.

Reuse, Recycle: Rethink Research. Reviewing related research can let you avoid wasting time and money re-testing things that other studies have already demonstrated. Here, we introduce tips and resources that you can use to review the evidence in a cost-effective and reliable way.

Strategic Planning: A New State of the Art. By Steve Wallis and Kent Frese

Strategic Planning 3.0. White paper. Describes four levels of strategic planning and how each supports effective decision making.

When to Use a Randomized Controlled Trial and When Not to. White paper. A randomized controlled trial (RCT) to measure your outcomes is expensive. You do not want to conduct one unless it is going to be useful. This resource guide introduces ways you can tell what approach might work for your situation.

 

Short Case Studies and Tip Sheets

You’re Worth the Investment: Now Prove It. Six Steps to Effective Evaluation Planning. Tip sheet. Your evaluation plan can make or break your grant funding proposals and chances for future funding and growth. This tip sheet by Bernadette Wright and Ladel Lewis presents six steps to planning an evaluation that gets you useful results.

Strategic Knowledge Maps in Action. Case study. Strategic Knowledge Mapping is a new way to assess and improve a map, or a set of interconnected ideas, such as a strategic plan, a program, or a law. We can also use Integrative Propositional Analysis (IPA) to link together related maps. This 3-page case study shows the new method in action to assess and find ways to improve a sample map for a coalition to reduce drug abuse.

When Is a Good Time for Evaluation? Tip sheet. Incorporating evaluation and strategic research from the start of a projects provides leaders with valuable unique insights that can enable them increase their competitiveness and positive impact, in addition to demonstrating their results to funders and stakeholders. This tip sheet provides a chart of example questions that are often relevant in each stage of a project and example uses of the evaluation findings.

Checklist for Choosing an Internal Evaluation or External Evaluator. Tip sheet. Will you evaluate your project in-house or work with an external evaluator? Whether you are seeking to identify effective ways to design a new project (formative evaluation), demonstrate what your activities accomplished (summative evaluation), or both, this checklist will help you decide.

Asking Useful Evaluation Questions. Tip sheet. “So what?” is a question that every evaluation must answer. This tip sheet provides considerations and a flow chart that can help you decide if a question would be useful to answer, to ensure you get information you can use to benefit your activities.

On the Road to Ethical Research. Tip sheet. As you write that winning proposal, it’s important to anticipate ethical issues that may arise during your project. These issues arise in specifying the research problem; identifying a purpose statement and research questions; and collecting, analyzing, and disseminating the results of the data. This resource provides examples.

Ethics Anyone? A Guide to Ethics in Social Research. Tip sheet. Whether you are evaluating your organization’s activities in-house, commissioning an external evaluation, or working as an evaluation consultant, ethical questions are likely to be an issue. This resource guide describes professional guidelines/publications and suggested further reading on ethics in evaluation and social research.

Case Studies in Action. Tip sheet. Case studies are a frequently misunderstood, valuable research method. This tip sheet describes how you can use case studies for:

  1. marketing and messaging,
  2. demonstrating impact, and
  3. identifying the best strategies for your specific circumstances.

Academic Papers with Key Ideas for Improving Theories

Overview

Wallis, S. (2019). The Missing Piece of the Integrative Studies Puzzle. Interdisciplinary Science Reviews 44(3-4).

Wallis, S. E. (2016). The science of conceptual systems: A progress report. Foundations of Science, 21(4), 579–602.

Wallis, S. E. (2014). Evaluating Explanations through their Conceptual Structures. In M. Lissack, & A. Graber (Eds.), Modes of Explanation: Affordances for Action and Prediction  New York: Palgrave MacMillan.

Key Insights into Structure and Abstraction

Wallis, S. E. (in press). Evaluating and improving theory using conceptual loops: A science of conceptual systems (SOCS) approach. Cybernetics and Human Knowing.

Wallis, S. E. (2020). Understanding and improving the usefulness of conceptual systems: An Integrative Propositional Analysis-based perspective on levels of structure and emergence. Systems Research and Behavioral Science (in press).  

Wallis, S. E. (2020). Exceeding the Limits: Commentary on ‘The Limits of Transparency’. Systems Research and Behavioral Science, 37(1), 170-173.

Wallis, S. E. (2020). Commentary on Roth: Adding a conceptual systems perspective. Systems Research and Behavioral Science, 37(1), 178-181.

Wallis, S. (2020). Orthogonality: Developing a structural/perspectival approach for improving theoretical models. Systems Research and Behavioral Science, 37(2), 345-359.

Wallis, S. (2019). Understanding the Systemic Structure of Models: A Missing Piece of the Design Thinking Puzzle (commentary on Michael Lissack’s two-part article: “Understanding is a Design Problem: Cognizing from a Designerly Thinking Perspective.”). She Ji: The Journal of Design, Economics, and Innovation 5(4).

Wallis, S. E. (2016). Structures of logic in policy and theory: Identifying sub-systemic bricks for investigating, building, and understanding conceptual systems. Foundations of Science, 20(3), 213-231.

Wallis SE (2014) Abstraction and insight: Building better conceptual systems to support more effective social change Foundations of Science 19:353-362.

Wallis, S. E. (2009). Seeking the robust core of organisational learning theory. International Journal of Collaborative Enterprise, 1(2), 180-193.

Some Earlier Insights

Wallis, S. E. (2010). Toward a science of metatheory. Integral Review, 6 (Special Issue: “Emerging Perspectives of Metatheory and Theory”).

Wallis SE. (2008). Validation of theory: Exploring and reframing Popper’s worlds. Integral Review 4(2): 71-91.

Society and Sytems

Wallis, S. E., & Valentinov, V. (2017). What Is Sustainable Theory? A Luhmannian Perspective on the Science of Conceptual Systems. Foundations of Science, 22(4), 733-747.

Wallis, S. E., & Valentinov, V. (2016). The imperviance of conceptual systems: Cognitive and moral aspects. Kybernetes, 45(9).

Wallis, S. E., & Valentinov, V. (2016). A limit to our thinking and some unanticipated moral consequences: A science of conceptual systems perspective with some potential solutions. Systemic Practice and Action Research, 30(2), 103-116.

Seminal Study

Wallis, S. E. (2010). The structure of theory and the structure of scientific revolutions: What constitutes an advance in theory? In S. E. Wallis (Ed.), Cybernetics and systems theory in management: Views, tools, and advancements (pp. 151-174). Hershey, PA: IGI Global.

Evaluation Focus

Re-Structuring Evaluation Findings into Useful Knowledge. Danielle Houston, Bernadette Wright, & Steven E. Wallis. Journal of MultiDisciplinary Evaluation (JMDE), September, 2017.

Table-top mapping focus

Strategic Knowledge Mapping: The Co-Creation of Useful Knowledge. Steven E. Wallis & Bernadette Wright. Developments in Business Simulation and Experiential Learning, Vol 42, 2015. Winner of 2015 Best Paper Award, Innovations and Future Directions in Education Track, 42nd Annual ABSEL (Association for Business Simulation and Experiential Learning) Meeting, Las Vegas, March 2015.

Topic Foci: Sample / Example Papers

In these papers, IPA and related tools have been used to evaluate and/or synthesize/integrate multiple theoretical perspectives within and/or between disciplines.

Business / Entrepreneurship

Wright, B. & Wallis, S.E. (2015). Using Integrative Propositional Analysis for Evaluating Entrepreneurship Theories. SAGE Open, 5(3).

Panetti, E., Parmentola, A., Wallis, S. E., & Ferretti, M. (2018). What drives technology transitions? An integration of different approaches within transition studies. Technology Analysis & Strategic Management, (online), 1-22.

Wallis, S. E. (2012). The right tool for the job: Philosophy’s evolving role in advancing management theory. Philosophy of Management, 11(3), 67-99.

Education

Goltz, S. M. (2017). Enhancing simulation learning with team mental model mapping. Management Teaching Review, 2(3), 211-224.

Russell D, Wallis SE (2015). Designing a Learning Analytic System for Assessing Immersive Virtual Learning Environments. In: Russell D, Laffey J (eds) Handbook of Research on Gaming Trends in P-12 Education. IGI Global, Hershey, PA.

Research Institutions

Wallis, S. E. (2019). Integrative Propositional Analysis for Developing Capacity in an Academic Research Institution by Improving Strategic Planning. Systems Research and Behavioral Science 37 (1).

Ethics

Wallis, S. (2010). Developing effective ethics for effective behavior. Social Responsibility Journal, 6 (4).

Interdisciplinary Studies

Wallis, S. E. (2018). Actionable knowledge mapping to accelerate interdisciplinary collaborations for research and practice. International Society for the Systems Sciences (ISSS). July 22-27. Corvallis, OR.

Wallis, S. E. (2012). Existing and Emerging Methods for Integrating Theories Within and Between Disciplines. 56th annual meeting of the International Society for Systems Sciences (ISSS). July 15-22, 2012. San Jose, California, p. 23.

Wallis SE (2014). Existing and emerging methods for integrating theories within and between disciplines. Organisational Transformation and Social Change 11:3-24.

Wallis, S. E. (2014). A systems approach to understanding theory: Finding the core, identifying opportunities for improvement, and integrating fragmented fields. Systems Research and Behavioral Science, 31(1), 23-31.

Wallis, S. E., & Johnson, L. (2018). Using Integrative Propositional Analysis to Understand and Integrate Four Theories of Social Power Systems. Journal on Policy and Complex Systems, 4(1), 169-194.

Legislation / Laws

Wallis S., Wright B. (2016). Integrative Propositional Analysis: The Missing Link for Creating More Effective Laws. The Science of Laws Journal Vol. 2, 10-15.

Policy

De Wee, Guswin (2020). An Evaluation of the Integrative Propositional Analysis Model for Complex Policy Environments: The Case of the Drakenstein Housing Policy, 2010-2017. Master’s Thesis, Nelson Mandela University.

Wallis, S., Wright, B. (2018). Integrative Propositional Analysis for Understanding and Reducing Poverty. Kybernetes 48 (6).

Steven E. Wallis, Bernadette Wright, & Franklin David Nash (2016).Using Integrative Propositional Analysis to Evaluate and Integrate Economic Policies of U.S. Presidential Candidates. SciPolicy Watchdog white paper.

Wallis S., Wright B. (2016). Integrative Propositional Analysis: The Missing Link for Creating More Effective Laws. The Science of Laws Journal, 2, 10-15.

Shackelford, C. (2014). Propositional Analysis, Policy Creation, and Complex Environments in the United States’ 2009 Afghanistan/Pakistan Policy. Doctoral Dissertation, Walden University.

Wallis, S. (2013). How to Choose Between Policy Proposals: A Simple Tool Based on Systems Thinking and Complexity Theory. E:CO – Emergence: Complexity and Organizations, 15(3).

Wallis, S. (2011). Avoiding Policy Failure: A Workable Approach. Litchfield Park, AZ: Emergent Publications.

Wallis, S. E. (2010). Towards the development of more robust policy models. Integral Review, 6(1), 153-160.

Research & Evaluation

Wallis, S. (2019). Improving our theory of evaluation through an African-made process: An ubuntu answer to the decolonising question. Administratio Publica, Vol. 27(4).

Houston, D., Wright, B., Wallis, S., (2017). Re-structuring evaluation findings into useful knowledge. Journal of Multi-Disciplinary Evaluation, Vol 30 (29).

Psychology

Wallis, S. E. (2015). Integrative Propositional Analysis: A new quantitative method for evaluating theories in psychology. Review of General Psychology, 19(3), 365-380.

Sociology

Wallis, S. (2015). Are theories of conflict improving? Using propositional analysis to determine the structure of conflict theories over the course of a century. E:CO – Emergence: Complexity and Organizations. 17(4).

Systems Thinking

Wallis, S. E. (2008). From reductive to robust: Seeking the core of complex adaptive systems theory. In A. Yang, & Y. Shan (Eds.), Intelligent Complex Adaptive Systems (pp. 1-25). Hershey, PA: IGI Publishing.

Wallis, S. E. (2008). Emerging order in CAS theory: Mapping some perspectives. Kybernetes, 38(7).

Wallis, S. E. (2009). The complexity of complexity theory: An innovative analysis. Emergence: Complexity and Organization, 11(4), 26-38.

Places to Go for Advancing Science

http://fasterforwardfund.org/

https://www.aaas.org/

http://www.scienceoflaws.org/

https://psysciacc.org/

Videos to See

Dealing with Messy Maps

Wright, B. & Wallis, S.E. Washington Evaluators Virtual Brown Bag: Causal Knowledge Mapping for More Useful Evaluation. June 27, 2017.

Wallis, S.E. (2017). Hidden Logics: New Templates for Intercultural Understanding? SIETAR Europa Webinar.

Wallis, S.E. & Wright, B (2017). Knowledge Mapping for Greater Social Good. Data Analysts for Social Good.

Wallis, S.E. & Wright. B. (2016). Network Knowledge Mapping: Mapping the Known, Discovering the Unknown. School of Public Affairs, University of Colorado Denver.

Presenting the Results of your Research

 A few key (free) sites to get you started:

 Plectica – Easy to use, very intuitive.

 Insight Maker – Great for turning diagrams into computer models.

 KUMU – For making great presentations.

 Some tips for working in Kumu

 Using KUMU for Visualizing Interview Data by Bernadette Wright. American Evaluation Association AEA365 blog, 3/23/17.

 A longer list of online mapping resources

Tools for Mapping Your Strategic Plan.

Reading Lists

Reading List: Integrative Propositional Analysis, Causal Knowledge Mapping, & Assessing Research Quality. Integrative Propositional Analysis (IPA) and Causal Knowledge Mapping are methods for improving understanding of and ability to address policies, programs, and systems.The materials in this reading list detail the research behind these techniques, examples of how they are applies, software/tools, tools for assessing the quality of evidence, and more. This is part of the handouts for participants at our workshop at the American Evaluation Association 2018 Summer Evaluation Institute.

Suggested Books on Research and Evaluation. Here, we summed up some suggested books on applied research and evaluation methods. These books correct common misconceptions about evaluation and provide good information on the basics of results-focused evaluation.