Archive for November, 2013

Assessing Digital Literacy and Problem solving in Technology-Rich Environments

November 14, 2013

A recently released U.S. report from the Organisation for Economic Co-operation and Development (OECD), the Survey of Adult Skills, also known as the PIAAC (Programme for the International Assessment of Adult Competencies) assessment, has Problem solving in Technology-Rich environments (PS-TRE) as one of its three assessment areas.  PS-TRE is defined as  using digital technology, communication tools, and networks to acquire and evaluate information, communicate with others, and perform practical tasks.” Although the U.S. assessment is a survey of a small, scientifically-selected sample of adults, not specifically of adult learners, it does call our attention to the need for formative and summative assessments of digital literacy and problem solving skills in technology-rich environments that are taught in adult literacy programs.  In this article, I will list some of the relevant technology-related questions and, a few of the pertinent formative and summative assessments for adult digital literacy and problem solving skills, and will provide some basic information on the PIAAC PS-TRE assessment.

Several Good Adult Digital Literacy Skills Questions

(These could easily be turned into learning objectives.)

  • Can the learner describe basic features of computers and the Internet such as “how to use command names, drop-down menus, naming protocols for files and folders, and links in a web page?” (from the PIAAC Survey of Adult Skills)
  • Can the learner demonstrate the needed level of keyboarding (typing) skills? E.g. 40-60 wpm.?
  • Can the learner demonstrate good skills to find specific needed information online?
  • Can s/he describe what to look for in judging the accuracy, quality and appropriateness of the information found?
  • Is the learner fearless, or at least reasonably comfortable, in attempting to solve technology-related problems?
  • When given a task to solve in a digital environment, can the learner analyze the task requirements, identify goals for the task, identify the tools and other resources available, develop a plan to solve it, and successfully monitor the progress to solution of the problem?
  • Can the learner effectively and efficiently use digital tools such as menus, search tools, sort tools, and word processing tools, in accomplishing the task or solving the problem?
  • Does the learner “create and communicate” using digital technology? For example does s/he create or add to web sites or blogs, join and contribute to threaded discussions or “listservs,” add to wikis, comment on online news articles, and/or help to create a class online newsletter?
  • When faced with computer or Internet hardware or software tool technical problems (e.g. in word processing, spreadsheets, databases, file management, statistical packages, graphics, web browsers, or email,) can the learner suggest strategies that might lead to a solution?

Four Adult Learner-oriented Digital Literacy Assessments

  • Northstar  http://digitalliteracyassessment.org is a free, online digital literacy assessment designed for adult learners by a state-level partnership of adult education, workforce development and libraries in Minnesota. It assesses basic skills needed to perform tasks on computers and online through six self-guided online modules, measuring Basic Computer Use, Internet, Windows Operating System, Mac OS, Email, and Word Processing (MSWord). Northstar could be used for formative or summative assessment. A passing score of 80% is needed to demonstrate competency on a module. The learner may take the assessment as many times as needed, and in between can learn skills needed through a variety of ways. One way, for states where this is available, is through a Learner Web digital literacy Learning Plan. (For more information on Learner Web, go to http://learnerweb.org. There is a high degree of congruence between the Learner Web digital literacy learning plan instruction and this assessment. Other free digital literacy learning resources, ranging from mouse and keyboard skills and writing letters and words while learning basic word processing skills, to spreadsheet, database and Internet problem-solving skills will be found on The Literacy List at http://home.comcast.net/~djrosen/newsome/litlist/complit.html
  • A Teacher-made Computer Skills Assessment  Several years ago teacher, technology field specialist, and adult curriculum writer, Kenneth Tamarkin, wrote an article for Adventures in Assessment, a Massachusetts SABES publication in which he included an adult-focused computer skills assessment. You will find it at http://www.sabes.org/resources/adventures/vol10/10tamarkin.htm
  • Self-assessing One’s Online Learning skills  Many colleges and universities offer free, online assessments for students to determine if e-learning or online learning is right for them. To find these, search for “Is eLearning Right for You?” (Be sure to put the whole phrase in quotes.)  For example, Carroll Community College offers a free self-assessment for those who want to assess their readiness for online learning. It will be found at http://www.carrollcc.edu/courses/online/assessment/skills_assessment.asp
  • PIAAC Digital Literacy (PS-TRE) Assessment  According to the U.S. National Center for Education Statistics Website, http://nces.ed.gov/surveys/piaac/cba.asp, the PIAAC Survey of Adult Skills assesses an “Information communication technology (ICT) core: A set of easy, basic computer tasks to assess basic functional computer skills necessary to take the main assessment on the computer” and includes 14 problem solving in technology-rich environmentsitems based on the PIAAC problem solving framework (See below for more information about this framework.) The PIAAC Assessments, including the PS-TRE assessment, will be made available online in January, 2014. Education & Skills Online, an online assessment tool, will enable individuals and organizations to assess key literacy, numeracy and PS-TRE competencies in real time, on demand. You will find more information about this at http://www.oecd.org/site/piaac/ENG_Brochure%20Education%20and%20Skills%20Online%20SAS_Oct%2013.pdf Retrieved 11.11.13.

More on the PIAAC Assessment

The PIAAC Problem Solving Framework

Specifically, it assesses the cognitive processes of problem solving–goal setting, planning, selecting, evaluating, organizing, and communicating results. The environment in which PS-TRE assesses these processes is meant to reflect the reality that digital technology has revolutionized access to information and communication capabilities over the past decades. In particular, the internet has immensely increased instantaneous access to large amounts of information and has expanded capabilities of instant voice, text, and graphics communication across the globe. In order to effectively operate in this environment, it is necessary to have (a) knowledge of how the environment is structured (e.g., an understanding of the basics of the environment, including how to use command names, drop-down menus, naming protocols for files and folders, and links in a web page), and (b) the ability to interact effectively with digital information. Such interaction involves understanding electronic texts, images, graphics and numerical data, as well as locating, evaluating, and critically judging the validity, accuracy, and appropriateness of the accessed information. These skills constitute the core aspects of the PIAAC PS-TRE assessment.

PS-TRE items present tasks of varying difficulty to be performed in simulated software applications using commands and functions commonly found in the technology environments of email, web pages, and spreadsheets. These tasks range from purchasing particular goods or services online and finding interactive health information to managing personal information and business finances.

PIAAC recognizes the diversity of digital technologies and the fact that they are evolving at a rapid pace, but due to implementation constraints the first round of PIAAC will be limited to using computers and computer networks. The PS-TRE assessment will only be computer-administered. Retrieved 11.11.13 from http://nces.ed.gov/surveys/piaac/problem-solving.asp

PIAAC Sample PS-TRE Items

Exhibit B-6. Examples of problem solving in technology-rich environments items

Items that exemplify the pertinent features of the proficiency levels in the domain of problem solving in technology-rich environments are described below.

Level 3: Meeting Rooms (Item ID: U02)

Difficulty score: 346

This task involves managing requests to reserve a meeting room on a particular date using a reservation system. Upon discovering that one of the reservation requests cannot be accommodated, the test-taker has to send an email message declining the request. Successfully completing the task involves taking into account multiple constraints (e.g., the number of rooms available and existing reservations). Impasses exist, as the initial constraints generate a conflict (one of the demands for a room reservation cannot be satisfied). The impasse has to be resolved by initiating a new sub-goal, i.e., issuing a standard message to decline one of the requests. Two applications are present in the environment: an email interface with a number of emails stored in an inbox containing the room reservation requests, and a web-based reservation tool that allows the user to assign rooms to meetings at certain times. The item requires the test-taker to use information from a novel web application and several email messages, establish and apply criteria to solve a scheduling problem where an impasse must be resolved, and communicate the outcome. The task involves multiple applications, a large number of steps, a  built-in impasse, and the discovery and use of ad hoc commands in a novel environment. The test-taker has to establish a plan and monitor its implementation in order to minimize the number of conflicts. In addition, the test-taker has to transfer information from one application (email) to another (the room-reservation tool).

Level 2: Club Membership (Item ID: U19b)

Difficulty score: 296

This task involves responding to a request for information by locating information in a spreadsheet and emailing the requested information to the person who asked for it. The

test-taker is presented with a word-processor page containing a request to identify members of a bike club who meet two conditions, and a spreadsheet containing 200 entries in which the relevant information can be found. The required information has to be extracted by using a sort function. The item requires the test-taker to organize large

amounts of information in a multiple-column spreadsheet using multiple explicit criteria and locate and mark relevant entries. The task requires switching between two different applications and involves multiple steps and operators. It also requires some amount of monitoring. Making use of the available tools greatly facilitates identifying the relevant entries.

Level 1: Party Invitations (Item ID: U01A)

Difficulty score: 286

This task involves sorting emails into pre-existing folders. An email interface is presented with five emails in an inbox. These emails are responses to a party invitation. The test-taker is asked to place the response emails into a pre-existing folder to keep track of who can and cannot attend a party. The item requires the test-taker to categorize a small number of messages in an email application in existing folders according to a single criterion. The task is performed in a single and familiar environment and the goal is explicitly stated in operational terms. Solving the problem requires a relatively small number of steps and the use of a restricted range of operators and does not demand a significant amount of monitoring across a large number of actions.

(From Appendix, Page B12 of Literacy, Numeracy, and Problem Solving in Technology-Rich Environments Among U.S. Adults: Results from the Program for the International Assessment of Adult Competencies 2012 First Look. Retrieved 11.11.13 from http://nces.ed.gov/pubs2014/2014008.pdf )

—–