ESRC Research Methods Programme

The ESRC Research Methods Programme funds a considerable number of projects that are developing online resources. In order to establish and explore some of the issues that this raises, we organised a workshop on 15 – 16 December 2004 at the University of Manchester. This was designed to bring together developers and users of online materials to find out what the experts could tell us about the design and storage of online materials.

Full programme and speakers

Introduction

There are major issues in ensuring that online resources are developed so that they:

• are easily located by potential users

• are accessible

• are easy to use, quality assured and fit for purpose

• meet copyright, IPR and ethical standards

Much development work is in the context of undergraduate distance learning and there is a need to adapt knowledge to the requirements of research training. Some of the key issues emerging from the meeting are summarised below:

Online resources are best seen as ‘learning objects’. Definitions of a learning object include:

‘an entity, digital or non-digital, that may be used for learning, education or training’

‘a Learning Object is an aggregation of one or more digital assets, incorporating meta-data, which represent an educationally meaningful stand-alone unit’

For more on learning objects and the source of these quotes go to Steve Rogers’ presentation: A learning object scenario.

Repositories are means of storing/hosting learning objects (and information about them) so that they can be used by the wider community. JISC are funding the development of Jorum as an online repository for learning and teaching materials. For information on why it is being set up, what materials it will hold, where they will come from and how JORUM will work go to Steve Rogers’ presentation: Metadata to facilitate searching and IPR matters in sharing (overheads 9 onwards) and notes on the session.

Jorum will not be going live until autumn 2005. It has adopted international standards for metadata. It will be developing guidelines for potential depositors, e.g. in terms of quality assurance, metadata. Notes on Steve Rogers' contribution to the technical session are here.

For some tips on planning materials to go into a digital repository, see Laura Bond's Tips on planning materials for placement into a digital repository .

To be effective, a repository needs to support interoperability, defined as: “… the ability of two or more systems or components to exchange information and to use the information that has been exchanged”. This means that users are not locked into proprietary systems; content can be moved to and from different systems and materials can be reused and re-purposed.

Meta-data is vital if online materials are to be catalogued and picked up by search engines. There are various metadata standards, including IEEE LOM (Learning Object Metadata) Standard, developed specifically to describe learning resources and which can be mapped to Dublin Core. Use of this standard, or sub-sets of it, mean that content can be shared between distributed systems and metadata can be “harvested” by portals to direct users to content.

For more details of the metadata standards required by JORUM go to Steve Rogers’ overheads.

It is recommended that all online resources should deposit metadata with JORUM. Some developers may wish to deposit their entire resource; however, JORUM will be setting particular requirements before accepting resources. It will retain materials in the form deposited; it will not update or revise materials.

Jim Petch presented a session on managing the demand side of repositories, a note of which is also available.

Licensing and Intellectual Property Rights are important areas for producers and users of online materials. Steve Rogers identified two kinds of licence: a licence that allows you to modify materials, extract and replicate and use with other materials and a licence that does not allow you to do any of these things.

A crucial question is who holds IPR rights and the key to this is usually in the employment contract. A guide to IPR written by John Casey is available at: http://www.jisclegal.ac.uk/publications/johncasey_1.htm

Practical consideration for producers of online materials include:

i. setting up an IPR framework from the start of any project and taking legal advice at an early stage.

ii. clarifying the distinction between copyrights, that can be sold and transferred and moral rights, that cannot be transferred from the original author but must be respected when any plans to modify content are made.

iii. clarifying who owns materials. Is it the funding body, e.g. ESRC; the employer, e.g. the university or the author, the academic/researcher. The answers may vary depending in individual circumstances.

iv. are materials designed to be available to all or only to HE/FE?

v. can users re-write materials; use small bits etc?

John Casey’s presentation The Role of IPR in content development and sustaining a community of practice and a note of his talk are available here.

Damon Berridge provided a note of his enquiry to software suppliers over use of screen shots in teaching materials and use of packages for training courses.

Designing effective online resources

Gráinne Conole presented a six-step framework for e-learning. She identified the gap between the potential of the technologies (confusion over how they can be used) and application of good pedagogical principles (confusion over which models to use). She also introduced 'The Learning Activity Toolkit which can be used to provide:

• Guidance

• On the development of learning activity

• Mapping pedagogy to tools and resources

• Repurposing

• Query database of existing learning activities

• Research

• Development of new e-learning models

• Quality assurance

Evaluation of materials needs to be built in to the development of online materials from the start. Evaluation can produce documentary evidence of assumptions, processes and outputs and also provide a source of evidence for QA. It can also increase innovation and transfer of good practice. More details can be found at: Design effective online resources.

Evaluation is particularly important with online resources where, unlike face-to-face teaching, there is no feedback from users and you cannot control what users do with materials.

Rob Shaw and Tristram Hooley discussed the use of evaluation in their project to develop online resources. They stressed the importance of understanding how resources are used; by whom and in what circumstances; the need for good web design – need to find ways of keeping the user’s attention and the need to avoid over-reacting to comments from one or two users. Different aspects of a resource may be evaluated in different ways. For example, academic content may be evaluated by academic peer review; the design/look of the pages by user feedback and evaluation questionnaire and the usability of the materials by expert evaluation and by observation of users and cognitive walkthrough (watching users and getting immediate feedback). For more details go to: The evaluation life-cycle .

Ethical and confidentiality issues arise across a wide range of materials and apply especially when access to resources is unrestricted. Bruce Mason and Matthew Williams described their ethnographic research in a multi-media science centre where many of the participants are children. They raised difficulties in achieving active informed consent: for example, should it be opt in or opt out? how much information should be provided (the more provided, the more concerned respondents become)? How can you allow participants to withdraw consent in the future? The reuse & repurposing of data are both issues in this example. Unresolved questions include: when is data personal under Data Protection Act and when is it anonymised? Can you identify someone from their image on a screen? Or do you need additional information, e.g. interview transcript or speech? For more details go to: Online Ethnographic Research: Method, Ethics and Practice .

Louse Corti outlined principles and procedures used by Qualidata at the Data Archive .

Consent for reuse – many people are very happy with this.
Consent for archiving must be obtained.
Anonymity & privacy have to be respected.
People have to be informed about the nature of the research (for example that information will be shared)
Copyright waiver must be obtained.
Some people want their names attached (especially in oral histories - need to respect this).
Researchers have a duty of confidentiality

Notes taken during Session 3a are available here.

Methods of sharing information

Cormac Connolly of ESRC delivered a presentation on sharing online resources, introducing the history behind ESRC Society Today and the Information Centre and illustrating how it may look once it goes live. The first half of the presentation, Methods for sharing online resources, focusses on delivering an overview of the technology that sits behind the project is available. For further information, please contact Cormac Connolly.

Notes on Session 3b are available here.

Disclosure problems with design information for surveys

Confidentiality and disclosure issues also arise with online materials that need to make survey design variables available. These may include primary sampling unit identifiers; information on strata used (e.g. local authority) and weights.

Disclosure can occur if the location of individual clusters is known; where a stratum is small and a large proportion of the stratum is sampled and where there is the means of linking the data on the web back to the full data source.

Steps to prevent disclosure include changing cluster identifiers so they no longer reveal location; changing IDs so they cannot link back; adding noise to the weights so they do not identify individuals; and concealing the details of how the strata are defined. For alternative methods, e.g. replicate weights, go to Gillian Raab’s presentation at: Disclosure problems in providing design information for surveys .

Joining up online resources

The ESRC’s Information Centre (Society Today) is designed to provide a first port of call for anyone interested in the UK’s society and economy. It will be a collaborative international platform for UK social and economic research. It will provide easy access to the latest knowledge generated from the ESRC’s research (and others) into a wide range of issues affecting modern British society. Details of the presentation by Astrid Wissenburg and Cormac Connelly are at: The ESRC Information Centre . Later phases of the development will include teaching and learning materials, including existing materials already produced in the community.

Long term storage

Long term storage of teaching and learning materials is vitally important. Michael Dodds presented a view of JORUM as a longer-term solution for learning resources - JORUM as a longer-term solution for learning resources . However, JORUM assumes a static resource. Most resources have a finite shelf-life before they need re-developing. However, after an initial period – say three years – the amount of use made of a resource would provide the evidence needed to decide whether to support redevelopment

The Digital Curation Centre (DCC) provides advice to HEIs on curation and preservation of online resources. It aims to work with repositories, rather than being one . Preservation options for the online resource creator include:

Personal custodianship – (lifetime limit?)

Deposition in a ‘trusted repository’, either discipline-based, object-type based or institutional-based

Sharing your data: LOCKSS

Being captured in a web archive:

• http://www.archive.org/ , Wayback machine

• UK Web Archiving Consortium (BL, JISC …)

Full details of Robin Rice’s presentation: Curation and preservation of online resources: the DCC and options for resource creators .

What the future may hold

Resource Discovery for Researchers in e-Social Science (ReDRESS) Rob Crouchley

REDRESS is developing a resource discovery tool (some Portal software) that constructs a tutorial (Learning Design/content sequence) appropriate to a users needs. It will integrate content created by most standard authoring systems (including video) that is visible on the web. It uses the grid and portal software, incorporating many developments from the USA.

The Grid can offer huge benefits in terms of: high-performance computation; collaborative visualization between distant researchers (Access Grid); moving large volumes of data; real-time monitoring (e.g. traffic flows, electronic transactions); orchestrating distributed content (educational grid). However, Rob Crouchley argued that, critical to these developments, are the creation of open source and platform independent software.

Portals are a framework for using deploy e-tools; they allow component integration, so that tools can work together closely and seem to really be parts of a larger “tool”. Portals have a lot of features that come ‘out of the box’, like chat, file sharing, etc. This makes them useful straight away. They allow customisation, preferences and improve accessibility. Portals can p rovide increased functionality, web pages and resources tailored to the needs of the individual and thus provide an ideal framework for e-Research tools and for resource discovery; they can provide an interface to many e-Research tools/services and resources and bridge the gap between e-Learning and e-Research.

There are exciting developments in the USA; e.g. Sakai/CHEF has already been adopted as the open source hosting framework of choice for Virtual Learning Environments (VLEs) and VREs development in the USA. However, in order to fully benefit from these developments we need to be moving away from proprietary software and towards open source software.

Use of the Grid for analysis of video data

Jon Hindmarsh and Katie Best presented their research into the use of the grid for collaborating over the analysis of video data. The grid supports analysis where a number of analysts are co-located or where data is shared remotely. More information at: Towards Remote Collaboration Over Video Data: The VidGrid Project .