CoronaNet Research Project Manual


This document contains thorough instructions for project implementation for CoronaNet. Given the complexity of this project, this documents aims thoroughly document all of our procedures so that we can ensure communication to promote quality control and produce as scientifically rigorous dataset as possible.



Data Collection and Entry


What is data collection and entry?

Data collection and entry forms the foundation on which the rest of the project is built. The ultimate goal of the project is to produce an accurate and complete record of all policies governments have taken in response to COVID-19. This entails 1) identifying relevant policies that governments have taken in response to COVID-19 2) documenting them in a standardized format.

What should I do before starting with data collection and entry?

Before entering in data, you should:

  1. Watch the required general RA onboarding video before starting to collect the data.
  2. Watch the required training on zoom on how to do updates to the data.
  3. Read through the Variable Definitions in the codebook and the corresponding examples provided below.
  4. Look through the RA FAQs in the Shiny App for commonly asked questions.

The value of this data is only as good as its quality so please be sure to take your time with documenting these policies, and pose your questions to the Slack Channel or the PIs if in doubt!

How do I actually do data collection and entry?

Assess, Plan, Code, Clean!

Doing data collection and entry requires having knowledge about 1) what the policies are in the region that you are responsible for and 2) how to document those policies in our survey instrument. To get a handle on both of these issues, it helps to first assess what the policies are in the region you are coding, make a plan for how to code them in the survey, code the policies into the survey and clean any previous entries or inconsistencies in the data. In the below, we provide more guidance on what this all means:

Assess

In order to assess the current state of the data and to learn more about their country before they start coding, RAs are asked to create a country overview template. This enables everyone to get a good understanding of the policies that have already been entered in their country/region, so that there is no duplication of policies, to see what policies might need to be ended or updated and also what policies have not been entered at all. This should be finished and sent to the Regional Manager within 7-10 days.

  • Video how to fill out country overview template (Password f5G6^AY7).
  • Shinyapp guide.
  • How to use the Shinyapp video.

Plan

Next RAs are asked to create a plan for how they want to enter their first policies. This helps see how policies are connected to each other with updates and gives the Regional Manager a chance to offer some feedback and tips on how to better code policies. Once the plan is submitted to the regional manager, they will set up a zoom or skype meeting for some one-on-one feedback.

Code

The process of coding policies is different for every country and region, some countries have relatively straight forward policies and others have fine grained and confusing policies. When in doubt on how to code a policy, we recommend that RAs:

  • Ask their question in the ra-chat, regional/country channel, or ask their regional or country manager.
  • Check out the coding guide.
  • For more general support check out our Zoom hangout room where RAs can code together or have a chat (Password 12345).

Clean

In order to achieve the project goal of having clean and complete policies we are focusing on the policy end dates. We have currently a weekly missing end dates competition in this video here to encourage RAs to bring these up to date as soon as possible. Learn more about the missing end dates competition in this video here and find out which policies are missing end dates for your country/region here.

You can use the automated data quality checklist to double check whether some of the data for your country needs to be updated. Remember to be in touch with your regional or country manager to do a full assessment and make a plan as to how to best bring the data up to date.

What counts as a ‘policy’?

These policies can be thought as being in one of two categories:

  1. A new policy
  2. An updated policy

A new policy is a policy in a category that the government has not addressed before, such as the imposition of a quarantine on the country. An updated policy is a policy change in which the details of the policy change, but not whom the policy is targeted at. For example, if Germany announces a quarantine for one province, and then later extends the quarantine for two weeks, that would count as an update. But if Germany announces a quarantine covering the whole country, that would count as a new policy. The same logic goes for travel bans – making travel bans cover more countries/areas counts as new policies. Changing the way existing travel bans are enforced would be an update. For more information, please reference the codebook below. Please watch the required training on zoom on how to do updates to the data for more information and refer to the accompanying slides.

If in doubt, ask on the #ra-chat Slack channel!


In the Qualtrics survey, you must select either new or updated. To update a policy, find the previous record either:

  1. The Record Look up sheet or
  2. The Shiny Web Application

Moving forward, we would want you to use the web application as it contains the same data as the Record Lookup sheet, but in a format that is easier to visualize. Also, you can do some corrections in the app itself.

To update, click on the ‘update’ link that is on the left-hand side of the table/google sheet. See pictures:

How do I document a ‘policy’?

The below gives a textual overview of how to document a policy. Please refer to the required general training on Zoom for more information.

Use the following checklist:

  1. For each country, check the data sources, in the order given below in the “Data Sources” section, for relevant information on government policy toward combating the coronavirus.

  2. Enter the relevant information into your country specific dataset. It should always be possible to answer the following request/questions for a particular policy:

    • What type of entry is this? (e.g. new, correction, update)
    • Description of the government policy.
    • What is the policy category? (e.g. travel ban)
    • From what level of government does this policy originate from?
    • What kind of enforcement does this policy have (if any)?
    • Which organizations or entities are in charge of enforcing a policy?
    • When was the policy announced?
    • What information sources did you use to identify this policy?


    Some, but not all policies,[^1] will also require you to identify:

    • Which geographical entity is the target of this policy?
    • What or whom is the policy targeted toward?
    • Is this policy inbound, outbound or both?
    • What kind of transportation is this policy targeted toward?

    When it is not possible to collect this information, please note the reason for the difficulty at the end of the survey.

    Please document each source by answering the according survey question. Currently, the dataset allows you to code up to 10 sources. Note also that it is possible that you will need more than one source to collect all the information for each event.

  3. Save a .pdf for each data source that you use. As the file name, use the following format: [Date Collected]_[Title of article].pdf where the [Date Collected] should follow the format MM-DD-YYYY.

  4. You can upload the file within the Qualtrics survey (be sure to choose the correct country).

How do I find a policy?

The rest of this document tells you how to go about finding policies. Of course you can just use your own knowledge and Google, but we have some great resources we’ve identified below you should use to get started.

  1. We have a list of policies that are coded in other sources external to the project. This external data should be seen as an additional source of information that we hope will make it easier and faster for RAs to identify policies to document for their country or subnational region. You can check the list of these policies in the CoronaNet Data Integration Sheet. In the section titled “Data Integration” later in this document the entire Data Integration process will be explained in detail.

  1. We now have a source of news articles possibly related to the coronavirus by country compiled by the data science company Starsift. Please see this link for a list of news articles (articles that are more likely to contain a policy are marked in the “policy” column, though all of the articles could contain some information. Once you’ve worked through that dataset, you should do some searching on your own. Of course Google/Google News is a good asset, but there are other available sources:

  1. Check the U.S. Embassy website for your country.

What is this?

Most U.S. Embassies’ have a designated webpage for disseminatingCOVD-19. Information for a particular country. For example, the U.S. Embassy & Consulates in Germany’s web page for this information can be found here.

How to access this source for your country:

You should be able to find this page for your website by Googling “U.S. Embassy [Your Country Name] COVID-19 Information”.

How to use this source in the dataset:

The U.S. Embassy’s designated webpage on the coronavirus in a particular countries will often list information on, for example, entry and exit requirements or quarantine information. Often the U.S. Embassy will also link to the primary home government source for this information.

  When links to the primary source of information are made available:     
  1. Click on the link.
  2. Corroborate the information from the primary source.
  3. Use this link as the primary source of in
  Use the U.S. Embassy page as the primary source if:     
  • There is no link or.
  • The information on the linked page is in a language you do not speak.

Please especially make sure to save a .pdf of the relevant page for your country for the day you collect the information if using the U.S. Embassy page as a source because it may update relatively frequently with new information.


  1. Check Wikipedia for your country’s response to the corona pandemic

What is this?

For many countries, there is a designated Wikipedia page for how a country has dealt with the coronavirus pandemic. For example, the “2020 coronavirus pandemic in the Dominican Republic” can be found here.

How to access this source for your country:

You should be able to find this page for your website by Googling “Wikipedia [Your Country Name] corona”.

How to use this source in the dataset:

You should treat the Wiki as a platform that aggregates information sources, not as an information source itself. That is, you should NOT cite the Wikipedia page as your news source. When entering information from Wikipedia into the dataset, please do the following:

  1. For every claim made on Wikipedia, visit the link of cited source in the footnotes.
  2. orroborate the information stated on Wikipedia in the information source.
  3. Add additional relevant information from the primary source to the dataset if you find it.
  4. If there is no source given for a claim made on Wikipedia or if you cannot corroborate the source, Google it — if you still do not find a corroborating source, do not enter the information into the dataset.

  1. Check Government websites for your country

What is this?

Different countries will have different government ministries issuing information in response to the crisis. In the course of gathering information on your country from going through the relevant U.S. Embassy and Wikipedia web pages, you may be able to identify the most relevant government ministries for your particular country. For example, the Taiwan CDC has up to date information on government policy toward the coronavirus.

How to access this source for your country:

If you have identified the relevant government ministries in charge of responding to the corona crisis for your country, Google: “[relevant ministry or department] corona”.

If, however, you have been unable to identify a relevant government body from the relevant U.S. Embassy and Wikipedia web pages, then:

  1. Identify/Google the executive body for your country and your country’s government bodies in charge of health, foreign affairs, or internal affairs and;
  2. Google: “[relevant ministry or department] corona”.

How to use this source in the dataset:

We consider information that comes directly from the government as a primary source. Identify any relevant information from the government websites and enter them into the dataset.

Please especially make sure to save a .pdf of the relevant page for your country if using the a government page as a source because it may update relatively frequently with new information.


  1. Check Newspaper coverage on the coronavirus in your country

What is this?

LexisNexis, Factiva are platforms which aggregate, among other things, newspaper articles around the world.

How to access this source for your country:

Use your home institution to access these datasets.

For NYU students, you can do so here:

For the Technical University of Munich, you can do so here:

How to use this source in the dataset:

You can search these articles for different search terms such as “quarantine” and “travel ban” to see what pops up and when. Please limit your search to December 31, 2019 (when China first reported the coronavirus to the WHO) to the present.

We consider information that comes from newspaper articles as a primary source. Identify any relevant information from the newspaper article and enter them into the dataset. However, if you have doubts or qualms about the rigor of a particular publication outlet, please do not hesitate to raise your concern with us.


  1. Other resources

We will update this section accordingly as more information becomes available. For now, some other resources you might consider include:

  • Check Facebook and Twitter accounts for political leaders in the country (or for health ministries).
  • This is a crowd-sourced list of government actions against the coronavirus by the Open Government Partnership (OGP). Note please be sure to vet the sources here carefully as the OGP themselves do not control for the quality of the entries made here.
  • This website is a crowd-sourced compendium of information about the coronavirus. For now these appear to deal largely with the health outcomes associated with coronavirus but may be useful insofar as includes many links to in-country websites.

What to do if I need to delete an entry?

Please post the entry that you need to be deleted in the Policy Delition sheet and provide a reason for why the entry needs to be deleted.

With regards to incomplete records, please note:

  • We DO include in the dataset records for which the RA filled everything out except they didn’t hit the final submit button; all other records are not included.
  • If, however, you are seeing a record in the record sheet that looks incomplete, please let us know (those are supposed to be filtered out before the record sheet is populated). Note however, if the record is is a correction, it may look incomplete because a correction may only correct one part of the original record.


Data Validation


Why do we validate data?

Data validation is required to ensure our dataset is reliable and accurate. At the moment we have over 500 hundred hard working research assistants that are entering more than 17,000 policies, updates and corrections. With so many minds, there is bound to be a difference of opinion and skill on how policies should be coded. This difference gives rise to conceptual errors and policies that have been entered that differ to our coding guidelines. For example, is a policy about quarantining or lockdown? Should a policy be entered as one, or broken down into smaller policies? Ideally, we should have high intercoder reliability, which means that two independent people will code the same source in the same way, but with a complex survey this is increasingly difficult. To increase intercoder reliability, we have designed a validation process.

How do we validate data?

Validation is split into two main stages; recoding and reconciliation. Recoding is carried out by experienced coders, from a source chosen out of the randomly selected validation dataset. Reconciliation is done by the validation PIs, who take the original policy and the recoded entries and try to find a match, determining if two independent coders, who had the same source, found the same policy and coded it in the same way.

How do I join the validation team?

  1. Send an email to Joan (joan.barcelo@nyu.edu) to signal your interest in becoming a validator.

  2. We will then send you a link to a quick test survey that helps us check your coding level. We do this to ensure that everyone who joins the team has a comparable level of coding skills. The score threshold for being added to the validation team is 70% accuracy.

Getting started with validation

  1. Once you pass the test, Joan will add you to the ra-validation slack channel.

  2. Pinned to the slack channel is a Google Sheet listing all observations to be validated (also known as the validation dataset) and the link to the validation survey. Click on it and follow the instructions below.

Assigning an observation

  1. Look for an observation in the Google Sheet validation dataset that is not assigned to anyone else.


  1. Click on the source to see the country/language.


  1. If you have the skills to code that observation (and you did not code that observation in the main dataset), then put your name on the RA_validate column in the Google Sheet.

  2. After assigning an observation to yourself, click the validation survey link, select “Validation” and insert the corresponding “record_id,” which you can find in the spreadsheet. Identify yourself using the dropdown menu. If your name is not in the list, please send an email to joan.barcelo@nyu.edu and they will add you.

  3. Complete the survey for as many entries as you think it is necessary to have a complete picture of the policies announced in that country on that date from that source - in case of doubt, too many is better than too few.

  4. Once done, put a 1 on the column “Validation_complete”.

Note: If the link to a source does not work, is missing, or links to an updating source, please message the PIs in the ra-validation slack channel to ask for the original PDF.

Second Round of Validation: “Checking”

Every entry in the validation set will be validated twice. The second round of validation - termed checking for clarity - will help us determine the most appropriate coding for each policy. The procedure for checking is _nearly _identical to the first round of validation described above. Two important things to note:

  1. You should not check an observation that you coded in the original dataset.

  2. You should not check an observation that you validated in the first round of validation.

It is important that each round (original entry, validation, and checking) be done by three different individuals. This ensures that our original coding, validation, and checking processes are independent of each other and makes the quality of our data more reliable (statistically speaking)! What follows is a description of how to get involved in checking the data.

Getting started with checking

  1. You should email Joan with your intention to join the validation/checking team and pass the same coding survey test described in the validation section with a score of 70% or higher.

  2. Once you are added to the ra-validation Slack channel, open the validation spreadsheet and look for an observation that is blank or NA in the column “RA_checker”.

  3. Click on the source/s to see country/language.

  4. If you have the skills to code that observation (and you did not code that observation for the main dataset and for the validation dataset), then put your name on the RA_checker column (you are assigning this observation to you).

  5. After assigning an observation to you, then click on the survey link and complete the survey for as many entries as you think it is necessary to have a complete picture of the policies announced in that country on that date from that source—in case of doubt, too many is better than too few.

  6. Please, find your name in the list of contributors within the Survey (if your name is not in the list, send an email to joan.barcelo@nyu.edu).

  7. Select “Checker Entry” from the multiple choice options and enter the record ID that you are checking.

  8. Once done, put a 1 on the column “checker_complete”.

Data Integration


This section contains instructions for data integration. Before reading this, you should make sure you have access to:


What is Data Integration?

Data integration is assessing the overlap between data external to CoronaNet and recoding data that is currently missing from CoronaNet into the CoronaNet taxonomy, while giving proper attribution to the original coders. To see the list of external datasets we are integrating data from, see the Data Integration Guidelines.There are in total around 150,000 policies that we have identified in the external data which we can potentially integrate into the CoronaNet dataset. We have done our best to map this external data to the CoronaNet taxonomy but there are some things that we cannot control and we need your help to work through.

The external data should be seen as an additional source of information that we hope will make it easier and faster for RAs to identify policies to document for their country or subnational region. However, like all sources of information we use, we still need to vet this information carefully before incorporating it into the CoronaNet data. While there are around 150,000 observations in the external data, that doesn’t mean we aim to increase the CoronaNet database by 150,000 observations. This is for a variety of reasons.

How to integrate data?

First of all, make sure you read the Data Integration Guidelines and you have access to the CoronaNet Data Integration Sheet:

  1. Identify data that other datasets have documented but which are not currently in the CoronaNet dataset. Fill in this information in the overlap_assess column in the CoronaNet data Integration sheet.


  1. Recode the data that is not currently in CoronaNet:

    1. To recode this data, RAs should adhere to the following steps:

      1. Identify policies that are not currently in the CoronaNet dataset by searching for rows that have the value ‘No’ in the overlap_assess column

      2. Click on the ‘link’ or ‘pdf_link’ for that observation and read through the information in the raw data source.

      3. Code the information that you find in that link into the CoronaNet Qualtrics survey. You can use the other column information which maps in the CoronaNet Integration Sheet as a guide for how you can code certain fields.

      4. At the end of the CoronaNet Qualtrics Survey, you will be asked two questions:

        1. [collab] “If (one of) the sources that you used to document this policy came from another dataset, please note which dataset’ ➪ Information about the dataset that you are integrating will be found in the ‘data’ column in the CoronaNet Data; if you used a source/link that you yourself found, please choose ‘I found this source myself’ instead.
        2. [collab_id] ‘Please copy and paste the unique id of the record that you used from the other dataset in the text entry below’ ➪ If you use a source from an external dataset, in this field, copy and paste the ‘unique_id’ found in the ‘unique_id’ column for this observation found in the CoronaNet Data Integration sheet.
      5. In the ‘integrated’ column in the CoronaNet Data Integration sheet, please choose one of the following:

        1. ‘Integrated’; this means you have identified a policy that was in another dataset and recoded it into the CoronaNet taxonomy.
        2. ‘Integrated with additional original research’: You may have to do some additional research for any number of reasons. E.g. the information that you receive from the link or pdf_link of the external dataset may be unclear or require additional context/knowledge to code well. In such cases, please note what additional research you had to do in the ‘Notes’ column and click this option.
        3. ‘Integrated with additional work to find a new link’ means that the original link for the policy as noted in the CoronaNet Data Integration sheet is dead but that the RA was able to find a new link that corroborates the information described in the ‘description’ column. In this case please choose ‘I found this source myself’ option in 2.4 and click this option in the data integration sheet. Note that if you were able to find the information from the original link using the Way Back Machine then choose the ‘Integrated or ‘Integrated with additional original research’ option as appropriate.
        4. ‘Integrated with additional original research AND with additional work to find a new link’: means fulfilled both the criterion under: ‘Integrated with additional original research’ and ‘Integrated with additional work to find a new link’. See above for more information.
        5. ‘Duplicated policy’: this means that there were multiple external policies that were duplicates of each other. In this case, please only integrate one of them (and choose one of the ‘integrated’ options above). When you click this option, this means you do not integrate this particular policy because it is a duplicate. If the data is already in CoronaNet, pick one policy to mark as ‘Yes’ in the overlap_assessment and find the corresponding record_id to paste into the matched_record_id column. In general, for duplicated data, the overlap_assessment should be ‘No’.
        6. ‘Not a relevant policy’: this means that after having taken a closer look at the link for the observation is not one that we would code in CoronaNet. The corresponding overlap_assessment should be ‘No’ in this case.
        7. ‘Link dead, no other link found’ means that the original link for the policy as noted in the CoronaNet Data Integration sheet is dead and the RA was unable to i) use the Way Back Machine to find the original data ii) find another link to corroborate this information. In this case, please do not recode this data.

This might seem like an overwhelming process at first so don’t worry if you make a mistake. Check out the Data Integration Errors sheet to see if you might have made an error and get credit for your work when the mistake is fixed!

Please make sure to read the Data Integration Guidelines for further tips and things to watch out for when doing data integration.



Data Science Team


The data science team exists to manage the export and cleaning of Qualtrics survey data, and also to visualize and analyze the data. This team is led by Tim Model (Chief Data Scientist) and Cindy Cheng. The members of this team may have some data entry responsibilities but their primary goal is to ensure that our data pipeline is as smooth as possible, and also to employ data-analytic tools to check the data and extract insights.

Regular tasks assigned to the team include:

  1. Daily exports of the Qualtrics data to our Github site, including cleaning the data, uploading for the RA record sheet/RShiny Data Validation App.
  2. Retrieving changed records from the RShiny Data Validation App and merging into the released data.
  3. Maintenance of the RShiny Data Validation App.
  4. Help with adjusting the Qualtrics JavaScript backend.
  5. Extract insights from the data through analysis and visualization.
  6. Monitoring the #orphaned-records channel and adding deleted records to removal lists (and also identify any record issues outstanding).

The data science team currently consists of five groups:

  1. Qualtrics: developing, managing, and maintaining the Qualtrics survey.
  2. Data Pipeline: data cleaning and wrangling, as well as releasing data to record lookup, AWS, and public Github repository.
  3. Shiny App: developing, managing, and maintaining the Shiny app used for data validation by RAs.
  4. Issue Triage and QC: identifying source of errors reported in Slack and elsewhere, including bad record IDs .
  5. Data Visualization: developing visualizations for public-facing presentations of our data.

If you are interested in joining the Data Science team, please message Tim Model or Cindy Cheng on Slack.



Qualtrics


Once an RA is onboarded, they will receive a link to the Qualtrics survey they should use to enter new records.

RA’s should select their name from the drop-down list (Hint: typing your name once the drop-down list is open makes it easier to find yourself in the list).

The CoronaNet Data Codebook can be used to help fill out the Qualtrics survey.

The penultimate page of the Qualtrics survey is a review of all the information about the policy that has been entered, RA’s should scroll down and read this page before finally clicking on the ‘advance arrow’ to ensure the entry gets submitted to the dataset.

How to retrieve a PDF Source?

If you are working on validating an entry, you can request a PDF source by posting in the #ra-validation channel and someone with access to the survey will retrieve it for you. You can also use this list, which will take you to your countries PDF source list.

Note that only those with permission to view/edit the Qualtrics survey will be able to retrieve a PDF Source. You will only be granted access if you are actively working on editing the Qualtrics survey or actively working on the raw Qualtrics output. Validators should not request to view the raw survey, as this will compromise the blind validation process.

If you do have permission to view/edit the survey, you can retrieve a source by following this procedure:

Click on the dataset > Data & Analysis > Add Filter > Embedded data, record ID > Once you have the record you want, click Export & Import > Export > User Submitted Files. This will produce a .zip file of the desired source pdf(s) for that record ID.



Slack


What is Slack?

Slack is a collaboration software tool that enables users to share messages, tools, and files with their fellow team members.

It helps increase productivity amongst large, remote teams and saves everybody from getting lost in endless email chains.

Key Functionalities

Channels

Channels are where the work happens!

Each different part of the project has its own dedicated channel, where all the people involved in that part can easily communicate.

All the messages shared in these channels are searchable, so you will never lose out on information.


Threads

If somebody posts a message in a channel that you want to add to, start a thread specific to the original message.

[](images/manual_img/9. Slack threads.png{width=70%}


Direct Messages

Perhaps you need to contact somebody within the project personally, but there is no need for a specific channel, utilise the direct message function instead.


Notifications / Mute Channel

Maybe it’s the weekend or you’re taking a break; pause all notifications here!

You can also customize notification settings or mute specific channels by clicking on the ‘Details’ tab of the channel.


You can also customize notification settings or mute specific channels by clicking on the ‘Details’ tab of the channel.


Pinned Items

Important/useful messages in a channel may be ‘pinned’.

‘Pinned’ items can be viewed here!


CoronaNet Workspace

All the information you need regarding the CoronaNet project can be found in our Slack workspace. In the rare circumstance you can’t find what you’re looking for, there will be somebody on-hand in Slack to help you answer your question.

Queries regarding policy coding, data validation, data cleaning, existing records, using the CoronaNet app, Hogwarts houses, Zoom team meetings, even birthday wishes for your CoronaNet colleagues, can all be found in Slack. It is the beating heart of CoronaNet!

Slack workspaces have subsets of different channels, the more aspects of the project you are involved in, the more channels you will be invited to join.

To help you navigate your way through the different channels, below you will find a brief summary of each one.

# random With such a global team, spread across multiple time-zones, it’s impossible for us all to gather around the water cooler for a short conversation. Utilise this channel as a substitute! Also, a good place to wish people a happy birthday!

# shinyapp-cornedit Direct any question you have about the Shinnyapp here

# orphaned-records Use this channel to talk about any records you are having trouble locating (or otherwise need help with).

# research-ideas An open forum where you can share any research ideas you may have as you are entering data. Have you noticed anything peculiar / interesting / innovative about certain Countries / Regions approach to policy making? This is the channel to share those thoughts in.

# Country / Region Once you are assigned a Country to work on, you will be invited to the corresponding channel. These channels are a platform for you to coordinate with the other RA’s working on the same Country / Region as you and provide an easy way for you to contact your Country/Regional Manager regarding any specific questions you have.

# Gryffindor / # Hufflepuff / # Ravenclaw / # Slytherin Once you are registered as a member of CoronaNet, the sorting hat will assign you a house! Use this channel to meet your fellow house members and consider it an additional support resource for any questions you may have.

Slack ‘Office Hours’

We have Slack ‘Office Hours’ in the #ra-chat Slack channel from 8am CEST to 4pm CEST every day to answer your questions and provide feedback (while we may also be on at other times, we are making a commitment to being there during those times in particular). Please feel free to post to the channel in off-hours as well! We will get to your questions and comments as soon as we can.

We have some great RAs who have been working with us on this project for a long time who will be stepping in to help out with the #ra-chat questions, they are:

  • Monday: Marco Waldbauer and Hans-Peter Novak.
  • Tuesday: Paloma Laye and Kader Saigili.
  • Wednesday: Celine Heng and Silvia Biagioli.
  • Thursday: Whitney Burkes.
  • Friday: Isaac Bravo and Jurgen Kadriaj.

Note that a PI will always be monitoring these exchanges and a PI might step in if there is a tricky question, but you’ll be in very competent hands with the team above!

Shiny Web Application Guide


In this section, we describe some of the useful features and tools of the web application for updating records, in addition to providing some helpful background for how it works.

Shiny App

The Shiny App can be found here. In order to access the app, you will need to get permission. You should get a message from “shinyapps.io” in your email to get access to the app. Please search your email for “shinyapps.io” if you did not receive the invitation. If you still do not receive one a few days after joining the project, please message in the #record-tracker-app channel on Slack.

What do I use the app for?

You can use it to:
* Make corrections to policy entries (NOT updates) * Find links to make corrections or updates to a policy entry * Look at what policies have currently been entered for certain countries * See a timeline visualization of policies * Find FAQs for internal use about the project

How often is the data updated in the app?

The data is updated approximately every 24 hours. In the “Recorded Data” section of the app, it will note the most recent data currently in the app.

.png){width=60%}


How do I use the app?

You can filter the data in the Policy Table in the app in two ways:

  1. You can use the filters for country, RA name, policy type, etc.
  2. You can click on a policy in the Timeline Visualization.

Filtering using country, RA name, etc:

  1. Select the Country and/or City/Province of the records you are trying to locate (one or more Countries can be selected simultaneously). You MUST select at least one country.
[](images/manual_img/17. Shiny select country.png{width=60%}


  1. To further refine your search, the specific policy category can also be defined


  1. “Policy Records Table” displays a full list of all available records that match the criteria entered above
  • The “Correct” and “Update” buttons, shown in the table above, are a direct link to the Qualtrics survey for that individual record. The ‘record_id’ and ‘policy_id’ are also easily identifiable in this table.


  1. “Timeline Visualization” shows the timeline in which the policies are active, and can be used to monitor ongoing policies and see if they need to be updated/ended.


  • By hovering the cursor over one of the blue tiles for a few seconds, the record description will appear.
  • By clicking on one of the blue tiles, that record will become ‘selected’. The record_id’s of the policies selected by the user will appear at the top of the page (top left of the screenshot above).
  • If the user then clicks on the ‘Policy Records Table’ tab again, only the records selected by the user will now be visible in the list (as shown below).


  • Utilise the ‘Clear records selected’ button to clear this filter and repeat the process from the beginning.


Project Administration


Organisational Flow Chart

Click here for high resolution version


  • All questions about how to do something in the project/where to go to help, please contact the prefects.
  • Any questions about your region, sources for entering your policy or if you need a break, please contact regional managers.
  • Please see the table on the next page for name of contacts.


CoronaNet organizes its coding in regions and countries. Find out what region your country belongs to by checking out the map below or visiting this link for an interactive version of this map!


PI Responsibilities


The Principal Investigators (PIs) are in charge of charting the overall course of the CoronaNet Project and for providing guidance and direction for RAs working on different parts of the project. They are also the points of contact for collaborations with other research endeavors or for handling media requests.

In addition, the PIs also take responsibility over certain parts of the project:

Main Qualtrics Survey

Cindy takes primary charge of making changes to the main Qualtrics survey in consultation with the other PIs.

Data Validation

Joan Barcelo

Cindy also provides support for the cleaning process and makes changes to the validation survey to mirror the changes made to the Qualtrics survey

R Shiny App

Robert Kubinec

Project Management

Luca Messerschmidt

Task Force Managers Responsibilities

Task Force managers are responsible for providing oversight and support for the specific tasks their group is focusing on.

Organization

  • Task Force will be added to the #task-force-managers Slack channel.
  • In addition to the weekly meetings that Task Force managers should have with the RAs under their supervision (see more below in the ‘Key Responsibilities’ section), there will be periodic meetings for task force managers to provide feedback and exchange information with each other and the PIs. These meetings will be announced in advanced in the #task-force-managers Slack channel.

Scope of Responsibilities

  • Task Force managers are responsible for fulfilling the ‘Key responsibilities’ listed below for their given task areas

Onboarding new RAs

Assess, Plan, Code, Clean!

Doing data collection and entry requires having knowledge about 1) what the policies are in the region that you are responsible for and 2) how to document those policies in our survey instrument. To get a handle on both of these issues, it helps to first assess what the policies are in the region you are coding, make a plan for how to code them in the survey, code the policies into the survey and clean any previous entries or inconsistencies in the data. In the below, we provide more guidance on what this all means:

Assess

In order to assess the current state of the data and to learn more about their country before they start coding, RAs are asked to create a country overview template. This enables everyone to get a good understanding of the policies that have already been entered in their country/region, so that there is no duplication of policies, to see what policies might need to be ended or updated and also what policies have not been entered at all. This should be finished and sent to the Regional Manager within 7-10 days.

  • Video how to fill out country overview template (Password f5G6^AY7).
  • Shinyapp guide.
  • How to use the Shinyapp video.

Plan

Next RAs are asked to create a plan for how they want to enter their first policies. This helps see how policies are connected to each other with updates and gives the Regional Manager a chance to offer some feedback and tips on how to better code policies. Once the plan is submitted to the regional manager, they will set up a zoom or skype meeting for some one-on-one feedback.

Code

The process of coding policies is different for every country and region, some countries have relatively straight forward policies and others have fine grained and confusing policies. When in doubt on how to code a policy, we recommend that RAs:

  • Ask their question in the ra-chat, regional/country channel, or ask their regional or country manager.
  • Check out the coding guide.
  • For more general support check out our Zoom hangout room where RAs can code together or have a chat (Password 12345).

Clean

In order to achieve the project goal of having clean and complete policies up until October 1, 2020 by our deadline of January 1, 2021 we are focusing on cleaning different policy types each week. RAs are to fill out their regional/country checklist once a week. This checklist breaks the cleaning process into small tasks for each policy type.

  • Regional country checklist.
  • Video on how to fill out the checklist (Password tCQ0%8wC).

All these links, resources and more can be found on our CoronaNet Research Assistant Dashboard.

Key responsibilities

  1. Once a month please fill out the RM/CM Monthly Survey where we ask you some basic questions about what goals you’ve set for your country or subnational region and how RA activity has been like for each country or subnational region.

    1. Each manager should feel free to fill out this survey separately for just the jurisdictions they are responsible for. If this is the case for you, just choose the number of countries/subnational regions that you are responsible for in the dropdown menu.
    2. It may be the case that because there is no RA currently active in a given country or subnational region, it makes no sense to report goals. There is an option in the survey for this situation.
    3. Finally, in the next survey we will send around next month, we will ask you about your progress towards your goals. At that time, you will also be given the option to adjust your goals depending on how it went over the last month.
    4. Additionally, please contact the project management team to provide any additional information that you think is important to report. This can include:
      1. Whether there are any inactive RAs in your group.
      2. Whether there are any country reports or research notes you would like to be posted to the website.
  2. Check for up-to-dateness. This involves providing information about when an RA was last able to code policies for their country/region.

    1. By ‘up to date’ we mean, just knowing about when the last time it was that an RA was able to look at a particular country in a given week. It is fine if they didn’t have time, just having the information is in itself valuable
    2. Please have the RAs under your responsibility fill out the excel sheet: noting when they were able to document policies for a particular country.
      1. They should open the “ ‘Policy Up-to-Dateness” tab and use the pop-up calendars to indicate when they last updated policies for each of the policy areas.
    3. If the work is not up to date, please make sure that you know the reason why.
      1. If the RA does not have time for a given week but is otherwise stated a commitment to the project, then note this in the excel sheet by filling in “On a break” in the “Are the RA’s Active?” column.
      2. If the RA is unresponsive/no longer active, these tasks may need to be reallocated. You may also nudge and message or nudge RA in your region to be more responsive.
  3. Check for completeness and/or accuracy of records.

    1. By ‘completeness’, we mean assessing, to the best of your ability, whether
      1. All policy types for a particular region are coded:
        1. For example, maybe there are no policies about restrictions of mass gatherings because a) the RA didn’t code it or because b) there are simply no policies created by that government. Part of your job as a regional manager would be to communicate with the RA and figure out if the issue is a) or b).
      2. Each policy type is appropriately coded over time:
        1. For example, it may be the case that there are multiple policies about ‘restrictions on mass gatherings’ that overlap over time. This may be because a) the government has implemented sequential updates to one policy about restriction of mass gatherings such that the policies should not in fact, overlap in time and corrections need to be done to fix the timeline of a given policy or b) the government has implemented multiple policies for restrictions of mass gatherings that should indeed be coded separately and there is no adjustment that needs to be made. Part of your job as regional manager would be to assess if the issue is a) or b) and communicate with the RA as necessary.
    2. The RShiny App is primarily useful for comparing policy types and countries to pin down which policy types in which countries appear to be out of date. You can also use it to see if policies are no longer enforced (end date has passed) but may still be in force, and which policies are in force but may have expired.
    3. Document the completeness of the records for each week in the excel sheet in the ‘ ‘Policy Completeness Form’. Note we expect that checking for ‘completeness’ will be a medium-term task, such that in the beginning, it might take one week to check the completeness of a just a few policies such that it might take 4-6 weeks to do an initial assessment of the completeness of the policies for the regions under the regional/country managers’ purview. After this initial assessment is done, going forward, assessing completeness should take less time/be comparable to the time it takes to check for up-to-dateness. This is a ‘joint’ task in that both RAs and regional/country managers can fill in this spreadsheet, but regional/country managers are responsible for verifying the RAs ‘completeness’ assessment.
    4. If the work is incomplete, contains errors or requires attention, please contact the RA directly to resolve any issues. If the RA is no longer active, these tasks may need to be reallocated. Regional managers may also message or nudge an RA in their region to be more responsive.
    5. If you find repeated errors with the coding in your region, leave a note in the “RA Activity+Notes” tab.
  4. Delegate tasks within their region. If a new RA is allocated to their region, they can delegate states or tasks to them as needed.

  5. Check and give approval for country reports for their region.

  6. Weekly meetings with their team. Each Regional and Country manager should organize a short weekly Zoom meeting with your team (suggested length of time is half an hour but regional/country managers have flexibility to decide how long they want their meetings to be). Ideally regional and country managers would not go over the updates and completeness line by line in the meeting for each team member. Rather the meeting should be focused on discussing and resolving any issues in terms of the updateness or completeness of regions under your responsibility.

For country managers: to to allocate and designate tasks for their subnational regions, please make use of this sheet.

SOPs for Inactive RAs.

  1. Check if the RA is marked inactive in the RA Allocation Sheet: If the RA is colored in orange and has number “2” marked in the inactive column, this means that the RA is no longer active with the CoronaNet Project and you should proceed to step 4.
  2. If the RA is not marked inactive, message the RA on Slack. If you do not receive a response in a reasonable time, follow up with the RA via email.
  3. If there is no response from email or Slack, mark “no” in the “Are All RA’s Active?” column in the “RA Activity+Notes” tab and put their name down in the “Names of Inactive RAs” column.
  4. If possible, the task may need to be reallocated to other active RAs.

How do I become a regional/country manager?

Research assistants are invited to become regional/country managers based on some combination of their expertise in the region/country, their familiarity with the CoronaNet project and their previous experience managing and coordinating people.

There are a limited number of regional/country manager positions available and when one becomes available, we invite a research assistant to fill the vacancy. Note, while all countries have a regional manager, not all countries have country managers (that is, we do not make a systematic effort to collect subnational data for all countries) though we may expand the number of country managers on a limited basis in the future.

If you are interested in becoming a regional or country manager, or if you would like to take on the country manager role for a country that currently is not assigned one, please contact the project management team at at admin@coronanet-project.org!

RA Responsibilities

After an RA has completed their preliminary training (i.e. they have passed the training survey and have been welcomed onto the project), each RA will initially be assigned a cleaning task to familiarize themselves with the data.

Once the initial cleaning task has been completed, each RA will be assigned a task to work on. They will either continue to deepen their knowledge of the cleaning process or move onto learning about data integration.They will be added to the relevant Task Force channels and should introduce themselves to their fellow RA’s in the task Force

They can use these channels to ask any questions they have about getting started and to communicate with the Task Force Manager. Other resources available are.

Once an RA has been assigned a particular cleaning or integration task, they commit to completing and updating the records for that assignment. The RA must also enter information into the task-specific spreadsheets that will be made available to them by the Task Force Manger Manager. This information concerns the date in which they have coded policies up to and the completeness of the records – important information for the Task Force managers and should be updated regularly.

Please focus on coding policies at the national level. Once those policies are up to date and complete, then RAs may focus on gathering subnational data in consultation with your task force manager. Note that in this case — that is, national policies are mostly up to date/complete and you would like to start focusing on gathering subnational data — please, in consultation with your task force manager, choose a couple of states/provinces that you would like to focus on first rather than trying to code them all at once. In order to decide which states/provinces to focus on, it may help to first do a little bit of leg work to get an overall sense of how actively certain provinces/states are, and then choose provinces/states based on 1) how active they are 2) how important they are to the country (e.g. by GDP or population). If you would like to make the case for coding all subnational policies for your country, please let your task force manager know. We will then assess the situation and decide if we should assign more RAs and a task force manager for that particular country to capture sub-national policies systematically.

There is no requirement on the number of policies coded per day/week by each RA. Each policy is unique in their approach and we understand that finding and coding policies vary in the time it takes. It is recommended you take your time with your research before coding a policy, to ensure you code as accurately as possible. RA’s should not hesitate in asking questions in the slack channels if they are unsure. If an RA feels overwhelmed and feels they need extra support, they should contact their Task Force Managers to let them know.

RA’s should also participate in the weekly meeting with their Task Force Managers – which will be set up by the Task Force Managers and links posted in the Slack channel.

In addition, RA’s should participate in/watch the recording of the general RA meeting where updates about the whole project are shared. These meetings take place on Wednesdays at 4.30pm CET at this link.

Summary of Main Tasks

  • Research policies thoroughly and enter them using the Qualtrics questionnaire.
  • Based on the number of hours you are asked to commit to the project, we expect RAs to do 25 overlap assessments or 5 coded policies each week.
  • You have to fill out the RA internal and Well Being surveys every 3 months. * When the time comes, you will get the link to the survey via email and it will also be available on the #general channel on slack.
  • Keep policy records up-to-date (enter records chronologically in order to make it easier to update/end policies as they develop).
  • Attend weekly RA meetings with the CoronaNet team (Wednesdays 4.30pm CEST, weekly).
  • Attend weekly meetings with the Task Force Managers (to be set up by the Managers and communicated in Slack).

RA points of contact?

  • Task Force Manager
    • Specific questions regarding process in the task force.
    • General point of contact/support.
    • Questions regarding the task-specific spreadsheets.
  • #ra-chat
    • Questions regarding how policies should be coded in Qualtrics.

Need a break? / Vacation

We understand nobody wants to be coding whilst they are on vacation! And there is zero expectation for you to do so, this is a voluntary project and we pride ourselves on our flexibility.

If you have a holiday booked and/or need a break due to other commitments, please get in touch with the Task Force Manager for the country you are responsible for and let them know.

They can arrange for another RA to cover you whilst you take a break, and ensure that the records remain up-to-date whilst you are away.

RA Onboarding Process


CoronaNet Learning Platform


CoronaNet also has a website dedicated to providing learning resources and free tutorials to anybody who wants to learn the software “R”.

The learning platform offers an introduction to “R” for new users and is paired with webinars and tutorials that utilise the CoronaNet data. So users can learn how to use/import/analyse/visualise data in “R” using the CoronaNet data as we collect it!