IBM Watson™ Ideas

Welcome to the IBM Watson™ Ideas Portal


We welcome and appreciate your feedback on IBM Watson™ Products to help make them even better than they are today!


If you are looking for troubleshooting help or wondering how to use our products and services, please check the IBM Watson™ documentation. Please do not use the Ideas Portal for reporting bugs - we ask that you report bugs or issues with the product by contacting IBM support.


Before you submit an idea, please perform a search first as a similar idea may have already been reported in the portal.


If a related idea is not yet listed, please create a new idea and include with it a description which includes expected behavior as well as why having this feature would improve the service and how it would address your use case.

Transferring ownership of Annotation Sets

Problem

When annotation sets are created by WKS and assigned to the selected (by the Project Manager) human annotators (HA), the ownership of the annotation sets cannot be changed. This means that if a HA is temporarily or permanently unavailable to finish the assigned (via annotation set) annotation work, the work cannot be transfered to another HA and it cannot be done.

Consequences:

  • When the HA is temporarily unavailable to finish the work, the entire Annotation Task (a group of one or more annotation sets, whose progress is tracked as a whole) is blocked and the entire annotation team is slowed down.
  • When the HA becomes permanently unavailable, the entire Annotation Task is in limbo. The workaround is to redo a lot of work in a new Annotation Task - not an acceptable solution. 

Several options (ideas)

Allow transferring of Annotation Set ownership.

Several ideas:

  1. Allow Project Manager to re-assign annotation sets - relatively safe (controlled) way to transfer ownership, but still may create issues with the fidelity of the annotation overlaps, as these overlaps were created by WKS with the original HAs in "mind."
    • PROS: Relatively easy and safe to implement
    • CONS: Side effects are expected; addresses a specific issue, but does not improve much the overall process.
  2. Dynamically adjusted hidden annotation sets - this would be radical change of the annotation workflow and UX, but it may be a promising one:
    • Annotation sets can be maintained internally by the system, only for the purpose of ensuring the overlapping document annotations, but no need to be explicitly managed by Proj Mgr and assigned to specific HAs, etc. In this case, the workflow could be something like:
      1. Project Manager uploads the training documents and organizes them in Document Sets (Btw, the management of Document Sets also needs improvement.)
      2. The Project Manager creates Annotation Tasks and adds Document Sets to them
      3. The HAs signup for (pull, not push approach) one of more available (unfinished) Annotation Tasks
      4. For each HA in an Annotation Task, WKS automatically creates a pool of documents, making sure that the Project Manager's setting for annotation overlap are met
      5. Each HA is presented with their own pool of documents for annotation (that is the automatically generated hidden Annotation Set)
      6. As HAs work at different speed, and join and leave the Annotation Task (the main unit of work progress that is tracked by the Project Manager), and as Project Manager adds/removes documents from the Annotation Tasks (yes, in real life this happens and should be supported by WKS), WKS dynamically adjusts the pool of documents for each HA, to ensure proper annotation overlap and to maximize the annotation speed based on the different speed of the HAs.
      7. Ta-da!
    • PROS: Dynamic optimization of the annotation process by WKS should yield best overall results (hypothesis to be validated yet)
    • CONS: Radical change - longer and more costly to execute; higher risk (compared to 1) as this is real innovation.
  • STEFAN TZANEV
  • Jun 15 2018
  • Needs review
Why is it useful?

Once implemented, this enhancement/fix will prevent expensive project delays and rework.

Who would benefit from this IDEA? Both WKS Project Managers and WKS Human Annotators
How should it work?

See description below.

Idea Priority
Priority Justification
Customer Name
Submitting Organization
Submitter Tags
  • Attach files