IBM Watson™ Ideas

Welcome to the IBM Watson™ Ideas Portal

We welcome and appreciate your feedback on IBM Watson™ Products to help make them even better than they are today!

If you are looking for troubleshooting help or wondering how to use our products and services, please check the IBM Watson™ documentation. Please do not use the Ideas Portal for reporting bugs - we ask that you report bugs or issues with the product by contacting IBM support.

Before you submit an idea, please perform a search first as a similar idea may have already been reported in the portal.

If a related idea is not yet listed, please create a new idea and include with it a description which includes expected behavior as well as why having this feature would improve the service and how it would address your use case.

Search documents by keywords/entity/relation

Ability to search document by keywords, entity names, relation names, etc is required.

During annotations, there is a case where annotators want to see how they annotated the documents that are similar to what annotators are annotating now, or a case where same word came up in a different document and want to see how they annotated that word then. In such cases it would be very convenient to be able to search the documents by keywords, entity names, relation names, etc. Also the human annotation work will be more efficient.

  • Guest
  • Jul 5 2018
  • Will not implement
Why is it useful?
Who would benefit from this IDEA? Human Annotators
How should it work?
Idea Priority
Priority Justification
Customer Name
Submitting Organization
Submitter Tags
  • Attach files
  • Admin
    STEFAN TZANEV commented
    December 11, 2018 19:54

    This enhancement request goes against the Knowledge Studio methodology and will introduce inter-annotator bias. Knowledge Studio isolates the work of the individual human annotators (HA) on purpose.

    The proper way to train the human annotators is to have clearly documented IAA (inter-annotator agreement). The IAA should not only list the Type System entries, but should also provide examples of proper annotation of the TS mentions.

    The other tool for educating the HAs is the discussions that happen around adjudication (conflict resolution.)

  • Guest commented
    December 12, 2018 00:02

    Thanks for them comment. I understand that looking at other human annotator's annotation will introduce bias.

    What about searching within your own annotated documents? I want to check the consistency of annotation within the documents that I annotated. To make sure that I didn't miss anything, didn't annotated words differently, and so on. When annotating many many documents, it's difficult to check....