DOI: 10.1038/s41592-018-0069-0. Quanti.us: a tool for rapid, flexible, crowd-based annotation of images

Quanti.us: a tool for rapid, flexible, crowd-based annotation of images

10.1038/s41592-018-0069-0

Crossref journal-article

Springer Science and Business Media LLC

Nature Methods (297)

Bibliography

Hughes, A. J., Mornin, J. D., Biswas, S. K., Beck, L. E., Bauer, D. P., Raj, A., Bianco, S., & Gartner, Z. J. (2018). Quanti.us: a tool for rapid, flexible, crowd-based annotation of images. Nature Methods, 15(8), 587â590.

Authors 8

Alex J. Hughes (first)
Joseph D. Mornin (additional)
Sujoy K. Biswas (additional)
Lauren E. Beck (additional)
David P. Bauer (additional)
Arjun Raj (additional)
Simone Bianco (additional)
Zev J. Gartner (additional)

References 28 Referenced 55

Kim, J. S. et al. Nature 509, 331–336 (2014). (10.1038/nature13240) / Nature by JS Kim (2014)
Chen, F. et al. Nat. Methods 13, 679–684 (2016). (10.1038/nmeth.3899) / Nat. Methods by F Chen (2016)
Lou, X., Kang, M., Xenopoulos, P., Muñoz-Descalzo, S. & Hadjantonakis, A.-K. Stem Cell Rep. 2, 382–397 (2014). (10.1016/j.stemcr.2014.01.010) / Stem Cell Rep. by X Lou (2014)
Ruhnow, F., Zwicker, D. & Diez, S. Biophys. J. 100, 2820–2828 (2011). (10.1016/j.bpj.2011.04.023) / Biophys. J. by F Ruhnow (2011)
Esteva, A. et al. Nature 542, 115–118 (2017). (10.1038/nature21056) / Nature by A Esteva (2017)
Goodfellow, I. J. et al. in Proc. Advances in Neural Information Processing Systems 27 (eds. Ghahramani, Z. et al.) 2672–2680 (NIPS, La Jolla, CA, 2014).
Salimans, T. et al. arXiv Preprint at https://arxiv.org/abs/1606.03498 (2016).
Thul, P. J. et al. Science 356, eaal3321 (2017). (10.1126/science.aal3321) / Science by PJ Thul (2017)
Simpson, R., Page, K. R. & De Roure, D. in Proc. 23rd International Conference on World Wide Web 1049–1054 (ACM, New York, 2014). (10.1145/2567948.2579215)
Sauermann, H. & Franzoni, C. Proc. Natl. Acad. Sci. USA 112, 679–684 (2015). (10.1073/pnas.1408907112) / Proc. Natl. Acad. Sci. USA by H Sauermann (2015)
Hitlin, P. Research in the Crowdsourcing Age, A Case Study (Pew Research Center, Washington, DC, 2016).
Bruggemann, J., Lander, G. C. & Su, A. I. bioRxiv Preprint at https://www.biorxiv.org/content/early/2017/11/15/220145 (2017).
Galton, F. Nature 75, 450–451 (1907). (10.1038/075450a0) / Nature by F Galton (1907)
Ipeirotis, P. G., Provost, F. & Wang, J. in Proc. ACM SIGKDD Workshop on Human Computation 64–67 (ACM, New York, 2010). (10.1145/1837885.1837906)
Zhai, S., Kong, J. & Ren, X. Int. J. Hum. Comput. Stud. 61, 823–856 (2004). (10.1016/j.ijhcs.2004.09.007) / Int. J. Hum. Comput. Stud. by S Zhai (2004)
Ipeirotis, P. G. XRDS 17, 16–21 (2010). (10.1145/1869086.1869094) / XRDS by PG Ipeirotis (2010)
Scharrel, L., Ma, R., Schneider, R., Jülicher, F. & Diez, S. Biophys. J. 107, 365–372 (2014). (10.1016/j.bpj.2014.06.014) / Biophys. J. by L Scharrel (2014)
Sadanandan, S. K., Ranefall, P., Le Guyader, S. & Wählby, C. Sci. Rep. 7, 7860 (2017). (10.1038/s41598-017-07599-6) / Sci. Rep. by SK Sadanandan (2017)
Xie, W., Noble, J. A. & Zisserman, A. Comput. Methods Biomech. Biomed. Eng. Imaging Vis. 6, 283–292 (2016). (10.1080/21681163.2016.1149104) / Comput. Methods Biomech. Biomed. Eng. Imaging Vis. by W Xie (2016)
Vedaldi, A. & Lenc, K. in Proc. 23rd ACM International Conference on Multimedia 689–692 (ACM, New York, 2015). (10.1145/2733373.2807412)
Talpalar, A. E. et al. Nature 500, 85–88 (2013). (10.1038/nature12286) / Nature by AE Talpalar (2013)
Marquardt, D. W. J. Soc. Ind. Appl. Math. 11, 431–441 (1963). (10.1137/0111030) / J. Soc. Ind. Appl. Math. by DW Marquardt (1963)
Tinevez, J. Y. et al. Methods 115, 80–90 (2017). (10.1016/j.ymeth.2016.09.016) / Methods by JY Tinevez (2017)
Adcock, R. J. Anal. (Lond.) 5, 53 (1878). (10.2307/2635758)
Arteta, C., Lempitsky, V., Noble, J. A. & Zisserman, A. Med. Image Anal. 27, 3–16 (2016). (10.1016/j.media.2015.03.002) / Med. Image Anal. by C Arteta (2016)
LeCun, Y., Bengio, Y. & Hinton, G. Nature 521, 436–444 (2015). (10.1038/nature14539) / Nature by Y LeCun (2015)
Vedaldi, A. & Fulkerson, B. in Proc. 18th ACM International Conference on Multimedia 1469–1472 (ACM, New York, 2010). / Proc. 18th ACM International Conference on Multimedia by A Vedaldi (2010)
Glorot, X. & Bengio, Y. in Proc. 13th International Conference on Artificial Intelligence and Statistics (eds. Teh, Y. W. & Titterington, M.) 249–256 (MLR Press/Microtome Publishing, Brookline, MA, 2010).

Dates

Type	When
Created	7 years, 1 month ago (July 19, 2018, 10:03 a.m.)
Deposited	1 month, 3 weeks ago (July 5, 2025, 6:11 p.m.)
Indexed	1 month ago (July 25, 2025, 6:16 a.m.)
Issued	7 years ago (July 31, 2018)
Published	7 years ago (July 31, 2018)
Published Online	7 years ago (July 31, 2018)
Published Print	7 years ago (Aug. 1, 2018)

Funders 0

None

BibTeX

@article{Hughes_2018, title={Quanti.us: a tool for rapid, flexible, crowd-based annotation of images}, volume={15}, ISSN={1548-7105}, url={http://dx.doi.org/10.1038/s41592-018-0069-0}, DOI={10.1038/s41592-018-0069-0}, number={8}, journal={Nature Methods}, publisher={Springer Science and Business Media LLC}, author={Hughes, Alex J. and Mornin, Joseph D. and Biswas, Sujoy K. and Beck, Lauren E. and Bauer, David P. and Raj, Arjun and Bianco, Simone and Gartner, Zev J.}, year={2018}, month=jul, pages={587–590} }

JSON

{
  "indexed": {
    "date-parts": [
      [
        2025,
        7,
        25
      ]
    ],
    "date-time": "2025-07-25T10:16:40Z",
    "timestamp": 1753438600238,
    "version": "3.41.0"
  },
  "reference-count": 28,
  "publisher": "Springer Science and Business Media LLC",
  "issue": "8",
  "license": [
    {
      "start": {
        "date-parts": [
          [
            2018,
            7,
            31
          ]
        ],
        "date-time": "2018-07-31T00:00:00Z",
        "timestamp": 1532995200000
      },
      "content-version": "tdm",
      "delay-in-days": 0,
      "URL": "http://www.springer.com/tdm"
    }
  ],
  "content-domain": {
    "domain": [
      "link.springer.com"
    ],
    "crossmark-restriction": false
  },
  "published-print": {
    "date-parts": [
      [
        2018,
        8
      ]
    ]
  },
  "DOI": "10.1038/s41592-018-0069-0",
  "type": "journal-article",
  "created": {
    "date-parts": [
      [
        2018,
        7,
        19
      ]
    ],
    "date-time": "2018-07-19T14:03:44Z",
    "timestamp": 1532009024000
  },
  "page": "587-590",
  "update-policy": "https://doi.org/10.1007/springer_crossmark_policy",
  "source": "Crossref",
  "is-referenced-by-count": 55,
  "title": "Quanti.us: a tool for rapid, flexible, crowd-based annotation of images",
  "prefix": "10.1038",
  "volume": "15",
  "author": [
    {
      "ORCID": "https://orcid.org/0000-0001-6801-3455",
      "authenticated-orcid": false,
      "given": "Alex J.",
      "family": "Hughes",
      "sequence": "first",
      "affiliation": []
    },
    {
      "given": "Joseph D.",
      "family": "Mornin",
      "sequence": "additional",
      "affiliation": []
    },
    {
      "given": "Sujoy K.",
      "family": "Biswas",
      "sequence": "additional",
      "affiliation": []
    },
    {
      "given": "Lauren E.",
      "family": "Beck",
      "sequence": "additional",
      "affiliation": []
    },
    {
      "given": "David P.",
      "family": "Bauer",
      "sequence": "additional",
      "affiliation": []
    },
    {
      "ORCID": "https://orcid.org/0000-0002-2915-6960",
      "authenticated-orcid": false,
      "given": "Arjun",
      "family": "Raj",
      "sequence": "additional",
      "affiliation": []
    },
    {
      "ORCID": "https://orcid.org/0000-0003-3367-2084",
      "authenticated-orcid": false,
      "given": "Simone",
      "family": "Bianco",
      "sequence": "additional",
      "affiliation": []
    },
    {
      "ORCID": "https://orcid.org/0000-0001-7803-1219",
      "authenticated-orcid": false,
      "given": "Zev J.",
      "family": "Gartner",
      "sequence": "additional",
      "affiliation": []
    }
  ],
  "member": "297",
  "published-online": {
    "date-parts": [
      [
        2018,
        7,
        31
      ]
    ]
  },
  "reference": [
    {
      "key": "69_CR1",
      "doi-asserted-by": "publisher",
      "first-page": "331",
      "DOI": "10.1038/nature13240",
      "volume": "509",
      "author": "JS Kim",
      "year": "2014",
      "unstructured": "Kim, J. S. et al. Nature 509, 331\u2013336 (2014).",
      "journal-title": "Nature"
    },
    {
      "key": "69_CR2",
      "doi-asserted-by": "publisher",
      "first-page": "679",
      "DOI": "10.1038/nmeth.3899",
      "volume": "13",
      "author": "F Chen",
      "year": "2016",
      "unstructured": "Chen, F. et al. Nat. Methods 13, 679\u2013684 (2016).",
      "journal-title": "Nat. Methods"
    },
    {
      "key": "69_CR3",
      "doi-asserted-by": "publisher",
      "first-page": "382",
      "DOI": "10.1016/j.stemcr.2014.01.010",
      "volume": "2",
      "author": "X Lou",
      "year": "2014",
      "unstructured": "Lou, X., Kang, M., Xenopoulos, P., Mu\u00f1oz-Descalzo, S. & Hadjantonakis, A.-K. Stem Cell Rep. 2, 382\u2013397 (2014).",
      "journal-title": "Stem Cell Rep."
    },
    {
      "key": "69_CR4",
      "doi-asserted-by": "publisher",
      "first-page": "2820",
      "DOI": "10.1016/j.bpj.2011.04.023",
      "volume": "100",
      "author": "F Ruhnow",
      "year": "2011",
      "unstructured": "Ruhnow, F., Zwicker, D. & Diez, S. Biophys. J. 100, 2820\u20132828 (2011).",
      "journal-title": "Biophys. J."
    },
    {
      "key": "69_CR5",
      "doi-asserted-by": "publisher",
      "first-page": "115",
      "DOI": "10.1038/nature21056",
      "volume": "542",
      "author": "A Esteva",
      "year": "2017",
      "unstructured": "Esteva, A. et al. Nature 542, 115\u2013118 (2017).",
      "journal-title": "Nature"
    },
    {
      "key": "69_CR6",
      "unstructured": "Goodfellow, I. J. et al. in Proc. Advances in Neural Information Processing Systems 27 (eds. Ghahramani, Z. et al.) 2672\u20132680 (NIPS, La Jolla, CA, 2014)."
    },
    {
      "key": "69_CR7",
      "unstructured": "Salimans, T. et al. arXiv Preprint at https://arxiv.org/abs/1606.03498 (2016)."
    },
    {
      "key": "69_CR8",
      "doi-asserted-by": "publisher",
      "first-page": "eaal3321",
      "DOI": "10.1126/science.aal3321",
      "volume": "356",
      "author": "PJ Thul",
      "year": "2017",
      "unstructured": "Thul, P. J. et al. Science 356, eaal3321 (2017).",
      "journal-title": "Science"
    },
    {
      "key": "69_CR9",
      "doi-asserted-by": "crossref",
      "unstructured": "Simpson, R., Page, K. R. & De Roure, D. in Proc. 23rd International Conference on World Wide Web 1049\u20131054 (ACM, New York, 2014).",
      "DOI": "10.1145/2567948.2579215"
    },
    {
      "key": "69_CR10",
      "doi-asserted-by": "publisher",
      "first-page": "679",
      "DOI": "10.1073/pnas.1408907112",
      "volume": "112",
      "author": "H Sauermann",
      "year": "2015",
      "unstructured": "Sauermann, H. & Franzoni, C. Proc. Natl. Acad. Sci. USA 112, 679\u2013684 (2015).",
      "journal-title": "Proc. Natl. Acad. Sci. USA"
    },
    {
      "key": "69_CR11",
      "unstructured": "Hitlin, P. Research in the Crowdsourcing Age, A Case Study (Pew Research Center, Washington, DC, 2016)."
    },
    {
      "key": "69_CR12",
      "unstructured": "Bruggemann, J., Lander, G. C. & Su, A. I. bioRxiv Preprint at https://www.biorxiv.org/content/early/2017/11/15/220145 (2017)."
    },
    {
      "key": "69_CR13",
      "doi-asserted-by": "publisher",
      "first-page": "450",
      "DOI": "10.1038/075450a0",
      "volume": "75",
      "author": "F Galton",
      "year": "1907",
      "unstructured": "Galton, F. Nature 75, 450\u2013451 (1907).",
      "journal-title": "Nature"
    },
    {
      "key": "69_CR14",
      "doi-asserted-by": "crossref",
      "unstructured": "Ipeirotis, P. G., Provost, F. & Wang, J. in Proc. ACM SIGKDD Workshop on Human Computation 64\u201367 (ACM, New York, 2010).",
      "DOI": "10.1145/1837885.1837906"
    },
    {
      "key": "69_CR15",
      "doi-asserted-by": "publisher",
      "first-page": "823",
      "DOI": "10.1016/j.ijhcs.2004.09.007",
      "volume": "61",
      "author": "S Zhai",
      "year": "2004",
      "unstructured": "Zhai, S., Kong, J. & Ren, X. Int. J. Hum. Comput. Stud. 61, 823\u2013856 (2004).",
      "journal-title": "Int. J. Hum. Comput. Stud."
    },
    {
      "key": "69_CR16",
      "doi-asserted-by": "publisher",
      "first-page": "16",
      "DOI": "10.1145/1869086.1869094",
      "volume": "17",
      "author": "PG Ipeirotis",
      "year": "2010",
      "unstructured": "Ipeirotis, P. G. XRDS 17, 16\u201321 (2010).",
      "journal-title": "XRDS"
    },
    {
      "key": "69_CR17",
      "doi-asserted-by": "publisher",
      "first-page": "365",
      "DOI": "10.1016/j.bpj.2014.06.014",
      "volume": "107",
      "author": "L Scharrel",
      "year": "2014",
      "unstructured": "Scharrel, L., Ma, R., Schneider, R., J\u00fclicher, F. & Diez, S. Biophys. J. 107, 365\u2013372 (2014).",
      "journal-title": "Biophys. J."
    },
    {
      "key": "69_CR18",
      "doi-asserted-by": "publisher",
      "DOI": "10.1038/s41598-017-07599-6",
      "volume": "7",
      "author": "SK Sadanandan",
      "year": "2017",
      "unstructured": "Sadanandan, S. K., Ranefall, P., Le Guyader, S. & W\u00e4hlby, C. Sci. Rep. 7, 7860 (2017).",
      "journal-title": "Sci. Rep."
    },
    {
      "key": "69_CR19",
      "doi-asserted-by": "publisher",
      "first-page": "283",
      "DOI": "10.1080/21681163.2016.1149104",
      "volume": "6",
      "author": "W Xie",
      "year": "2016",
      "unstructured": "Xie, W., Noble, J. A. & Zisserman, A. Comput. Methods Biomech. Biomed. Eng. Imaging Vis. 6, 283\u2013292 (2016).",
      "journal-title": "Comput. Methods Biomech. Biomed. Eng. Imaging Vis."
    },
    {
      "key": "69_CR20",
      "doi-asserted-by": "crossref",
      "unstructured": "Vedaldi, A. & Lenc, K. in Proc. 23rd ACM International Conference on Multimedia 689\u2013692 (ACM, New York, 2015).",
      "DOI": "10.1145/2733373.2807412"
    },
    {
      "key": "69_CR21",
      "doi-asserted-by": "publisher",
      "first-page": "85",
      "DOI": "10.1038/nature12286",
      "volume": "500",
      "author": "AE Talpalar",
      "year": "2013",
      "unstructured": "Talpalar, A. E. et al. Nature 500, 85\u201388 (2013).",
      "journal-title": "Nature"
    },
    {
      "key": "69_CR22",
      "doi-asserted-by": "publisher",
      "first-page": "431",
      "DOI": "10.1137/0111030",
      "volume": "11",
      "author": "DW Marquardt",
      "year": "1963",
      "unstructured": "Marquardt, D. W. J. Soc. Ind. Appl. Math. 11, 431\u2013441 (1963).",
      "journal-title": "J. Soc. Ind. Appl. Math."
    },
    {
      "key": "69_CR23",
      "doi-asserted-by": "publisher",
      "first-page": "80",
      "DOI": "10.1016/j.ymeth.2016.09.016",
      "volume": "115",
      "author": "JY Tinevez",
      "year": "2017",
      "unstructured": "Tinevez, J. Y. et al. Methods 115, 80\u201390 (2017).",
      "journal-title": "Methods"
    },
    {
      "key": "69_CR24",
      "doi-asserted-by": "crossref",
      "unstructured": "Adcock, R. J. Anal. (Lond.) 5, 53 (1878).",
      "DOI": "10.2307/2635758"
    },
    {
      "key": "69_CR25",
      "doi-asserted-by": "publisher",
      "first-page": "3",
      "DOI": "10.1016/j.media.2015.03.002",
      "volume": "27",
      "author": "C Arteta",
      "year": "2016",
      "unstructured": "Arteta, C., Lempitsky, V., Noble, J. A. & Zisserman, A. Med. Image Anal. 27, 3\u201316 (2016).",
      "journal-title": "Med. Image Anal."
    },
    {
      "key": "69_CR26",
      "doi-asserted-by": "publisher",
      "first-page": "436",
      "DOI": "10.1038/nature14539",
      "volume": "521",
      "author": "Y LeCun",
      "year": "2015",
      "unstructured": "LeCun, Y., Bengio, Y. & Hinton, G. Nature 521, 436\u2013444 (2015).",
      "journal-title": "Nature"
    },
    {
      "key": "69_CR27",
      "volume-title": "Proc. 18th ACM International Conference on Multimedia",
      "author": "A Vedaldi",
      "year": "2010",
      "unstructured": "Vedaldi, A. & Fulkerson, B. in Proc. 18th ACM International Conference on Multimedia 1469\u20131472 (ACM, New York, 2010)."
    },
    {
      "key": "69_CR28",
      "unstructured": "Glorot, X. & Bengio, Y. in Proc. 13th International Conference on Artificial Intelligence and Statistics (eds. Teh, Y. W. & Titterington, M.) 249\u2013256 (MLR Press/Microtome Publishing, Brookline, MA, 2010)."
    }
  ],
  "container-title": "Nature Methods",
  "original-title": [],
  "language": "en",
  "link": [
    {
      "URL": "http://www.nature.com/articles/s41592-018-0069-0",
      "content-type": "text/html",
      "content-version": "vor",
      "intended-application": "text-mining"
    },
    {
      "URL": "http://www.nature.com/articles/s41592-018-0069-0.pdf",
      "content-type": "application/pdf",
      "content-version": "vor",
      "intended-application": "text-mining"
    },
    {
      "URL": "http://www.nature.com/articles/s41592-018-0069-0.pdf",
      "content-type": "application/pdf",
      "content-version": "vor",
      "intended-application": "similarity-checking"
    }
  ],
  "deposited": {
    "date-parts": [
      [
        2025,
        7,
        5
      ]
    ],
    "date-time": "2025-07-05T22:11:16Z",
    "timestamp": 1751753476000
  },
  "score": 1,
  "resource": {
    "primary": {
      "URL": "https://www.nature.com/articles/s41592-018-0069-0"
    }
  },
  "subtitle": [],
  "short-title": [],
  "issued": {
    "date-parts": [
      [
        2018,
        7,
        31
      ]
    ]
  },
  "references-count": 28,
  "journal-issue": {
    "issue": "8",
    "published-print": {
      "date-parts": [
        [
          2018,
          8
        ]
      ]
    }
  },
  "alternative-id": [
    "69"
  ],
  "URL": "http://dx.doi.org/10.1038/s41592-018-0069-0",
  "relation": {},
  "ISSN": [
    "1548-7091",
    "1548-7105"
  ],
  "subject": [],
  "container-title-short": "Nat Methods",
  "published": {
    "date-parts": [
      [
        2018,
        7,
        31
      ]
    ]
  },
  "assertion": [
    {
      "value": "25 September 2017",
      "order": 1,
      "name": "received",
      "label": "Received",
      "group": {
        "name": "ArticleHistory",
        "label": "Article History"
      }
    },
    {
      "value": "22 May 2018",
      "order": 2,
      "name": "accepted",
      "label": "Accepted",
      "group": {
        "name": "ArticleHistory",
        "label": "Article History"
      }
    },
    {
      "value": "31 July 2018",
      "order": 3,
      "name": "first_online",
      "label": "First Online",
      "group": {
        "name": "ArticleHistory",
        "label": "Article History"
      }
    },
    {
      "value": "J.D.M. holds an equity interest in Quanti.us LLC. Quanti.us passes payments from users to Amazon Mechanical Turk, which then distributes these payments to workers.",
      "order": 1,
      "name": "Ethics",
      "group": {
        "name": "EthicsHeading",
        "label": "Competing interests"
      }
    }
  ]
}