Zac Hatfield-Dodds

Deriving Semantics-Aware Fuzzers from Web API Schemas

2022 · 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)

We present Schemathesis, a tool for finding semantic errors and crashes in OpenAPI or GraphQL web APIs through property-based testing. Our evaluation, thirty independent runs of eight tools against sixteen containerized open-source web services, shows …

Discovering Language Model Behaviors with Model-Written Evaluations

2022 · arXiv (Cornell University)

As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data …