How Data Made Me a Believer in New York Restaurant Grades

Reviews and grades can be nasty, capricious, biased, unscientific - and occasionally useful, to say nothing of accurate.

Business owners and executives may believe that the welter of regulations and self-appointed/annointed experts on the worthiness of their business are often destructive. And much of that can be true. But as the following article explains, with a strong process and tested metrics, they can work to the enterprise's advantage. JL

Kaiser Fung reports in the 538 blog:

Did one tiny mouse really close the trendy bakery?

One morning in April, the neighbors of Dominique Ansel Bakery, whose owner invented the “cronut” (the pastry famed for combining the croissant and donut), were amused. For the first time in a long time, the tourists who commonly mobbed the little Manhattan bakery had disappeared. Were we finally hearing the popping of the cronut bubble?
Not quite.
A makeshift sign on the door disclosed that the bakery was temporarily closed for failing an inspection by New York City’s health department, which made waves during Mayor Michael Bloomberg’s tenure for initiating a letter grading system for restaurants. The April inspection, Dominique Ansel’s second in seven months, was triggered by an act of citizen journalism. The day before, the Gothamist blog had published a video shot by a customer of a small mouse scurrying inside the store.
In response, the bakery sent a note to its cronut-deprived fans: “Due to the video that was released showing a small mouse running across the screen for five seconds, the health department used it as evidence to ask us to re-cement and closed down the bakery for extermination. As a small one-shop bakery, we often feel like we’re being looked at under a tremendous microscope.”
Did one tiny mouse really close the trendy bakery? I used data from the health department to investigate this question, and the short answer is no. I ran an algorithm to analyze historical data, deducing the relative importance of different factors, including vermin, on restaurant health grades. While certain chefs have complained about specific health inspection rules, after examining the data, I accept the logic behind the grades, and feel pretty confident referencing them now.
Here’s how New York City’s restaurant grading system works. There are 68 possible violations, ranging from “Evidence of rats or live rats present in facility’s food and/or non-food areas” to “Tobacco use, eating or drinking from open container in food preparation, food storage or dishwashing area.” Restaurants rack up points for violations — and receive an A if they have fewer than 14 points, a B if they have between 14 and 27 points and a C for 28 points or more. Each violation can carry a range of points depending on its severity. Only four of the 68 violations relate to physical evidence of vermin — rats, mice, roaches and flies.
As it turns out, to my unpleasant surprise, a restaurant can earn an A while harboring mice or roaches, provided that it has few or no violations of other types. Records show that last fall health inspectors went to a Brooklyn café where they found droppings from mice, rats, roaches and flies. The restaurant amassed 11 points — good enough for an A! Since June 2011, two other restaurants have managed to pull off this act of escape.
Having one type of vermin could be bad luck; having all four looks like total carelessness to me. Recent social-media commotion around the mouse at Dominique Ansel, roaches at Blue Ribbon Sushi, and a rat rattling a pastry case at Dunkin Donuts indicate that many diners feel the same way. According to Food Safety Magazine, mice and rats urinate and defecate frequently, spreading diseases such as hantavirus and salmonellosis; flies carry more than 100 kinds of disease-causing germs; and cockroaches transfer bacteria, parasitic worms and human pathogens.
Using the health department’s data, I created a vermin-only grading scale, and compared it to the official A-B-C restaurant ratings.¹ The vermin grades are 0, 1, 2 or 3+, corresponding to the number of types of vermin found during an inspection.

As you can see, my vermin grade and the city’s health grades are strongly correlated. Nonetheless, about 24 percent of inspections uncovering two vermin-related violations resulted in the restaurants receiving As. Among restaurants with one such violation, 40 percent got As. So, contrary to what Dominique Ansel believes, one mouse is unlikely to ruin an inspection.
So if vermin only explain some of a restaurant’s grade, what factors account for the rest?
The most recent restaurant grades data set contains all inspections from Aug. 2, 2010, to July 15, 2014, including the name of the restaurant, the neighborhood, the cuisine,² the violations, the point score and the grade. I removed data before June 1, 2011, since the low frequency of visits suggests that some grades were not recorded. I also excluded inspections after May 2014, because some 20 percent to 30 percent of those have yet to be resolved. (Restaurants have the option to engage in a lengthy re-inspection and arbitration process if they receive a grade of B or C.³) Ungraded inspections were also dropped. In total, my database comprises just under 63,000 inspections of about 17,300 New York City restaurants.
The health department’s 68 violation types spread the information too thinly, so I aggregated them to seven key violation groups: vermin, temperature, facilities, food handling, hygiene, contamination and regulatory. An example of a temperature-related violation is “Hot food item not held at or above 140 F”; an example of a facilities-related violation is “Sewage disposal system improper or unapproved”; and a food handling violation might be “Raw food not properly washed prior to serving.” A few violations belong to multiple groups — for example, “‘Wash hands’ sign not posted at hand-washing facility” concerns facilities as well as regulation.
I wanted to explore how different factors — such as violation groups, cuisine groups, year and month of inspection, and neighborhood — contribute to the A-B-C grading system.⁴ The below chart lists the variables sorted by their influence on the grade.

Vermin-related violations have the biggest influence on restaurant grades. The next most harmful violation group is temperature-related offenses, which rates only slightly below vermin. Violations related to food handling are also influential, while relatively minor infractions include those grouped under facilities, contamination, regulatory and hygiene.
I used the date of inspection to control for season. According to my model, the month of inspection contributes almost nothing to explaining grades, although the year does have a moderate effect. The type of cuisine and the location of the restaurant (i.e. neighborhood) also don’t appear to predict health inspection failure.⁵
A more detailed analysis reveals that restaurants started getting better grades in the second half of 2013. In 2012, the second year of A-B-C grading, about 70 percent of restaurants in my database earned As; by the first quarter of 2014, this proportion exceeded 80 percent. The same trend is evident in all five boroughs.

I now believe the system in New York City makes sense. The health grade incorporates a variety of metrics, with reasonable weights. These weights are not explicitly specified in a formula, and they reflect the average behavior of all inspectors, but it is possible to use a statistical model to deduce their values, as I have here. My one complaint: If I were heading the New York City health department, I would shift more consideration to the rats, mice, roaches and flies.