What is a simulation

A simulation is the imitation of the operation of a real-world process or system over time

INTRODUCTION TO SIMULATION, Jerry Banks

I think of a simulations as trying to reproduce reality in a simpler, modifiable setting. A very tangible example of a simulation is the Mississippi River Basin model. This was built in the 1940s to aid systematic understanding of flood control measures that had been built on the Mississippi river in the previous decade. These locks, run-off channels and levees could prevent local flooding but may increase flooding in other areas so it was clear that a big picture understanding of how to implement these measures was needed. Various segments of the Mississippi River became operational in the simulation throughout the 1950s, helping to avoid damage of an estimated $65 million in 1952 along (almost a billion dollars in today’s money) Tests on individual problems were conducted until 1971 but high costs and growth of computer modelling meant that the facility was put on standby.

We can use this example to explore the different parts that make up a simulation.

What makes up a simulation

It’s useful to separate the components of a simulation into three different parts: Input, Mechanism, and Output

For example in the Mississippi River Basin Model, the Input is the amount of water added at upstream points, and the Output is the water heights at downstream points at various times. The Mechanism is a literal recreation of the landscape at reduced scales, and the simulation propagates along using real physics (gravity, hydrodynamics etc). Many successful simulations take advantage of re-using parts of the real system for their mechanism. An important part of the mechanism are the configuration parameters, which in this case would be the various settings of the dams and levees. Many different simulations can be run with various Inputs and Mechanism config parameters to obtain Outputs.

Depending on which part of the simulation is generating new data, the simulation is used for different purposes. For many practical applications, simulations are used for Prediction, i.e. the Mississippi River Basin Model was mostly used to predict what Outputs would arise from various rainfall levels and flood control measure settings.

For more theoretical applications, simulations are used for Explanation. Given Inputs that produce known Outputs in reality, if a Mechanism can reproduce these we can gain confidence that this corresponds to reality. This can be particularly useful in applications where fine-grained measurements of reality aren’t possible. Imagine in 1950s that scientists wanted to understand the mechanism of water flow in remote regions of the Mississippi basin. Before satellites this measurement would have been difficult, but using a simulation these scientists could examine this in fine detail. In computer simulations, scientists may try to simplify the mechanism to the bare minimum required to reproduce known Outputs from Inputs, thus gaining theoretical understanding of their domain.

A less common application is to use simulations for Retrodiction. An interesting example of this is the theory that a planet known as Theia smashed into Earth billions of years ago which resulted in the formation of our Moon. Many different Inputs (planet sizes, speeds etc) were trialed to find ones that match an Output of an Earth and Moon systems like ours (with the Mechanism being a physics engine)

Correspondence

It is the designer’s or user’s intentions that determine what a simulation is a simulation of and what features are to be taken as corresponding with reality.

How simulations fail

Regardless of the use of the simulations, there are questions being asked by the users of the simulation. Depending on the questions being asked, the simulation will need to more or less complex. There will be aspects of reality that are clearly not needed in the simulation (e.g. the phases of the moon don’t need to be considered in a molecular simulation) and aspects of reality that are clearly necessary in the simulation (e.g. the electromagnetic force is necessary for a molecular simulation). The craft of building a simulation arises in determining the aspects of reality that aren’t needed in the simulation and can be ignored, which generally allows the simulation to run for longer or be calibrated more easily (or simply make it feasible to build at all). The degree to which your simulation contains the aspects of reality required to answer your question is known as the correspondence of your simulation.

To examine this idea of correspondence, we can look at three different methods of simulating the spread of infectious diseases.

Time Series

In a time series, we predict the Output, the incidence of disease in the population, using a predictive model fit on some training data (historical Inputs and Outputs), see this study as an example. While this satisfies the technical definition of a simulation, with Inputs, (an extremely simple) ‘Mechanism’, and Outputs, and can answer important questions about the future of a disease, it doesn’t feel like the type of think we think of when we think of simulations. Let’s examine some other methods before we examine why this doesn’t seem like a good simulation.

Compartmental Model

In compartmental models, a population of people is modelled, and each person in the population is assigned to one of three buckets: Susceptible, Infectious, or Recovered. At the beginning of the simulation all but a small number of people are in the Susceptible bucket. At each timestep of the simulation, there a probability that a Susceptible person will move to the Infected bucket, defined by the parameter β. There is also a probability that a person in the Infected bucket will move to Recovered, defined by the parameter γ.

As this simulation moves through time, we will see an exponential increase in the number of people in the Infected bucket, until it reaches a peak and then decays towards zero, as can be seen in the red line in the figure above. This was the infamous curve that we all wanted to flatten at the beginning of the COVID pandemic, which can be flattened in the simulation by decreasing the parameter β. In fact, you may remember at the beginning of the COVID pandemic that we all kept very close watch on R0, the basic reproduction number of the disease, and R0 is simply calculated from β / γ.

By varying the parameters β and γ diseases of different infectivities and populations with different behaviours can be modelled. ¹ This allows researchers and public health officials predict what the likely outcome of various treatments or government measures would be on the spread of the disease. There are many more extensions that can be made to this compartmental model to better match the details of the disease of interest, but this one will do for our discussion.

Agent-based Model

https://pubmed.ncbi.nlm.nih.gov/16642006/ In agent-based models, a population of people are again modelled, but this time each person’s locations, movements, activities, and interactions with others etc. are all modeled. This results in the same curve of infection we have seen in the compartmental model, however the large increase in complexity means it much more closely models reality. This complexity allows the answering of much more complex questions. In a 2006 paper a agent-based study looked at the effectiveness of various prevention and containment strategies for an Influenza epidemic. They found that

• border restrictions and/or internal travel restrictions are unlikely to delay spread by more than 2–3 weeks unless more than 99% effective.
• School closure during the peak of a pandemic can reduce peak attack rates by up to 40%, but has little impact on overall attack rates,
• Case isolation or household quarantine could have a significant impact, if feasible

This level of detailed prescription likely helped give confidence to governments around the world to implement stay-at-home orders that would damage their economies. The complexity of these simulations come with a hefty price however, with the many parameters making them extremely difficult to calibrate and the level of detail causing them to be quite expensive to run.

Correspondence

Clearly these three types of simulations have very different levels of correspondence to reality. For infectious disease modelling, clearly we need to have some representation of individuals and movement of disease between individuals at the very least.

• Time series models have no correspondence to these things that matter, so don’t let us ask very many questions that matter
• Compartment models have the bare minimum of correspondence, so we can only ask the bare minimum of questions
• Agent based models have correspondence with many more parts of reality, and so we can ask much more complex questions, at the cost of complexity

In tension with this increasing ability to answer important questions about the system of interest, is the ability to set up a calibrated simulation and to run the many different realisations of the simulation needed for exploration of the problem space and for statistical significance.

So the design of a simulation comes down to deciding on the questions that need to be answered, what aspects of reality matter to answering those questions. These aspects of reality clearly must be represented faithfully in the mechanism.

For example in the compartmental model, even though it is an extremely simple simulation, the two aspects of the problem that matter in reality do correspond in the simulation. What about an aspect of disease spread that we know is important in reality, for example how much people move around and interact with other people. In the compartmental model, this is baked into the β parameter, so this doesn’t correspond to reality. However this is an intentional choice by the designers of the simulation. Many important parts of real-life disease spread are abstracted into the β parameter, allowing epidemiologists to study the dynamics of diseases more generally and extremely efficiently. If we do want to ask questions about people’s movement, then we need to ensure this is represented faithfully in the simulation, as is done in the agent-based model.

We can therefore break up the aspects of reality into the parts that matter to answering the question and the parts that don’t. In the simulation, the parts that matter must correspond in the simulation to gain insights, otherwise you will get erroneous results. For the parts of reality that don’t matter, making them correspond will generally lead to wasted resources in simulation setup and runtime. These parts can either be ignored if they are truly not relevant (e.g. we don’t need to simulate tectonic plates in a disease simulation), or abstracted into parameters or components of the simulation. For example, the β parameter in the compartmental model doesn’t correspond to anything concrete in reality, but stands in for many different aspects of disease spread from infected to non-infected people (e.g. contact rates, disease infectivity, mask wearing etc.) It’s important to note that correspondence to reality is subjective, and even the agent-based models have been criticized for simplifying and unrealistic assumptions

Why use simulations

• Make predictions about the future
• Too risky/expensive to do in reality
• Floods of areas
• Too slow to do in reality
• Simulate the cosmic web
• Need to run too many experiments
• Waymo has driven 1000x more in simulations than in reality
• Reinforcement learning requires more data than reality can provide
• Impossible to make measurements of real system
• Protein simulations allow visualization of the movement of proteins

Conclusion

You should simulate too. If you want to, look into reinforcement learning papers to find how they’ve simulated since it’s so data hungry. Simulation is used in many real-world applications, and we in the data/software field should take note of the potential for our domains.

There are many other aspects of running a simulation that are interesting to discuss which I may add in future parts, including calibration of simulations and avoiding “production bias” in simulation results, as well as more on the “SimOps” of running and analysing simulations.

People without dirty hands are wrong.

Cult of Done Manifesto

I haven’t been lucky enough to build any Generative AI products other than playing around on my own time. I was keen to learn from those who have built these and deployed them to customers, so I attended the LLMs in Production conference back in April. This blog is a summary of what I learned, mostly from the viewpoint of someone working at a company that is well versed in ML but is feeling pressure to “do something with Generative AI”.

One thing to keep in mind there are many biases in the views given in topics like these, pessimistic ones from those fearful of change, optimistic ones from those selling picks and shovels in a gold rush, and overoptimistic ones from the future-gazers that come out in every hype cycle. I liked this conference as it seemed to balance these viewpoints well, and it was pretty clear which persona the speakers fell into (and thankfully very few of the overoptimistic ones).

Intro

LLMs are the most successful type of model commonly known as a foundation models, trained to be able to predict what the next word in a sentence would be. The major differentiator of these models is their architecture allows them to be trained on HUGE amounts of data, basically the entire internet. In learning how to do this simple task (and particularly with the addition of instruction fine-tuning), researchers discovered these models could reason, solve problems, and create new, creative-seeming pieces of work. These instruction-tuned foundation models were branded with the label Generative AI, most commonly seen with ChatGPT, DALL-E and Copilot.

Putting this in context, we have the broad base of traditional ML that will likely still be required for quantitative, low-latency, or interpretable use cases. The public will have had the most interaction with copilot-type (i.e. a chat based interface) Generative AI, however it’s likely that the short-term successes for many non-data-rich companies will come from using these foundation models for very targeted applications (e.g. summarise this document, what is this user’s need etc.) which, as we’ll see later, can be easily tackled by distilled versions.

Beyond copilots, full automation of these models is still being worked on, and the long term future is to develop agents, capable of achieving a task e.g. ”go buy me a blue couch for my living room for less than $1000”

ML Product Development

The main benefit of LLMs that kept being brought up at the conference was the ability to reach Version 1 of your product much quicker.

A traditional ML product has generally required the sourcing or creation of datasets, a long process of model training and evaluation, plus hosting of the model for inference. There has been a growing number of off-the-shelf solutions, but these have been for limited use cases, and so were difficult to use as differentiators for your business. Due to the zero-shot/few-shot capabilities of LLMs (and the fact they’re hosted behind a relatively inexpensive API), suddenly we have a model capable of solving a huge variety of tasks with just a bit of prompt tuning (and maybe the addition of some context). While this version of your product may be too expensive or high-latency to use at scale, it crucially allows you to evaluate product-market fit i.e. does the customer even want to use this. Too many times in the past we have gone through the traditional ML product lifecycle only to discover that the customer has no interest in the product!

What does V1 look like

The user’s input (e.g. their query to your system) is wrapped in a prompt and sent to an LLM API (e.g. OpenAI). The response can then be parsed before sending the desired result to the user. Evaluation of the response (i.e. is this a good or bad response) allows for finding prompts that work best. Evaluation can be quite tricky for complex tasks, so you can get away with LGTM@few for V1, but the earlier you implement evaluation the better your product will be. What if the model doesn’t know anything about the task? For example, requires knowledge of data internal to your company or information from after the model was trained (e.g. the score from today’s football game). Then you want to somehow retrieve that relevant information and provide it to the model, also known as Retrieval Augmented Generation.

Retrieval Augmented Generation (RAG)

It is useful to think of the model as a reasoning engine rather than a knowledge store. This shift in thinking can help get the best out of foundation models, as, due to their training process, they are likely to respond with any answer at all rather than say “I don’t know” i.e.s hallucinate. Therefore, we will get more success from asking a question and providing relevant context, which is even true of human interactions. Compare the question “What is the best bed?” to the question “What is the best bed out of these 100 options, given that I am a 36-year-old male redecorating my daughter’s room” Providing relevant information allows the model to respond with information not in its training data and also helps prevent fabrication of plausible but non-existent information. The main limitation is the quality of the search system, if you have low recall you will not be providing the model with the facts it requires for its reasoning.

Most descriptions of RAG assume using a vector search engine for this step, presumably because practitioners familiar with LLMs are more comfortable working with embeddings of text than using information retrieval methods. However, there’s no good reason to prefer semantic search over keyword search for this application, and generally the best choice for V1 is the search system that’s most prevalent at your company (in other words don’t spend your innovation tokens on a new search system since you likely need to spend them elsewhere in this novel product).

Beyond V1

If you get to the lucky place that V1 is actually used by customers, there are likely to be two major issues to contend with:

• Accuracy isn’t sufficient
• Cost/latency isn’t sufficient

The main solutions for these problems outlined in this great panel discussion were:

• Accuracy
  • Fine tuning
• Cost/Latency
  • Use in-house model
  • Distill/quantize in-house model

Accuracy

There are a lot of accuracy improvements to be obtained with just the base API model. Improvements to RAG context can be obtained with better search systems. Prompt engineering (e.g. few shot prompting, chain-of-thought prompting) can make large improvements but changes to the model behind the API can render this work obsolete very quickly. At a certain point however, fine-tuning is required, particularly in domain specific applications which may not be well represented in training data for the API models e.g. legal, finance, health.

This great talk highlighted the ability for LLMs to work out-of-the-box for generic prototypes, but falling down in more complex domains. He showed 4 different use-cases in the banking, pharma, ecommerce, and legal domains with accuracies of 40-60% from off the shelf models which rose 20-30 percentage points with fine-tuning.

Fine-tuning

• Freeze most/all weights of the foundation model, fit an additional set of weights in the model
• Can be done at Open AI/GCP, but in the context of cost/latency section maybe doesn’t make sense
  • Relatively cheap to finetune on these platforms so could be a good V1.1
• Numerous steps
  • Labelling
  • Evaluation
  • Training
• Requires: Labeled examples (data) and evaluation metrics
• Nice to have: OS foundation model, self-hosting

Cost & Latency

Anecdotally, the cost and latency of OpenAI APIs are too high for production applications, particularly for high-volume automated tasks or non-chat user facing applications. High-volume automated tasks are also the tasks most suitable for training smaller expert models that can be self-hosted. An example task from this great panel discussion was summarising web pages for a web search index. The OpenAI API allowed them to evaluate whether LLMs were even capable of task, then a self-hosted, distilled open-source model allowed them to summarise every web page in their index.

In-house model hosting

• LLMs are BIG
• Naive self-hosting potentially increases the cost of latency
• Need techniques to reduce the size of model while retaining performance

Note even shrinking these models 1000x will require engineering prowess to host effectively

Quantisation

• Keep model size, reduce resolution of model weights
• For example, float32to float16 or float32 to int8
• Recently been extended to allow this to work during fine-tuning, reducing 780GB memory yo a more manageable 48GB

Distillation

• Use ”teacher” model (billion parameter) to generate training data to train “student” model (million parameter)
  • Good overview from google
• Note these “small” models are still FAR larger than anything currently hosted at many companies

What does V2 and beyond look like

Using some or all of these techniques, we can reach a state where the accuracy, cost, and latency/throughput of the system is acceptable for production use cases. Note that we have re-introduced many of the steps from the ML product lifecycle we were able to leave out in the creation of V1.0, which highlights the fact that there is no free lunch in bringing ML products to production. This was a recurring theme of the conference, that current gen LLMs are excellent for generic applications and proof-of-concepts, however non-generic domains/application and production workloads are going to require a lot of custom work. The idea of simply throwing your problems to an LLM API and walking away is heavily marketed by the API providers, but the current reality doesn’t seem to bear this out.

a16z (who are invested in a lot of “pick and shovel” companies, so take it with a pinch of salt) have created this architecture diagram for LLM use cases. The details aren’t important, what is most noteworthy is how little of the diagram is focused on the model, and how much is taken up by the supporting infrastructure. This is reminiscent of the infamous Figure 1 in the Hidden Technical Debt in ML Systems paper which has ML as a tiny (yet central) component in the context of the supporting infrastructure.

What Generative AI has really done is accelerated the trend of ML products moving from needing science as the lynchpin to needing engineering as the lynchpin. If your company has been investing in its MLOps capabilities, none of the above should sound worrying to you, however if they haven’t your company is at risk of falling further behind.

Next part

In part 2, we will examine some of the details of the Engineering, Science, and Data needed for succeeding with GenAI products.

• Good Science necessary but not sufficient
  • Designing good evaluation
  • Training and fine-tuning large models is a craft
• Mostly an Engineering problem
  • This is a trend in traditional ML as seen in the recent history of MLOps
  • Generative models exaggerates this due to greater capabilities and risks
• Data
  • Lots to discuss!

Everyone who’s lived in Chicago knows about its weather. We get all four seasons, and not necessarily in the expected order. The “feels like” temperature last year ranged from -40 to 95 degrees Fahrenheit. The Windy City lived up to its name with gusts up to 55 mph and the rain and snow was ever present. Throughout all this though, Divvy cyclists powered through, cycling on all but two days of the year, and that’s only because the system was shut down!! Let’s explore how the various forms of weather affected Divvy riders last year.

Temperature

Temperature and Divvy trips for every day in 2014

Obviously no-one likes cycling in cold weather, so a good place to see how weather affects Divvyers is the temperature. The graph below shows the average “feels like” temperature in Chicago for every day in 2014. You can see the long, cold period we had in the first few months, then a lovely mild summer with no sweltering hot days, followed by a relatively mild beginning of winter at the end of 2014. On the same graph, I’ve plotted the number of Divvy trips taken each day in 2014 and, as you’d expect, when the weather warms up throughout the year, people use Divvy much more. It’s especially interesting to look at the times in March when the temperature first gets above 40 degrees, Divvy ridership spikes correspondingly, just like the first time the temperature spikes above 60 degrees in May. From seeing this overall trend, let’s see how well temperature and ridership are correlated. Figure 1: The number of trips taken each day of 2014 tracks very well with the temperature for the day. This is especially true for days when there is a significant increase in temperature. There’s nothing like a sunny day in March to make you want to cycle down the lakeshore!!

Temperature vs Number of Trips

In the next figure, I’ve plotted the temperature and number of trips for each day against each other. In case you’re unfamiliar with this type of visualization, each point on the graph is an individual day. The x-position for the point is the temperature for that day and the y-position is the number of Divvy trips taken that day. As you can see, as the temperature increases, the number of trips taken increases rapidly. The overall trend roughly follows (temperature)2 i.e. when the temperature doubles, the number of trips taken quadruples. It’s also possible to fit the data to two straight lines, one below freezing and one above freezing since it’s pretty easy to see that there is an elbow in the graph at around 32 degrees F. Unfortunately (for the purposes of analysis, not for real life!) this summer was particularly mild, so we can’t see what effect really hot weather has on Divvy ridership. I would imagine that the number of trips per day would start to fall off as the temperature got above 100 degrees F, but we can’t see that from the 2014 data. There’s maybe a hint of this effect in the fact that the top 5 days for Divvy ridership all occured below 75 degrees F, but we’ll need more hot weather data to be sure. Figure 2: Temperature and number of trips taken correlate very well, following both a (temperature)2 trend or two linear trends above and below freezing. We don’t see any falloff in number of trips taken in very warm weather just yet.

Divvy includes in the data about each trip whether the person taking the trip has a Divvy membership or whether they just have a daily Divvy pass. My assumption about these two groups of people is that Divvy members primarily use it for their commutes and non-members are tourists or pleasure-bikers, and the temperature data helps to bear that out. The plot below is the same as Figure 2, but now I have split the data into whether the person taking a trip is a member or non-member. Looking at the non-member data, we can see that if the temperature is below freezing, there are barely any non-members using Divvy, and as the weather warms up there is an explosion in trips taken, up to 12,000 trips per day. Members however are much hardier creatures, with thousands of trips per day still being taken in negative temperatures!! There isn’t a much better testament to the toughness of Chicagoans than the fact there were over 400 trips taken on days when the temperature felt like negative 20 degrees fahrenheit!!! Thinking about the lunatics who were out cycling in that type of temperature made me wonder how long they were suffering out there, and then how long trips taken on Divvy are in general, and whether they get shorter as the temperature drops. Figure 3: Members (those with yearly passes) and non-members (those with daily passes) show very different behaviour. Members tend to use Divvy for commuting and so travel even in very cold weather, whereas non-members use Divvy for pleasure, and there’s nothing pleasurable about cycling in below freezing weather, so they take very few trips below 32 degrees F.

Temperature vs Length of Trips

In this figure I’ve plotted the average trip duration vs the temperature. It’s interesting to note that the maximum length of time you can have a Divvy bike out before you are charged extra is 30 minutes, and the average length of trips for every day is less than 30 minutes. People really don’t want to pay extra! More interestingly for this discussion, we can see a clear relationship between temperature and average length of trip. The average length of a trip at 80 degrees F is roughly 5 minutes longer that the average length of a trip at 0 degrees F. We can drill down into the main cause of this by examining the differences between members and non-members in the next figure. Figure 4: As temperature increases, the average length of a Divvy trip increases.

In this figure you can see the large difference between the behavior of members vs non-members. The average length of a member trip is about 10 minutes, which jibes well with the idea that most members use Divvy for the last part of their commute, e.g. from the train to their house. The average length of a non-member trip is much closer to 30 minutes, the maximum allowed time you can take a bike out before extra charges. Clearly non-members want to get the most “value” from their daily pass and are taking much longer trips. In addition, both types of behavior change very little as temperature changes. Members consistently average about 10 minutes per trip in cold weather or warm, with a slight increase of about 2 minutes over the range of temperatures we saw. Non-members show a slightly larger increase in average trip length, but the large increase we see in average trip length we saw in Figure 4 has more to to with the shift in the types of users at different temperatures we saw in Figure 3, with more members at low temperature, and a 50/50 mix of members and non-members at higher temperatures. Figure 5: Members and non-members show very different behavior. Members take short trips in all weather but non-members take much longer trips and their trips get shorter as the weather cools down.

Rain (well, actually clouds!)

As there was great correlation between temperature and people’s Divvy riding habits, I was excited to see how other types of weather would affect Divvy ridership. Again, no-one likes riding in the rain, so I expected to see much fewer trips on days when it rained. In the figure below, we can see there’s no correlation between the probability of rain and Divvy ridership. There tended to be just as many trips on days where there was a 100% probability of rain as there were on days where there was a 0% probability of rain. Figure 6: No correlation is seen between the probability of precipitation and number of Divvy riders. This is somewhat difficult to see in the points alone, so I’ve fit a straight line to the data to show that there is almost no decrease in the number of trips taken per day as the precipitation probability increases.

When we look at cloud cover though, (where 0 = blue skies and 1 is a totally cloudy sky) we can see there is a definite decrease in the average number of trips taken per day when the cloud cover increases. This suggests that people use the clouds as their own rain forecast. Similar to temperature, we see that high cloud cover really cuts the non-member trips down, while members are more happy to risk getting caught in a shower. Figure 7: There is a strong correlation between cloud cover and number of trips taken, especially for non-members. If the cloud cover is above 75%, there are never more than 2,000 non-members trips, where there are up to 12,000 non-member trips on clearer days.

The Windy City

A discussion about the effects of weather in Chicago would not be complete without talking about the wind! The wind data gives the average daily wind speed in miles per hour, with average speeds up to 25 mph seen in 2014. It should be noted that these daily averages will be lower than the max wind speeds seen. As we can see in the figure, wind speed has very little effect on the number of trips taken when it’s below about 15 mph, with no clear correlation between wind speed and number of trips taken in a day. However, once the average wind speed gets above 15 mph, the number of trips taken per day drops off dramatically. 15 mph corresponds to a moderate to fresh breeze at which point “dust and loose paper raised, small branches begin to move” according to wikipedia, so it clearly doesn’t take much of an average wind speed to stop people Divvying. It’s also interesting to note that the drop off in number of trips taken happens at 15 mph for both members and non-members. For both temperature and rain/cloud cover, we saw that members were willing to take their short 10 minute cycles in worse weather than non-members did. Wind is the great equalizer though it seems, no-one wants to battle through a windy day on a Divvy. Figure 8: There is no correlation between wind speed and number of trips taken when the wind speed is below about 15 mph. Once the average wind speed for the day gets above about 15 mph, the number of trips for the day drops to much lower values.

There is one clear outlier on this figure however, the member point at wind speed = 23 mph, number of trips = 4000. It got me worrying whether I had made some mistake in processing my data, as I didn’t think it was reasonable for 4000 people to be willing to cycle on one of the windiest days Chicago had last year! Once I looked at the day for that data point I realized what was going on. That data point represents October 31st 2014, Hallowe’en, so clearly people were willing to cycle to their costume parties whatever the weather. I still wanted to make sure I hadn’t made some error in processing the data, so I checked the weather in Chicago for Hallowe’en in 2014 and I came across this video.

Nothing embodies the ridiculous levels that Divvy riders went last year better than this video. Not only did someone walk up to a Divvy station, see the bikes blowing in the wind like flags and still hop on and cycle off, but thousands of other someones did the exact same thing!! Either Chicagoans DGAF about the weather at Hallowe’en or we had 4000 extremely commited Wicked Witches of the West.

Conclusion

Overall we’ve seen that the weather has a strong effect on Divvy riding. Non-members show the most sensitivity to the weather, which fits with their usage of Divvy for pleasure. Members tend to be more consistent in their riding patterns, which fits with their usage of Divvy for short commutes. We also saw that temperature has by far the strongest effect on ridership, with strong winds and cloudy skys also having an effect. Finally, we have proof for the claim that Chicagoans are don’t care about bad weather, with many trips taken by members on days with negative Fahrenheit temperatures and 50 mph gusts of wind.

If anyone would like to see how I analyzed the Divvy and weather data, I have all my code on github where you can download it and play with it yourself. All weather data is from forecast.io