So there are many data sets, many websites that are kind of specializing on data and make a lot of data available.
One that I really like is the 538,
the data set that was started by Nate Silver, the data set, the website that was started by Nate Silver.
Five, thirty, eight, stands for what?
Number of electoral college votes, though.
who been a sort of a statistician for baseball specialists,
moved to using aggregating the polls to think about giving probability of success in various elections, has been quite successful at it.
he was the only person that he went anywhere that I read,
who was reasonably who put a reasonably high probability on the fact that the 2016 action went the way it did.
So we had a lot of very interesting discussion on how to use data for politics and for
But the website is much much much much more than politics in particular.
It has a lot of sports data But it is just a of like data about
for example last year a student in this class did a paper on
Flight it's get there was some like survey of people on what do they think?
What do they think is polite or impolite to do in a flight?
You'd be surprised by the number of people who think that it's impolite to bring a kid in the flight
So this was what I remember from former people So 538 I really would go and visit first of all if you're interested in data.
There lots of interesting discussion And so there were lots of data sets for this class.
These are set usually not very big,
kind of easy to go around, kind of it could be a good place to start and play with some time.
Recently, so last year about this time, Yahoo started the data dump.
which you can find there, which is at the time was the largest data set available in one terabyte of data.
how people relate to the news,
so I haven't really looked into it,
but if you want to play with big data, that's big data right there that you can just use and download.
Very, very soon, so Uber, I think it's fired by Yahoo, decided that why can't we do that, too?
And then they have started to make available from now, it's for city planner.
there will be an extract for the general public, and can put your name on the web on the waiting list.
It might not be enough for this class, but it might help you.
This is going to, it's called Uber movement, and it's going to be based on a very, very large data set on people.
So people driving movement, so there's probably going to be a lot to learn from that.
I think when we teach this, I this year from this one.
Over time, we'll have more of those.
getting used to seeing how this data looks like is there.
Two other data sets usually, a lot of people in this class like to work with bots data.
You're welcome to, you don't have to.
And if you can, then they are a website that sort of specialized in integrating lots of sports data.
Something which is very, very helpful for some projects is the Wayback Machine, which is basically an archive of the Internet.
you can find all of the headlines of the New York Times for,
I mean, all of the front page of the New York Times from any, from, you know, going back a long period of time.
So, the way back machine can be useful.
I'll show you a project today where they were interested I'm in
I'm interested your how the price of used books has changed over time with the introduction of more and more online sellers of those books,
etc., so that's the paper.
So, they use the way back machine to display a bunch of web pages for searches for a particular
book and then they use the some data technique, the scraping technique that I'm going to show you in a minute to extract them.
So, the way back machine is like something that is very good to be aware of that's more in combination with web scraping method.
And this is just like a snippet.
You can search in library catalog.
If interested in something, at the movement, like access for many projects, you'd find something that might be appropriate.
