Now, let's think about harvesting data.
So two ways to harvest data.
First, you harvest data that's already there.
You go with your software.
One you need to find is like a combined tractor or whatever.
We don't need to plant it.
And then sometimes you actually need to plant it first.
That takes a little bit more time sometimes, but that's also useful.
So let me go over these two.
So what's grape scraping?
It could be full data from some data from some It could be calling an entire webpage, maybe from form to form.
It could be a set of forms running in the background,
so I'll give you an example where you keep a sort of querying,
for example, the of books, and any of the above in an ongoing fashion where you keep sending.
Crawling means searching all over the website.
So in a page, you could just find one page and extract what you want from one page.
Or you could just keep going to the website and extract,
like, go through, for example, If you want to look for the, you want to construct the database of the price of
light, calling will involve going from, you know, keep querying.
This is, you know, Boston L.A.
And then Boston to Phoenix and Boston to San Diego.
So you're calling the site to find out what it is.
So to get data from the internet, that's not already in available form.
There are two big classes of methods to do that.
First of all, many of the big websites maintain what's called an API.
our application program interface, which basically help a particular program or a particular application to communicate with other applications.
So if you want data from Twitter,
or you want data from Facebook,
or you want data from Google Map,
you will not physically go on each of the website and then copy the information and put it in your database.
I mean, you could, and I'll show you a little bit how to do it later.
But it's not typically the way you would do it.
Typically something like Google Map,
I'll show you an example with Google Map,
but will have an API which is, they will directly give you the data that you need upon receiving a query from you.
sometimes it is, you have to pay, sometimes it is free for a little while, and then you have to pay for the rest.
Sometimes you first need authorization from the person, from the data you want, etc.
a lot of the data is public anyway, because once you can follow someone and once you follow them, you can get their news.
So a lot of data from Twitter is perfectly fine.
And way you would do that is not by actually doing it.
scraping from the Twitter page the way you would do that by communicating with Twitter to an API and telling exactly what you need.
Say you want all of the tweet for
