[0:00] hey there how's it going everybody in
[0:01] this video we'll be learning how to
[0:02] scrape websites using the beautifulsoup
[0:04] library now if you don't know what it
[0:06] means to scrape websites basically this
[0:08] means parsing the content from my
[0:10] website and pulling out exactly the
[0:12] information that you want so for example
[0:14] maybe you want to pull down some
[0:15] headlines from my new site or grab some
[0:18] scores from a sports website or monitor
[0:20] the prices of some items in an online
[0:23] store or something like that now to show
[0:25] an example of this let's take a look at
[0:27] the finished product that we'll be
[0:28] building in this video and then we will
[0:30] learn how to build it so I'm here on my
[0:32] personal website and on my home page
[0:34] here I have a lot of different posts of
[0:36] my most recent videos and every post
[0:39] that I have has a title here that is a
[0:42] big heading tag and then I have a text
[0:45] summary of the video here and then I
[0:48] have a link to the video so let's say
[0:50] that we wanted to write a scraper that
[0:52] would go out and scrape all of this
[0:54] information so I wanted to grab all of
[0:56] the post titles and summaries and links
[0:59] to the videos from my home page and I
[1:02] wanted to ignore all this other
[1:03] information so to do this I have a
[1:05] finished version of what we'll be
[1:07] building in this video and then we'll
[1:09] learn how to build it so right now I can
[1:10] just run this with Python and this is
[1:13] called CMS scrape pie so if I run this
[1:16] then this went out and scraped all of
[1:19] the titles and summaries and links so we
[1:21] can see here we have a title so this is
[1:23] my CSV module video and then we have the
[1:27] summary text here and then we have the
[1:30] link text here now not only did this go
[1:32] out and scrape this information from the
[1:35] website and print it out here in the
[1:36] terminal but it also created a CSV of
[1:40] all this information as well so if I
[1:42] open up this CMS scrape dot CSV this
[1:47] should open it up in numbers but you
[1:48] could also open it up in Excel
[1:51] now this isn't very readable right now
[1:53] but if I make these columns a little bit
[1:57] smaller here and then wrap this text
[2:01] then we should be able to read this so
[2:03] you can see we have a column that has
[2:05] all of our headlines for all the
[2:06] articles on that homepage and then all
[2:09] the text summaries and then a link to
[2:10] each video so that is what web scraping
[2:13] is
[2:13] it goes out and it pulls down all of the
[2:16] information that you want from a
[2:17] specific website so now I am going to
[2:20] clear those out and pull these back up
[2:22] now if you're trying to parse out that
[2:24] information with something that you had
[2:25] built in Python yourself then you
[2:27] probably run into a lot of issues but
[2:29] luckily there's the beautiful soup
[2:30] library that makes parsing out all this
[2:32] information a lot easier to do now we'll
[2:34] also be using the request library in
[2:36] this video to make our web request now
[2:38] you could use the built-in URL Lib
[2:40] module but the request library is
[2:42] extremely popular for fetching websites
[2:44] so we're gonna go ahead and use that so
[2:46] let's go ahead and get started and see
[2:48] how to do this
[2:49] so first of all let's make sure that we
[2:50] have everything installed that we need
[2:52] so to install beautifulsoup you can just
[2:55] use the pip install command so to do
[2:57] this we can just say pip install and
[2:59] this is beautiful soup and this is
[3:03] beautiful soup for so you can see that I
[3:06] already had that installed but if you
[3:08] don't have that installed then your
[3:09] should just go through the installation
[3:10] at that point now you definitely want to
[3:12] install beautiful soup 4 because there
[3:15] is an older version just called
[3:16] beautiful soup but beautiful soup 4 will
[3:19] give you one that's most up-to-date so
[3:21] once that's installed then we need to
[3:23] make sure that we have a parser to parse
[3:25] our HTML now I won't go deep into the
[3:27] details of these parsers but there are
[3:29] some small differences between the
[3:31] parsers and they could return different
[3:32] results depending on the HTML that
[3:34] you're trying to parse now if you're
[3:36] trying to parse perfectly formed HTML
[3:38] then those differences aren't going to
[3:40] matter but if there are mistakes in the
[3:42] HTML then the different parsers will try
[3:44] to fill in missing information
[3:46] differently so beautiful soup has a
[3:48] section in their documentation about the
[3:50] differences between those parsers and
[3:52] basically they suggest installing and
[3:54] using the L XML parser so that's what
[3:57] we're going to use in this video now
[3:59] they also say that the html5 Lib parser
[4:02] uses techniques that are part of the
[4:04] html5 standard so you could use that one
[4:06] too
[4:07] but most of the time the choice between
[4:09] the parsers isn't really going to matter
[4:11] all that much as long as you're working
[4:12] with good HTML but I'll go ahead and
[4:14] leave a link to the differences between
[4:15] those parsers and the description
[4:17] section below if you want to read more
[4:18] about those so to make sure that we have
[4:20] the L XML parser installed we can
[4:23] install it with pip also so we could
[4:25] just say pip install
[4:27] and that is El XML so if we run that
[4:30] then I already have that installed but
[4:32] yours will install there if you don't
[4:33] already have that now if you want the
[4:35] html5 Lib parser then you can just do a
[4:38] pip install html5 Lib so like I said
[4:42] we'll be using El XML in this video but
[4:45] the html5 Lib is popular as well so now
[4:48] we also need the request library and
[4:51] just the same we can do a pip install
[4:54] request and run that you can see that
[4:57] mine's already installed but if you
[4:58] don't have it then yours should get
[5:00] pulled down right there okay so now that
[5:02] we have those installed let me clear
[5:03] that out and now let's take a look at
[5:06] what we can use these for now you don't
[5:07] have to be extremely familiar with HTML
[5:09] in order to scrape websites but it
[5:11] definitely helps to know so basically
[5:14] hTML is structured in a way where all
[5:16] the information is contained within
[5:18] certain tags and if you're at all
[5:20] familiar with XML then it's very similar
[5:22] to that now I have a very extremely
[5:25] basic HTML file open here in my browser
[5:28] so we can see that this small example
[5:29] just has one big hit header here that
[5:32] says test website and then we have two
[5:35] large links here for articles and one is
[5:38] the article 1 headline and then it has a
[5:40] small text summary here below that and
[5:43] then we have a big article 2 headline
[5:44] here with a text summary below that and
[5:47] then we have a footer down here at the
[5:49] bottom now this is how browsers display
[5:51] HTML we are using the Chrome browser
[5:53] right now but in the background the
[5:55] source code looks a bit different so I
[5:58] have the source code for this very basic
[5:59] website pulled Oh pulled up over here on
[6:02] the right side of my screen so let me
[6:04] make this a little smaller here and then
[6:06] I will stretch this over so that we can
[6:08] better see the source code so we can see
[6:11] how this is structured so we have these
[6:13] tags throughout our document and there
[6:16] are opening tags that are surrounded by
[6:18] these angle brackets here so we have
[6:20] this head tag that opens the tag and
[6:23] they also have closing brackets down
[6:26] here which are the same except they have
[6:28] a forward slash after the first angle
[6:30] bracket so the close of our head tag
[6:32] will be this line here and everything
[6:35] all this content is within this head tag
[6:38] so all this here is a single head
[6:41] and all of these tags can be nested so
[6:43] if we want to find our article headline
[6:45] and article summaries then we can look
[6:47] down here in our body tag so we have an
[6:49] opening body tag here and within the
[6:52] body we have our you know test website
[6:55] h1 here which is a heading and then we
[6:58] have a div tag here which has a class of
[7:01] article and within this div we have our
[7:04] h2 tag and h2 is another heading a
[7:08] subheading and within that h2 we have a
[7:11] link these a tags which are anchor tags
[7:14] these are links so this is the text to
[7:16] the link here article 1 headline that's
[7:19] what gets displayed over here in the
[7:20] actual website we can see article 1
[7:22] headline and this href this is actually
[7:25] where this links to so this links to a
[7:28] page article 1 dot HTML now these
[7:31] classes here how this has a class of
[7:33] article these are mainly used for CSS
[7:36] styling and can also be used within
[7:38] JavaScript to identify specific elements
[7:41] now below that heading tag that we
[7:42] looked at then we just have a paragraph
[7:44] tag here which is just a P and this is
[7:46] the text summary of that article so we
[7:49] can see here that this is this entire
[7:51] div with the class of article has our h2
[7:54] heading and then our paragraph for the
[7:56] summary and then this is just repeated
[7:58] down here so for our second article we
[8:00] have another div with the class article
[8:02] and then another h2 but this one is for
[8:05] the article - head headline and the
[8:06] article - link and then the summary text
[8:10] for article - and then lastly we have a
[8:12] footer down here at the bottom that is
[8:14] just a div with the class of footer and
[8:16] that has a paragraph tag within there
[8:18] with some text so everything else in
[8:21] here is just extra information so we
[8:23] have some scripts and up here at the top
[8:25] we have some style sheets and things
[8:27] like that but all of this and the body
[8:30] is what gets displayed over here in the
[8:31] website so let's use this very simple
[8:33] example to see how we can parse out
[8:35] information using beautifulsoup
[8:37] so I'm going to open up a file here
[8:39] called scrape pie now all we have in
[8:41] here so far are our imports for a
[8:43] beautiful soup and requests so we have
[8:46] from bs4 import beautiful soup and then
[8:50] we're also importing requests so let's
[8:52] say that we wanted to parse out the
[8:53] article headline
[8:54] and the summaries from our very simple
[8:57] website over here so in this example
[8:59] it's just article one and it's summary
[9:01] text and then article two headline and
[9:04] it's summary text so first things first
[9:05] let's pass our HTML in the beautiful
[9:08] soup so that we can get a beautiful soup
[9:10] object now there are a couple ways to do
[9:12] this we can either pass in the HTML as a
[9:14] string which is what we'll do in a
[9:16] minute when we parse our website from
[9:18] the internet but we can also pass in an
[9:20] HTML file and in our case we have this
[9:22] sample HTML file within our current
[9:25] directory so let's go ahead and just
[9:27] open up this file and pass it in to
[9:29] beautifulsoup
[9:30] so to open up this file we can just say
[9:32] with open and this HTML file is called
[9:36] simple dot HTML and it's within the same
[9:39] directory of our script so we don't have
[9:42] to specify a path and then we're just
[9:45] going to read that in so read is the
[9:47] default so we don't have to do anything
[9:48] there and I'll just say as HTML file and
[9:52] then the pass dot HTML file into
[9:54] beautifulsoup
[9:55] we can just say soup equals beautiful
[9:58] soup and then we will pass in that HTML
[10:00] file and now we need to specify our
[10:02] parser and like I said for this video we
[10:05] are going to use the L XML parser now if
[10:08] working with files is new to you and you
[10:09] want to know more about this like with
[10:12] open statement and things like that then
[10:14] I do have a video specifically on
[10:16] working with file objects and I'll leave
[10:18] a link to that in the description
[10:19] section below ok so now we have this
[10:21] soup variable which is a beautiful soup
[10:23] object of our parsed HTML so let's just
[10:26] print this out and see what we get so we
[10:29] can just print out soup so if I save
[10:31] that and run it then let me make this a
[10:34] little bit bigger here so we can see
[10:36] that this just prints out all of the
[10:38] HTML and so it's very similar to what we
[10:41] just looked at now this HTML isn't
[10:43] formatted in a very readable way it's
[10:45] all you know pushed over to the left if
[10:47] we actually look at that simple dot HTML
[10:49] file we can see that it's nice and
[10:51] indented so in order to format this to
[10:54] where we can more clearly see which tags
[10:56] are nested within each other then we can
[10:58] just use the prettify method to clean
[11:00] this up a bit so if we say soup dot
[11:03] prettify and that is a method so we have
[11:06] to put in
[11:07] if we save that and run it now we can
[11:09] see that it in dents these two where we
[11:12] can see what tags are nested within each
[11:14] other so here is that head tag that we
[11:16] saw before and then everything that is
[11:18] indented within that head head tag is
[11:20] belongs to that head tag okay so now
[11:23] let's see how to grab information from
[11:25] this HTML so the easiest way to get
[11:27] information from a tag is to just access
[11:30] it like an attribute so if we wanted to
[11:32] grab the title of our HTML page and if I
[11:35] look here at our HTML this should just
[11:38] be test a sample website is our title
[11:40] for this so the easiest way to get that
[11:43] is to just access it like an attribute
[11:45] so I will say match equals and we will
[11:47] do soup dot title and then we will just
[11:51] print out that match so I'll save that
[11:54] and run it then we can see that it
[11:55] parsed out that title tag now it still
[11:57] has the title tags around the text so if
[12:00] we only wanted to grab the text of the
[12:02] title tag then we can access the text
[12:04] attribute of that tag so we can just add
[12:06] that to the end here so we'll say dot
[12:09] title dot text so if I save that and run
[12:11] it then you can see that now we only get
[12:14] the text of that title tag now searching
[12:16] for a tag like we did here by accessing
[12:18] it like an attribute by saying dot title
[12:20] that will get the first title tag on the
[12:23] page but the first tag on the page not
[12:25] all might not always be what we want so
[12:28] we can use the fine method to do
[12:30] something similar but it will also allow
[12:32] us to pass in some arguments that we can
[12:34] find the exact tag that we're looking
[12:36] for so for example if I use this dot
[12:39] access to find the first div on the page
[12:41] and I do soup dot div if I save that and
[12:45] run it then we can see that it got the
[12:46] first div tag on our page with all of
[12:48] its child tags which is everything for
[12:51] that first article but if we wanted to
[12:54] grab the div tag that has a class of
[12:56] footer for example then we'll have to
[12:58] use that fine method and pass in some
[13:01] arguments so let's use that fine method
[13:03] so we'll do soup dot find and now we
[13:06] will search for a div now if I save that
[13:09] and run that right there then we just
[13:11] get the same thing we just get the first
[13:14] div on the page but with this fine
[13:16] method we can pass in arguments of
[13:18] attributes that narrow down exactly
[13:20] what tag we want to find so for example
[13:23] I can pass in an argument of class and
[13:26] then after class we need an underscore
[13:29] class underscore equals footer now these
[13:33] arguments can match any attributes that
[13:35] your tag might have and most of the time
[13:37] you can just pass in arguments just like
[13:39] they are in the HTML so if you wanted to
[13:41] match a div with an ID of footer then
[13:43] you could just pass in an argument of ID
[13:45] equals footer but the reason that we
[13:47] need an underscore after class is
[13:49] because class is a special keyword in
[13:51] Python so they use class underscore
[13:53] instead so if you were confused about
[13:55] that then that's why they have that so
[13:57] if we save that and run it then we can
[13:59] see that now we're not getting the first
[14:01] div on the page we're actually getting
[14:03] the div with the class of footer okay so
[14:06] now let's say that we wanted to parse
[14:07] the HTML and get all the article
[14:10] headlines and summaries from our page
[14:11] now anytime that we want to get multiple
[14:14] things from a page a good way to start
[14:16] is to just get one of whatever it is
[14:18] that you're trying to parse so for
[14:19] example if I wanted to get grab the
[14:21] headline and snip it from each article
[14:24] on our page over here then let me start
[14:26] by first grabbing that information for
[14:28] one article and once we have that
[14:31] working then we can apply the same logic
[14:32] to all of our articles so if we go back
[14:35] here to our browser and look at our page
[14:37] now in order to dig down into the HTML
[14:39] and find exactly where our article
[14:41] headline and summary is within the
[14:43] chrome browser we can just right click
[14:45] on whatever it is that we want to parse
[14:47] and then click on inspect now I know
[14:51] this is a little small let me make this
[14:52] just a little bit bigger here and then
[14:54] we'll walk through a little bit of how
[14:56] to use this so I'm using chrome here but
[14:59] pretty much every major browser anymore
[15:00] has something like this and this is
[15:03] really useful for finding exactly what
[15:05] you want so within the inspect here if I
[15:07] just hover over our div class of article
[15:10] then you can see that in the top part
[15:13] here it's actually highlighting that
[15:15] entire everything that is within that
[15:17] div and if I go down to the h2 then it
[15:21] only highlights that h2 and then if I
[15:23] hover over the href then it highlights
[15:25] that link and if I hover over the
[15:28] paragraph it highlights that paragraph
[15:29] so we can see exactly what is what and
[15:32] the same with the second article if I go
[15:34] down
[15:34] here and hover over this article that I
[15:36] can see that that has the article to
[15:38] headline and summary text and I can
[15:41] click on this little arrow here to
[15:42] expand this and then it shows me
[15:44] everything that is within that div there
[15:46] so we have the h2 they ate the anchor
[15:49] tag and then the paragraph tag with the
[15:51] summary text so just like we saw before
[15:53] in the source code our article headlines
[15:55] are within a div with a class of article
[15:58] and then an h2 and then an anchor tag so
[16:01] let's go ahead and grab the article div
[16:04] so let me make this little smaller here
[16:07] so that we can see this so let's grab
[16:09] that first div with the class of article
[16:12] so I'm going to change this variable
[16:13] name here over to article and then print
[16:16] that out now this is going to be a div
[16:18] with the class of article so if we save
[16:22] that and run it then we can see that now
[16:24] we have that first article and we can
[16:26] search that matched tag just like we
[16:28] searched the entire HTML document so we
[16:31] can access child tags with the dot
[16:33] access like an attribute or we can use
[16:36] the find method so for example if we
[16:38] wanted to dig down into the text of the
[16:40] headline then we could say headline is
[16:43] equal to and now we don't want to use
[16:47] that entire soup which is the entire
[16:49] HTML we only want to search within this
[16:51] article so now we'll say article dot h2
[16:55] and within that h2 we want to access the
[16:58] anchor tag so that is dot a and now we
[17:01] want the text of that acre tag so we can
[17:04] just string all of that together so with
[17:06] all that strung together if I print out
[17:09] that headline and save that let me
[17:11] comment out this entire article for now
[17:13] so if I save that and run it then we can
[17:16] see that it grabbed the text of that
[17:17] first articles headline and we can do
[17:20] the same thing with the article summary
[17:22] so it's just a paragraph within our
[17:25] article so if we go down a couple lines
[17:28] here then we could say summary is equal
[17:30] to and that is article dot P so just to
[17:34] grab that paragraph dot text so if we
[17:37] print out that summary save that and run
[17:41] it then we can see that now we have that
[17:43] article one headline text and then we
[17:46] have the text summary of that our
[17:48] as well okay so now we have the code
[17:50] here for grabbing a headline and a
[17:53] summary from a single article so now
[17:56] that we have this information for one
[17:57] article we can most likely use this
[17:59] reuse this information to parse the
[18:02] information from all of our articles so
[18:04] right now we're using this fine method
[18:06] to just get the first article but now we
[18:08] need to loop through all of the articles
[18:10] so to get all of the articles we instead
[18:13] of using find we can just use the fine
[18:16] all method now with find all instead of
[18:19] just returning the first tag that
[18:21] matches these arguments it will instead
[18:23] return a list of all the tags that match
[18:25] those arguments so instead of just
[18:27] setting this variable we can now loop
[18:29] over the list that that returns so
[18:32] instead of saying article equals we can
[18:34] just create a for loop so we can say for
[18:36] article in soup dot find all since this
[18:40] returns a list so now we have a for loop
[18:43] there I'm just going to get rid of that
[18:45] print article line and then I'm going to
[18:48] put this logic here for grabbing the
[18:51] headline and summary from an article
[18:53] within that for loop and now it'll loop
[18:56] through all the articles which in this
[18:57] case is just the two of them and we'll
[18:59] get the information for both of those
[19:01] and also let me put in one more blank
[19:04] print statement here within our loop so
[19:07] that at the end we have a blank line
[19:09] between our articles so if I save that
[19:11] and run it then now we can see that we
[19:13] have the article one headline and the
[19:16] summary for that article we also have
[19:18] the article two headline and the summary
[19:20] for that article okay so this is good so
[19:22] we're starting to see how this would be
[19:24] useful for getting information from
[19:25] websites so now let's do something
[19:27] similar but with an actual website so
[19:30] like we saw before I have my personal
[19:33] website pulled up here in the browser
[19:34] and like we saw if I scroll down then we
[19:37] can see that we have a lot of video
[19:38] headlines and summaries and the embedded
[19:41] videos that themselves so let's say that
[19:44] we wanted to grab these titles and these
[19:46] summaries and links to the videos so
[19:49] first things first let me just delete
[19:51] what we had so far with our simple HTML
[19:56] file that we used and I'm also going to
[19:59] get rid of where we are
[20:02] in that file so first things first we
[20:05] want to get the source code from my
[20:08] website using the request library and to
[20:10] do this we can just say source equals
[20:13] request dot get and now we want to get
[20:17] my website which is just HTTP kori MS
[20:22] com
[20:23] now this request dot get will return a
[20:25] response object and to get the source
[20:28] code from that response object we can
[20:30] just add on dot text to the end so now
[20:33] this source variable should be equal to
[20:35] the HTML of my website so now we can
[20:39] pass this in to beautiful soup so now
[20:42] let's see if that worked so if we print
[20:45] out soup dot prettify like we saw before
[20:48] then this should print out the formatted
[20:51] code for my website so it looked like
[20:53] that worked if I scroll up here we can
[20:55] see that this does look like HTML it's
[20:57] kind of a mess because it's a larger
[20:59] website but we can see that you know
[21:02] these links seem to be coming from my
[21:04] website so it looked like that worked so
[21:06] now we can start parsing out the
[21:07] information that we want now just like
[21:09] before let's start off by grabbing one
[21:12] videos information and then it will loop
[21:14] through to get the information for all
[21:16] the videos so to grab the first headline
[21:19] and snippet for the first post on my
[21:21] page let's inspect my website and see if
[21:25] we can figure out what the structure is
[21:27] so I'm going to make this a little
[21:28] larger here and now I'm going to use
[21:29] that inspect functionality again within
[21:32] our browser to see if we can pinpoint
[21:34] exactly where this information is that
[21:37] we want to parse so if I hover over my
[21:40] headline and right-click on that and go
[21:42] to inspect then we can see that it is a
[21:46] link inside of an h2 here with a class
[21:49] of entry title now if I go up a little
[21:52] more we're trying to find something that
[21:54] encompasses all of our headline and our
[21:57] summary text and our video now if I
[21:59] hover over this article here with all
[22:02] these different classes if I scrolled
[22:04] down then we can see that that article
[22:07] encompasses our headline and our summary
[22:10] text and our embedded video now if I
[22:12] scroll down a little bit more than we
[22:14] can see that it stops after
[22:16] that first post so this is likely going
[22:18] to be our starting point since this
[22:20] contains all of the information within
[22:22] this first post so if I scroll back up
[22:24] within this article we have this h2 with
[22:28] entry title that has our header there
[22:31] now if I expand this paragraph here then
[22:35] go down a little bit okay so that's just
[22:37] metadata for the entry if I go over this
[22:40] entry content that seems to have the
[22:43] summary text and the embedded video so
[22:46] if I expand that then this first
[22:48] paragraph here is our summary text and
[22:52] the second paragraph here has the
[22:54] information for our embedded video okay
[22:56] so this is a good starting point so
[22:58] let's start off by first grabbing this
[23:00] entire first article that contains all
[23:03] of this information so now I'm going to
[23:06] close the inspector and take this down
[23:09] to size a little bit so that we can see
[23:11] that at the same time that we're working
[23:13] okay so to grab that first article let's
[23:16] just say article is equal to soup dot
[23:20] find and then we will search for article
[23:24] so if I save that and now let's also
[23:27] print out this article and put in a
[23:30] space there and run that now this is all
[23:33] kind of a mess here so we can actually
[23:36] pretty PHY these tags as well so if I do
[23:39] a pretty Phi on this tag and save that
[23:44] and run it now we can see that this tag
[23:46] is well structured as well so now we can
[23:48] see that we got all the HTML for that
[23:50] first article so we can see that we have
[23:53] the link here that contains the title
[23:56] for that so this is a video about Python
[23:59] regular expressions and then within if
[24:02] we go down here a little bit more then
[24:05] we have the text summary for that and we
[24:08] also have the embedded YouTube video so
[24:10] we have all the information for that
[24:12] first article where we can begin parsing
[24:14] out the headline and summary and video
[24:17] so first let's grab the headline so if
[24:20] we look in the HTML we have our h2 and
[24:23] within that h2 we have a link and the
[24:27] text of that link contains the headline
[24:29] so for now let's just comment out where
[24:33] we're printing out the HTML for that
[24:35] article and now let's just say headline
[24:38] is equal to and we want to do use the
[24:41] article HTML here and not the entire
[24:43] soup so let's say article dot H to dot a
[24:47] to grab that anchor tag and then text to
[24:50] grab the text out of that anchor tag so
[24:52] now let's print out that headline so if
[24:55] I save that and run it then we can see
[24:57] that we did get the title of that latest
[24:59] post which is that tutorial on regular
[25:02] expressions now I think that this
[25:04] headline link here is actually the first
[25:06] link within our article so I don't think
[25:08] we actually needed this H to parent tag
[25:10] here so if we just an article dot a dot
[25:13] text then I believe that we would have
[25:15] gotten the same result but it doesn't
[25:17] hurt to be a little overly specific here
[25:20] but you just don't want to get carried
[25:22] away and put in every single parent tag
[25:24] because then that's going to stretch
[25:25] your line out far longer than it needs
[25:28] to be and just look more confusing than
[25:30] it needs to be so it's okay to be a
[25:31] little overly specific but just don't
[25:33] get carried away okay so now that we've
[25:35] got the headline of this latest post now
[25:38] let's get the summary text for this post
[25:40] so I'm going to comment out where we got
[25:42] the headline and uncomment out our
[25:44] prettified article HTML and reprint this
[25:48] back out so that we can look and see
[25:51] where this summary text is so our
[25:54] summary text is within a paragraph tag
[25:56] and that paragraph tag is within a div
[25:59] with a class of entry content so to grab
[26:03] that let's comment out our article dot
[26:06] prettify again and below our headline
[26:08] let's just say summary is equal to
[26:11] article dot and we're going to use that
[26:14] fine method because we're going to be
[26:16] searching for a div with a specific
[26:18] class so we want to find a div and to
[26:22] search for a specific class we can just
[26:24] pass that in as an argument so we can
[26:27] say class and that's going to be class
[26:28] underscore is equal to entry - content
[26:33] so all of this here is going to return
[26:36] the tag for this div here so it's going
[26:39] to return all of this information so
[26:42] within this
[26:43] div we want to parse out the first
[26:47] paragraph so we can just do dot P and
[26:49] now within that paragraph we want the
[26:52] text of that paragraph so we can just
[26:54] string all this together so dot P dot
[26:57] text so now if we print that out and
[27:00] save that and run it then we can see
[27:03] that we correctly parsed out the summary
[27:05] text for that post okay so lastly we
[27:08] need to get the link to the video for
[27:10] this post now this one is going to be a
[27:12] little more difficult but I want to show
[27:14] you this because sometimes parsing
[27:16] information can be a little ugly and
[27:18] required you to take several steps
[27:20] before getting to your final desired
[27:21] result so on this website these videos
[27:24] are embedded so if we comment out our
[27:26] summary here and then uncomment out our
[27:30] article dot prettify HTML if we run this
[27:34] and then find our video that is embedded
[27:38] it should be in an iframe which is right
[27:42] here so the source attribute of this
[27:46] iframe is to the embedded version of the
[27:50] video it's not the direct link to the
[27:52] video itself but if you know how YouTube
[27:55] videos work they all have a video ID and
[27:58] the ID for this video is actually right
[28:01] here I just highlighted it now the
[28:04] question mark in the URL it specifies
[28:06] where the query parameters start so it's
[28:09] not part of that video ID so with that
[28:12] ID we could actually create the link to
[28:15] the video ourselves so we need to parse
[28:18] that ID from that URL so first we need
[28:21] to grab the URL from the iframe so just
[28:24] like before let's comment out our
[28:26] article HTML go down below our summary
[28:30] and let's go ahead and just grab this
[28:33] and we'll say video source is equal to
[28:36] article dot find because we want to find
[28:40] a an iframe with a specific class we can
[28:43] see that this iframe has a class of
[28:45] YouTube player so I'm just going to copy
[28:48] this so we will find an iframe with a
[28:53] class and remember that underscore
[28:56] class equal to YouTube player so now
[28:59] let's just print out what we have so far
[29:01] so I'm going to get rid of those spaces
[29:03] so let's print out and this should be
[29:05] the HTML for that iframe so let's run
[29:09] this we can see that we have the HTML
[29:12] for that iframe now unlike what we've
[29:14] been doing before we don't want to grab
[29:16] the text from this tag what we really
[29:18] want is the value of that source
[29:20] attribute from the tag now if you want
[29:23] to get that value from an attribute of a
[29:25] tag then you can access it like a
[29:28] dictionary so at the end here after we
[29:31] grab that iframe we can just access this
[29:34] like a dictionary and say that we want
[29:36] the source attribute of that tag so now
[29:40] if I save that and run it now we can see
[29:43] that we got the link to that embedded
[29:44] video so now we're going to have to
[29:47] parse this URL string to grab the ID of
[29:49] that video and we'll break this up into
[29:51] several lines so first we can see that
[29:54] the ID comes after a forward slash here
[29:58] so let's split up this string based on
[30:01] forward slashes so if I go down another
[30:04] line here I can say vid ID is equal to
[30:08] our vid source dot split and we want to
[30:13] split on a forged slash and now let's
[30:15] let me take this down a little bit here
[30:18] and now let's print this out so you can
[30:20] see what this does and let me actually
[30:23] comment out the vid source there save
[30:26] that and rerun it now if you've never
[30:27] used the split method on a string then
[30:29] basically it just splits the string into
[30:32] a list of values based on the character
[30:34] that you specify so we can see that now
[30:37] our URL is broken to a broken into a
[30:40] list of several parts based on where
[30:42] those forward slashes were so if we look
[30:44] at the items in our list then our video
[30:48] ID is right here because it was right
[30:50] after a forged slash so that is an index
[30:53] so this is index 0 1 2 3 4 so this is in
[30:59] index 4 so let's specify that we want
[31:02] the fourth index of that returned list
[31:05] so after that split method we can just
[31:08] say that we want index 4
[31:10] so now if we run this then we can see
[31:12] that we're getting closer so we have the
[31:14] video ID here and then we have these
[31:16] query parameters here at the end so like
[31:19] I said before the question mark
[31:20] specifies where the parameters for the
[31:23] URL begin and the video ID is before
[31:26] that so if we do another split on the
[31:29] question mark then it should separate
[31:31] those out so I'll go to a new line so
[31:34] that we're not making this one too long
[31:36] or too complicated and we can just say
[31:38] vid ID is equal to vid ID and now we
[31:43] want to split that based on the question
[31:47] mark so now if we save that and run it
[31:50] then now that got split up and our video
[31:52] ID is the first item of that list and
[31:55] the query parameters are the second item
[31:57] of that list so to grab the video ID we
[32:00] can just get the 0 index of that
[32:02] returned list so right after that I'll
[32:05] just say that I want the 0 index so now
[32:07] if I save that and run it then we can
[32:09] see that there we got the video ID now I
[32:12] know that that was a lot of parsing but
[32:13] sometimes website source code doesn't
[32:16] have the information that you want in
[32:18] the most accessible way so I wanted to
[32:20] show you how you might go about getting
[32:22] the data that you want even if it's a
[32:24] little bit messy ok so now we can create
[32:26] our own YouTube link using this video ID
[32:29] so the way YouTube links are formatted
[32:33] are like this so I'll comment out the
[32:35] video ID for now and scroll down here a
[32:38] little bit we can just call this
[32:39] variable youtube link and we will set
[32:42] this equal to we'll just do a formatted
[32:44] string here this will be HTTPS then
[32:48] youtube.com then Forge slash and the
[32:52] watch route and then the query parameter
[32:54] here is going to be a question mark with
[32:56] V which stands for video V equal to and
[32:59] we want to set that V equal to that
[33:01] video ID so I will just put in a
[33:04] placeholder there with that video ID so
[33:06] if we print out this YouTube link that
[33:10] we just created if I save that and run
[33:12] it then you can see that now we have
[33:15] this YouTube link now I used F strings
[33:18] to format that string but those are only
[33:20] available in Python 3.6 and above
[33:23] if you're using an
[33:24] older version of Python then you can use
[33:27] the format method on that string to
[33:29] insert that placeholder and I have a
[33:31] separate video on how to format strings
[33:33] if anyone needs to see how to do that
[33:35] and I'll leave a link to that video in
[33:36] the description section below but now
[33:38] that we've run this and got this link
[33:40] that we created so now if I copy this
[33:43] and paste this into my browser over here
[33:46] then we can see that that does go to
[33:49] that video goes directly to that video
[33:50] that we specified ok so perfect so we've
[33:53] scraped all the information that we
[33:55] wanted from that first article so just
[33:58] like in our earlier example with the
[34:00] simple HTML now that we've got the
[34:02] information for one article now we can
[34:04] loop over all the articles and get that
[34:06] information for all of them so to do
[34:08] that we can just uncomment out the code
[34:12] that we grabbed here for the summary so
[34:15] I'll uncomment out that I'll uncomment
[34:18] out the code for the headline and I can
[34:21] remove our comment into out print
[34:23] statements here just to clean things up
[34:25] a bit
[34:25] let me remove our prettify article print
[34:29] statement there okay so just like we did
[34:31] before instead of just finding the first
[34:34] article now we want to find all of the
[34:36] articles so now we can just use the
[34:38] final method instead and remember this
[34:41] returns a list of all of those articles
[34:43] so instead of just setting that equal to
[34:46] one variable called article we can do
[34:48] put in a for loop so we can say for
[34:50] article in that list then be sure we put
[34:54] in that colon there and now we have to
[34:56] put all of this information within our
[34:58] for loop so we will index or indent that
[35:01] over and save that and just like I did
[35:05] in our earlier example right here at the
[35:07] bottom I'm also going to put a blank
[35:09] print statement just so that it
[35:11] separates out the information from all
[35:13] of our articles so now if I run this
[35:16] then let me pull our output up here a
[35:19] little bit and scroll up to the top so
[35:21] we can see that we got the headline for
[35:24] our first article and the text summary
[35:26] for our first article and the link to
[35:28] that YouTube video and we did this for
[35:31] all the articles on the web page okay
[35:33] perfect okay so now we can see that that
[35:35] works getting all the information from
[35:37] the latest Artic
[35:38] on the homepage of the website now we're
[35:41] almost finished up but let me show you a
[35:43] couple more things so sometimes you'll
[35:45] run into situations where you're missing
[35:47] some data and if that happens then it
[35:49] could break our scraper
[35:51] now maybe you're pulling down a list of
[35:53] items and one is missing an image or
[35:55] something like that that you thought
[35:57] would be there so to show what this
[35:59] looks like I'm going to edit one of my
[36:01] posts here and remove the link to one of
[36:05] the YouTube videos so instead of having
[36:07] you watch me log in to my webpage to do
[36:10] this I'm just going to fast forward this
[36:11] video a bit and skip to the point where
[36:14] I've edited this post okay so I logged
[36:16] in and edited my page so that there is
[36:19] no longer a video link for the post a
[36:22] couple of numbers down here so you can
[36:23] see that this post here does not have a
[36:26] video associated with the post so now if
[36:29] I go back to our code that was just
[36:31] working before and I try to rerun this
[36:34] then we can see that it gets the first
[36:37] post just fine it gets the title and the
[36:40] summary text and the YouTube video link
[36:42] but for the second post here it gets the
[36:45] title and it gets the summary text but
[36:47] when it gets to the youtube link it
[36:49] breaks our script and it says that none
[36:52] type object is not subscript Abul and
[36:54] some weird errors there basically it's
[36:56] breaking on this line here where it's
[36:58] trying to find that iframe with the
[37:01] YouTube player class so if you run into
[37:03] something like this and you just want to
[37:05] skip by any missing information then
[37:07] what we can do is put that part of the
[37:09] code into a try except block so I'm
[37:12] going to pull down our output a little
[37:14] bit here now here at the bottom I'm just
[37:16] going to create a try except block and
[37:20] within sublime text this has
[37:21] autocomplete so I just click there for
[37:24] the try except and this gave me a little
[37:26] template here so within the try we want
[37:29] to take all of the code that gets that
[37:31] video information and we want to put
[37:34] that within our try block so I would
[37:38] just paste all that in and indent it
[37:40] correctly there and I meant to cut that
[37:44] out so I need to delete all of that and
[37:48] let's get this print here and
[37:51] that out and we will put that below the
[37:55] try/except block okay so the way that we
[37:57] have this set up right now this youtube
[38:00] link variable will only get set if this
[38:02] succeeds here now in our exception if
[38:06] this fails then it's going to go to our
[38:08] exception block here now sometimes
[38:10] people will just put in pass if they
[38:12] just want to skip over this but in our
[38:15] case we still want this youtube link
[38:18] variable to be set so instead of just
[38:20] passing here let's set this youtube link
[38:23] variable equal to none just to say that
[38:26] we couldn't get that youtube link okay
[38:29] so now with that code within a try
[38:31] except block let me make our output a
[38:33] little bit larger here so now if we save
[38:36] that and run it then we should get all
[38:39] of the information on our page so our
[38:41] top post here still works fine we got
[38:44] the title
[38:44] we got the summary text and we got the
[38:46] youtube link and for our second post
[38:49] which has the missing video we still
[38:52] have the title and we have the summary
[38:54] text and then the video is just set to
[38:56] none that variable set to none and then
[38:58] it just continues on with the other post
[39:01] after that so that's what we wanted the
[39:03] video was missing but it didn't break
[39:05] our program it still went and got the
[39:07] information for all the other posts on
[39:09] the page okay so now we're done
[39:11] scraping the information so now I'm just
[39:13] going to up the sublime text here so
[39:17] that we can see everything a little bit
[39:19] larger here and scroll up here to the
[39:21] top so now that we've scraped the
[39:24] information that we want from our web
[39:25] page now we can save this in any way
[39:28] that we'd like so right now we're just
[39:30] printing this information out to the
[39:32] screen and maybe that's fine for your
[39:34] needs but you can also you know save it
[39:36] to a file or say it save it to a CSV or
[39:39] anything that you'd like so for example
[39:42] real quick let's say that we wanted to
[39:43] scrape this page and save that
[39:45] information to a CSV file so we've
[39:47] already done the hard part of getting
[39:49] the information that we want from the
[39:51] web page now to save it to a CSV file we
[39:54] could simply import the CSV module so
[39:57] we'll import CSV then here at the top
[40:01] right before our for loop we can
[40:05] open a CSV file so we'll just create a
[40:07] variable here called CSV file we'll set
[40:10] this equal to open and we want to call
[40:13] this CMS scrape dot CSV you can call
[40:17] that whatever you'd like and we want to
[40:19] write to this file so we'll pass in a
[40:21] w-4 that now this video isn't about
[40:23] working with files or CSV s I do have a
[40:26] separate video going into detail about
[40:28] how to work with CSVs but for this video
[40:30] we'll just walk through really quickly
[40:32] so I'm not going to go into much detail
[40:34] here but we could use a context manager
[40:37] here but the way that we currently have
[40:39] our script setup I think it'll just be a
[40:41] little quicker to just set this variable
[40:43] and open the file like this so now we
[40:45] can write some lines to set up our CSV
[40:48] file and again I'm not going to go into
[40:50] a lot of detail here I have a separate
[40:52] video on this if you're interested so we
[40:54] can say a CSV writer is equal to CSV dot
[40:59] right so the right or a writer method of
[41:02] that CSV module and we want to pass in
[41:05] that CSV file that we just opened and
[41:08] now we want to write the headers of this
[41:12] CSV file so we can say CSV writer that
[41:17] we just created and we can do a dot
[41:20] write row and we can pass in a list of
[41:25] values that we want to write to this row
[41:26] so we can create a list and we just are
[41:29] passing in the headers for now
[41:31] so our headers are going to be headline
[41:33] and summary and we need to pass that in
[41:37] as text and also video link so those are
[41:42] the headers to our CSV file which are
[41:45] basically the column names that's the
[41:46] data that we're going to be saving to
[41:48] this CSV and now within our for loop
[41:50] where we're getting that scraped
[41:51] information we can just write that
[41:54] information to our CSV file so at the
[41:57] very bottom of our loop after we print
[42:00] that blank line we can just write that
[42:03] data to our CSV with each iteration
[42:05] through our for loop so we can say CSV
[42:08] writer dot write row and we're going to
[42:12] pass in a list here and the values that
[42:15] we want to pass in are going to be
[42:17] our headline first and then our summary
[42:23] second and then our YouTube blink third
[42:27] and lastly at the very end of our script
[42:30] outside of the for-loop since we didn't
[42:32] use a context manager to open that file
[42:34] before we need to close our file here at
[42:37] the end of the script so we can say CSV
[42:39] file not CSV writer this is the actual
[42:41] CSV file I can say CSV file dot close so
[42:45] now if I run this code then you can see
[42:48] that it prints out all the information
[42:49] like it did before but now if I open up
[42:52] my sidebar here we can see that now we
[42:55] have this CMS dot CSV file here in the
[43:00] side so I'm gonna open this within
[43:02] finder which is just within the file
[43:04] system and now I'm going to open this
[43:07] with any kind of spreadsheet application
[43:10] now mine is numbers but yours might be
[43:12] Excel so now we can see that we have all
[43:14] this data available within our
[43:16] spreadsheet so let me maximize this here
[43:18] and make this to where it's a little bit
[43:21] more readable so I'll make the columns a
[43:24] little bit smaller there and then wrap
[43:26] the text in all of our cells so we can
[43:28] see that we have all this information so
[43:30] here are our headers here headline
[43:33] summary and video link here are all of
[43:36] our headlines parsed out for us and our
[43:38] summaries and then you can see here in
[43:40] the video links with that second post
[43:43] where the video was missing this got
[43:46] posted in as blank there so there's a
[43:48] none value there okay so now I can exit
[43:51] out of that and pull back up our script
[43:56] here okay so I think that is going to do
[43:58] it for this video hopefully now you have
[44:00] a pretty good idea for how you can go
[44:02] out and scrape information from websites
[44:04] now one thing that I do want to mention
[44:06] is if you want data from a large website
[44:09] like Twitter or Facebook or YouTube or
[44:11] something like that then it may be
[44:13] beneficial for you to see whether or not
[44:15] they have a public API public API is
[44:17] allow those sites to serve up data to
[44:20] you in a more efficient way and
[44:21] sometimes they don't appreciate if you
[44:24] try to you know scrape their data
[44:26] manually they'd rather you go through
[44:28] the public API but it's usually those
[44:29] larger websites that have those public
[44:31] P is so if you want data from you know a
[44:35] small or medium size website then likely
[44:37] you'll have to go through and do
[44:38] something like we did here now also I
[44:40] should point out that you should be
[44:42] considerate when scraping websites so
[44:44] computer programs allow us to send a lot
[44:46] of requests very quickly so be aware
[44:48] that you might be bogging down someone's
[44:50] server if you aren't careful so try to
[44:53] keep that in mind so you know after this
[44:55] tutorial try not to go out and you know
[44:57] hammer my website with you know tons of
[45:00] requests through your program and that
[45:02] goes for other websites too some
[45:03] websites will even you know monitor if
[45:05] they're getting hit quickly and they may
[45:07] even block your program if you're
[45:09] hitting them too fast but other than
[45:11] that if anyone has any questions about
[45:12] what we covered in this video then feel
[45:14] free to ask in the comment section below
[45:15] and I'll do my best to answer those and
[45:17] if you enjoy these tutorials and would
[45:19] like to support them then there are
[45:20] several ways you can do that the easiest
[45:21] way is to simply like the video and give
[45:23] it a thumbs up also it's a huge help to
[45:25] share these videos with anyone who you
[45:26] think would find them useful and if you
[45:28] have the means you can contribute
[45:29] through patreon and there's a link to
[45:30] that page in the description section
[45:31] below be sure to subscribe for future
[45:33] videos and thank you all for watching
[45:46] you