Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

Python Scraping Data From the Web A World Full of Spiders Crawling Spiders

Posted by Jason Tran
Jason Tran
7,393 Points

Spider not found: ike

Hi, I'm having a little trouble with this. For some reason when I run the scrapy project via the terminal the result is the following KeyError.

KeyError: 'Spider not found: ike'

However, the name property for my class is set to 'ike'.

import scrapy

class HorseSpider(scrapy.Spider):
    name = 'ike'

    def start_requests(self):
        urls = ['https://treehouse-projects.github.io/horse-land/index.html',
                'https://treehouse-projects.github.io/horse-land/mustang.html']
        return [scrapy.Request(url=url, callback=self.parse) for url in urls]

    def parse(self, response):
        url = response.url
        page = url.split('/')[-1]
        filename = 'horses-%s' % page
        print('URL is: {}'.format(url))
        with open(filename, 'wb') as file:
            file.write(response.body)
        print('Saved file %s' % filename)

I'm spider project ("horses") is saved as a jupyter notebook. I'm not sure if that matters. I would be greatly appreciated if someone could help me with this problem.

Thanks!

3 Answers

Josh Keenan
Josh Keenan
20,315 Points
Josh Keenan
Josh Keenan
20,315 Points

Is it possible you had the ' on the end of the command when you tried to run it?

Jason Tran
Jason Tran
7,393 Points

Hi Josh, Thanks for replying. An apostrophe (') was not added on the end of the command. This is what I had written in the terminal:

scrapy crawl ike

Josh Keenan
Josh Keenan
20,315 Points
Josh Keenan
Josh Keenan
20,315 Points

The only difference I can see between yours and mine is this:

    def parse(self, response):
        url = response.url
        page = url.split('/')[-1]
        filename = "horses-{}s".format(page)          # this line using .format() instead of % formatting
        print("URL is: {}".format(url))
        with open(filename, 'wb') as file:
            file.write(response.body)
        print("Saved file {}".format(filename))
Josh Keenan
Josh Keenan
20,315 Points
Josh Keenan
Josh Keenan
20,315 Points

And your code is running fine for me as well too sorry should have lead with that

Richard Button
Richard Button
12,968 Points
Richard Button
Richard Button
12,968 Points

I accidentally created the file in the root folder. It should be in the spiders folder. Once I moved it there it resolved this KeyError for me.