Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

General Discussion

Carl Blatcher
Carl Blatcher
854 Points
Posted by Carl Blatcher
Carl Blatcher
854 Points

Project in mind but not sure where to start.

I want to create some software that if two companies use the same product code for an item, then the company can be alerted when the competition changes the price of the item. Basically input all product codes and then when there is a change in price on a competitors site you will be alerted. Is this possible and if so what programming language do I need to do learn in order to program this?

Thanks in advance

1 Answer

Brendan Whiting
seal-mask
.a{fill-rule:evenodd;}techdegree seal-36
Brendan Whiting
Front End Web Development Techdegree Graduate 84,736 Points
Brendan Whiting
.a{fill-rule:evenodd;}techdegree seal-36
Brendan Whiting
Front End Web Development Techdegree Graduate 84,736 Points

You probably want to use a web scraper. Here's a course on it in Python but can do this in most popular languages. So your web scraper is going to pull certain values out of a website. You'll need a way to run your script periodically to see if the value has changed. You could use a cron job, and you can set one up with AWS Lambda. Be prepared that the website you're scraping might change its layout which might break your web scraper, or they might not like having their website scraped and find a way to block you.

Then you'll have to figure out what to do with that info. Display it in a web interface for your user? Send emails informing them of what prices have changed?

Carl Blatcher
Carl Blatcher
854 Points
Carl Blatcher
Carl Blatcher
854 Points

Brendan Whiting I'd probably like to display it in a web interface and then send an email saying 'Please look at the web interface, prices have changed' or something to that effect.

So, if I did create one of these the company would be able to see I was scraping their website and be able to tell what location the scraping was coming from?

Carl Blatcher
Carl Blatcher
854 Points
Carl Blatcher
Carl Blatcher
854 Points

OK thanks. I will read that soon. So can every site tell they're being scraped or do they have to have certain code in order to pick up on scraping? eg. could it go unnoticed that their site was being scraped?

Brendan Whiting
seal-mask
.a{fill-rule:evenodd;}techdegree seal-36
Brendan Whiting
Front End Web Development Techdegree Graduate 84,736 Points
Brendan Whiting
.a{fill-rule:evenodd;}techdegree seal-36
Brendan Whiting
Front End Web Development Techdegree Graduate 84,736 Points

I think it's possible to make a request that looks close enough to a human request that the website won't be able to tell that it's a bot. But there are ways that they can detect bot behavior and may block you. It's kind of a cat and mouse game.

Carl Blatcher
Carl Blatcher
854 Points
Carl Blatcher
Carl Blatcher
854 Points

OK. Thanks for your help, I will look into this and comment back if I have any more questions. :)