# Lyft Web Scraping



## elelegido (Sep 24, 2014)

I need some help from a web programmer.

I'm writing some code in Excel VBA to scrape Lyft trip data from their website and put it into a spreadsheet. 

The scraper code works fine, but I have not been able to automate the login process to Lyft's domain. At the moment I am getting the session ID and CSRF tokens from a web browser and then pasting them into Excel; the code then reads them and spoofs the browser.

I'm not a web programmer, so I don't know - how do I get the session ID and CSRF tokens programatically using VBA?


----------



## Undermensch (Oct 21, 2015)

Use Fiddler4 (free on the web) to capture the messages that get sent when you login. It's probably one to two messages. You should see everything you need to reproduce the login process.


----------



## elelegido (Sep 24, 2014)

Thanks; I just used a browser to pick up the network traffic; works ok. Fiddler 4 looks good, though; better interface than Opera anyway.

It was just a case of not thinking enough about the storage of cookies - if an old unexpired session ID is still stored on the client then there is no need to request a new one.  It was just a case of clearing out the old ID and token, forcing a new connection to Lyft and then capturing the URL and headers from the network traffic to be able to replicate it with code.


----------



## beezlewaxin (Feb 10, 2015)

Post the code that you're having trouble with, and I can get it working for you.


----------



## dcselinacz (Dec 17, 2017)

elelegido said:


> Thanks; I just used a browser to pick up the network traffic; works ok. Fiddler 4 looks good, though; better interface than Opera anyway.
> 
> It was just a case of not thinking enough about the storage of cookies - if an old unexpired session ID is still stored on the client then there is no need to request a new one.  It was just a case of clearing out the old ID and token, forcing a new connection to Lyft and then capturing the URL and headers from the network traffic to be able to replicate it with code.


I saw a couple question it, have you considered sharing your spreadsheet? I like the ease of it. Would greatly appreciate it since I'm not knowledgeable enough to create such a file.


----------

