Getting 999 when hitting public profile pages in linkedIn.

[ladlakaran2607]

When I hit public profiles I get 999. I guess after 4 to 5 continous hits, linkedIn blocks the ip address. So changed the ip, but still 999 exists. Is it possible that they are able to get local machine ip address?
How to solve this error without login?

I'm stuck with this. Please help.

[adolfopeccin]

Same here

[rochenka]

@Karan-Daiict @adolfopeccin same for me until I found an API that avoids the 999, i make GET requests to that API which calls Linkedin and returns the data, the API is from proxycrawl.

[ladlakaran2607]

Hi rochenka, You won't be able to access all the data from LinkedIn APIs. I solved it using headless browsers. Works like a charm. Thanks, Karan Ladla

…

On Wed 10 Oct, 2018, 11:46 AM rochenka, ***@***.***> wrote: @Karan-Daiict <https://github.com/Karan-Daiict> @adolfopeccin <https://github.com/adolfopeccin> same for me until I found an API that avoids the 999, i make GET requests to that API which calls Linkedin and returns the data, the API is from proxycrawl. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANTVYc92DRhKjDC7MttcGZh2rga8MIq9ks5ujZDEgaJpZM4T06HQ> .

[rochenka]

@Karan-Daiict I do not use Linkedin API, i use ProxyCrawl API which calls linkedin internally. https://proxycrawl.com/scraping-api-avoid-captchas-blocks

[ladlakaran2607]

Ohh okay, that is good. Scraping with proxy built in. But you'll have to pay for it.

…

On Wed 10 Oct, 2018, 11:59 AM rochenka, ***@***.***> wrote: @Karan-Daiict <https://github.com/Karan-Daiict> I do not use Linkedin API, i use ProxyCrawl API which calls linkedin internally. https://proxycrawl.com/scraping-api-avoid-captchas-blocks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANTVYbsazsjlI5ficQdrR72kNov-oLFpks5ujZPWgaJpZM4T06HQ> .

[rochenka]

@Karan-Daiict yes I do not want to manage headless browser infrastructure or whatever tricks comes along the way from Linkedin, especially that i am building a project for recruiting so I need data constantly, that is why I pay for it, the other option is to find proxies that work for Linkedin which is another difficult task to do and also I have to pay for it, so I went for an easy good solution. Do you use proxies or how do you get the data?

[ladlakaran2607]

Yeah. Basically I created some proxies and ran them in round robin fashion. Does the API give complete data and is the signature easy to use for proxy crawl? Also I think LinkedIn will block based on creds. Do check that.

…

On Wed 10 Oct, 2018, 12:29 PM rochenka, ***@***.***> wrote: @Karan-Daiict <https://github.com/Karan-Daiict> yes I do not want to manage headless browser infrastructure or whatever tricks comes along the way from Linkedin, especially that i am building a project for recruiting so I need data constantly, that is why I pay for it, the other option is to find proxies that work for Linkedin which is another difficult task to do and also I have to pay for it, so I went for an easy good solution. Do you use proxies or how do you get the data? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANTVYXqCr2oO49uXNZQ63bviCU4zbG5Xks5ujZrrgaJpZM4T06HQ> .

[rochenka]

I do not really know what they do internally, i just receive the html data that is public, i do not think they support private data crawling from linkedin though.

[ladlakaran2607]

Yeah, nobody supports private crawling. For public data and static data, you can use http client too. But fine, proxy crawl is good.👍

…

On Wed 10 Oct, 2018, 12:45 PM rochenka, ***@***.***> wrote: I do not really know what they do internally, i just receive the html data that is public, i do not think they support private data crawling from linkedin though. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANTVYT-KX1h17OLFvdXF9QLpmzY1f_zhks5ujZ6cgaJpZM4T06HQ> .

[adolfopeccin]

@rochenka how did you implemented the scraping for linkedin data using proxycrawl?

[rochenka]

@adolfopeccin Its basically simple, proxycrawl does the crawling with their API, you need to have a token. I think you can get one token for 1000 free requests. So, proxycrawl gives me the raw html data response and I then use linkedin-scraper to scrape the html content.

[abhishmitra]

@rochenka

Hi,
I spoke to proxycrawl but they have certain limits and stuff. Did they impose those recently? How are they working for you?

[Bilal815]

@rochenka

Hi, I spoke to proxycrawl but they have certain limits and stuff. Did they impose those recently? How are they working for you?

I recently tried Crawlbase which was ProxyCrawl previously. They say that with JS-enabled for dynamic content which is a need for LinkedIn, you need to pay them a little. I tried my luck and it got interesting here https://crawlbase.com/docs/crawling-api/scrapers/#linkedin.