Feed aggregator

Sonic Hedgehog Protein

Hacker News - Wed, 04/02/2025 - 1:22pm
Categories: Hacker News

Strategies to download data constantly changing via API

Hacker News - Wed, 04/02/2025 - 1:21pm

I have to download a dataset through one API (WFS provided by geoserver) that tells me the total amount of items and delivers at maximum 1000 items per request and I can sort by one field and offset the requests start index. The layer has ~1Million items. I can use at maximum 5 parallel request before API gets overloaded.

Problem is that items are being added and removed in real time, so at the end of the copy process I already have stale data copied and there are new items to be copied over. So what would you do, or have done in this situation? Start a never ending loop to crawl data all day long would be something evil or is it something to be fixed on provider side?

The api url is https://geoserver.car.gov.br/geoserver/sicar/wfs

Source data website: https://consultapublica.car.gov.br/publico/imoveis/index

Comments URL: https://news.ycombinator.com/item?id=43559034

Points: 2

# Comments: 0

Categories: Hacker News

Best Tennis Balls to Up Your Game in 2025

CNET Feed - Wed, 04/02/2025 - 1:19pm
Choosing the right tennis ball can make all the difference, whether you’re a beginner or a Grand Slam champ. We rounded up the best options from top brands.
Categories: CNET

SN 1019: EU OS - Troy Hunt Phished, Ransomware List, InControl

Security Now - Tue, 04/01/2025 - 11:15pm
  • Kuala Lumpur International Airport says no to a ransom attack, switches to whiteboard.
  • A tired and jet-lagged Troy Hunt got Phished then listed himself on his own site.
  • Cloudflare completely pulls the plug on port 80 (HTTP) API access.
  • Malware is switching to obscure languages to avoid detection. FORTH, anyone?
  • Password reuse doesn't appear to be dropping. Cloudflare has numbers.
  • A listener shares his log of malicious Microsoft login attempts. Why no geofencing?
  • 23andMe down for the count (reminder).
  • A sobering Ransomware attack & victim listing website. Gulp!
  • "InControl" keeps VR planes aloft.
  • And the European Union gets serious about a switch to Linux

Show Notes - https://www.grc.com/sn/SN-1019-Notes.pdf

Hosts: Steve Gibson and Leo Laporte

Download or subscribe to Security Now at https://twit.tv/shows/security-now.

You can submit a question to Security Now at the GRC Feedback Page.

For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: grc.com, also the home of the best disk maintenance and recovery utility ever written Spinrite 6.

Join Club TWiT for Ad-Free Podcasts!
Support what you love and get ad-free shows, a members-only Discord, and behind-the-scenes access. Join today: https://twit.tv/clubtwit

Sponsors:

Categories: Security Now

Pages