NEW Search Engine Stuff: CBP & USAO Press Releases and Fast Access to CBP Tweeted Images

 Every day three (3) tasks require overnight attention

  • Update the Search Engine results (DAILY: DL 15m/Process 1+hr)
  • Scrape 70+ CBP Twitter accounts for text, pictures and videos (DAILY: DL 1+hr/Process 2+hr)
  • Download CBP Weekday Press releases (WEEKDAYS: DL 15m/Process 1-3hr)

This does not include maintenance of in-house software used, random tweets added, and 8-12 hours of videos and podcast playing in the background.

Four (4) NEW THINGS

  1. Soon we will restart processing USAO (United States Attorneys' Offices) press releases. Details as the process restarts.
  2. Soon our search engine website will have a "dedicated" webpage for links to CBP press releases. Details as the process rolls into place.
  3. In a few days, we will describe - in detail - how our search engine works and how you can help.
  4. On the near horizon is access to CBP Tweeted Images. Details below

TL;DR (Too Long; Didn't Read)

Daily, #BorderObserver will scrap 70+ CBP twitter accounts for the text, images and videos, and make those easily accessible from webpages on our search engine website. Those webpages will include categories, hashtags & timeline threads.

A LITTLE DETAIL

For sometime now, when CBP does a press release the images are not always great. So what we have had to do is search for a CBP tweet that very likely has a better quality image. This could take from 2 to 20 minutes to find an image. When this fails, then the fallback plan is to backfill with a stock image and/or a map.

Just after midnight EST, 70+ CBP related accounts will be scrapped for the tweets of the previous day; including any pictures and/or videos. Then those tweets will be condensed on to a single webpage on our search engine website. You will then be able to see that previous day's tweets on one webpage. In addition, if you click on a picture or video you will be taken to our Flicker repository where you can download the picture or video.

Tweets and images of interest will be promoted on the search engine results page. A separate webpage will have the daily scraping of CBP tweets.

MORE DETAILS

There are some side effects to this process.

  1. If CBP or Texas DPS tweet about an effective stop, it is possible that that effective stop can be timeline-traced through to a USAO or local DA press release - which might take 6 months to 3+ years (to sentencing, if found guilty).
  2. CBP and CBP-related tweeted images, pictures and videos will be part of an ever growing organized repository of STOCK IMAGES.
  3. If a CBP account tweets about a press release, that tweet will be directly pointed to on the webpage.
  4. Sometime in the near future, tweeted image related to a press release will be made directly available on that webpage.

A QUICK WORD ON THE SEARCH ENGINE

A few months back when launching the press-release-database, it became painfully obvious that we had missed the mark. An obvious take away was that a majority of people are STILL USING the current "search engines" (Google, Bing, DuckDuckGo).

However, the real stories are often at the front lines. In Texas & New Mexico, it is Brownsville, Laredo, Eagle Pass, and El Paso. In Arizona, it is Douglas, Nogales, and Yuma. In California, it is Calexico, Tecate, and San Diego.

Nationally, the "search engine" uses - to name a few:

breitbart.com
dailysignal.com
foxnews.com
freebeacon.com
newsmax.com
newsnationnow.com
nypost.com
thegatewaypundit.com
thepostmillennial.com
washingtontimes.com

AT the bottom of the results page for the search engine is the complete list of websites we check. In addition it contains:

  • Keywords List - Words we search for in the Title
  • Fetch List - A complete list of news outlets we check - in alphabetic order.
  • Display List - A list of news outlets we check - in display order. The order we use on the front page.

SEARCH ENGINE RESULTS PAGE

https://dailybordernews.github.io/


EXAMPLE DAILY TWITTER PAGE SCRAPE

 

Comments

Post a Comment

Popular posts from this blog

Kamala Harris was NEVER assigned to be the "Border Czar" - Video and written evidence support this

Image
UPDATE (Oct 14, 2024) We realized after this blog that there may be a video *somewhere* of Joe Biden actually naming the VP as the "Border Czar," and the we video used as evidence here may be him "correcting himself." If you happen to know of a copy of such a video where Joe actually says or uses the words "Border Czar," please let us know. Like many of you reading this blog, I thought I heard Kamala be assigned to be the "Border Czar." I even recall watching the video where Biden said something to that nature. But I was wrong. To be clear, I wanted to hear Kamala and Joe say those words or something like that, but in their usual political mumbo jumbo, COVID masks, and Joe Biden's mumbling, their message was lots with me -- at the time. Again to be very clear, I wanted to find Kamala Harris being named "Border Czar." The truth is we found no evidence of her being named "Border Czar," but there is evidence to what they ...
Read more

Massive Changes in Tweets from @BorderObserver

Image
TL;DR Less tweets from the Search Engine on a daily basis, more tweets from press releases. Some Context Some of you  that have been around awhile know @borderObserver is an effort to make a workstation for news reporters . The first effort was to make a system whereby the basis for most news reporting was catalogued and reported on. To be clear, the "first effort" was to report on the latest press releases. Why? because that is where 90+% of all news starts from. The days of on-the-ground of gumshoe reporting are mostly gone. By all logically means, having a groud news reporter show up for a "local news briefing" — even for a city council meeting — is too expensive. As such, now-a-days reporters show up and take the printed press release. And in many such cases, they just gather the online press release via newsletters. I want to make this very clear. No actual person shows up for a "local news briefing." Unless, the "briefing" is extra-ordinar...
Read more

What Kash Patel's Confirmation as Head of the FBI Means for Enforcement of Illegal Border Crossers and Illicit Drugs

Image
Kash Patel is likely to be Confirmed this week . What you may not know is how it affects HSI  (Homeland Security Investigations) - the investigative branch of CBP (U.S. Customs and Border Protection) and USBP (United States Border Patrol) First, HSI is a law enforcement agency within the Department of Homeland Security (DHS) and as such it frequently completes criminal investigations for the CBP and USBP. One of reason it existing is because at one point in time greater than 50% of cases referred from CBP and USBP to the FBI did not get resolved _or_ took too long. More recently, a joint survey and report (dated: July 31, 2019) from the OIG (Office of the Inspector General) of both the DOJ and the DHS finds While 63 percent of survey respondents did not report any cooperation failures, 37 percent of survey respondents and many interviewees reported cooperation failures that resulted in negative impacts on investigations and operations. Let's be clear, 37% of the time the FBI was ...
Read more