Tag Bash

Web Scraping Basics:
Understanding the World of Scrapers

Web scraping basics refer to the fundamental techniques and tools used to extract data from websites. This powerful process enables users to gather large amounts of data automatically from the internet, transforming unstructured content into structured formats for analysis, research, or use in various applications.

At its core, web scraping involves sending an HTTP request to a website, downloading the page, and then parsing the HTML to extract useful information. The extracted data can range from text and images to links and tables. Popular programming languages like Python, along with libraries like BeautifulSoup, Scrapy, and Selenium, are often used to build scrapers that automate this process.

The importance of web scraping basics lies in its ability to collect data from numerous sources efficiently. Businesses, data scientists, marketers, and researchers rely on scraping to gather competitive intelligence, track market trends, scrape product details, and monitor changes across websites.

However, web scraping is not without its challenges. Websites often use anti-scraping technologies like CAPTCHAs, rate-limiting, or IP blocking to prevent unauthorized scraping. To overcome these hurdles, scrapers employ techniques like rotating IPs, using proxies, and simulating human-like browsing behavior to avoid detection.

Understanding the ethical and legal implications of web scraping is equally important. Many websites have terms of service that prohibit scraping, and violating these terms can lead to legal consequences. It’s crucial to always respect website policies and use scraping responsibly.

In conclusion, web scraping basics provide the foundation for harnessing the power of automated data extraction. By mastering the techniques and tools involved, you can unlock valuable insights from vast amounts of online data, all while navigating the challenges and ethical considerations in the world of scrapers.

Web Scraping Basics:
Best Resources for Learning Web Scraping

Web scraping is a popular topic, and there are many excellent resources available for learning. Here are some of the best places where you can find comprehensive and high-quality resources on web scraping:

1. Online Courses

• Udemy:
  • “Web Scraping with Python” by Andrei Neagoie: Covers Python libraries like BeautifulSoup, Selenium, and requests.
  • “Python Web Scraping” by Jose Portilla: A complete beginner’s guide to web scraping.
• Coursera:
  • “Data Science and Python for Web Scraping”: This course provides a great mix of Python and web scraping with practical applications.
• edX:
  • Many universities, like Harvard and MIT, offer courses that include web scraping topics, especially related to data science.

2. Books

• “Web Scraping with Python” by Ryan Mitchell: This is one of the best books for beginners and intermediates, providing in-depth tutorials using popular libraries like BeautifulSoup, Scrapy, and Selenium.
• “Python for Data Analysis” by Wes McKinney: Although it’s primarily about data analysis, it includes sections on web scraping using Python.
• “Automate the Boring Stuff with Python” by Al Sweigart: A beginner-friendly book that includes a great section on web scraping.

3. Websites & Tutorials

• Real Python:
  • Offers high-quality tutorials on web scraping with Python, including articles on using BeautifulSoup, Scrapy, and Selenium.
• Scrapy Documentation: Scrapy is one of the most powerful frameworks for web scraping, and its documentation provides a step-by-step guide to getting started.
• BeautifulSoup Documentation: BeautifulSoup is one of the most widely used libraries, and its documentation has plenty of examples to follow.
• Python Requests Library: The Requests library is essential for making HTTP requests, and its documentation has clear, concise examples.

4. YouTube Channels

• Tech with Tim: Offers great beginner tutorials on Python and web scraping.
• Code Bullet: Focuses on programming projects, including some that involve web scraping.
• Sentdex: Sentdex has a great web scraping series that covers tools like BeautifulSoup and Selenium.

5. Community Forums

• Stack Overflow: There’s a large community of web scraping experts here. You can find answers to almost any problem related to web scraping.
• Reddit – r/webscraping: A community dedicated to web scraping with discussions, tips, and resources.
• GitHub: There are many open-source web scraping projects on GitHub that you can explore for reference or use.

6. Tools and Libraries

• BeautifulSoup (Python): One of the most popular libraries for HTML parsing. It’s easy to use and great for beginners.
• Scrapy (Python): A more advanced, powerful framework for large-scale web scraping. Scrapy is excellent for handling complex scraping tasks.
• Selenium (Python/JavaScript): Primarily used for automating browsers. Selenium is great for scraping dynamic websites (like those that use JavaScript heavily).
• Puppeteer (JavaScript): If you’re working in JavaScript, Puppeteer is a great choice for scraping dynamic content.

7. Web Scraping Blogs

• Scrapinghub Blog: Articles on best practices, tutorials, and new scraping techniques using Scrapy and other tools.
• Dataquest Blog: Offers tutorials and guides that include web scraping for data science projects.
• Towards Data Science: This Medium publication regularly features web scraping tutorials with Python and other languages.

8. Legal and Ethical Considerations

• It’s important to understand the ethical and legal aspects of web scraping. Resources on this topic include:
  • Scrapinghub Legal Section: Includes guides and articles on the legal implications of web scraping.
  • Legal Risks of Web Scraping (Blog): Discusses the dos and don’ts of web scraping to ensure you are compliant with laws like GDPR and site-specific Terms of Service.

9. Practice Sites

• Web Scraper.io: A web scraping tool that also offers tutorials and practice datasets.
• BeautifulSoup Practice: Hands-on exercises specifically for web scraping.
• Scrapingbee: Provides an API for scraping websites and a blog with tutorials.

With these resources, you should be able to build a solid foundation in web scraping and advance to more complex tasks as you become more experienced.

SSH-Snake, a self-modifying worm that leverages SSH credentials.

Original Article : The Hacker News

A recently open-sourced network mapping tool called SSH-Snake has been repurposed by threat actors to conduct malicious activities.

“SSH-Snake is a self-modifying worm that leverages SSH credentials discovered on a compromised system to start spreading itself throughout the network,” Sysdig researcher Miguel Hernández said.

“The worm automatically searches through known credential locations and shell history files to determine its next move.”

SSH-Snake was first released on GitHub in early January 2024, and is described by its developer as a “powerful tool” to carry out automatic network traversal using SSH private keys discovered on systems.

In doing so, it creates a comprehensive map of a network and its dependencies, helping determine the extent to which a network can be compromised using SSH and SSH private keys starting from a particular host. It also supports resolution of domains which have multiple IPv4 addresses.

“It’s completely self-replicating and self-propagating – and completely fileless,” according to the project’s description. “In many ways, SSH-Snake is actually a worm: It replicates itself and spreads itself from one system to another as far as it can.”

BotNet CNC Control Hacker Infiltrates & Exploits Vulnerabilities Vie SSH TCP Both Hardware Software Exploited

Sysdig said the shell script not only facilitates lateral movement, but also provides additional stealth and flexibility than other typical SSH worms.

The cloud security company said it observed threat actors deploying SSH-Snake in real-world attacks to harvest credentials, the IP addresses of the targets, and the bash command history following the discovery of a command-and-control (C2) server hosting the data.

How Does It Work?

These attacks involve active exploitation of known security vulnerabilities in Apache ActiveMQ and Atlassian Confluence instances in order to gain initial access and deploy SSH-Snake.
“The usage of SSH keys is a recommended practice that SSH-Snake tries to take advantage of in order to spread,” Hernández said. “It is smarter and more reliable which will allow threat actors to reach farther into a network once they gain a foothold.”

When reached for comment, Joshua Rogers, the developer of SSH-Snake, told The Hacker News that the tool offers legitimate system owners a way to identify weaknesses in their infrastructure before attackers do, urging companies to use SSH-Snake to “discover the attack paths that exist – and fix them.”

“It seems to be commonly believed that cyber terrorism ‘just happens’ all of a sudden to systems, which solely requires a reactive approach to security,” Rogers said. “Instead, in my experience, systems should be designed and maintained with comprehensive security measures.”

Netcat file transfer chat utility. Easily Send & Receive Files Local & Remote.

“If a cyber terrorist is able to run SSH-Snake on your infrastructure and access thousands of servers, focus should be put on the people that are in charge of the infrastructure, with a goal of revitalizing the infrastructure such that the compromise of a single host can’t be replicated across thousands of others.”

Rogers also called attention to the “negligent operations” by companies that design and implement insecure infrastructure, which can be easily taken over by a simple shell script.

“If systems were designed and maintained in a sane manner and system owners/companies actually cared about security, the fallout from such a script being executed would be minimized – as well as if the actions taken by SSH-Snake were manually performed by an attacker,” Rogers added.

“Instead of reading privacy policies and performing data entry, security teams of companies worried about this type of script taking over their entire infrastructure should be performing total re-architecture of their systems by trained security specialists – not those that created the architecture in the first place.”

The disclosure comes as Aqua uncovered a new botnet campaign named Lucifer that exploits misconfigurations and existing flaws in Apache Hadoop and Apache Druid to corral them into a network for mining cryptocurrency and staging distributed denial-of-service (DDoS) attacks.

The hybrid cryptojacking malware was first documented by Palo Alto Networks Unit 42 in June 2020, calling attention to its ability to exploit known security flaws to compromise Windows endpoints.
As many as 3,000 distinct attacks aimed at the Apache big data stack have been detected over the past month, the cloud security firm said. This also comprises those that single out susceptible Apache Flink instances to deploy miners and rootkits.

“The attacker implements the attack by exploiting existing misconfigurations and vulnerabilities in those services,” security researcher Nitzan Yaakov said.

Apache Vulnerability Update Available!

“Apache open-source solutions are widely used by many users and contributors. Attackers may view this extensive use as an opportunity to have inexhaustible resources for implementing their attacks on them.”

Hacker News:
Russian Hackers Have Infiltrated U.S. Household and Small Business Routers, FBI Warns
Original Article: MSN News

The FBI has recently thwarted a large-scale cyberattack orchestrated by Russian operatives, targeting hundreds of routers in home offices and small businesses, including those in the United States.

These compromised routers were used to form “botnets”, which were then employed in cyber operations worldwide.

The United States Department of Justice has attributed this cyberattack to the Russian GRU Military Unit 26165. Countermeasures undertaken by authorities ensured that the GRU operators were expelled from the routers and denied further access, ABC News reported.

The GRU deployed a specialized malware called “Moobot,” associated with a known criminal group, to seize control of susceptible home and small office routers, converting them into “botnets” — a network of remotely controlled systems.

The Justice Department, in an official statement, explained, “Non-GRU cybercriminals installed the Moobot malware on Ubiquiti Edge OS routers that still used publicly known default administrator passwords. GRU hackers then used the Moobot malware to install their own bespoke scripts and files that repurposed the botnet, turning it into a global cyber espionage platform.”

Utilizing this botnet, Russian hackers engaged in various illicit activities, including extensive “spearphishing” campaigns and credential harvesting campaigns against targets of intelligence interest to the Russian government, such as governmental, military, security and corporate entities in the United States and abroad.

Botnets pose a significant challenge for intelligence agencies, hindering their ability to detect foreign intrusions into their computer networks, Reuters notes.

In January 2024, the FBI executed a court-approved operation dubbed “Operation Dying Ember” to disrupt the hacking campaign. According to the Department of Justice, the FBI employed malware to copy and erase the malicious data from the routers, restoring full access to the owners while preventing further unauthorized access by GRU hackers.

FEDOR was designed as an android able to replace humans in high-risk areas, such as rescue operations,” Andrey Grigoriev, director of Russia’s Advanced Research Fund, said.

Quick Nmap Scanning Utility Framework

This script provides a basic framework for a quick and easy Nmap scanning utility. Designed for rapid security checkups, it leverages the Zenity tool to create a graphical user interface (GUI) that simplifies the process of running common Nmap scans. This script does not require sudo privileges, making it suitable for environments where elevated permissions are restricted. However, it does have a minor bug that affects user interaction with the script descriptions.

• Options Array: Defines a list of common Nmap scan options, each associated with a descriptive label.
• Zenity Dialogs:
  • The zenity --list command presents a GUI list for selecting Nmap options.
  • The zenity --entry command prompts the user to input a URL.
• Command Execution:
  • Constructs the full Nmap command using the selected options and entered URL.
  • Uses eval to execute the constructed Nmap command.
  • Displays the command being executed using another Zenity dialog.

The Code:

#!/bin/bash
# Quick Nmap - K0NxT3D
# Here's The Framework For A Project I Put
# Together For Quick Response Security Checkups.
# BUGS: Clicking The Description Will Process As Command.
# Click The Actual Command In This Example & Then The URL.

# Function to display error message and exit
    show_error() {
    zenity --error --text="$1" --title="Error"
    exit 1
}

# Function to display Nmap options list and prompt for URL
    get_nmap_options() {
# List of Nmap options
    options=(
    "[Ping Remote Host]" " -p 22,113,139" \
    "[Quick scan]" " -F" \
    "[Intense scan, all TCP ports]" " -p 1-65535 -T4 -A -v" \
    "[Scan all TCP ports (SYN scan)]" " -p- -sS -T4 -A -v" \
    "[Scan UDP ports]" " -sU -p 1-65535" \
    "[Full Scan, OS Detection, Version]" " -A" \
    "[Scan All Ports On Host]" " -sT -n -p-" \
    "[Scan with default NSE Scripts]:" " -sC" \
    "[TCP SYN port scan]" " -sS" \
    "[UDP Port Scan]" " -sU" \
    "[Scan For HTTP Vulnerabilities]" " -Pn -sV -p80 --script=vulners" \
    "[Nmap Help]" " -h")

# Display list of options and prompt for selection
    selected_option=$(zenity --list --title="Quick Nmap - K0NxT3D" --column="Options" "${options[@]}" --height 500 --width 500 --text="Select Options:")
        [ -z "$selected_option" ] && show_error "No Option Selected."

# Prompt for URL
    url=$(zenity --entry --title="Enter URL" --text="Enter URL To Scan:")
        [ -z "$url" ] && show_error "URL Not Provided."

# Execute Nmap command
    nmap_command="nmap $selected_option $url"
    echo "Executing Command: $nmap_command"
    zenity --info --text="Executing Nmap command:\n$nmap_command"
    eval "$nmap_command"
}

# Display GUI for Nmap options and URL input
get_nmap_options

Bug Description

• Description Bug: The script’s current implementation has a bug where clicking on a description in the Zenity list triggers an attempt to run the description as a command first. This results in an error message being displayed before the actual Nmap command is executed. While this does not significantly affect the performance or functionality of the script, it is noted as a minor inconvenience.

Advanced Usage

• Enhanced Functionality: Users who are familiar with Nmap can modify and extend this framework to include additional scanning options or integrate more advanced features.
• Proxy and Anonymity: The script is compatible with tools like torsocks and proxychains for executing Nmap scans through proxies, enhancing privacy and bypassing geographical restrictions.

This script serves as a convenient starting point for running common Nmap scans with a user-friendly interface, while also allowing for customization and enhancement based on individual needs.

Knowledge For Life Volume I

The Omniverse Library:
A diverse reading list from several topics.
The Omniverse Library boasts an extensive collection of resources covering a wide range of subjects, including science, history, philosophy, and the occult. Users can access a plethora of articles, books, research papers, manuscripts, and multimedia content curated from reputable sources worldwide.

Continuous Enrichment: The Omniverse Library is a dynamic platform continually enriched with new additions and updates. With regular contributions from experts, scholars, and content creators, the library remains a vital source of knowledge, fostering intellectual growth and exploration in an ever-evolving world.

Join the Quest for Knowledge: Embark on a journey of discovery and enlightenment with The Omniverse Library—an unparalleled digital repository where the boundaries of human understanding are transcended, and the pursuit of truth knows no bounds.

American & World History – Science – Philosophy – The Occult – Survival & Of Course.. some Miscreant Materials.
Carl Sagan – Isaac Newton – Nikola Tesla – Sun Tzu – Aleister Crowley – Karl Marx – Anarchist Cookbook – Bushcraft

Main Menu

Random HTTP Request Generator – “generator.php”

This generates the Header Request Information to be sent to a Destination URL.
For Testing Purposes Only – Some Files Have Been Excluded.
The Destination URL tracks incoming HTTP Requests and filters them for “bad data” or
“Spoofed Requests” such as the requests generated here.

This is pretty basic and I like it that way.
Using ffmpeg to capture the integrated web cam on my laptop and my USB webcam plugged in and then creating an archive to store subsequent photos in.
Part of a bigger project.

#!/bin/bash
# Set date for file naming
date=$(date +"%Y-%m-%d_%H%M%S")
    # Take photo using Integrated Webcam
      ffmpeg -f v4l2 -video_size 1280x720 -i /dev/video0 -frames 1 int.$date.jpg

    # Take photo using USB Webcam
      ffmpeg -f v4l2 -video_size 1280x720 -i /dev/video1 -frames 1 usb.$date.jpg

    # Add all .jpg files to payload.zip
      zip payload.zip *.jpg

    # Remove all .jpg files now
      rm *.jpg

    # Set time between photos
      sleep 10

    # Exit and start over
./$(basename $0) && exit

Using Netcat may be “Old School”, but so am I, so I love using Netcat for simple tasks or just chatting without Big Brother paying too much attention. I love using Bold Text too.

These are two separate scripts, one for use on a server, “server.sh” (home pc/Pi/laptop or and server that allows you to use Netcat) and “client.sh”, which you can use on your Android or Laptop etc from a mobile location.
Of course you’re going to have to set permissions and run them. I highly suggest editing out the sleep function and using cron if you’re savvy as this is really meant to update files such as remote sensors, cameras etc.

*Edit the IP address to your server in client.sh.

server.sh

#!/bin/bash
clear
    echo "Server Running."
        mkdir incoming
    date="$(date +'%Y-%m-%d_%H-%M')"
    file="incoming/payload.file"
# Set the Servers Port To Listen On
    echo $(nc -l 1234 > $file)
        mv $file "incoming/$date.payload"
    echo "File Recieved."
    sleep 10
./$(basename $0) && exit

client.sh

#!/bin/bash
clear
mkdir outgoing
    echo "Client Running."
        file="outgoing/payload.file"
# For Demo Only
    touch $file
    echo "Some Data" >> $file
# Set The Server IP and Port To Connect To
    echo $(nc -w 3 192.168.1.XXX 1234 < $file)
    echo "File Sent."
    sleep 60
./$(basename $0) && exit

Example script using OpenSSL AES 256 with Salt and a random generated password.
It’s the little things.

#!/bin/bash
clear
echo "Input String:"
    read input
        pass=$(echo cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 1024 | head -n 1)
        encrypt="$(echo -e $input | openssl aes-256-cbc -pbkdf2 -iter 20000 -salt -a -e -k $pass)"
        decrypt="$(echo -e $encrypt | openssl aes-256-cbc -pbkdf2 -iter 20000 -salt -a -d -k $pass)"
    echo -e "Encrypted String: "$encrypt
    echo -e "Decrypted String: "$decrypt
    echo "Hit Any Key.."
  read anykey
./$(basename $0) && exit

BashKat is pretty straight forward and really easy to use.
I made sure to add some “cute” to it with the emojis.
This bot will scrape from user input or a file using the wget function (example: urls.txt) and it’s Super Fun when using Proxychains.

#!/usr/bin/env bash
# BashKat Version 1.0.2
# K0NxT3D

# Variables
BotOptions="Url File Quit"

# Welcome Banner
clear
printf "✨ BashKat 1.0 ✨\nScrape Single URL/IP or Multiple From File.\n\n" && sleep 1

# Bot Options Menu
select option in $BotOptions; do

# Single URL Scrape
   if [ "$option" = "Url" ];
    then
      printf "URL To Scrape: "
       read scrapeurl
     mkdir -p data/
    wget -P data/ \
     -4 \
     -w 0 \
     -t 3 \
     -rkpN -e robots=off \
     --header="Accept: text/html" \
     --user-agent="BashKat/1.0 (BashKat 1.0 Web Scraper Utility +http://www.bashkat.bot/)" \
     --referer="http://www.bashkat.bot" \
     --random-wait \
     --recursive \
     --no-clobber \
     --page-requisites \
     --convert-links \
     --restrict-file-names=windows \
     --domains $scrapeurl \
     --no-parent \
         $scrapeurl

      printf "🏁Scrape Complete.\nHit Enter To Continue.👍"
       read anykey
./$(basename $0) && exit

  elif [ "$option" = "File" ];
   then
      printf "Path To File: "
       read filepath
     while IFS= read -r scrapeurl
      do
     mkdir -p data/
    wget -P data/ \
     -4 \
     -w 0 \
     -t 3 \
     -rkpN -e robots=off \
     --header="Accept: text/html" \
     --user-agent="BashKat/1.0 (BashKat 1.0 Web Scraper Utility +http://www.bashkat.bot/)" \
     --referer="http://www.bashkat.bot" \
     --random-wait \
     --recursive \
     --no-clobber \
     --page-requisites \
     --convert-links \
     --restrict-file-names=windows \
     --domains $scrapeurl \
     --no-parent \
         $scrapeurl 
     done < "$filepath"
      printf "🏁Scrape Complete.\nHit Enter To Continue.👍"
       read anykey
./$(basename $0) && exit

 elif [ "$option" = "Quit" ];
 then
   printf "Quitting🏳"
    sleep 1
     clear
      exit
# ERRORS
  else
   clear
    printf "❌"
    sleep 1
   ./$(basename $0) && exit
  fi
 exit
done