!bash@lemmy.ml
Talk about the Bash Shell and Bash scripting
!bash
@lemmy.mlI made grebuntu to merge all of the separate scripts into one script for all distros, but it doesn't work. The individual scripts do, I tested in VMs. What could have caused the issue? the script in question is tsubuntu.sh btw, can be found in the repo The original scripts are available at https://github.com/Tsu-gu/tsubuntu
Hi all,
I'm trying to put a command together to download a bz2 archive containing an img file and decompress it immediately, basically without saving it to the filesystem. Can this be done?
This is what I've come up with so far, but it's incomplete: wget -qO- "https://opnsense.com/.../img.bz2" | bzip2 -dv
Background: Trying to install OPNsense on Linode. Their hacky official guide says the best way to install FreeBSD is via the rescue mode. But FreeBSD posts their images as .img, so the filesystem size limitation of 1GB for the rescue image isn't an issue. But with OPNsense I need to decompress it.
I have a few different options on how to install this but I see it as a good reason to learn more about stdin/out, piping commands, etc.
Thanks in advance.
After a long process of roaming the web, re-runs and troubleshoot the script with this wonderful community, the script is functional and does what it's intended to do. The script itself is probably even further improvable in terms of efficiency/logic, but I lack the necessary skills/knowledge to do so, feel free to copy, edit or even propose a more efficient way of doing the same thing.
I'm greatly thankful to @AernaLingus@hexbear.net, @GenderNeutralBro@lemmy.sdf.org, @hydroptic@sopuli.xyz and Phil Harvey (exiftool) for their help, time and all the great idea's (and spoon-feeding me with simple and comprehensive examples ! )
Prerequisites:
parallel
package installed on your distributionCopy/past the below script in a file and make it executable. Change the start_range/end_range
to your needs and install the parallel
package depending on your OS and run the following command:
time find /path/to/your/image/directory/ -type f | parallel ./script-name.sh
This will order only the pictures from your specified time range into the following structure YEAR/MONTH
in your current directory from 5 different time tag/timestamps (DateTimeOriginal, CreateDate, FileModifyDate, ModifyDate, DateAcquired).
You may want to swap ModifyDate
and FileModifyDate
in the script, because ModifyDate
is more accurate in a sense that FileModifyDate
is easily changeable (as soon as you make some modification to the pictures, this will change to your current date). I needed that order for my specific use case.
From:
'-directory<$DateAcquired/' '-directory<$ModifyDate/' '-directory<$FileModifyDate/' '-directory<$CreateDate/' '-directory<$DateTimeOriginal/'
To:
'-directory<$DateAcquired/' '-directory<$FileModifyDate/' '-directory<$ModifyDate/' '-directory<$CreateDate/' '-directory<$DateTimeOriginal/'
As per exfitool's documentation:
ExifTool evaluates the command-line arguments left to right, and latter assignments to the same tag override earlier ones.
#!/bin/bash
if [ $# -eq 0 ]; then
echo "Usage: $0 <filename>"
exit 1
fi
# Concatenate all arguments into one string for the filename, so calling "./script.sh /path/with spaces.jpg" should work without quoting
filename="$*"
start_range=20170101
end_range=20201230
FIRST_DATE=$(exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$filename" | tr -d '-' | awk '{print $1}')
if [[ "$FIRST_DATE" != '' ]] && [[ "$FIRST_DATE" -gt $start_range ]] && [[ "$FIRST_DATE" -lt $end_range ]]; then
exiftool -api QuickTimeUTC -d %Y/%B '-directory<$DateAcquired/' '-directory<$ModifyDate/' '-directory<$FileModifyDate/' '-directory<$CreateDate/' '-directory<$DateTimeOriginal/' '-FileName=%f%-c.%e' "$filename"
else
echo "Not in the specified time range"
fi
Hi everyone !
Please no bash-shaming
, I did my outmost best to somehow put everything together and make it somehow work without any prior bash programming knowledge. It took me a lot of effort and time.
While I'm pretty happy with the result, I find the execution time very slow: 16min for 2288 files
.
On a big folder with approximately 50,062 files, this would take over 6 hours !!!
If someone could have a look and give me some easy to understand hints, I would greatly appreciate it.
Create a bash script that use exiftool
to stripe the date from images in a readable format (20240101) and compare it with an end_range
to order only images from that specific date range
(ex: 2020-01-01 -> 2020-12-30).
Also, some images lost some EXIF data, so I have to loop through specific time fields:
#!/bin/bash
shopt -s globstar
folder_name=/home/user/Pictures
start_range=20170101
end_range=20180130
for filename in $folder_name/**/*; do
if [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateTimeOriginal "$filename") =~ ^[0-9]+$ ]]; then
DateTimeOriginal=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -DateTimeOriginal "$filename")
if [ "$DateTimeOriginal" -gt $start_range ] && [ "$DateTimeOriginal" -lt $end_range ]; then
/usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$DateTimeOriginal/' '-FileName=%f%-c.%e' "$filename"
echo "Found a value"
echo "Okay its $(tput setab 22)DateTimeOriginal$(tput sgr0)"
fi
elif [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -CreateDate "$filename") =~ ^[0-9]+$ ]]; then
CreateDate=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -CreateDate "$filename")
if [ "$CreateDate" -gt $start_range ] && [ "$CreateDate" -lt $end_range ]; then
/usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$CreateDate/' '-FileName=%f%-c.%e' "$filename"
echo "Found a value"
echo "Okay its $(tput setab 27)CreateDate$(tput sgr0)"
fi
elif [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -FileModifyDate "$filename") =~ ^[0-9]+$ ]]; then
FileModifyDate=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -FileModifyDate "$filename")
if [ "$FileModifyDate" -gt $start_range ] && [ "$FileModifyDate" -lt $end_range ]; then
/usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$FileModifyDate/' '-FileName=%f%-c.%e' "$filename"
echo "Found a value"
echo "Okay its $(tput setab 202)FileModifyDate$(tput sgr0)"
fi
elif [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateAcquired "$filename") =~ ^[0-9]+$ ]]; then
DateAcquired=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -DateAcquired "$filename")
if [ "$DateAcquired" -gt $start_range ] && [ "$DateAcquired" -lt $end_range ]; then
/usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$DateAcquired/' '-FileName=%f%-c.%e' "$filename"
echo "Found a value"
echo "Okay its $(tput setab 172)DateAcquired(tput sgr0)"
fi
elif [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -ModifyDate "$filename") =~ ^[0-9]+$ ]]; then
ModifyDate=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -ModifyDate "$filename")
if [ "$ModifyDate" -gt $start_range ] && [ "$ModifyDate" -lt $end_range ]; then
/usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$ModifyDate/' '-FileName=%f%-c.%e' "$filename"
echo "Found a value"
echo "Okay its $(tput setab 135)ModifyDate(tput sgr0)"
fi
else
echo "No EXIF field found"
done
if
callsBut it didn't much improve the execution time (maybe a few ms?). The syntax looks way less readable but what I did, was to add a lot of or ( || ) in the syntax to reduce to a single if
call. It's not finished, I just gave it a test drive with 2 EXIF fields (DateTimeOriginal and CreateDate) to see if it could somehow improve time. But meeeh :/.
#!/bin/bash
shopt -s globstar
folder_name=/home/user/Pictures
start_range=20170101
end_range=20201230
for filename in $folder_name/**/*; do
if [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateTimeOriginal "$filename") =~ ^[0-9]+$ ]] || [[ $(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -CreateDate "$filename") =~ ^[0-9]+$ ]]; then
DateTimeOriginal=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -DateTimeOriginal "$filename")
CreateDate=$(/usr/bin/vendor_perl/exiftool -d '%Y%m%d' -T -CreateDate "$filename")
if [ "$DateTimeOriginal" -gt $start_range ] && [ "$DateTimeOriginal" -lt $end_range ] || [ "$CreateDate" -gt $start_range ] && [ "$CreateDate" -lt $end_range ]; then
/usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$DateTimeOriginal/' '-directory<$CreateDate/' '-FileName=%f%-c.%e' "$filename"
echo "Found a value"
echo "Okay its $(tput setab 22)DateTimeOriginal$(tput sgr0)"
else
echo "FINISH YOUR SYNTAX !!"
fi
fi
done
To recursively find my image files in all my folders I first tried the find
function, but that gave me a lot of headaches... When my image file name had some spaces in it, it just broke the image path strangely... And all answers I found on the web were gibberish, and I couldn't make it work in my script properly... Lost over 4 yours only on that specific issue !
To overcome the hurdle someone suggest to use shopt -s globstar
with for filename in $folder_name/**/*
and this works perfectly. But I have no idea If this could be the culprit of slow execution time?
[ ]
into [[ ]]
That also didn't do the trick.
I have no Idea if it's related to my script or the exiftool call that makes the script so slow. This isn't that much of a complicated script, I mean, it's a comparison between 2 integers not a hashing of complex numbers.
I hope someone could guide me in the right direction :)
Thanks !
I was able to setup a debugger using a launch mode using Visual Studio Code with the Bash Debug extension. Is it possible to setup the debugger in VSCode to be able to debug a bash script using a attach debug mode?
For debugging scripts on the host machine and scripts inside a docker container?
https://github.com/yazgoo/fuzzysh/
minimalist selector in shell, inspired by fzf. Contribute to yazgoo/fuzzysh development by creating an account on GitHub.
cross-posted from: https://lemm.ee/post/23155648
Here is the script.
#!/usr/bin/env bash # Download and search youtube subs # deps yt-dlp ,awk, perl, any one or more of either ugrep, ripgrep, grep # usage "script youtube_url" main() { url="$@" check_if_url get_video_id search_for_downloaded_matching_files set_download_boolean_flag download_subs read_and_format_transcript_file echo_description_file user_search } # Iterate over the array and add items to the new array if they match the regex check_if_url() { local regex='^https://[^[:space:]]+$' if ! [[ $url =~ $regex ]]; then echo "Invalid input. Valid input is a url matching regex ${regex}" exit 1 fi } get_video_id() { video_id=$(echo "$url" | sed -n 's/.*v=\([^&]*\).*/\1/p') } search_for_downloaded_matching_files() { # Find newest created files matching the video_id transcript_file="$( /usr/bin/ls -t --time=creation "$PWD"/*${video_id}*\.vtt 2>/dev/null | head -n 1 )" description_file="$( /usr/bin/ls -t --time=creation "$PWD"/*${video_id}*\.description 2>/dev/null | head -n 1 )" } set_download_boolean_flag() { if [ -n "$transcript_file" ] && [ -n "$description_file" ]; then download=0 # FALSE else download=1 # TRUE fi } download_subs() { if [ "$download" -eq 1 ]; then yt-dlp --restrict-filenames --write-auto-sub --skip-download "${url}" yt-dlp --restrict-filenames --sub-langs=eng --write-subs --skip-download "${url}" yt-dlp --restrict-filenames --write-description --skip-download "${url}" # Search files again since they were just downloaded search_for_downloaded_matching_files fi } read_and_format_transcript_file() { perl_removed_dupes="$(perl -0777 -pe 's/^\d\d.*\n.*\n.*<\/c>//gm' <"${transcript_file}")" local prefix="https://www.youtube.com/watch?v=${video_id}&t=" local suffix="s" formated_transcript_file="$(awk -v pre="$prefix" -v suf="$suffix" ' /^([0-9]{2}:){2}[0-9]{2}\.[0-9]{3}/ { split($1, a, /[:.]/); $1 = pre (int(a[1]*3600 + a[2]*60 + a[3]) - 3) suf; sub(/ --> [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}/, ""); sub(/ align:start position:0%$/, ""); print; next; } { sub(/ align:start position:0%$/, ""); print; } ' <<<"${perl_removed_dupes}")" #CRLF for ugrep to avoid ?bug? where before lines are not all outputted formated_transcript_file_CRLF=$(printf '%b' "$formated_transcript_file" | sed 's/$/\r/') } echo_description_file() { cat "${description_file}" } user_search() { echo -e "\n\n" read -rp "Enter regex (read as raw input): " search_term : ${app_count:=0} if command -v ug >/dev/null 2>&1; then echo -e "\n\n\n\n" echo "Ugrep output" ug --pretty=never -B2 -A1 -i -Z+-~1 -e "${search_term}" --andnot "^https?:\/\/" <<<"$formated_transcript_file_CRLF" ((app_count++)) fi if command -v rg >/dev/null 2>&1; then echo -e "\n\n\n\n" echo "Ripgrep output" rg -iP -B2 -A7 "^(?!https?:\/\/).*\K${search_term}" <<<"$formated_transcript_file" ((app_count++)) fi if [ "$app_count" -eq 0 ]; then echo -e "\n\n\n\n" echo "Grep output" grep -iP -B2 -A1 "${search_term}" <<<"$formated_transcript_file" echo -e "\n\n" echo "Consider installing ripgrep and ugrep for better search" ((app_count++)) fi } main "$@"
I made a script that downloads from youtube super fast using a custom aria2 build.
Aria2 https://github.com/P3TERX/Aria2-Pro-Core/releases
ffmpeg build https://github.com/yt-dlp/FFmpeg-Builds/releases
I choose ffmpeg-master-latest-linux64-gpl.tar.xz
#!/usr/bin/env bash
#set -x
if [[ -z $@ ]]; then
echo "specify download url"
exit
fi
dir_dl="$PWD"
url="$@"
ffmpeg_dir="$HOME/.local/bin.notpath/"
download_archive_dir="$HOME/Videos/yt-dlp/"
download_archive_filename=".yt-dlp-archived-done.txt"
mkdir -p "$download_archive_dir"
youtube_match_regex='^.*(youtube[.]com|youtu[.]be|youtube-nocookie[.]com).*$'
if [[ "$1" =~ $youtube_match_regex ]]; then
url="$(echo "$@" | perl -pe 's/((?:http:|https:)*?\/\/(?:www\.|)(?:youtube\.com|m\.youtube\.com|youtu\.|#youtube-nocookie\.com).*(?:c(?:hannel)?\/|u(?:ser)?\/|v=|v%3D|v\/|(?:a|p)\/(?:a|u)\/\d.*\/|watch\?|vi(?:=|\/)|\/#embed\/|oembed\?|be\/|e\/)([^&?%#\/\n]+)).*/$1/gm')"
yt-dlp \
--check-formats \
--clean-info-json \
--download-archive "$download_archive_dir$download_archive_filename" \
--embed-chapters \
--embed-info-json \
--embed-metadata \
--embed-thumbnail \
--external-downloader aria2c \
--downloader-args \
"aria2c: \
--allow-piece-length-change=true \
--check-certificate=false \
--console-log-level=notice \
--content-disposition-default-utf8=true \
--continue=true \
--disk-cache=8192 \
--download-result=full \
--enable-mmap \
--file-allocation=falloc \
--lowest-speed-limit=100K \
--max-concurrent-downloads=16 \
--max-connection-per-server=64 \
--max-mmap-limit=8192M \
--max-resume-failure-tries=5 \
--max-file-not-found=2 \
--max-tries=3 \
--min-split-size=64K \
--no-file-allocation-limit=8192M \
--piece-length=64k \
--realtime-chunk-checksum=false \
--retry-on-400=true \
--retry-on-403=true \
--retry-on-406=true \
--retry-on-unknown=true \
--retry-wait=1 \
--split=32 \
--stream-piece-selector=geom \
--summary-interval=0 " \
--ffmpeg-location "$ffmpeg_dir" \
--output "$dir_dl"'/%(channel)s/%(title)s_%(channel)s_%(upload_date>%Y-%m-%d)s_%(duration>%H-%M-%S)s_%(resolution)s.%(ext)s' \
--prefer-free-formats \
--remux-video mkv \
--restrict-filenames \
--sponsorblock-remove "filler,interaction,intro,music_offtopic,outro,preview,selfpromo,sponsor" \
--sub-langs "en.*,live_chat" \
--write-auto-subs \
--write-description \
--write-info-json \
--write-playlist-metafiles \
--write-subs \
--write-thumbnail \
"$url"
else
yt-dlp \
--download-archive "$download_archive_dir$download_archive_filename" \
--embed-chapters \
--ffmpeg-location "$ffmpeg_dir" \
--http-chunk-size 10M \
--output "$dir_dl/%(title)s_%(duration>%H-%M-%S)s_%(upload_date>%Y-%m-%d)s_%(resolution)s_URL_(%(id)s).%(ext)s" \
--prefer-free-formats \
--restrict-filenames \
"$url"
fi
https://cmdchallenge.com/
One-line shell challenges, to help improve your skill on the commandline.
[SOLVED] Solution: https://lemmy.ml/comment/4317564
I am doing all of this using ChatGPT, I know enough bash to understand the script partially, but not enough to write the script myself.
I recently posted bash script(click) to download songs on spotify
using Spotdl
Updated Post
This is all good, but I am not trying to download whole playlists and I want to make sure to
URL
to the Archieve_file
so it doesn't download it again. i.e., compare and if present, exit loop, if not present in file, download and add link to the file.This was easier when I was dealing with only song
links and not playlist
links. But now, playlists complicates the equation, but if I can achieve this, I can basically add this script to crontab
and make sure I always have a local copy of the songs in my playlist and these playlists would be checked regularly for new downloads and new songs would be downloaded. This is really cool!
Now, the complications I am facing,
I don't know how to get the URLs of the spotify songs
out of spotify playlists
using spotdl
or any cli package. If I can do this then, I can make the script go through each link clean
the link of unncessary attributes and then download it or if it's downloaded already, move on to the next song.
Now, I don't know how to do it. It would be very helpful if you guys could share any scripts that you have which will achieve this effect or help me get the song URL's from a playlist using a cli package.
PS: I might need some time to reply, I might sleep rn Thank you for your help! ___
I would rather prefer that you would buy spotify premium
if you can. But, till you can, there's always spotdl
which can allow you to download your songs
Installation of spotdl (github page click here): https://i.imgur.com/5g6uUgD.png
https://paste.debian.net/plain/1293528 a very simple script, you don't actually need the script, but it makes it easier to download songs.
btw, I am using file manager nemo here, if you want something else, change it to your default file manager, if you don't want to open folder, well remove last but one line.
execute this command first and when nano text edit appears, paste the debian pastebin in there. ::: spoiler spoiler
:::