Compare commits

...

16 Commits

Author SHA1 Message Date
Samuel Ortion 97c22977d0 db: Add mysql support for birdnet_observation database 2022-08-24 14:12:42 +02:00
Samuel Ortion 7a175e7f0a plot: Fix date in chart title 2022-08-24 12:00:47 +02:00
Samuel Ortion 4f09a2dd4e daemon: Remove birdnet_miner and call birdnet_output_to_sql at each new model execution 2022-08-24 11:58:15 +02:00
Samuel Ortion 39233fe937 install: Fix merge 2022-08-24 08:03:26 +02:00
Samuel Ortion fba68c6c90 docs: Updated CHANGELOG and README for better installation méthod 2022-08-24 08:01:52 +02:00
Samuel Ortion 0d696c8399 www: Add try catch to avoid exception on missing database 2022-08-24 07:23:47 +02:00
Samuel Ortion 30b1b44876 www: Removed home made log controller 2022-08-24 07:13:35 +02:00
Samuel Ortion 089acdffd9 docker: Symfony seems to be running (still mysql issues) 2022-08-23 17:15:40 +02:00
Samuel Ortion daffa3ff96 install: Fix typo on -b <branch> git clone 2022-08-23 13:36:57 +02:00
Samuel Ortion d8ca495471 docker: Working on mysql database and symfony dependencies 2022-08-23 13:25:10 +02:00
Samuel Ortion 1654885838 docker: Working database creation 2022-08-23 06:06:18 +02:00
Samuel Ortion 082823892c Remove instance specific .env file 2022-08-23 05:06:17 +02:00
Samuel Ortion a557723b35 docker: Symfony webapp seems to work (no database connection yet) 2022-08-22 20:27:33 +02:00
Samuel Ortion c1af47eb26 docker: Update docker-compose and ./docker Dockerfiles 2022-08-22 18:49:40 +02:00
Samuel Ortion 048e93fef0 Spliiting to far may not be the solution 2022-08-22 06:22:30 +02:00
Samuel Ortion 9d0cb08792 install: Adding `loginctl enable-linger` for user recording using pulseaudio 2022-08-22 05:48:45 +02:00
41 changed files with 941 additions and 377 deletions

View File

@ -1,4 +1,7 @@
var
.venv
.github
.ideas
/var
/.venv
/.github
/.ideas
/media
/daemon/systemd
/analyzer

1
.env
View File

@ -1 +0,0 @@
CUDA_VISIBLE_DEVICES=""

12
.env.example Normal file
View File

@ -0,0 +1,12 @@
# BirdNET-Analyzer environment
CUDA_VISIBLE_DEVICES=""
# BirdNET-stream environment
DATABASE_USER="birdnet"
DATABASE_PASSWORD="secret" # change this
DATABASE_PORT="3306"
DATABASE_ROOT_PASSWORD="secret" # change this
RECORDS_DIR="/media/data/birdnet/records"
CHARTS_DIR="/media/data/birdnet/charts"
SERVER_NAME="birdnet.local"

5
.gitignore vendored
View File

@ -1,7 +1,10 @@
var/
/.venv/
.env
/.env
/.env.local
!.env.local.example
!.env.example
species_list.txt

View File

@ -1,6 +1,11 @@
# Changelog
## v0.0.1-alpha
## v0.0.1-rc
- Add docker compose port
- Improve install script
- Add base uninstall script (need deeper work)
- Add ttyd for systemd logging
## v0.0.1-rc (2022-08-18)
- Integrate BirdNET-Analyzer as submodule
- Add birdnet_recording service
@ -11,4 +16,4 @@
- Add /today/species and /today/{date}/species/{id} endpoints
- Add records deletion button and /records/delete endpoint as well as bulk deletion (select all button on /today/species/{id} endpoint)
- Add systemd status page /status
- Add i18n for webapp (not species name), en|fr only for the moment
- Add i18n for webapp (not species name), en|fr only for the moment

View File

@ -79,17 +79,23 @@ sudo systemctl enable --now birdnet_recording.service birdnet_analyzis.service b
#### Check if services are working
```bash
# Sercices status
sudo systemctl status birdnet_recording.service birdnet_analyzis.service
# Timers status
sudo systemctl status birdnet_miner.timer
# Sercices and timers status
sudo systemctl status birdnet_\*
```
```bash
# BirdNET-stream logs
sudo journalctl -feu {birdnet_recording,birdnet_analyzis}.service
sudo journalctl -feu birdnet_\*
```
#### Enable `loginctl-linger` for the user that runs the servuces
Running:
```bash
loginctl enable-linger
```
This allows to use `/run/user/1000/pulse` to record audio using PulseAudio in birdnet_recording.sh.
## Setup BirdNET-stream symfony webapp
### Install php 8.1
@ -123,7 +129,7 @@ sudo mv /composer.phar /usr/local/bin/composer
```bash
cd www
composer install
composer install --no-dev --prefer-dist --optimize-autoloader
```
### Install nodejs and npm
@ -141,7 +147,7 @@ nvm use 16
```
```bash
sudo dnf install npm
sudo apt-get install npm
```
```bash

View File

@ -8,7 +8,7 @@
## Introduction
BirdNET-stream records sound 24/7 on any Linux computer with a microphone and analyze it using BirdNET algorithm by [**@kahst**](https://github.com/kahst).
BirdNET-stream records sound 24/7 on any Linux computer with a microphone and analyze it using BirdNET algorithm by [**@kahst**](https://github.com/kahst).
Bird contacts are stored in a database and are made accessible in a webapp.
@ -28,19 +28,45 @@ It should work on a Raspberry Pi (or other Single Board Computer) with a USB mic
## Installation
On debian based system, you can install BirdNET-stream with the following command:
> **Warning** BirdNET-stream is in early development, and may not work properly...
<!-- On debian based system, you can install BirdNET-stream with the following command:
```bash
curl -sL https://raw.githubusercontent.com/UncleSamulus/BirdNET-stream/main/install.sh | bash
``` -->
On debian based systems (tested on Debian Bullseye), the following command should allow you to install the base components without too much trouble:
```bash
# Change to your installation directory here, /home/$USER/Documents/BirdNET-stream for instance, or /opt/birdnet-stream, or whatever
cd /path/to/installation/directory
# Download installation script
curl -0 https://raw.githubusercontent.com/UncleSamulus/BirdNET-stream/main/install.sh
# Run installation script:
chmod +x ./install.sh
./install.sh
```
For finer control, or to adapt to your system, you can follow the instructions in the [INSTALL.md](./INSTALL.md) file.
I recommend to add `DEBUG=1` before this command to see the installation steps:
```bash
DEBUG=1 ./install.sh
```
To install from a specific git branch, add `BRANCH=<branch>` before the command, for instance:
```bash
BRANCH=dev DEBUG=1 ./install.sh
```
For finer control, or to adapt to your system, you can follow the instructions in the [INSTALL.md](./INSTALL.md) file (it may unfortunatly not be accurate for your system).
## Usage
- BirdNET-stream web application can be accessed on any web browser at [https://birdnet.home](https://birdnet.home), from your local network, or at any other hostname you set in nginx configuration, if your public IP is accessible from the internet.
- See the species detected
- See the species detected
## Acknoledgments
@ -49,4 +75,4 @@ For finer control, or to adapt to your system, you can follow the instructions i
## License
BirdNET-stream is licensed under the GNU General Public License v3.0, see [./LICENSE](./LICENSE) for more details.
BirdNET-stream is licensed under the GNU General Public License v3.0, see [./LICENSE](./LICENSE) for more details.

1
TODO
View File

@ -1,5 +1,4 @@
- Fix service manager
- Add docker support
- Species i18n
- File purge policy
- Add and test RTSP support

View File

@ -19,6 +19,4 @@ PYTHON_VENV="./.venv/birdnet-stream"
WORKDIR="/home/$USER/BirdNET-stream"
# Database location
DATABASE="./var/db.sqlite"
DAEMON_USER="birdnet"
DAEMON_PASSWORD="secret"
# DATABASE="mysql://birdnet:secret@localhost:3306/birdnet_observations" # uncomment and change 'secret' if you want to use a mariadb (mysql) database instea of sqlite

View File

@ -3,9 +3,7 @@ set -e
DEBUG=${DEBUG:-1}
debug() {
if [ $DEBUG -eq 1 ]; then
echo "$1"
fi
[[ $DEBUG -eq 1 ]] && echo "$@"
}
config_filepath="./config/birdnet.conf"
@ -64,7 +62,9 @@ check_prerequisites() {
# Get array of audio chunks to be processed
get_chunk_list() {
find "${CHUNK_FOLDER}/in" -type f -name '*.wav' -exec basename {} \; ! -size 0 | sort
chunk_list=($(ls ${CHUNK_FOLDER}/in))
echo "${chunk_list}"
# find "${CHUNK_FOLDER}/in" -type f -name '*.wav' -exec basename {} \; ! -size 0 | sort
}
# Perform audio chunk analysis on one chunk
@ -75,13 +75,22 @@ analyze_chunk() {
mkdir -p "$output_dir"
date=$(echo $chunk_name | cut -d'_' -f2)
week=$(./daemon/weekof.sh $date)
$PYTHON_EXECUTABLE ./analyzer/analyze.py --i $chunk_path --o "$output_dir/model.out.csv" --lat $LATITUDE --lon $LONGITUDE --week $week --min_conf $CONFIDENCE --threads 4 --rtype csv
if [[ ! -z "${THREADS}" ]]; then
threads="--threads ${THREADS}"
else
threads=""
fi
$PYTHON_EXECUTABLE ./analyzer/analyze.py --i $chunk_path --o "$output_dir/model.out.csv" --lat $LATITUDE --lon $LONGITUDE --week $week --min_conf $CONFIDENCE $threads --rtype csv
debug "Model output written to $output_dir/model.out.csv"
bash ./daemon/birdnet_output_to_sql.sh "$output_dir/model.out.csv"
debug "Dumped to SQL database"
}
# Perform audio chunk analysis on all recorded chunks
analyze_chunks() {
for chunk_name in $(get_chunk_list); do
local chunks
chunks="${1}"
for chunk_name in "${chunks}"; do
if [[ -f "${CHUNK_FOLDER}/out/$chunk_name.d/model.out.csv" ]]; then
debug "Skipping $chunk_name, as it has already been analyzed"
else
@ -98,4 +107,4 @@ check_prerequisites
chunks=$(get_chunk_list)
# Analyze all chunks in working directory
analyze_chunks $chunks
analyze_chunks "$chunks"

View File

@ -1,15 +1,13 @@
#! /usr/bin/env bash
# Extract observations from a model output folder
# Extract observations from a model output file into SQL database
#
DEBUG=${DEBUG:-1}
set -e
# set -x
DEBUG=${DEBUG:-1}
debug() {
if [ $DEBUG -eq 1 ]; then
echo "$1"
fi
[[ $DEBUG -eq 1 ]] && echo "$@"
}
# Load bash library to deal with BirdNET-stream database
@ -18,16 +16,6 @@ source ./daemon/database/scripts/database.sh
# Load config
source ./config/birdnet.conf
# Check config
if [[ -z ${CHUNK_FOLDER} ]]; then
echo "CHUNK_FOLDER is not set"
exit 1
else
if [[ ! -d ${CHUNK_FOLDER}/out ]]; then
echo "CHUNK_FOLDER does not exist: ${CHUNK_FOLDER}/out"
echo "Cannot extract observations."
exit 1
fi
fi
if [[ -z ${LATITUDE} ]]; then
echo "LATITUDE is not set"
@ -39,10 +27,6 @@ if [[ -z ${LONGITUDE} ]]; then
exit 1
fi
model_outputs() {
ls ${CHUNK_FOLDER}/out/*/model.out.csv
}
source_wav() {
model_output_path=$1
model_output_dir=$(dirname $model_output_path)
@ -107,13 +91,6 @@ save_observations() {
done
}
main() {
# # Remove all junk observations
# ./daemon/birdnet_clean.sh
# Get model outputs
for model_output in $(model_outputs); do
save_observations $model_output
done
}
model_output_path="$1"
main
save_observations $model_output_path

View File

@ -3,6 +3,7 @@
DEBUG=${DEBUG:-1}
export PULSE_RUNTIME_PATH="/run/user/$(id -u)/pulse/"
FFMPEG_OPTIONS="-nostdin -hide_banner -loglevel error -nostats -vn -acodec pcm_s16le -ac 1 -ar 48000"
debug() {
if [ $DEBUG -eq 1 ]; then
@ -27,20 +28,25 @@ record_loop() {
done
}
FFMPEG_OPTIONS="-nostdin -hide_banner -loglevel error -nostats -vn -acodec pcm_s16le -ac 1 -ar 48000 "
record_stream() {
local STREAM=$1
local DURATION=$2
local debug "Recording from $STREAM for $DURATION seconds"
ffmpeg $FFMPEG_OPTIONS -f -i ${DEVICE} -t ${DURATION} file:${CHUNK_FOLDER}/in/birdnet_$(date "+%Y%m%d_%H%M%S").wav
ffmpeg $FFMPEG_OPTIONS -i ${STREAM} -t ${DURATION} file:${CHUNK_FOLDER}/in/birdnet_$(date "+%Y%m%d_%H%M%S").wav
}
record_device() {
DEVICE=$1
DURATION=$2
debug "Recording from $DEVICE for $DURATION seconds"
ffmpeg $FFMPEG_OPTIONS -f pulse -i ${DEVICE} -t ${DURATION} -af "volume=$RECORDING_AMPLIFY" file:${CHUNK_FOLDER}/in/birdnet_$(date "+%Y%m%d_%H%M%S").wav
local ffmpeg_input
if [[ "$AUDIO_USE_PULSE" = "true" ]]; then
ffmpeg_input="-f pulse -i ${DEVICE}"
else
ffmpeg_input="-f alsa -i ${DEVICE}"
fi
ffmpeg $FFMPEG_OPTIONS $ffmpeg_input -t ${DURATION} -af "volume=$RECORDING_AMPLIFY" file:${CHUNK_FOLDER}/in/birdnet_$(date "+%Y%m%d_%H%M%S").wav
}
config_filepath="./config/birdnet.conf"

View File

@ -17,5 +17,16 @@ if [ -z "$DATABASE" ]; then
DATABASE="./var/db.sqlite"
fi
# Create database according to schema in structure.sql
sqlite3 "$DATABASE" < ./daemon/database/structure.sql
if [[ $DATABASE = "mysql://"* ]]; then
# Split mysql uri into user, password, host, port, and database
MYSQL_ADDRESS=$(echo "$DATABASE" | sed 's/mysql:\/\///g')
MYSQL_CREDENTIALS=$(echo "$MYSQL_ADDRESS" | cut -d@ -f1)
MYSQL_USER=$(echo "$MYSQL_CREDENTIALS" | cut -d: -f1)
MYSQL_PASSWORD=$(echo "$MYSQL_CREDENTIALS" | cut -d: -f2)
MYSQL_HOST=$(echo "$MYSQL_ADDRESS" | cut -d@ -f2 | cut -d: -f1)
MYSQL_PORT=$(echo "$MYSQL_ADDRESS" | cut -d@ -f2 | cut -d: -f2 | cut -d/ -f1)
MYSQL_DATABASE=$(echo "$MYSQL_ADDRESS" | cut -d/ -f2)
mysql -u$MYSQL_USER -p$MYSQL_PASSWORD -h$MYSQL_HOST -P$MYSQL_PORT -D$MYSQL_DATABASE < ./daemon/database/structure-mysql.sql
else
sqlite3 $DATABASE < ./daemon/database/structure-sqlite.sql
fi

View File

@ -8,10 +8,30 @@ source ./config/birdnet.conf
# Create database in case it was not created yet
./daemon/database/scripts/create.sh
DATABASE=${DATABASE:-"./var/db.sqlite"}
# Check if database location is specified
if [ -z "$DATABASE" ]; then
echo "DATABASE location not specified"
echo "Defaults to ./var/db.sqlite"
DATABASE="./var/db.sqlite"
fi
query() {
sqlite3 -cmd ".timeout 1000" $DATABASE "$1"
local stmt
stmt="$1"
if [[ $DATABASE = "mysql://"* ]]; then
# Split mysql uri into user, password, host, port, and database
MYSQL_ADDRESS=$(echo "$DATABASE" | sed 's/mysql:\/\///g')
MYSQL_CREDENTIALS=$(echo "$MYSQL_ADDRESS" | cut -d@ -f1)
MYSQL_USER=$(echo "$MYSQL_CREDENTIALS" | cut -d: -f1)
MYSQL_PASSWORD=$(echo "$MYSQL_CREDENTIALS" | cut -d: -f2)
MYSQL_HOST=$(echo "$MYSQL_ADDRESS" | cut -d@ -f2 | cut -d: -f1)
MYSQL_PORT=$(echo "$MYSQL_ADDRESS" | cut -d@ -f2 | cut -d: -f2 | cut -d/ -f1)
MYSQL_DATABASE=$(echo "$MYSQL_ADDRESS" | cut -d/ -f2)
mysql -u$MYSQL_USER -p$MYSQL_PASSWORD -h$MYSQL_HOST -P$MYSQL_PORT -D$MYSQL_DATABASE -e "$stmt"
else
sqlite3 -cmd ".timeout 1000" "$DATABASE" "$stmt"
fi
}
get_location_id() {
@ -37,4 +57,4 @@ insert_observation() {
# Check if the observation already exists in the database
observation_exists() {
query "SELECT EXISTS(SELECT observation_id FROM observation WHERE audio_file='$1' AND start='$2' AND end='$3' AND taxon_id='$4' AND location_id='$5')"
}
}

View File

@ -0,0 +1,31 @@
/** Database structure for BirdNET-stream SQLite*/
/** Taxon table */
CREATE TABLE IF NOT EXISTS taxon (
taxon_id INTEGER PRIMARY KEY NOT NULL AUTO_INCREMENT,
scientific_name TEXT NOT NULL,
common_name TEXT NOT NULL
);
/** Location table */
CREATE TABLE IF NOT EXISTS location (
location_id INTEGER PRIMARY KEY NOT NULL AUTO_INCREMENT,
latitude REAL NOT NULL,
longitude REAL NOT NULL
);
/** Observation table */
CREATE TABLE IF NOT EXISTS observation (
`observation_id` INTEGER PRIMARY KEY NOT NULL AUTO_INCREMENT,
`audio_file` TEXT NOT NULL,
`start` REAL NOT NULL,
`end` REAL NOT NULL,
`taxon_id` INTEGER NOT NULL,
`location_id` INTEGER NOT NULL,
`date` TEXT NOT NULL,
`notes` TEXT,
`confidence` REAL NOT NULL,
`verified` BOOLEAN DEFAULT 0 CHECK (`verified` IN (0, 1)),
FOREIGN KEY(taxon_id) REFERENCES taxon(taxon_id),
FOREIGN KEY(location_id) REFERENCES location(location_id)
);

View File

@ -2,21 +2,21 @@
/** Taxon table */
CREATE TABLE IF NOT EXISTS taxon (
taxon_id INTEGER PRIMARY KEY,
taxon_id INTEGER PRIMARY KEY NOT NULL,
scientific_name TEXT NOT NULL,
common_name TEXT NOT NULL
);
/** Location table */
CREATE TABLE IF NOT EXISTS location (
location_id INTEGER PRIMARY KEY,
location_id INTEGER PRIMARY KEY NOT NULL,
latitude REAL NOT NULL,
longitude REAL NOT NULL
);
/** Observation table */
CREATE TABLE IF NOT EXISTS observation (
`observation_id` INTEGER PRIMARY KEY,
`observation_id` INTEGER PRIMARY KEY NOT NULL,
`audio_file` TEXT NOT NULL,
`start` REAL NOT NULL,
`end` REAL NOT NULL,
@ -25,7 +25,7 @@ CREATE TABLE IF NOT EXISTS observation (
`date` TEXT NOT NULL,
`notes` TEXT,
`confidence` REAL NOT NULL,
`verified` BOOLEAN NOT NULL CHECK (`verified` IN (0, 1)) DEFAULT 0,
`verified` BOOLEAN DEFAULT 0 CHECK (`verified` IN (0, 1)),
FOREIGN KEY(taxon_id) REFERENCES taxon(taxon_id),
FOREIGN KEY(location_id) REFERENCES location(location_id)
);

View File

@ -6,90 +6,119 @@ import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm
import seaborn as sns
from datetime import datetime
import os
import glob
CONFIG = {
"readings": 10,
"palette": "Greens",
"db": "./var/db.sqlite",
"date": datetime.now().strftime("%Y-%m-%d")
# "date": "2022-08-15"
"date": datetime.now().strftime("%Y-%m-%d"),
"charts_dir": "./var/charts"
}
db = sqlite3.connect(CONFIG['db'])
db = None
df = pd.read_sql_query("""SELECT common_name, date, location_id, confidence
FROM observation
INNER JOIN taxon
ON observation.taxon_id = taxon.taxon_id""", db)
df['date'] = pd.to_datetime(df['date'])
df['hour'] = df['date'].dt.hour
df['date'] = df['date'].dt.date
df['date'] = df['date'].astype(str)
df_on_date = df[df['date'] == CONFIG['date']]
top_on_date = (df_on_date['common_name'].value_counts()[:CONFIG['readings']])
if top_on_date.empty:
print("No observations on {}".format(CONFIG['date']))
exit()
def get_database():
global db
if db is None:
db = sqlite3.connect(CONFIG["db"])
return db
df_top_on_date = df_on_date[df_on_date['common_name'].isin(top_on_date.index)]
def chart(date):
db = get_database()
df = pd.read_sql_query(f"""SELECT common_name, date, location_id, confidence
FROM observation
INNER JOIN taxon
ON observation.taxon_id = taxon.taxon_id
WHERE STRFTIME("%Y-%m-%d", `date`) = '{date}'""", db)
df['date'] = pd.to_datetime(df['date'])
df['hour'] = df['date'].dt.hour
df['date'] = df['date'].dt.date
df['date'] = df['date'].astype(str)
df_on_date = df[df['date'] == date]
top_on_date = (df_on_date['common_name'].value_counts()[:CONFIG['readings']])
if top_on_date.empty:
print("No observations on {}".format(date))
return
else:
print(f"Found observations on {date}")
df_top_on_date = df_on_date[df_on_date['common_name'].isin(top_on_date.index)]
# Create a figure with 2 subplots
fig, axs = plt.subplots(1, 2, figsize=(20, 5), gridspec_kw=dict(
width_ratios=[2, 6]))
plt.subplots_adjust(left=None, bottom=None, right=None,
top=None, wspace=0, hspace=0)
# Create a figure with 2 subplots
fig, axs = plt.subplots(1, 2, figsize=(20, 5), gridspec_kw=dict(
width_ratios=[2, 6]))
plt.subplots_adjust(left=None, bottom=None, right=None,
top=None, wspace=0, hspace=0)
# Get species frequencies
frequencies_order = pd.value_counts(df_top_on_date['common_name']).iloc[:CONFIG['readings']].index
# Get min max confidences
confidence_minmax = df_top_on_date.groupby('common_name')['confidence'].max()
confidence_minmax = confidence_minmax.reindex(frequencies_order)
# Norm values for color palette
norm = plt.Normalize(confidence_minmax.values.min(),
confidence_minmax.values.max())
# Get species frequencies
frequencies_order = pd.value_counts(df_top_on_date['common_name']).iloc[:CONFIG['readings']].index
# Get min max confidences
confidence_minmax = df_top_on_date.groupby('common_name')['confidence'].max()
confidence_minmax = confidence_minmax.reindex(frequencies_order)
# Norm values for color palette
norm = plt.Normalize(confidence_minmax.values.min(),
confidence_minmax.values.max())
colors = plt.cm.Greens(norm(confidence_minmax))
plot = sns.countplot(y='common_name', data=df_top_on_date, palette=colors, order=frequencies_order, ax=axs[0])
colors = plt.cm.Greens(norm(confidence_minmax))
plot = sns.countplot(y='common_name', data=df_top_on_date, palette=colors, order=frequencies_order, ax=axs[0])
plot.set(ylabel=None)
plot.set(xlabel="Detections")
plot.set(ylabel=None)
plot.set(xlabel="Detections")
heat = pd.crosstab(df_top_on_date['common_name'], df_top_on_date['hour'])
# Order heatmap Birds by frequency of occurrance
heat.index = pd.CategoricalIndex(heat.index, categories=frequencies_order)
heat.sort_index(level=0, inplace=True)
heat = pd.crosstab(df_top_on_date['common_name'], df_top_on_date['hour'])
# Order heatmap Birds by frequency of occurrance
heat.index = pd.CategoricalIndex(heat.index, categories=frequencies_order)
heat.sort_index(level=0, inplace=True)
hours_in_day = pd.Series(data=range(0, 24))
heat_frame = pd.DataFrame(data=0, index=heat.index, columns=hours_in_day)
heat = (heat + heat_frame).fillna(0)
hours_in_day = pd.Series(data=range(0, 24))
heat_frame = pd.DataFrame(data=0, index=heat.index, columns=hours_in_day)
heat = (heat + heat_frame).fillna(0)
# Generate heatmap plot
plot = sns.heatmap(
heat,
norm=LogNorm(),
annot=True,
annot_kws={
"fontsize": 7
},
fmt="g",
cmap=CONFIG['palette'],
square=False,
cbar=False,
linewidth=0.5,
linecolor="Grey",
ax=axs[1],
yticklabels=False)
plot.set_xticklabels(plot.get_xticklabels(), rotation=0, size=7)
# Generate heatmap plot
plot = sns.heatmap(
heat,
norm=LogNorm(),
annot=True,
annot_kws={
"fontsize": 7
},
fmt="g",
cmap=CONFIG['palette'],
square=False,
cbar=False,
linewidth=0.5,
linecolor="Grey",
ax=axs[1],
yticklabels=False)
plot.set_xticklabels(plot.get_xticklabels(), rotation=0, size=7)
for _, spine in plot.spines.items():
spine.set_visible(True)
for _, spine in plot.spines.items():
spine.set_visible(True)
plot.set(ylabel=None)
plot.set(xlabel="Hour of day")
plt.suptitle(f"Top {CONFIG['readings']} species on {CONFIG['date']}", fontsize=14)
plt.text(15, 11, f"(Updated on {datetime.now().strftime('%Y/%m-%d %H:%M')})")
plt.savefig(f"./var/charts/chart_{CONFIG['date']}.png", dpi=300)
plt.close()
plot.set(ylabel=None)
plot.set(xlabel="Hour of day")
plt.suptitle(f"Top {CONFIG['readings']} species on {date}", fontsize=14)
plt.text(15, 11, f"(Updated on {datetime.now().strftime('%Y/%m-%d %H:%M')})")
plt.savefig(f"./var/charts/chart_{date}.png", dpi=300)
print(f"Plot for {date} saved.")
plt.close()
db.close()
def main():
done_charts = glob.glob(f"{CONFIG['charts_dir']}/*.png")
last_modified = max(done_charts, key=os.path.getctime)
last_modified_date = last_modified.split("_")[-1].split(".")[0]
missing_dates = pd.date_range(start=last_modified_date, end=CONFIG['date'], freq='D')
print(missing_dates)
for missing_date in missing_dates:
date = missing_date.strftime("%Y-%m-%d")
chart(date)
chart(CONFIG['date'])
if db is not None:
db.close()
print("Done.")
if __name__ == "__main__":
main()

View File

@ -1,13 +0,0 @@
[Unit]
Description=BirdNET-stream miner service
[Service]
Type=simple
User=<USER>
Group=<GROUP>
WorkingDirectory=<DIR>
ExecStart=bash ./daemon/birdnet_miner.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

View File

@ -1,9 +0,0 @@
[Unit]
Description=BirdNET-stream miner Timer
[Timer]
OnCalendar=*:0/15
Unit=birdnet_miner.service
[Install]
WantedBy=basic.target

View File

@ -1,36 +1,119 @@
version: '3.8'
networks:
birdnet_network:
version: '3.9'
services:
# database:
# container_name: birdnet_database
# image:
# recording:
# container_name: birdnet_recording
# build:
# context: .
# dockerfile: ./docker/recording/Dockerfile
# restart: unless-stopped
# environment:
# - CHUNK_FOLDER=${CHUNK_FOLDER:-/media/birdnet/records}
# volumes:
# - ${RECORDS_DIR:-/media/birdnet/records}:${RECORS_FOLDER:-/media/birdnet/records}
# # Allow container to access to the hosts microphone
# devices:
# - /dev/snd:/dev/snd
# analyzer:
# container_name: birdnet_analyzer
# build:
# context: ./analyzer/
# dockerfile: ./Dockerfile
php:
container_name: birdnet_php
image: php:8.1-fpm
db:
container_name: birdnet_database
image: mariadb:latest
command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
ports:
- "${PHP_FPM_PORT:-9001}:9000"
- ${DATABASE_PORT:-3306}:3306
networks:
- birdnet_network
environment:
MYSQL_ROOT_PASSWORD: ${DATABASE_ROOT_PASSWORD:-secret}'
MYSQL_USER: ${DATABASE_USER:-birdnet}
MYSQL_PASSWORD: ${DATABASE_PASSWORD:-secret}
volumes:
- ./docker/database/init:/docker-entrypoint-initdb.d
restart: unless-stopped
php-fpm:
container_name: birdnet_php-fpm
build:
context: .
dockerfile: ./docker/php-fpm/Dockerfile
ports:
- '${PHP_FPM_PORT:-9000}:9000'
networks:
- birdnet_network
environment:
- APP_ENV=${APP_ENV:-prod}
- APP_DEBUG=${APP_DEBUG:-true}
- DATABASE_DEFAULT_URL=mysql://${DATABASE_USER:-birdnet}:${DATABASE_PASSWORD:-secret}@${DATABASE_HOST:-birdnet_database}:${DATABASE_PORT:-3306}/birdnet_default
- DATABASE_OBSERVATIONS_URL=mysql://${DATABASE_USER:-birdnet}:${DATABASE_PASSWORD:-secret}@${DATABASE_HOST:-birdnet_database}:${DATABASE_PORT:-3306}/birdnet_observations
restart: unless-stopped
volumes:
- birdnet_app:${PROJECT_ROOT:-/opt/birdnet}
symfony:
container_name: birdnet_symfony
networks:
- birdnet_network
build:
context: .
dockerfile: ./docker/symfony/Dockerfile
args:
- DATABASE_DEFAULT_URL=mysql://${DATABASE_USER:-birdnet}:${DATABASE_PASSWORD:-secret}@${DATABASE_HOST:-birdnet_database}:${DATABASE_PORT:-3306}/birdnet_default
- DATABASE_OBSERVATIONS_URL=mysql://${DATABASE_USER:-birdnet}:${DATABASE_PASSWORD:-secret}@${DATABASE_HOST:-birdnet_database}:${DATABASE_PORT:-3306}/birdnet_observations
- RECORDS_DIR=/media/birdnet/records
- CHARTS_DIR=/media/birdnet/charts
restart: on-failure
volumes:
- birdnet_app:${PROJECT_ROOT:-/opt/birdnet}
- birdnet_records:${RECORDS_DIR:-/media/birdnet/records}
depends_on:
- db
nginx:
container_name: birdnet_nginx
hostname: ${SERVER_NAME:-birdnet.local}
build:
context: ./docker/
environment:
SERVER_NAME: ${SERVER_NAME:-birdnet.local}
PHP_FPM_PORT: ${PHP_FPM_PORT:-9001}
restart: unless-stopped
volumes:
- ./www:/var/www/birdnet/
- ./www/nginx.conf:/etc/nginx/conf.d/birdnet.conf
context: .
dockerfile: ./docker/nginx/Dockerfile
args:
- SERVER_NAME=${SERVER_NAME:-birnet.local}
- SYMFONY_PUBLIC=/opt/birdnet/www/public
- CHARTS_DIR=/media/birdnet/charts
- RECORDS_DIR=/media/birdnet/records
- PHP_FPM_HOST=birdnet_php-fpm
- PHP_FPM_PORT=9000
ports:
- "81:80"
dependends_on:
- php
- ${HTTP_PORT:-80}:80
- ${HTTPS_PORT:-443}:443
volumes:
- birdnet_app:/opt/birdnet
- birdnet_records:/media/data/records
networks:
birdnet_network:
ipv4_address: ${IP_ADDRESS:-172.25.0.101}
aliases:
- ${SERVER_NAME:-birdnet.local}
restart: unless-stopped
depends_on:
- symfony
- php-fpm
birdnet:
container_name: birdnet_analyzer
image:
networks:
birdnet_network:
driver: bridge
ipam:
config:
- subnet: ${IP_SUBNET:-172.25.0.0/24}
volumes:
birdnet_app:
birdnet_records:
driver_opts:
type: none
device: ${RECORDS_DIR:-/media/data/records}
o: bind

View File

@ -4,19 +4,24 @@ FROM debian:bullseye
ENV REPOSITORY=${REPOSITORY:-https://github.com/UncleSamulus/BirdNET-stream.git}
# DEBUG defaults to 1 for descriptive DEBUG logs, 0 for error logs only
ENV DEBUG=${DEBUG:-1}
RUN useradd birdnet
WORKDIR /home/birdnet
RUN useradd -m -s /bin/bash -G sudo birdnet
USER birdnet
# Upgrade system
RUN apt-get update && apt-get upgrade -y
# Install sudo
# Install some dependencies
RUN apt-get install -y \
sudo \
git
git \
curl \
bash \
vim \
systemctl
COPY ./install.sh install.sh
RUN bash ./install.sh
RUN ./install.sh
USER birdnet
EXPOSE 443

View File

@ -0,0 +1,5 @@
CREATE DATABASE IF NOT EXISTS birdnet_default;
CREATE DATABASE IF NOT EXISTS birdnet_observations;
GRANT ALL ON `birdnet_observations`.* TO 'birdnet'@'%' IDENTIFIED BY 'secret';
GRANT ALL ON `birdnet_default`.* TO 'birdnet'@'%' IDENTIFIED BY 'secret';

31
docker/nginx/Dockerfile Normal file
View File

@ -0,0 +1,31 @@
FROM nginx
ARG SERVER_NAME
ARG PROJECT_ROOT
ARG SYMFONY_PUBLIC
ARG CHARTS_DIR
ARG RECORDS_DIR
ARG PHP_FPM_HOST
ARG PHP_FPM_PORT
RUN apt-get update && apt-get upgrade -y \
&& apt-get install -y nginx-full
USER root
COPY "docker/nginx/nginx.conf.template" "/etc/nginx/sites-available/birdnet.conf"
RUN ln -s /etc/nginx/sites-available/birdnet.conf /etc/nginx/sites-enabled/birdnet.conf
RUN sed -i "s|<SERVER_NAME>|${SERVER_NAME}|g" /etc/nginx/sites-available/birdnet.conf \
&& sed -i "s|<PHP_FPM_HOST>|${PHP_FPM_HOST}|g" /etc/nginx/sites-available/birdnet.conf \
&& sed -i "s|<PHP_FPM_PORT>|${PHP_FPM_PORT}|g" /etc/nginx/sites-available/birdnet.conf \
&& sed -i "s|<SYMFONY_PUBLIC>|${SYMFONY_PUBLIC}|g" /etc/nginx/sites-available/birdnet.conf \
&& sed -i "s|<RECORDS_DIR>|${RECORDS_DIR}|g" /etc/nginx/sites-available/birdnet.conf \
&& sed -i "s|<CHARTS_DIR>|${CHARTS_DIR}|g" /etc/nginx/sites-available/birdnet.conf
RUN mkdir -p /etc/nginx/certs/birdnet
WORKDIR /etc/nginx/certs/birdnet
RUN openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out fullchain.pem -sha256 -days 365 -nodes --subj "/CN=${SERVER_NAME}"
RUN sed -i "s|<CERTIFICATE>|/etc/nginx/certs/birdnet/fullchain.pem|g" /etc/nginx/sites-available/birdnet.conf \
&& sed -i "s|<PRIVATE_KEY>|/etc/nginx/certs/birdnet/privkey.pem|g" /etc/nginx/sites-available/birdnet.conf
EXPOSE 443
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

View File

@ -0,0 +1,52 @@
server {
listen 80;
server_name <SERVER_NAME>;
location / {
return 302 https://$host$request_uri;
}
location /.well-known/acme-challenge {
alias /var/www/html/.well-known/acme-challenge;
allow all;
}
}
server {
listen 443 ssl;
server_name <SERVER_NAME>;
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
root <SYMFONY_PUBLIC>;
ssl_certificate <CERTIFICATE>;
ssl_certificate_key <PRIVATE_KEY>;
index index.php;
location / {
try_files $uri $uri/ /index.php$is_args$args;
}
location ^~ /media/records {
autoindex on;
alias <RECORDS_DIR>;
}
location ^~ /media/charts {
autoindex on;
alias <CHARTS_DIR>;
}
location ~ \.php$ {
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass <PHP_FPM_HOST>:<PHP_FPM_PORT>;
fastcgi_index index.php;
include fastcgi.conf;
}
access_log /var/log/nginx/birdnet-access.log;
error_log /var/log/nginx/birdnet-error.log error;
}

View File

@ -0,0 +1,8 @@
ARG PHP_VERSION=${PHP_VERSION:-8.1}
FROM php:${PHP_VERSION}-fpm
RUN apt-get update && apt-get upgrade -y
RUN docker-php-ext-install pdo pdo_mysql
EXPOSE 9000

View File

@ -1,16 +1,25 @@
# Recording container for BirdNET-stream
# Reference: https://leimao.github.io/blog/Docker-Container-Audio/
# References:
# - https://leimao.github.io/blog/Docker-Container-Audio/
# - https://askubuntu.com/questions/972510/how-to-set-alsa-default-device-to-pulseaudio-sound-server-on-docker
FROM debian:bullseye
ENV DEBIAN_FRONTEND noninteractive
# Install packages dependencies
RUN apt-get update && \
apt-get install apt-utils \
&& apt-get install -y --no-install-recommends \
RUN apt-get update && apt-get upgrade -y \
&& apt-get install -y \
--no-install-recommends \
libasound2 \
alsa-utils \
libsndfile1-dev \
&& apt-get install -y ffmpeg \
&& apt-get clean
RUN mkdir -p /opt/birdnet/
WORKDIR /opt/birdnet/
COPY config ./config
COPY daemon/birdnet_recording.sh /usr/local/bin/birdnet_recording.sh
ENTRYPOINT ["/usr/local/bin/birdnet_recording.sh"]

75
docker/symfony/Dockerfile Normal file
View File

@ -0,0 +1,75 @@
ARG PHP_VERSION=${PHP_VERSION:-8.1}
FROM php:${PHP_VERSION}
ARG PROJECT_ROOT
ARG NODE_VERSION
ARG RECORDS_DIR
ARG CHARTS_DIR
ARG DATABASE_DEFAULT_URL
ARG DATABASE_OBSERVATIONS_URL
ENV PHP_VERSION=${PHP_VERSION:-8.1} \
NODE_VERSION=${NODE_VERSION:-16.17.0} \
PROJECT_ROOT=${PROJECT_ROOT:-/opt/birdnet} \
RECORDS_DIR=${RECORDS_DIR:-/media/data/birdnet/records} \
CHARTS_DIR=${CHARTS_DIR:-/media/data/birdnet/charts} \
DATABASE_DEFAULT_URL=${DATABASE_DEFAULT_URL:-mysql://birdnet:secret@birdnet_database/birdnet} \
DATABASE_OBSERVATIONS_URL=${DATABASE_OBSERVATIONS_URL:-mysql://birdnet:secret@birdnet_database/birdnet_observations}
ENV APP_ENV=${APP_ENV:-prod}
ENV APP_DEBUG=${APP_DEBUG:-0}
# RUN apt-get update && apt-get upgrade -y \
# && apt-get install -y \
# curl \
# zip \
# unzip \
# zlib1g-dev \
# libzip-dev \
# git \
# vim \
# && apt-get clean
# RUN docker-php-ext-install zip pdo_mysql
# # Install composer
# RUN php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" \
# && php composer-setup.php --install-dir=/usr/local/bin --filename=composer
# # Install nodejs and npm
# ENV NVM_DIR="/usr/local/nvm"
# RUN mkdir ${NVM_DIR}
# RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
# RUN . "$NVM_DIR/nvm.sh" \
# && nvm install ${NODE_VERSION} \
# && nvm use ${NODE_VERSION} \
# && nvm alias default ${NODE_VERSION} \
# && npm install -g yarn
# ENV PATH="$PATH:/usr/local/nvm/versions/node/v${NODE_VERSION}/bin"
# Change permissions for the home folder of www-data (used for composer cache)
RUN chown -R www-data:www-data /var/www
COPY . ${PROJECT_ROOT}
WORKDIR ${PROJECT_ROOT}/www
RUN rm -rf {vendor,node_modules}
RUN chown -R www-data:www-data .
USER www-data
# Setup .env
RUN cp .env.local.example .env.local
RUN sed -i "s/^APP_ENV=.*/APP_ENV=prod/g" .env.local \
&& sed -i "s/^APP_DEBUG=.*/APP_DEBUG=0/g" .env.local \
&& sed -i "s/^APP_SECRET=.*/APP_SECRET=${APP_SECRET}/g" .env.local \
&& sed -i "s|^DATABASE_DEFAULT_URL=.*|DATABASE_DEFAULT_URL=${DATABASE_DEFAULT_URL}|g" .env.local \
&& sed -i "s|^DATABASE_OBSERVATIONS_URL=.*|DATABASE_OBSERVATIONS_URL=${DATABASE_OBSERVATIONS_URL}|g" .env.local \
&& sed -i "s|^RECORDS_DIR=.*|RECORDS_DIR=${RECORDS_DIR}|g" .env.local \
&& sed -i "s|^CHARTS_DIR=.*|CHARTS_DIR=${CHARTS_DIR}|g" .env.local
# # # Install yarn dependencies
# RUN . "$NVM_DIR/nvm.sh" && yarn install && yarn build
# # Install composer dependencies
# RUN composer install --no-interaction --prefer-dist --optimize-autoloader
# RUN composer dump-env prod
# RUN composer dump-autoload

View File

58
docs/DATABASE.md Normal file
View File

@ -0,0 +1,58 @@
# Setting up the database
There is two database managment systems available: sqlite or mariadb (mysql).
## sqlite
To use sqlite, simply install the sqlite3 package, if it is not already installed on the machine that runs BirdNET-stream.
```bash
sudo apt-get install sqlite3
```
Then fill `config/birdnet.conf` with the proper DATABASE value (you may use any database location):
```bash
DATABASE="./var/db.sqlite"
```
## mariadb
To use mariadb, you need to install the mariadb-server package.
```bash
sudo apt-get install mariadb-server
```
Then, populate the `config/birdnet.conf` file with the proper DATABASE uri:
```bash
DATABASE="mysql://user:password@localhost/birdnet_observations"
```
## Symfony configuration
For both method you need to adapt the file `www/.env.local` to suit your new configuration.
```bash
cd www
# If .env.local does not exists:
cp .env.local.example .env.local
```
```text
# .env.local
# for sqlite (example)
DATABASE_DEFAULT_URL=sqlite:///%kernel.project_dir%/./var/db-default.sqlite
DATABASE_OBSERVATIONS_URL=sqlite:///%kernel.project_dir%/../var/db.sqlite
# for mariadb (example)
DATABASE_DEFAULT_URL=mysql://user:password@localhost/birdnet_default
DATABASE_OBSERVATIONS_URL=mysql://user:password@localhost/birdnet_observations
```
## PHP modules
For symfony to work, make sure you have the required modules according to each method:
- pdo_sqlite
- pdo_mysql

54
docs/DOCKER.md Normal file
View File

@ -0,0 +1,54 @@
# Use docker to run BirdNET-stream
There are two ways to run BirdNET-stream using docker: a "all in one" container, running all services on the same container, or using a splitted approach, running each service on a separate container.
## Prerequisites
- docker
- docker-compose (for splitted approach)
- git
## Using the all in one container (not working yet)
The all in one container is a container that runs all services on the same container.
You can follow the instructions in [./docker/all/README.md](./docker/all/README.md) to create this container.
## Using the splitted approach (recommended)
The splitted approach uses docker-compose and a docker container for each service.
This is the recommended approach to run BirdNET-stream while using docker.
First of of all, you need to clone the repository.
```bash
mkdir ~/Documents/BirdNET-stream
cd ~/Documents/BirdNET-stream
git clone -b main https://github.com/UncleSamulus/BirdNET-stream.git .
```
Then, create your dotenv file and populate it with your own configuration (for instance, generate random passwords and add them to .env credentials):
```bash
cp .env.example .env
```
You may need to adapt the listening ports of the services or other configuration parameters.
In general all variables stated with ${VARIABLE:-default} inside [../docker-compose.yml](../docker-compose.yml) can be override in the .env file using `VARIABLE=value`.
Once that is done, you can build and start docker services:
```bash
# Build images (first time only)
docker compose build
# Run
docker compose up # add `-d`, to run in background
# Stop
docker compose down
```
For a one liner:
```bash
docker compose up --build
```

View File

@ -48,7 +48,7 @@ install_birdnetstream() {
# Clone BirdNET-stream
cd "$WORKDIR"
debug "Cloning BirdNET-stream from $REPOSITORY"
git clone -b "$BRANCH"--recurse-submodules "$REPOSITORY" .
git clone -b "$BRANCH" --recurse-submodules "$REPOSITORY" .
debug "Creating python3 virtual environment $PYTHON_VENV"
python3 -m venv $PYTHON_VENV
debug "Activating $PYTHON_VENV"
@ -67,7 +67,7 @@ install_birdnetstream_services() {
DIR="$WORKDIR"
cd "$WORKDIR"
debug "Setting up BirdNET stream systemd services"
services="birdnet_recording.service birdnet_analyzis.service birdnet_miner.timer birdnet_miner.service birdnet_plotter.service birdnet_plotter.timer"
services="birdnet_recording.service birdnet_analyzis.service birdnet_plotter.service birdnet_plotter.timer"
read -r -a services_array <<<"$services"
for service in ${services_array[@]}; do
sudo cp "daemon/systemd/templates/$service" "/etc/systemd/system/"
@ -78,7 +78,7 @@ install_birdnetstream_services() {
done
sudo sed -i "s|<VENV>|$WORKDIR/$PYTHON_VENV|g" "/etc/systemd/system/birdnet_plotter.service"
sudo systemctl daemon-reload
enabled_services="birdnet_recording.service birdnet_analyzis.service birdnet_miner.timer birdnet_plotter.timer"
enabled_services="birdnet_recording.service birdnet_analyzis.service birdnet_plotter.timer"
read -r -a services_array <<<"$services"
for service in ${services_array[@]}; do
debug "Enabling $service"
@ -163,11 +163,11 @@ setup_http_server() {
fi
debug "Enable birdnet.lan domain"
sudo ln -s /etc/nginx/sites-available/birdnet-stream.conf /etc/nginx/sites-enabled/birdnet-stream.conf
debug "Info: Please edit /etc/nginx/sites-available/birdnet-stream.conf to set the correct server name and paths"
debug "INFO: Please edit /etc/nginx/sites-available/birdnet-stream.conf to set the correct server name and paths"
debug "Setup nginx variables the best way possible"
sudo sed -i "s|<SYMFONY_PUBLIC>|$WORKDIR/www/public/|g" /etc/nginx/sites-available/birdnet-stream.conf
sudo sed -i "s|<RECORDS_FOLDER>|$CHUNK_FOLDER/out|g" /etc/nginx/sites-available/birdnet-stream.conf
sudo sed -i "s|<CHARTS_FOLDER>|$WORKDIR/var/charts|g" /etc/nginx/sites-available/birdnet-stream.conf
sudo sed -i "s|<RECORDS_DIR>|$CHUNK_FOLDER/out|g" /etc/nginx/sites-available/birdnet-stream.conf
sudo sed -i "s|<CHARTS_DIR>|$WORKDIR/var/charts|g" /etc/nginx/sites-available/birdnet-stream.conf
debug "Generate self signed certificate"
CERTS_LOCATION="/etc/nginx/certs/birdnet"
sudo mkdir -p "$CERTS_LOCATION"
@ -219,6 +219,8 @@ main() {
install_web_interface
setup_http_server
install_config
debug "Run loginctl enable-linger for $USER"
loginctl enable-linger
update_permissions
debug "Installation done"
}

View File

@ -27,6 +27,7 @@ uninstall_birdnet_services() {
sudo systemctl stop "$service"
sudo systemctl disable "$service"
sudo rm -f "/etc/systemd/system/$service"
sudo systemctl daemon-reload
done
debug "Done removing systemd services"
}
@ -36,6 +37,13 @@ uninstall_webapp() {
debug "Removing nginx server configuration"
sudo unlink /etc/nginx/sites-enabled/birdnet-stream.conf
sudo systemctl restart nginx
debug "Removing webapp directory"
sudo rm -rf $WORKDIR
}
}
main() {
echo "WARNING: This will remove all BirdNET-stream related files and services. \
Note that it may forget some special configuration."
uninstall_webapp
uninstall_birdnet_services
}
main

1
www/.gitignore vendored
View File

@ -8,7 +8,6 @@
/public/bundles/
/var/
/vendor/
yarn.lock
###< symfony/framework-bundle ###
###> symfony/webpack-encore-bundle ###

42
www/composer.lock generated
View File

@ -174,26 +174,27 @@
},
{
"name": "doctrine/collections",
"version": "1.6.8",
"version": "1.7.0",
"source": {
"type": "git",
"url": "https://github.com/doctrine/collections.git",
"reference": "1958a744696c6bb3bb0d28db2611dc11610e78af"
"reference": "07d15c8a766e664ec271ae84e5dfc597aeeb03b1"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/doctrine/collections/zipball/1958a744696c6bb3bb0d28db2611dc11610e78af",
"reference": "1958a744696c6bb3bb0d28db2611dc11610e78af",
"url": "https://api.github.com/repos/doctrine/collections/zipball/07d15c8a766e664ec271ae84e5dfc597aeeb03b1",
"reference": "07d15c8a766e664ec271ae84e5dfc597aeeb03b1",
"shasum": ""
},
"require": {
"doctrine/deprecations": "^0.5.3 || ^1",
"php": "^7.1.3 || ^8.0"
},
"require-dev": {
"doctrine/coding-standard": "^9.0",
"phpstan/phpstan": "^0.12",
"phpstan/phpstan": "^1.4.8",
"phpunit/phpunit": "^7.5 || ^8.5 || ^9.1.5",
"vimeo/psalm": "^4.2.1"
"vimeo/psalm": "^4.22"
},
"type": "library",
"autoload": {
@ -237,22 +238,22 @@
],
"support": {
"issues": "https://github.com/doctrine/collections/issues",
"source": "https://github.com/doctrine/collections/tree/1.6.8"
"source": "https://github.com/doctrine/collections/tree/1.7.0"
},
"time": "2021-08-10T18:51:53+00:00"
"time": "2022-08-18T05:44:45+00:00"
},
{
"name": "doctrine/common",
"version": "3.3.0",
"version": "3.3.1",
"source": {
"type": "git",
"url": "https://github.com/doctrine/common.git",
"reference": "c824e95d4c83b7102d8bc60595445a6f7d540f96"
"reference": "6a76bd25b1030d35d6ba2bf2f69ca858a41fc580"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/doctrine/common/zipball/c824e95d4c83b7102d8bc60595445a6f7d540f96",
"reference": "c824e95d4c83b7102d8bc60595445a6f7d540f96",
"url": "https://api.github.com/repos/doctrine/common/zipball/6a76bd25b1030d35d6ba2bf2f69ca858a41fc580",
"reference": "6a76bd25b1030d35d6ba2bf2f69ca858a41fc580",
"shasum": ""
},
"require": {
@ -261,6 +262,7 @@
},
"require-dev": {
"doctrine/coding-standard": "^9.0",
"doctrine/collections": "^1",
"phpstan/phpstan": "^1.4.1",
"phpstan/phpstan-phpunit": "^1",
"phpunit/phpunit": "^7.5.20 || ^8.5 || ^9.0",
@ -313,7 +315,7 @@
],
"support": {
"issues": "https://github.com/doctrine/common/issues",
"source": "https://github.com/doctrine/common/tree/3.3.0"
"source": "https://github.com/doctrine/common/tree/3.3.1"
},
"funding": [
{
@ -329,20 +331,20 @@
"type": "tidelift"
}
],
"time": "2022-02-05T18:28:51+00:00"
"time": "2022-08-20T10:48:54+00:00"
},
{
"name": "doctrine/dbal",
"version": "3.4.0",
"version": "3.4.2",
"source": {
"type": "git",
"url": "https://github.com/doctrine/dbal.git",
"reference": "118a360e9437e88d49024f36283c8bcbd76105f5"
"reference": "22de295f10edbe00df74f517612f1fbd711131e2"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/doctrine/dbal/zipball/118a360e9437e88d49024f36283c8bcbd76105f5",
"reference": "118a360e9437e88d49024f36283c8bcbd76105f5",
"url": "https://api.github.com/repos/doctrine/dbal/zipball/22de295f10edbe00df74f517612f1fbd711131e2",
"reference": "22de295f10edbe00df74f517612f1fbd711131e2",
"shasum": ""
},
"require": {
@ -424,7 +426,7 @@
],
"support": {
"issues": "https://github.com/doctrine/dbal/issues",
"source": "https://github.com/doctrine/dbal/tree/3.4.0"
"source": "https://github.com/doctrine/dbal/tree/3.4.2"
},
"funding": [
{
@ -440,7 +442,7 @@
"type": "tidelift"
}
],
"time": "2022-08-06T20:35:57+00:00"
"time": "2022-08-21T14:21:06+00:00"
},
{
"name": "doctrine/deprecations",

View File

@ -1,6 +1,6 @@
server {
listen 80;
server_name birdnet.lab.home;
server_name <SERVER_NAME>;
location / {
return 302 https://$host$request_uri;
@ -14,7 +14,7 @@ server {
server {
listen 443 ssl;
server_name birdnet.lab.home;
server_name <SERVER_NAME>;
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
@ -32,12 +32,12 @@ server {
location ^~ /media/records {
autoindex on;
alias <RECORDS_FOLDER>;
alias <RECORDS_DIR>;
}
location ^~ /media/charts {
autoindex on;
alias <CHARTS_FOLDER>;
alias <CHARTS_DIR>;
}
location ~ \.php$ {

View File

@ -1,4 +1,5 @@
<?php
namespace App\Controller;
use Symfony\Component\HttpFoundation\Response;
@ -6,81 +7,124 @@ use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use App\AppBundle\ConnectionObservations;
use Psr\Log\LoggerInterface;
use Symfony\Component\HttpKernel\Log\Logger;
class HomeController extends AbstractController
{
private ConnectionObservations $connection;
private LoggerInterface $logger;
public function __construct(ConnectionObservations $connection)
public function __construct(ConnectionObservations $connection, LoggerInterface $logger)
{
$this->connection = $connection;
$this->logger = $logger;
}
/**
* @Route("", name="home")
* @Route("/{_locale<%app.supported_locales%>}/", name="home_i18n")
*/
public function index()
public function index(Request $request)
{
$date = $request->get("on");
if ($date == null) {
$date = date("Y-m-d");
}
return $this->render('index.html.twig', [
"stats" => $this->get_stats(),
"charts" => $this->last_chart_generated(),
"stats" => $this->get_stats($date),
"charts" => $this->last_chart_generated($date),
]);
}
/**
* @Route("/about", name="about")
* @Route("/{_locale<%app.supported_locales%>}/about", name="about_i18n")
*/
public function about()
{
return $this->render('about/index.html.twig', [
]);
return $this->render('about/index.html.twig', []);
}
private function get_stats()
private function get_stats($date)
{
$stats = array();
$stats["most-recorded-species"] = $this->get_most_recorded_species();
$stats["last-detected-species"] = $this->get_last_recorded_species();
$stats["number-of-species-detected"] = $this->get_number_of_species_detected($date);
return $stats;
}
private function get_most_recorded_species()
private function get_most_recorded_species()
{
$species = [];
$sql = "SELECT `scientific_name`, `common_name`, COUNT(*) AS contact_count
FROM `taxon`
INNER JOIN `observation`
ON `taxon`.`taxon_id` = `observation`.`taxon_id`
ORDER BY `contact_count` DESC LIMIT 1";
$stmt = $this->connection->prepare($sql);
$result = $stmt->executeQuery();
$species = $result->fetchAllAssociative();
return $species[0];
try {
$stmt = $this->connection->prepare($sql);
$result = $stmt->executeQuery();
$species = $result->fetchAllAssociative()[0];
} catch (\Exception $e) {
$this->logger->error($e->getMessage());
}
return $species;
}
private function get_last_recorded_species()
private function get_last_recorded_species()
{
$species = [];
$sql = "SELECT `scientific_name`, `common_name`, `date`, `audio_file`, `confidence`
FROM `observation`
INNER JOIN `taxon`
ON `observation`.`taxon_id` = `taxon`.`taxon_id`
ORDER BY `date` DESC LIMIT 1";
$stmt = $this->connection->prepare($sql);
$result = $stmt->executeQuery();
$species = $result->fetchAllAssociative();
return $species[0];
try {
$stmt = $this->connection->prepare($sql);
$result = $stmt->executeQuery();
$species = $result->fetchAllAssociative()[0];
} catch (\Exception $e) {
$this->logger->error($e->getMessage());
}
return $species;
}
private function last_chart_generated() {
private function get_number_of_species_detected($date)
{
$count = 0;
$sql = "SELECT COUNT(`taxon_id`) AS contact_count
FROM `observation`
WHERE STRFTIME('%Y-%m-%d', `date`) = :date
GROUP BY `taxon_id`";
try {
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(":date", $date);
$result = $stmt->executeQuery();
$output = $result->fetchAllAssociative();
if ($output != null) {
$count = $output[0]["contact_count"];
}
} catch (\Exception $e) {
$this->logger->error($e->getMessage());
}
return $count;
}
private function last_chart_generated()
{
$files = glob($this->getParameter('kernel.project_dir') . '/../var/charts/*.png');
usort($files, function($a, $b) {
return filemtime($b) - filemtime($a);
});
$last_chart = basename(array_pop($files));
return $last_chart;
if (count($files) > 0) {
usort($files, function ($a, $b) {
return filemtime($a) - filemtime($b);
});
$last_chart = basename(array_pop($files));
return $last_chart;
} else {
$this->logger->info("No charts found");
return "";
}
}
}
}

View File

@ -1,39 +0,0 @@
<?php
namespace App\Controller;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
class LogsController extends AbstractController
{
private $allowed_services = "recording analyzis miner plotter";
/**
* @Route("/logs/{service}", name="logs")
* @Route("/{_locale<%app.supported_locales%>}/logs/{service}", name="logs_i18n")
*/
public function logs($service = "all")
{
$logs = "";
if ($service === "all") {
foreach (explode(" ", $this->allowed_services) as $service) {
$logs .= $this->journal_logs($service);
}
} else if (str_contains($this->allowed_services, $service)) {
$logs .= $this->journal_logs($service);
} else {
return new Response("Service not found", Response::HTTP_BAD_REQUEST);
}
return $this->render('logs/logs.html.twig', [
'logs' => $logs
]);
}
private function journal_logs($service)
{
$logs = shell_exec("journalctl -u birdnet_recording -n 10");
return $logs;
}
}

View File

@ -6,15 +6,20 @@ use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use App\AppBundle\ConnectionObservations;
use Psr\Log\LoggerInterface;
class TodayController extends AbstractController
{ private ConnectionObservations $connection;
{
private ConnectionObservations $connection;
private LoggerInterface $logger;
public function __construct(ConnectionObservations $connection)
public function __construct(ConnectionObservations $connection, LoggerInterface $logger)
{
$this->connection = $connection;
$this->logger = $logger;
}
/**
* @Route("/today", name="today")
* @Route("/{_locale<%app.supported_locales%>}/today", name="today_i18n")
@ -88,28 +93,38 @@ class TodayController extends AbstractController
private function recorded_species_by_date($date)
{
$recorded_species = [];
$sql = "SELECT `taxon`.`taxon_id`, `scientific_name`, `common_name`, COUNT(*) AS `contact_count`, MAX(`confidence`) AS max_confidence
FROM observation
INNER JOIN taxon
ON observation.taxon_id = taxon.taxon_id
WHERE strftime('%Y-%m-%d', `observation`.`date`) = :date
GROUP BY observation.taxon_id";
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(':date', $date);
$result = $stmt->executeQuery();
return $result->fetchAllAssociative();
try {
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(':date', $date);
$result = $stmt->executeQuery();
$recorded_species = $result->fetchAllAssociative();
} catch (\Exception $e) {
$this->logger->error($e->getMessage());
}
return $recorded_species;
}
private function recorded_species_by_id_and_date($id, $date)
{
/* Get taxon even if there is no record this date */
$sql = "SELECT * FROM `taxon` WHERE `taxon_id` = :id";
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(':id', $id);
$result = $stmt->executeQuery();
$taxon = $result->fetchAllAssociative()[0];
if (!$taxon) {
return [];
$taxon = [];
$stat = [];
$records = [];
try {
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(':id', $id);
$result = $stmt->executeQuery();
$taxon = $result->fetchAllAssociative()[0];
} catch (\Exception $e) {
$this->logger->error($e->getMessage());
}
/* Get daily stats */
$sql = "SELECT COUNT(*) AS `contact_count`, MAX(`confidence`) AS `max_confidence`
@ -118,33 +133,47 @@ class TodayController extends AbstractController
ON `taxon`.`taxon_id` = `observation`.`taxon_id`
WHERE strftime('%Y-%m-%d', `observation`.`date`) = :date
AND `observation`.`taxon_id` = :id";
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(':id', $id);
$stmt->bindValue(':date', $date);
$result = $stmt->executeQuery();
$stat = $result->fetchAllAssociative();
try {
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(':id', $id);
$stmt->bindValue(':date', $date);
$result = $stmt->executeQuery();
$stat = $result->fetchAllAssociative();
} catch (\Exception $e) {
$this->logger->error($e->getMessage());
}
$sql = "SELECT * FROM `observation`
WHERE `taxon_id` = :id
AND strftime('%Y-%m-%d', `observation`.`date`) = :date
ORDER BY `observation`.`date` ASC";
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(':id', $id);
$stmt->bindValue(':date', $date);
$result = $stmt->executeQuery();
$records = $result->fetchAllAssociative();
try {
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(':id', $id);
$stmt->bindValue(':date', $date);
$result = $stmt->executeQuery();
$records = $result->fetchAllAssociative();
} catch (\Exception $e) {
$this->logger->error($e->getMessage());
}
return array("taxon" => $taxon, "stat" => $stat, "records" => $records);
}
private function best_confidence_today($id, $date)
{
$best_confidence = 0;
$sql = "SELECT MAX(`confidence`) AS confidence
FROM `observation`
WHERE strftime('%Y-%m-%d', `observation`.`date`) = :date
AND `taxon_id` = :id";
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(':id', $id);
$stmt->bindValue(':date', $date);
$result = $stmt->executeQuery();
return $result->fetchAllAssociative();
try {
$stmt = $this->connection->prepare($sql);
$stmt->bindValue(':id', $id);
$stmt->bindValue(':date', $date);
$result = $stmt->executeQuery();
$result->fetchAllAssociative();
} catch (\Exception $e) {
$this->logger->error($e->getMessage());
}
return $best_confidence;
}
}

View File

@ -41,10 +41,9 @@
<li class="dropdown">
<span class="dropdown-toggle">{{ "Tools"|trans }}</span>
<ul class="dropdown-content">
{% include 'utils/nav-item.html.twig' with {
route: 'logs',
text: 'View Logs'|trans
} %}
<li><a href="/ttyd">
{{ "Logs"|trans }}
</a></li>
{% include 'utils/nav-item.html.twig' with {
route: 'services_status',
text: 'Status'|trans

View File

@ -1,50 +1,78 @@
<div id="stats">
<h2>{{ "Quick Stats" | trans }}</h2>
<ul>
<li class="most-recorded-species">
{{ "Most recorded species" | trans }}:
{% if stats["most-recorded-species"] is defined %}
<span class="scientific-name">
{{ stats["most-recorded-species"]["scientific_name"] }}
</span>
(<span class="common_name">{{ stats["most-recorded-species"]["common_name"] }}</span>)
{{ "with" | trans }}
<span class="observation-count">
{{ stats["most-recorded-species"]["contact_count"] }}
</span>
{{ "contacts" | trans }}.
{% else %}
{{ "No species in database." | trans }}
{% endif %}
</li>
<li class="last-recorded-species">
{{ "Last detected species" | trans }}:
{% if stats["last-detected-species"] is defined %}
<span class="scientific-name">
{{ stats["last-detected-species"]["scientific_name"] }}
</span>
(<span class="common_name">{{ stats["last-detected-species"]["common_name"] }}</span>)
{{ "with" | trans }}
<span class="confidence">
{{ stats["last-detected-species"]["confidence"] }}
</span>
{{ "AI confidence" | trans }}
<span class="datetime">
{% set date = stats["last-detected-species"]["date"] %}
{% if date | date("Y-m-d") == "now" | date("Y-m-d") %}
{{ "today" | trans }}
{% else %}
{{ "on" | trans }}
{{ date | format_datetime("full", "none") }}
{% endif %}
at
<span class="time">
{{ date | date("H:i") }}
</span>
</span>.
{% else %}
{{ "No species in database" | trans }}
{% endif %}
</li>
</ul>
</div>
<h2>
{{ 'Quick Stats'|trans }}
</h2>
<ul>
<li class="stat">
{{ 'Most recorded species'|trans }}:{% if
stats['most-recorded-species'] is defined
and (stats['most-recorded-species']|length) > 0 %}
<span class="scientific-name">
{{ stats['most-recorded-species']['scientific_name'] }}
</span>
(<span class="common_name">
{{ stats['most-recorded-species']['common_name'] }}
</span>)
{{ 'with'|trans }}
<span class="observation-count">
{{ stats['most-recorded-species']['contact_count'] }}
</span>
{{ 'contacts'|trans }}.
{% else %}
{{ 'No species in database.'|trans }}
{% endif %}
</li>
<li class="stat">
{{ 'Last detected species'|trans }}:{% if
stats['last-detected-species'] is defined
and (stats['last-detected-species']|length) > 0 %}
<span class="scientific-name">
{{ stats['last-detected-species']['scientific_name'] }}
</span>
(<span class="common_name">
{{ stats['last-detected-species']['common_name'] }}
</span>)
{{ 'with'|trans }}
<span class="confidence">
{{ stats['last-detected-species']['confidence'] }}
</span>
{{ 'AI confidence'|trans }}
<span class="datetime">
{% set date = stats['last-detected-species']['date'] %}
{% if (date|date('Y-m-d')) == ('now'|date('Y-m-d')) %}
{{ 'today'|trans }}
{% else %}
{{ 'on'|trans }}
{{ date|format_datetime('full', 'none') }}
{% endif %}at
<span class="time">{{ date|date('H:i') }}</span>
</span>.
{% else %}
{{ 'No species in database'|trans }}
{% endif %}
</li>
<li class="stat">
{% set today = 'now'|date('Y-m-d') %}
{% set date = app.request.get('on') %}
{% if
stats['number-of-species-detected'] is defined
and stats['number-of-species-detected'] > 0 %}
{% if today == date %}
{{ 'Number of species detected today: '|trans }}
{% else %}
{{ 'Number of species detected on '|trans }}
{{ date|format_datetime('full', 'none') }}:
{% endif %}
<span>{{ stats['number-of-species-detected'] }}</span>.
{% else %}
{# {{ 'No species detected today'|trans }} #}
{% if today == date %}
{{ 'No species detected today.'|trans }}
{% else %}
{{ 'No species detected on '|trans }}
{{ date|format_datetime('full', 'none') }}
{% endif %}
{% endif %}
</li>
</ul>
</div>