mirror of
https://github.com/tgbot-collection/ytdlbot.git
synced 2024-09-20 06:55:55 +08:00
Add K8S Deloyment and Readme
This commit is contained in:
parent
1b245471dc
commit
c8c04b4321
157
README.md
157
README.md
|
@ -197,6 +197,7 @@ You can also limit CPU and RAM usage by adding an `deploy' key:
|
||||||
|
|
||||||
Be sure to use `--compatibility` when deploying.
|
Be sure to use `--compatibility` when deploying.
|
||||||
|
|
||||||
|
|
||||||
## 4. run
|
## 4. run
|
||||||
|
|
||||||
### 4.1. standalone mode
|
### 4.1. standalone mode
|
||||||
|
@ -231,6 +232,162 @@ docker-compose -f worker.yml up -d
|
||||||
|
|
||||||
**⚠️ Bear in mind don't publish redis directly on the internet! You can use WireGuard to wrap it up.**
|
**⚠️ Bear in mind don't publish redis directly on the internet! You can use WireGuard to wrap it up.**
|
||||||
|
|
||||||
|
## Kubernetes
|
||||||
|
|
||||||
|
Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications
|
||||||
|
|
||||||
|
# Complete deployment guide for k8s deloyment
|
||||||
|
|
||||||
|
* contains every functionality
|
||||||
|
* compatible with amd64, arm64 and armv7l
|
||||||
|
|
||||||
|
## First. Get all file in k8s folder
|
||||||
|
|
||||||
|
Download `k8s` file to a directory on your k8s server and go to this folder
|
||||||
|
|
||||||
|
## 1. Create Redis deloyment
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl apply -f 01.redis.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
This command will create ytdl namespace, redis pod and redis service
|
||||||
|
|
||||||
|
## 2. Creat MariaDB deloyment
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl apply -f 02.mariadb.yml
|
||||||
|
```
|
||||||
|
This deloyment will claim 10GB storage from storageClassName: longhorn. Please replace longhorn with your storageClassName before apply.
|
||||||
|
|
||||||
|
## 3. Set environment variables
|
||||||
|
|
||||||
|
Create configMap for env
|
||||||
|
### 3.1 Edit configmap.yml
|
||||||
|
|
||||||
|
```shell
|
||||||
|
vim 03.configmap.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
you can configure all the following environment variables:
|
||||||
|
|
||||||
|
* PYRO_WORKERS: number of workers for pyrogram, default is 100
|
||||||
|
* WORKERS: workers count for celery
|
||||||
|
* APP_ID: **REQUIRED**, get it from https://core.telegram.org/
|
||||||
|
* APP_HASH: **REQUIRED**
|
||||||
|
* TOKEN: **REQUIRED**
|
||||||
|
* REDIS: **REQUIRED if you need VIP mode and cache** ⚠️ Don't publish your redis server on the internet. ⚠️
|
||||||
|
|
||||||
|
* OWNER: owner username
|
||||||
|
* QUOTA: quota in bytes
|
||||||
|
* EX: quota expire time
|
||||||
|
* MULTIPLY: vip quota comparing to normal quota
|
||||||
|
* USD2CNY: exchange rate
|
||||||
|
* VIP: VIP mode, default: disable
|
||||||
|
* AFD_LINK
|
||||||
|
* COFFEE_LINK
|
||||||
|
* COFFEE_TOKEN
|
||||||
|
* AFD_TOKEN
|
||||||
|
* AFD_USER_ID
|
||||||
|
|
||||||
|
* AUTHORIZED_USER: users that could use this bot, user_id, separated with `,`
|
||||||
|
* REQUIRED_MEMBERSHIP: group or channel username, user must join this group to use the bot. Could be use with
|
||||||
|
above `AUTHORIZED_USER`
|
||||||
|
|
||||||
|
* ENABLE_CELERY: Distribution mode, default: disable. You'll can setup workers in different locations.
|
||||||
|
* ENABLE_FFMPEG: enable ffmpeg so Telegram can stream
|
||||||
|
* MYSQL_HOST: you'll have to setup MySQL if you enable VIP mode
|
||||||
|
* MYSQL_USER
|
||||||
|
* MYSQL_PASS
|
||||||
|
* GOOGLE_API_KEY: YouTube API key, required for YouTube video subscription.
|
||||||
|
* AUDIO_FORMAT: audio format, default is m4a. You can set to any known and supported format for ffmpeg. For
|
||||||
|
example,`mp3`, `flac`, etc. ⚠️ m4a is the fastest. Other formats may affect performance.
|
||||||
|
* ARCHIVE_ID: group or channel id/username. All downloads will send to this group first and then forward to end user.
|
||||||
|
**Inline button will be lost during the forwarding.**
|
||||||
|
|
||||||
|
### 3.2 Apply configMap for environment variables
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl apply -f 03.configmap.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Run Master Celery
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl apply -f 04.ytdl-master.yml
|
||||||
|
```
|
||||||
|
This deloyment will create ytdl-pvc PersistentVolumeClaim on storageClassName: longhorn. This clain will contain vnstat, cookies folder and flower database. Please replace longhorn with your storageClassName before apply
|
||||||
|
|
||||||
|
### 4.1 Setup instagram cookies
|
||||||
|
Required if you want to support instagram.
|
||||||
|
|
||||||
|
You can use this extension
|
||||||
|
[Get cookies.txt](https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid)
|
||||||
|
to get instagram cookies
|
||||||
|
|
||||||
|
Get pod running ytdl master:
|
||||||
|
```shell
|
||||||
|
kubectl get pods --namespace ytdl
|
||||||
|
```
|
||||||
|
Name should be ytdl-xxxxxxxx
|
||||||
|
|
||||||
|
Access to pod
|
||||||
|
```shell
|
||||||
|
kubectl --namespace=ytdl exec --stdin --tty ytdl-xxx -- sh
|
||||||
|
```
|
||||||
|
(replace ytdl-xxx by your pod name)
|
||||||
|
|
||||||
|
Go to ytdl-pvc mounted folder
|
||||||
|
```shell
|
||||||
|
cd /ytdlbot/ytdlbot/data/
|
||||||
|
vim instagram.com_cookies.txt
|
||||||
|
# paste your cookies
|
||||||
|
```
|
||||||
|
## 5. Run Worker Celery
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl apply -f 05.ytdl-worker.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. Run Flower image (OPTIONAL)
|
||||||
|
|
||||||
|
|
||||||
|
### 6.1 Setup flower db
|
||||||
|
|
||||||
|
Get pod running ytdl master:
|
||||||
|
```shell
|
||||||
|
kubectl get pods --namespace ytdl
|
||||||
|
```
|
||||||
|
Name should be ytdl-xxxxxxxx
|
||||||
|
|
||||||
|
Access to pod
|
||||||
|
```shell
|
||||||
|
kubectl --namespace=ytdl exec --stdin --tty ytdl-xxx -- sh
|
||||||
|
```
|
||||||
|
(replace ytdl-xxx by your pod name)
|
||||||
|
|
||||||
|
Go to ytdl-pvc mounted folder
|
||||||
|
```shel
|
||||||
|
cd /var/lib/vnstat/
|
||||||
|
```
|
||||||
|
|
||||||
|
Create flower database file
|
||||||
|
```shell
|
||||||
|
{} ~ python3
|
||||||
|
Python 3.9.9 (main, Nov 21 2021, 03:22:47)
|
||||||
|
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
|
||||||
|
Type "help", "copyright", "credits" or "license" for more information.
|
||||||
|
>>> import dbm;dbm.open("flower","n");exit()
|
||||||
|
```
|
||||||
|
### 6.2 Config Flower Ingress
|
||||||
|
This step need config ingress from line 51 with your ingress service for access from internet
|
||||||
|
YML file should be adjusted depending on your load balancing, ingress and network system
|
||||||
|
### 6.2 Apply Flower deloyment
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl apply -f 06.flower.yml
|
||||||
|
```
|
||||||
|
|
||||||
# Command
|
# Command
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
53
k8s/01.redis.yml
Normal file
53
k8s/01.redis.yml
Normal file
|
@ -0,0 +1,53 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: ytdl
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: redis
|
||||||
|
name: redis
|
||||||
|
namespace: ytdl
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
ytdl: redis
|
||||||
|
strategy: {}
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: redis
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- image: redis:7-alpine
|
||||||
|
name: redis
|
||||||
|
ports:
|
||||||
|
- containerPort: 6379
|
||||||
|
resources: {}
|
||||||
|
restartPolicy: Always
|
||||||
|
status: {}
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: redis
|
||||||
|
name: redis
|
||||||
|
namespace: ytdl
|
||||||
|
spec:
|
||||||
|
ports:
|
||||||
|
- name: "6379"
|
||||||
|
port: 6379
|
||||||
|
targetPort: 6379
|
||||||
|
selector:
|
||||||
|
ytdl: redis
|
||||||
|
status:
|
||||||
|
loadBalancer: {}
|
80
k8s/02.mariadb.yml
Normal file
80
k8s/02.mariadb.yml
Normal file
|
@ -0,0 +1,80 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: mariadb-pvc
|
||||||
|
name: mariadb-pvc
|
||||||
|
namespace: ytdl
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
storageClassName: longhorn
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 10Gi
|
||||||
|
status: {}
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
annotations:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: mariadb
|
||||||
|
name: mariadb
|
||||||
|
namespace: ytdl
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
ytdl: mariadb
|
||||||
|
strategy:
|
||||||
|
type: Recreate
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: mariadb
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- env:
|
||||||
|
- name: MYSQL_ROOT_PASSWORD
|
||||||
|
value: ro0tP4sSworD
|
||||||
|
- name: MYSQL_DATABASE
|
||||||
|
value: ytdl
|
||||||
|
image: mariadb:latest
|
||||||
|
name: mariadb
|
||||||
|
ports:
|
||||||
|
- containerPort: 3306
|
||||||
|
resources: {}
|
||||||
|
volumeMounts:
|
||||||
|
- mountPath: /var/lib/mysql
|
||||||
|
name: "mariadb-persistent-storage"
|
||||||
|
restartPolicy: Always
|
||||||
|
volumes:
|
||||||
|
- name: mariadb-persistent-storage
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: mariadb-pvc
|
||||||
|
status: {}
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: mariadb
|
||||||
|
name: mariadb-svc
|
||||||
|
namespace: ytdl
|
||||||
|
spec:
|
||||||
|
ports:
|
||||||
|
- name: "3306"
|
||||||
|
port: 3306
|
||||||
|
targetPort: 3306
|
||||||
|
selector:
|
||||||
|
ytdl: mariadb
|
||||||
|
status:
|
||||||
|
loadBalancer: {}
|
||||||
|
|
17
k8s/03.configmap.yml
Normal file
17
k8s/03.configmap.yml
Normal file
|
@ -0,0 +1,17 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: ytdlenv
|
||||||
|
namespace: ytdl
|
||||||
|
annotations:
|
||||||
|
data:
|
||||||
|
APP_HASH:
|
||||||
|
APP_ID:
|
||||||
|
TOKEN:
|
||||||
|
ARCHIVE_ID:
|
||||||
|
ENABLE_CELERY: 'True'
|
||||||
|
ENABLE_FFMPEG: 'True'
|
||||||
|
MYSQL_HOST: mariadb-svc
|
||||||
|
MYSQL_PASS: ro0tP4sSworD
|
||||||
|
MYSQL_USER: root
|
||||||
|
REDIS: redis
|
65
k8s/04.ytdl-master.yml
Normal file
65
k8s/04.ytdl-master.yml
Normal file
|
@ -0,0 +1,65 @@
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
name: ytdl-pvc
|
||||||
|
namespace: ytdl
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: ytdl-pvc
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteMany
|
||||||
|
storageClassName: longhorn
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 10Gi
|
||||||
|
status: {}
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: ytdl
|
||||||
|
namespace: ytdl
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: ytdl
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
ytdl: ytdl
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: ytdl
|
||||||
|
spec:
|
||||||
|
volumes:
|
||||||
|
- name: ytdl-pvc
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: ytdl-pvc
|
||||||
|
containers:
|
||||||
|
- name: ytdl
|
||||||
|
image: bennythink/ytdlbot
|
||||||
|
envFrom:
|
||||||
|
- configMapRef:
|
||||||
|
name: ytdlenv
|
||||||
|
resources: {}
|
||||||
|
volumeMounts:
|
||||||
|
- name: ytdl-pvc
|
||||||
|
mountPath: /var/lib/vnstat/
|
||||||
|
subPath: vnstat/
|
||||||
|
- name: ytdl-pvc
|
||||||
|
mountPath: /ytdlbot/ytdlbot/data/
|
||||||
|
subPath: data/
|
||||||
|
terminationMessagePath: /dev/termination-log
|
||||||
|
terminationMessagePolicy: File
|
||||||
|
imagePullPolicy: Always
|
||||||
|
restartPolicy: Always
|
||||||
|
terminationGracePeriodSeconds: 30
|
||||||
|
dnsPolicy: ClusterFirst
|
||||||
|
securityContext: {}
|
||||||
|
schedulerName: default-scheduler
|
||||||
|
status: {}
|
47
k8s/05.ytdl-worker.yml
Normal file
47
k8s/05.ytdl-worker.yml
Normal file
|
@ -0,0 +1,47 @@
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: ytdl-worker
|
||||||
|
name: ytdl-worker
|
||||||
|
namespace: ytdl
|
||||||
|
spec:
|
||||||
|
replicas: 4
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
ytdl: ytdl-worker
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: ytdl-worker
|
||||||
|
spec:
|
||||||
|
volumes:
|
||||||
|
- name: ytdl-pvc
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: ytdl-pvc
|
||||||
|
containers:
|
||||||
|
- name: ytdl-worker
|
||||||
|
image: bennythink/ytdlbot
|
||||||
|
args:
|
||||||
|
- /usr/local/bin/supervisord
|
||||||
|
- '-c'
|
||||||
|
- /ytdlbot/conf/supervisor_worker.conf
|
||||||
|
envFrom:
|
||||||
|
- configMapRef:
|
||||||
|
name: ytdlenv
|
||||||
|
resources: {}
|
||||||
|
volumeMounts:
|
||||||
|
- name: ytdl-pvc
|
||||||
|
mountPath: /ytdlbot/ytdlbot/data/
|
||||||
|
subPath: data/
|
||||||
|
terminationMessagePath: /dev/termination-log
|
||||||
|
terminationMessagePolicy: File
|
||||||
|
imagePullPolicy: Always
|
||||||
|
restartPolicy: Always
|
||||||
|
terminationGracePeriodSeconds: 30
|
||||||
|
dnsPolicy: ClusterFirst
|
||||||
|
securityContext: {}
|
||||||
|
schedulerName: default-scheduler
|
||||||
|
status: {}
|
101
k8s/06.flower.yml
Normal file
101
k8s/06.flower.yml
Normal file
|
@ -0,0 +1,101 @@
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: flower
|
||||||
|
name: flower
|
||||||
|
namespace: ytdl
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
ytdl: flower
|
||||||
|
strategy:
|
||||||
|
type: Recreate
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: flower
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- envFrom:
|
||||||
|
- configMapRef:
|
||||||
|
name: ytdlenv
|
||||||
|
args:
|
||||||
|
- /usr/local/bin/celery
|
||||||
|
- -A
|
||||||
|
- flower_tasks
|
||||||
|
- flower
|
||||||
|
- --basic_auth=bennythink:123456
|
||||||
|
- --address=0.0.0.0
|
||||||
|
- --persistent
|
||||||
|
- --purge_offline_workers=3600
|
||||||
|
image: bennythink/ytdlbot
|
||||||
|
name: flower
|
||||||
|
ports:
|
||||||
|
- containerPort: 5555
|
||||||
|
resources: {}
|
||||||
|
volumeMounts:
|
||||||
|
- name: ytdl-pvc
|
||||||
|
mountPath: /ytdlbot/ytdlbot/flower
|
||||||
|
subPath: vnstat/flower
|
||||||
|
restartPolicy: Always
|
||||||
|
volumes:
|
||||||
|
- name: ytdl-pvc
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: ytdl-pvc
|
||||||
|
status: {}
|
||||||
|
|
||||||
|
# THIS IS OPTION IF YOU WANT PUBLIC FLOWER PAGE TO INTERNET.
|
||||||
|
# should be adjusted depending on your load balancing system machine
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
labels:
|
||||||
|
ytdl: flower
|
||||||
|
name: flower-svc
|
||||||
|
namespace: ytdl
|
||||||
|
spec:
|
||||||
|
type: NodePort
|
||||||
|
ports:
|
||||||
|
- name: "5555"
|
||||||
|
protocol: TCP
|
||||||
|
port: 5555
|
||||||
|
targetPort: 5555
|
||||||
|
selector:
|
||||||
|
ytdl: flower
|
||||||
|
status:
|
||||||
|
loadBalancer: {}
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: Ingress
|
||||||
|
metadata:
|
||||||
|
name: nginx-flower-ingress
|
||||||
|
namespace: ytdl
|
||||||
|
annotations:
|
||||||
|
# cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||||
|
nginx.ingress.kubernetes.io/rewrite-target: /
|
||||||
|
# nginx.ingress.kubernetes.io/whitelist-source-range: 14.161.27.151 limit by ipaddresss
|
||||||
|
|
||||||
|
spec:
|
||||||
|
ingressClassName: nginx
|
||||||
|
tls:
|
||||||
|
- hosts:
|
||||||
|
- your-domain
|
||||||
|
secretName: flower-tls
|
||||||
|
rules:
|
||||||
|
- host: your-domain
|
||||||
|
http:
|
||||||
|
paths:
|
||||||
|
- path: /
|
||||||
|
pathType: Prefix
|
||||||
|
backend:
|
||||||
|
service:
|
||||||
|
name: flower-svc
|
||||||
|
port:
|
||||||
|
number: 5555
|
Loading…
Reference in a new issue