🏗️(ds_proxy) introduce how to use ds_proxy with Drive

We want to add documentation showing how to use DS_Proxy with Drive. With proxy is fully optionnal and is here if you want to an encryption layer between Drive and the object storage.
2026-04-25 17:15:19 +02:00 · 2026-01-15 14:38:36 +01:00
parent 84e7dba52e
commit ca6cbacff3
11 changed files with 185 additions and 7 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,10 @@ and this project adheres to

 ## [Unreleased]

+### Added
+
+- 🏗️ (ds_proxy) introduce how to use ds_proxy with Drive
+
 ## [v0.11.1] - 2026-01-13

 ### Fixed
--- a/compose.yaml
+++ b/compose.yaml
@@ -100,6 +100,17 @@ services:
      onlyoffice:
        condition: service_healthy

+  ds-proxy:
+    image: demarchenumerique/ds-proxy:v2.0.0-alpha.2
+    command: ["proxy", "--address", "0.0.0.0:4444", "--password-file", "/etc/dsproxy/PASSWORD", "--salt", "12345678901234567890123456789012", "--keyring-file", "/etc/dsproxy/keyring.toml", "--upstream-url", "http://minio:9000", "--local-encryption-directory", "/var/tmp/local_encryption/", "--s3-access-key", "drive", "--s3-secret-key", "password", "--s3-region", "us-east-1"]
+    ports:
+      - "4444:4444"
+    environment:
+      - RUST_LOG=debug,ds_proxy::http::handlers::fetch=trace,ds_proxy::http::handlers::forward=trace
+      - RUST_BACKTRACE=full
+    volumes:
+      - ./docker/files/development/ds_proxy/:/etc/dsproxy/:ro
+
  celery-dev:
    user: ${DOCKER_USER:-1000}
    image: drive:backend-development
--- a/docker/files/development/ds_proxy/PASSWORD
+++ b/docker/files/development/ds_proxy/PASSWORD
@@ -0,0 +1 @@
+a good password
--- a/docker/files/development/ds_proxy/keyring.toml
+++ b/docker/files/development/ds_proxy/keyring.toml
@@ -0,0 +1,2 @@
+[keys]
+    0 = "tC8AKtY34zcBArmuwcR8dakK6PhO67pv+5x3gQXG9dYQ+llXpsQZSOFIK8iPmwp3t/sLr8fqhNzNC6yiVObwY6fQDxDSDryD"
--- a/docker/files/development/etc/nginx/conf.d/default.conf
+++ b/docker/files/development/etc/nginx/conf.d/default.conf
@@ -20,6 +20,9 @@ server {
        # Get resource from Minio
        proxy_pass http://minio:9000/drive-media-storage/;
        proxy_set_header Host minio:9000;
+        # To use with ds_proxy
+        # proxy_pass http://ds-proxy:4444/upstream/drive-media-storage/;
+        # proxy_set_header Host ds-proxy:4444;
        add_header Content-Disposition "attachment";
    }

@@ -39,6 +42,9 @@ server {
        # Get resource from Minio
        proxy_pass http://minio:9000/drive-media-storage/;
        proxy_set_header Host minio:9000;
+        # To use with ds_proxy
+        # proxy_pass http://ds-proxy:4444/upstream/drive-media-storage/;
+        # proxy_set_header Host ds-proxy:4444;
    }

    location /media-auth {
--- a/docs/ds_proxy.md
+++ b/docs/ds_proxy.md
@@ -0,0 +1,70 @@
+# DS Proxy
+
+DS Proxy is an encryption HTTP Proxy compatible with the S3 API. Its goal is to encrypt all the objects you upload on your S3 compatible object storage and then decrypt it when you download them.
+
+This solution allow you to safely store the files uploaded on drive by just change the S3 configuration in your settings.
+
+You can find a complete description of this Proxy on its github repository (but in french) : [https://github.com/demarche-numerique/ds_proxy](https://github.com/demarche-numerique/ds_proxy)
+
+By default the compose environment and the development helm chart env do not enable ds_proxy. They are present as an example and can be easily enabled.
+
+### Known issues
+
+Drive is using [boto3](https://boto3.amazonaws.com), the official AWS SDK, as client to manage files with the S3 compatible object storage you decided to use.
+The SDK is using some improvements to enhance performance but they are not compatible with DS Proxy.
+The SDK is able to switch between single or multipart data for both the upload and the download.
+
+To disable this behaviour you have to configure boto3 using environment variable.
+
+For the upload, set these environment variables:
+
+```
+AWS_REQUEST_CHECKSUM_CALCULATION: when_required
+AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
+```
+
+For the download, set this environment variables:
+
+```
+S3_TRANSFER_CONFIG_USE_THREADS: False
+S3_TRANSFER_CONFIG_MULTIPART_THRESHOLD: "10737418240"
+S3_TRANSFER_CONFIG_MULTIPART_CHUNKSIZE: "10737418240"
+S3_TRANSFER_CONFIG_MAX_CONCURRENCY: 1
+```
+
+### Enable ds_proxy with docker compose
+
+You have to change the settings related to S3 in the `env.d/development/common.local` file:
+
+```
+AWS_S3_DOMAIN_REPLACE=http://localhost:4444/upstream
+AWS_S3_ENDPOINT_URL=http://ds-proxy:4444/upstream
+```
+
+You also have to change the nginx config present in `docker/files/development/etc/nginx/conf.d/default.conf`. In the file comment the minio config and uncomment the DS Proxy config. Present twice in `location /media/` and `location /media/preview` blocks:
+
+```
+# Get resource from Minio
+# proxy_pass http://minio:9000/drive-media-storage/;
+# proxy_set_header Host minio:9000;
+# To use with ds_proxy
+proxy_pass http://ds-proxy:4444/upstream/drive-media-storage/;
+proxy_set_header Host ds-proxy:4444;
+```
+
+Then start the django stack running `make run-backend`.
+Finally, start ds_proxy: `docker compose up -d ds-proxy`
+
+That's all, ds_proxy is running and Drive configured to use it. All the file uploaded will be encrypted and then decrypted when you download them.
+
+### Enable ds_proxy with tilt
+
+Once you have an up and running stack with tilt, you can enable ds_proxy and configure drive to use it.
+
+In the `src/helm/helmfile.yaml` file change the `ds_proxy.enabled` value to `true`.
+Then in the `src/helm/env.d/dev/values.drive.yaml.gotmpl` file you will have to comment/uncomment these variables:
+    - `AWS_S3_ENDPOINT_URL`
+    - `nginx.ingress.kubernetes.io/upstream-vhost` (present twice)
+    - `host` (in the `serviceMedia`)
+
+Reloading the tilt stack should deploy DS Proxy.
--- a/docs/env.md
+++ b/docs/env.md
@@ -90,6 +90,10 @@ This document lists all configurable environment variables for the Drive applica
 | `POSTHOG_KEY` | PostHog analytics API key | `None` |
 | `REDIS_URL` | Redis connection URL | `redis://redis:6379/0` |
 | `RESTRICT_UPLOAD_FILE_TYPE` | Boolean to enable or not upload restriction based on file type (extension + mimetype) | `True` |
+| `S3_TRANSFER_CONFIG_MULTIPART_THRESHOLD` | `multipart_threshold` value for the `TransferConfig` configuration | `8388608` (8MB) |
+| `S3_TRANSFER_CONFIG_MULTIPART_CHUNKSIZE` | `multipart_chunksize` value for the `TransferConfig` configuration | `8388608` (8MB) |
+| `S3_TRANSFER_CONFIG_MAX_CONCURRENCY` | `max_concurrency` value for the `TransferConfig` configuration | `10` |
+| `S3_TRANSFER_CONFIG_USE_THREADS` | `use_threads` value for the `TransfertConfig` configuration | `True` |
 | `SEARCH_INDEXER_ALLOWED_MIMETYPES` | Indexable files mimetypes | `["text/"]` |
 | `SEARCH_INDEXER_CLASS` | Class of the backend for item indexation & search ||
 | `SEARCH_INDEXER_BATCH_SIZE` | Size of each batch for indexation of all items | `1000` |
--- a/env.d/development/common
+++ b/env.d/development/common
@@ -23,11 +23,15 @@ DJANGO_EMAIL_PORT=1025

 # Media
 STORAGES_STATICFILES_BACKEND=django.contrib.staticfiles.storage.StaticFilesStorage
-AWS_S3_ENDPOINT_URL=http://minio:9000
 AWS_S3_ACCESS_KEY_ID=drive
 AWS_S3_SECRET_ACCESS_KEY=password
+AWS_S3_REGION_NAME=eu-east-1
 AWS_S3_SIGNATURE_VERSION=s3v4
 AWS_S3_DOMAIN_REPLACE=http://localhost:9000
+AWS_S3_ENDPOINT_URL=http://minio:9000
+# To use with ds-proxy
+# AWS_S3_DOMAIN_REPLACE=http://localhost:4444/upstream
+# AWS_S3_ENDPOINT_URL=http://ds-proxy:4444/upstream
 MEDIA_BASE_URL=http://localhost:8083

 # OIDC
--- a/src/backend/drive/settings.py
+++ b/src/backend/drive/settings.py
@@ -19,6 +19,7 @@ from django.utils.translation import gettext_lazy as _
 import dj_database_url
 import posthog
 import sentry_sdk
+from boto3.s3.transfer import TransferConfig
 from configurations import Configuration, values
 from lasuite.configuration.values import SecretFileValue
 from sentry_sdk.integrations.django import DjangoIntegration
@@ -29,6 +30,10 @@ from sentry_sdk.integrations.django import DjangoIntegration
 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 DATA_DIR = os.environ.get("DATA_DIR", os.path.join("/", "data"))

+KB = 1024
+MB = 1024 * KB
+GB = 1024 * MB
+

 def get_release():
    """
@@ -125,7 +130,7 @@ class Base(Configuration):
        default=50, environ_name="SEARCH_INDEXER_QUERY_LIMIT", environ_prefix=None
    )
    SEARCH_INDEXER_CONTENT_MAX_SIZE = values.PositiveIntegerValue(
-        2 * (2**20),  # 2MB
+        2 * MB,
        environ_name="SEARCH_INDEXER_CONTENT_MAX_SIZE",
        environ_prefix=None,
    )
@@ -150,6 +155,30 @@ class Base(Configuration):
    STORAGES = {
        "default": {
            "BACKEND": "storages.backends.s3.S3Storage",
+            "OPTIONS": {
+                "transfer_config": TransferConfig(
+                    use_threads=values.BooleanValue(
+                        default=True,
+                        environ_name="S3_TRANSFER_CONFIG_USE_THREADS",
+                        environ_prefix=None,
+                    ),
+                    multipart_threshold=values.PositiveIntegerValue(
+                        default=8 * MB,
+                        environ_name="S3_TRANSFER_CONFIG_MULTIPART_THRESHOLD",
+                        environ_prefix=None,
+                    ),
+                    multipart_chunksize=values.PositiveIntegerValue(
+                        default=8 * MB,
+                        environ_name="S3_TRANSFER_CONFIG_MULTIPART_CHUNKSIZE",
+                        environ_prefix=None,
+                    ),
+                    max_concurrency=values.PositiveIntegerValue(
+                        default=10,
+                        environ_name="S3_TRANSFER_CONFIG_MAX_CONCURRENCY",
+                        environ_prefix=None,
+                    ),
+                ),
+            },
        },
        "staticfiles": {
            "BACKEND": values.Value(
@@ -206,7 +235,7 @@ class Base(Configuration):
    # This is used to limit the size of the request body in memory.
    # This also limits the size of the file that can be uploaded to the server.
    DATA_UPLOAD_MAX_MEMORY_SIZE = values.PositiveIntegerValue(
-        2 * (2**30),  # 2GB
+        2 * GB,
        environ_name="DATA_UPLOAD_MAX_MEMORY_SIZE",
        environ_prefix=None,
    )
--- a/src/helm/env.d/dev/values.drive.yaml.gotmpl
+++ b/src/helm/env.d/dev/values.drive.yaml.gotmpl
@@ -57,11 +57,12 @@ backend:
    DB_PORT: 5432
    REDIS_URL: redis://user:pass@dev-backend-redis:6379/1
    DJANGO_CELERY_BROKER_URL: redis://user:pass@dev-backend-redis:6379/1
+    # AWS_S3_ENDPOINT_URL: https://drive-dsproxy.127.0.0.1.nip.io/upstream
    AWS_S3_ENDPOINT_URL: https://drive-minio.127.0.0.1.nip.io
    AWS_S3_ACCESS_KEY_ID: dinum
    AWS_S3_SECRET_ACCESS_KEY: password
-    AWS_STORAGE_BUCKET_NAME: drive-media-storage
    AWS_S3_SIGNATURE_VERSION: s3v4
+    AWS_S3_REGION_NAME: eu-east-1
    STORAGES_STATICFILES_BACKEND: django.contrib.staticfiles.storage.StaticFilesStorage
    MEDIA_BASE_URL: https://drive.127.0.0.1.nip.io
  migrate:
@@ -90,7 +91,7 @@ backend:
  # Extra volume mounts to manage our local custom CA and avoid to set ssl_verify: false
  extraVolumeMounts:
    - name: certs
-      mountPath: /usr/local/lib/python3.12/site-packages/certifi/cacert.pem
+      mountPath: /app/.venv/lib/python3.13/site-packages/certifi/cacert.pem
      subPath: cacert.pem

  # Exra volumes to manage our local custom CA and avoid to set ssl_verify: false
@@ -143,6 +144,7 @@ ingressMedia:
    nginx.ingress.kubernetes.io/auth-url: https://drive.127.0.0.1.nip.io/api/v1.0/items/media-auth/
    nginx.ingress.kubernetes.io/auth-response-headers: "Authorization, X-Amz-Date, X-Amz-Content-SHA256"
    nginx.ingress.kubernetes.io/upstream-vhost: drive-minio.127.0.0.1.nip.io
+    # nginx.ingress.kubernetes.io/upstream-vhost: drive-dsproxy.127.0.0.1.nip.io
    nginx.ingress.kubernetes.io/rewrite-target: /drive-media-storage/$1

 ingressMediaPreview:
@@ -153,8 +155,10 @@ ingressMediaPreview:
    nginx.ingress.kubernetes.io/auth-url: https://drive.127.0.0.1.nip.io/api/v1.0/items/media-auth/
    nginx.ingress.kubernetes.io/auth-response-headers: "Authorization, X-Amz-Date, X-Amz-Content-SHA256"
    nginx.ingress.kubernetes.io/upstream-vhost: drive-minio.127.0.0.1.nip.io
+    # nginx.ingress.kubernetes.io/upstream-vhost: drive-dsproxy.127.0.0.1.nip.io
    nginx.ingress.kubernetes.io/rewrite-target: /drive-media-storage/$1

 serviceMedia:
  host: dev-backend-minio.drive.svc.cluster.local
+  # host: drive-dsproxy.127.0.0.1.nip.io
  port: 80
--- a/src/helm/helmfile.yaml
+++ b/src/helm/helmfile.yaml
@@ -12,7 +12,7 @@ releases:
  - name: dev-backend
    namespace: {{ .Namespace }}
    chart: dev-backends/dev-backend
-    version: 0.0.3
+    version: 0.0.6
    values:
      - postgres:
          enabled: true
@@ -76,10 +76,53 @@ releases:
            username: drive
            password: drive
            email: drive@example.com
+      - dsproxy:
+          enabled: false
+          aws_url: https://drive-minio.127.0.0.1.nip.io
+          image: demarchenumerique/ds-proxy:v2.0.0-alpha.2
+          command:
+            - /bin/sh
+            - "-c"
+            - |
+              /dsproxy/ds_proxy proxy --address=0.0.0.0:4444 --verify-ssl-certificate=false --password-file /etc/dsproxy/PASSWORD \
+              --salt "$SALT" --keyring-file /etc/dsproxy/keyring.toml --upstream-url ${AWS_URL} \
+              --local-encryption-directory /var/tmp/local_encryption/ --s3-access-key ${AWS_ACCESS_KEY} \
+              --s3-secret-key ${AWS_SECRET_KEY} --s3-region ${AWS_REGION}
+            # Extra volume mounts to manage our local custom CA and avoid to set ssl_verify: false
+          ingress:
+            enabled: true
+            hostname: drive-dsproxy.127.0.0.1.nip.io
+            tls:
+              enabled: true
+          localSecret:
+            enabled: true
+          envVars:
+            RUST_LOG: debug,ds_proxy::http::handlers::fetch=trace,ds_proxy::http::handlers::forward=trace,ds_proxy::config=trace
+            RUST_BACKTRACE: full
+            SALT:
+              secretKeyRef:
+                name: dev-backend-dsproxy
+                key: SALT
+            AWS_ACCESS_KEY:
+              secretKeyRef:
+                name: dev-backend-dsproxy
+                key: AWS_ACCESS_KEY
+            AWS_SECRET_KEY:
+              secretKeyRef:
+                name: dev-backend-dsproxy
+                key: AWS_SECRET_KEY
+            AWS_URL:
+              secretKeyRef:
+                name: dev-backend-dsproxy
+                key: AWS_URL
+            AWS_REGION:
+              secretKeyRef:
+                name: dev-backend-dsproxy
+                key: AWS_REGION

  - name: drive
    version: {{ .Values.version }}
    namespace: {{ .Namespace }}
    chart: ./drive
    values:
-      - env.d/{{ .Environment.Name }}/values.drive.yaml.gotmpl
+      - env.d/{{ .Environment.Name }}/values.drive.yaml.gotmpl